[FFmpeg-devel] Allow interrupt callback for AVCodecContext

Don Moir donmoir at comcast.net
Tue Jan 7 05:50:22 CET 2014 

----- Original Message ----- 
From: "Don Moir" <donmoir at comcast.net>
To: "FFmpeg development discussions and patches" <ffmpeg-devel at ffmpeg.org>
Sent: Monday, January 06, 2014 5:12 PM
Subject: Re: [FFmpeg-devel] Allow interrupt callback for AVCodecContext


>
> ----- Original Message ----- 
> From: "Ronald S. Bultje" <rsbultje at gmail.com>
> To: "FFmpeg development discussions and patches" <ffmpeg-devel at ffmpeg.org>
> Sent: Monday, January 06, 2014 5:04 PM
> Subject: Re: [FFmpeg-devel] Allow interrupt callback for AVCodecContext
>
>
>> Hi,
>>
>> On Mon, Jan 6, 2014 at 4:08 PM, Don Moir <donmoir at comcast.net> wrote:
>>
>>>
>>> ----- Original Message ----- From: "Ronald S. Bultje" <rsbultje at gmail.com>
>>> To: "FFmpeg development discussions and patches" <ffmpeg-devel at ffmpeg.org>
>>> Sent: Monday, January 06, 2014 3:59 PM
>>>
>>> Subject: Re: [FFmpeg-devel] Allow interrupt callback for AVCodecContext
>>>
>>>
>>>  Hi,
>>>>
>>>> On Mon, Jan 6, 2014 at 2:48 PM, Don Moir <donmoir at comcast.net> wrote:
>>>>
>>>>
>>>>> ----- Original Message ----- From: "Ronald S. Bultje" <
>>>>> rsbultje at gmail.com>
>>>>> To: "FFmpeg development discussions and patches" <
>>>>> ffmpeg-devel at ffmpeg.org>
>>>>> Sent: Monday, January 06, 2014 9:52 AM
>>>>>
>>>>> Subject: Re: [FFmpeg-devel] Allow interrupt callback for AVCodecContext
>>>>>
>>>>>
>>>>>  Hi,
>>>>>
>>>>>>
>>>>>> On Mon, Jan 6, 2014 at 3:56 AM, Don Moir <donmoir at comcast.net> wrote:
>>>>>>
>>>>>>
>>>>>>  ----- Original Message ----- From: "Ronald S. Bultje" <
>>>>>>> rsbultje at gmail.com>
>>>>>>> To: "FFmpeg development discussions and patches" <
>>>>>>> ffmpeg-devel at ffmpeg.org>
>>>>>>> Sent: Monday, January 06, 2014 8:44 AM
>>>>>>>
>>>>>>> Subject: Re: [FFmpeg-devel] Allow interrupt callback for AVCodecContext
>>>>>>>
>>>>>>>
>>>>>>>  Hi,
>>>>>>>
>>>>>>>
>>>>>>>> On Mon, Dec 16, 2013 at 1:21 AM, Don Moir <donmoir at comcast.net>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>  ----- Original Message ----- From: "Don Moir" <donmoir at comcast.net>
>>>>>>>>
>>>>>>>>
>>>>>>>>>  To: "FFmpeg development discussions and patches" <
>>>>>>>>>
>>>>>>>>>> ffmpeg-devel at ffmpeg.org
>>>>>>>>>> >
>>>>>>>>>> Sent: Monday, December 16, 2013 2:03 AM
>>>>>>>>>> Subject: Re: [FFmpeg-devel] Allow interrupt callback for
>>>>>>>>>> AVCodecContext
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>  ----- Original Message ----- From: "Ronald S. Bultje" <
>>>>>>>>>>
>>>>>>>>>>  rsbultje at gmail.com>
>>>>>>>>>>> To: "FFmpeg development discussions and patches" <
>>>>>>>>>>> ffmpeg-devel at ffmpeg.org>
>>>>>>>>>>> Sent: Wednesday, January 01, 2014 11:14 AM
>>>>>>>>>>> Subject: Re: [FFmpeg-devel] Allow interrupt callback for
>>>>>>>>>>> AVCodecContext
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>  Hi,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>  On Mon, Dec 16, 2013 at 1:30 AM, Don Moir <donmoir at comcast.net>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>  ----- Original Message ----- From: "Ronald S. Bultje" <
>>>>>>>>>>>>
>>>>>>>>>>>>  rsbultje at gmail.com>
>>>>>>>>>>>>> To: "FFmpeg development discussions and patches" <
>>>>>>>>>>>>> ffmpeg-devel at ffmpeg.org>
>>>>>>>>>>>>> Sent: Wednesday, January 01, 2014 10:52 AM
>>>>>>>>>>>>> Subject: Re: [FFmpeg-devel] Allow interrupt callback for
>>>>>>>>>>>>> AVCodecContext
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>  Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>  On Mon, Dec 16, 2013 at 2:07 AM, Don Moir <donmoir at comcast.net>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  For now just seeing what you think about this. This is about
>>>>>>>>>>>>>> thread
>>>>>>>>>>>>>> based
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  apps where this makes sense.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> When attempting to do a new seek or waiting to close a video, I
>>>>>>>>>>>>>>> find
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> I am waiting on avcodec_decode_video2 to return before I can
>>>>>>>>>>>>>>> continue.
>>>>>>>>>>>>>>> Depending on machine and video, this wait time can be up to
>>>>>>>>>>>>>>> about
>>>>>>>>>>>>>>> 50ms
>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>> normally wait time about 5 to 20 ms or so.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> You might say 'so what' and I would agree for a simple player
>>>>>>>>>>>>>>> app
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>> does
>>>>>>>>>>>>>>> not make that much difference.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If you are trying to stay on a timeline, or in case of
>>>>>>>>>>>>>>> scrubbing,
>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>> editing apps, this wait time does make a difference. That is,
>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>> can not
>>>>>>>>>>>>>>> move on until avcodec_decode_video2 has returned.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I can pretty much seek instantly to zero for any file except
>>>>>>>>>>>>>>> when I
>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>> to wait on avcodec_decode_video2 if that be the case.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> For me, it's normal for any intense process like decoding to be
>>>>>>>>>>>>>>> interruptible but this is not the case for AVCodecContext in
>>>>>>>>>>>>>>> ffmpeg.
>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>> is strange, don't you think?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> For AVFormatContext you have:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> typedef struct AVIOInterruptCB {
>>>>>>>>>>>>>>>     int (*callback)(void*);
>>>>>>>>>>>>>>>     void *opaque;
>>>>>>>>>>>>>>> } AVIOInterruptCB;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I would use this model for AVCodecContext but change naming to:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> typedef struct AVInterruptCB {
>>>>>>>>>>>>>>>     int (*callback)(void*);
>>>>>>>>>>>>>>>     void *opaque;
>>>>>>>>>>>>>>> } AVInterruptCB;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Then make name changes to whereever and add to AVCodecContext.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This callback could be implemented piecemeal whereever needed
>>>>>>>>>>>>>>> over
>>>>>>>>>>>>>>> time,
>>>>>>>>>>>>>>> hitting the more intense processes first.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  Just open a (potentially pre-cached) new AVCodecContext, it'll
>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  even
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> faster than your solution.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ronald
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  hmmm. That's a thought for seeking I suppose but does not apply
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  waiting
>>>>>>>>>>>>> to close. Why do I care about close time ? Because another video
>>>>>>>>>>>>> has
>>>>>>>>>>>>> come
>>>>>>>>>>>>> in to replace it or variations of it. This can happen rapidly.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>  This is where more evolved languages have the concept of garbage
>>>>>>>>>>>> collection. For your purpose, you simply have a queue where you
>>>>>>>>>>>> push
>>>>>>>>>>>> "AVCodecContexts I don't need anymore" into, and while the
>>>>>>>>>>>> application
>>>>>>>>>>>> is
>>>>>>>>>>>> in the idle loop, you pop it and destroy items left in it.
>>>>>>>>>>>>
>>>>>>>>>>>> Really, I understand your use case, but you don't want to add all
>>>>>>>>>>>> kind
>>>>>>>>>>>> of
>>>>>>>>>>>> clever hacks in AVCodecContext to get this kind of stuff done.
>>>>>>>>>>>> You're
>>>>>>>>>>>> not
>>>>>>>>>>>> using a shared I/O resource that may be protected by a cookie or
>>>>>>>>>>>> worse
>>>>>>>>>>>> for
>>>>>>>>>>>> pay-per-view video, and you're not in any sort of kernel wait, so
>>>>>>>>>>>> there's
>>>>>>>>>>>> no reason to add these kind of hacks. It's a logical thought, but
>>>>>>>>>>>> this
>>>>>>>>>>>> problem has been solved already and there's better, easier and
>>>>>>>>>>>> faster
>>>>>>>>>>>> solutions out there that do not involve adding hacks in every
>>>>>>>>>>>> single
>>>>>>>>>>>> FFmpeg
>>>>>>>>>>>> decoder to actually support this.
>>>>>>>>>>>>
>>>>>>>>>>>> Ronald
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>  Yeah really did not like the notion of changing decoders and I
>>>>>>>>>>>>
>>>>>>>>>>> don't
>>>>>>>>>>> like adding anything that might not be needed, but I will see what
>>>>>>>>>>> I
>>>>>>>>>>> can do
>>>>>>>>>>> with your suggestions. I never know what ffmpeg can tolerate. I had
>>>>>>>>>>> asked
>>>>>>>>>>> in libav-user but get the same old BS there when asking about
>>>>>>>>>>> things
>>>>>>>>>>> like
>>>>>>>>>>> this.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>  Ok tried some test code allocating new context and that worked
>>>>>>>>>>>
>>>>>>>>>> pretty
>>>>>>>>>> well.
>>>>>>>>>>
>>>>>>>>>> I had to do this to get consistent results:
>>>>>>>>>>
>>>>>>>>>> AVCodec *codec ... already have it
>>>>>>>>>>
>>>>>>>>>> AVCodecContext *new_context = avcodec_alloc_context3 (NULL);
>>>>>>>>>> avcodec_copy_context (new_context,old_context);
>>>>>>>>>> avcodec_open2 (new_context,codec,NULL);
>>>>>>>>>>
>>>>>>>>>> The following worked for at least one file but for failed for others
>>>>>>>>>> like
>>>>>>>>>> Theora etc.
>>>>>>>>>>
>>>>>>>>>> AVCodecContext *new_context = avcodec_alloc_context3 (codec);
>>>>>>>>>> avcodec_open2 (new_context,codec,NULL);
>>>>>>>>>>
>>>>>>>>>> For Theora it failed in avcodec_open2 saying 'missing side data' or
>>>>>>>>>> similiar.
>>>>>>>>>>
>>>>>>>>>> Using a cached context the wait time is zero. Executing the 3
>>>>>>>>>> statements
>>>>>>>>>> above on slower machine is about 1 to 4 ms. It's also not like it
>>>>>>>>>> always
>>>>>>>>>> stuck in avcodec_decode_video2 either. In this case I don't need a
>>>>>>>>>> new
>>>>>>>>>> context and wait time is zero.
>>>>>>>>>>
>>>>>>>>>> Thanks Ronald.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>  Ronald says:
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  Just open a (potentially pre-cached) new AVCodecContext, it'll be
>>>>>>>>> even
>>>>>>>>>
>>>>>>>>>  faster than your solution.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>>>   and
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>  Really, I understand your use case, but you don't want to add all
>>>>>>>>> kind
>>>>>>>>> of
>>>>>>>>>
>>>>>>>>>  clever hacks in AVCodecContext to get this kind of stuff done.
>>>>>>>>> You're
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>  not
>>>>>>>>>>>
>>>>>>>>>>>> using a shared I/O resource that may be protected by a cookie or
>>>>>>>>>>>> worse
>>>>>>>>>>>> for
>>>>>>>>>>>> pay-per-view video, and you're not in any sort of kernel wait, so
>>>>>>>>>>>> there's
>>>>>>>>>>>> no reason to add these kind of hacks. It's a logical thought, but
>>>>>>>>>>>> this
>>>>>>>>>>>> problem has been solved already and there's better, easier and
>>>>>>>>>>>> faster
>>>>>>>>>>>> solutions out there that do not involve adding hacks in every
>>>>>>>>>>>> single
>>>>>>>>>>>> FFmpeg
>>>>>>>>>>>> decoder to actually support this.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>   Using a cached and open AVCodecContext does work but it's like
>>>>>>>>>>>>
>>>>>>>>>>> trying
>>>>>>>>>>>
>>>>>>>>>>>  to
>>>>>>>>>>
>>>>>>>>> kill an ant with a sledgehammer. Using a cached and not opened
>>>>>>>>> context
>>>>>>>>> helps some but you still loose time when opening it. So best results
>>>>>>>>> are
>>>>>>>>> when using a pre-cached open context.
>>>>>>>>>
>>>>>>>>> This means you will have allocated resources and most likely opened
>>>>>>>>> threads in your cached context that are not doing anything.
>>>>>>>>>
>>>>>>>>> So you have to ask what the real hack is. Keeping an opened cached
>>>>>>>>> context
>>>>>>>>> or having an interrupt callback. An interrupt callback does not use
>>>>>>>>> any
>>>>>>>>> additional resources but then it has to be implemented for every
>>>>>>>>> decoder.
>>>>>>>>> An opened context works now, but uses addtional resources and mostly
>>>>>>>>> opened
>>>>>>>>> threads that are more or less dormant. Having an interruptible
>>>>>>>>> intense
>>>>>>>>> process is normal and not a hack and you should not have to beat it
>>>>>>>>> to
>>>>>>>>> death to get it to work.
>>>>>>>>>
>>>>>>>>> I restrict the context to have at most 2 threads. Yes I could limit
>>>>>>>>> it
>>>>>>>>> to
>>>>>>>>> no new threads, but I get better results with 2. So you have to be
>>>>>>>>> careful.
>>>>>>>>> If the thread_count is zero, which is the default, then it will
>>>>>>>>> choose
>>>>>>>>> the
>>>>>>>>> number of threads based on the cpu count. You will have this number
>>>>>>>>> of
>>>>>>>>> opened threads in a cached opened context. On one of my machines this
>>>>>>>>> would
>>>>>>>>> be 8 opened and unused threads for cached open context, but I set
>>>>>>>>> thread_count to 2, getting diminishing returns on greater number of
>>>>>>>>> threads.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> I'll bite. Please do define expensive, as Reimar also said. Do you
>>>>>>>> mean
>>>>>>>> "cpu usage"? Or "memory usage"? Or something else?
>>>>>>>>
>>>>>>>>
>>>>>>>>  I was definning expensive as allocating threads and memory that do
>>>>>>> nothing
>>>>>>> for what should be a simple process.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> But isn't that a completely academic concern?
>>>>>>
>>>>>>
>>>>> Don't think so as something is not good somewhere. Did some more checking
>>>>> and tried to figure the cost of this more accurately. It varies on codec
>>>>> and some other things so trying to get a handle on it.
>>>>>
>>>>> With H264 file, 2 threads, 2 opened context, one in use and one cached.
>>>>>
>>>>> Initially all is good. Then when I seek, I swap the context. The 'in use'
>>>>> context which might still be in avcode_decode_video2 is flushed later but
>>>>> for test I waited on video and flushed both cached and original context.
>>>>>
>>>>> When the seek is complete, I see about double the amount of memory being
>>>>> used for the file. This a large video file and goes from about 30mb to
>>>>> 60mb
>>>>> after seek give or take.
>>>>>
>>>>
>>>>
>>>> Right, because you didn't flush the old context yet. You need to flush and
>>>> reset it for the reference frames to be released. Until then, it's as
>>>> alive
>>>> as it could be, consuming any bit as much memory as the running one.
>>>>
>>>>
>>> I guess you missed reading this:
>>>
>>>
>>>  The 'in use' context which might still be in avcode_decode_video2 is
>>>>> flushed later but
>>>>> for test I waited on video and flushed both cached and original context.
>>>>>
>>>>
>>> For test I waiting on video and flushed both the old context and new
>>> context. Normally the old context is flushed later (very soon later), but
>>> for testing I flushed both to be sure. In either case it did not make any
>>> difference.
>>
>>
>> Did you avcodec_close() it?
>
> No. That is almost pointless since the whole thing here is doing something fast. The re-open takes too much time. Caching in 
> itself doesn't make too much sense if you have to close and reopen each time. The allocation takes very little time. It can help a 
> little but not too much.

The way it is right now with cached context is extreme overkill to perform a fairly simple task. You have memory allocations per 
thread that can be quite large and that depends on codec. For h264 it can be large and for others not as much. The whole process of 
caching a context has some merit but the overhead and time required to close and reopen reduces it's effectiveness.

Essentially, everytime you want to use a cached context, you will be closing threads, freeing memory, etc and then reopening the 
threads and reallocating memory (silly). The reason for this is it's the only way to free the context accumulated memory. flush does 
not do this and I don't know whats up with that.

Backing off from this but of course would still like avcodec_decode_video2 to be interruptible or a reasonable way to avoid waiting 
on it. Cached context the way it stands now is anything but reasonable.  Possibly if flush actually flushed the memory it would be 
more attractive.

The AVCodecContext.execute function which can be set by user would almost work but that is only used for slices and it does not look 
like its complete to me.

More information about the ffmpeg-devel mailing list