[FFmpeg-devel] Allow interrupt callback for AVCodecContext

[Ronald S. Bultje]

Hi,

On Mon, Jan 6, 2014 at 4:08 PM, Don Moir <donmoir at comcast.net> wrote:

>
> ----- Original Message ----- From: "Ronald S. Bultje" <rsbultje at gmail.com>
> To: "FFmpeg development discussions and patches" <ffmpeg-devel at ffmpeg.org>
> Sent: Monday, January 06, 2014 3:59 PM
>
> Subject: Re: [FFmpeg-devel] Allow interrupt callback for AVCodecContext
>
>
>  Hi,
>>
>> On Mon, Jan 6, 2014 at 2:48 PM, Don Moir <donmoir at comcast.net> wrote:
>>
>>
>>> ----- Original Message ----- From: "Ronald S. Bultje" <
>>> rsbultje at gmail.com>
>>> To: "FFmpeg development discussions and patches" <
>>> ffmpeg-devel at ffmpeg.org>
>>> Sent: Monday, January 06, 2014 9:52 AM
>>>
>>> Subject: Re: [FFmpeg-devel] Allow interrupt callback for AVCodecContext
>>>
>>>
>>>  Hi,
>>>
>>>>
>>>> On Mon, Jan 6, 2014 at 3:56 AM, Don Moir <donmoir at comcast.net> wrote:
>>>>
>>>>
>>>>  ----- Original Message ----- From: "Ronald S. Bultje" <
>>>>> rsbultje at gmail.com>
>>>>> To: "FFmpeg development discussions and patches" <
>>>>> ffmpeg-devel at ffmpeg.org>
>>>>> Sent: Monday, January 06, 2014 8:44 AM
>>>>>
>>>>> Subject: Re: [FFmpeg-devel] Allow interrupt callback for AVCodecContext
>>>>>
>>>>>
>>>>>  Hi,
>>>>>
>>>>>
>>>>>> On Mon, Dec 16, 2013 at 1:21 AM, Don Moir <donmoir at comcast.net>
>>>>>> wrote:
>>>>>>
>>>>>>  ----- Original Message ----- From: "Don Moir" <donmoir at comcast.net>
>>>>>>
>>>>>>
>>>>>>>  To: "FFmpeg development discussions and patches" <
>>>>>>>
>>>>>>>> ffmpeg-devel at ffmpeg.org
>>>>>>>> >
>>>>>>>> Sent: Monday, December 16, 2013 2:03 AM
>>>>>>>> Subject: Re: [FFmpeg-devel] Allow interrupt callback for
>>>>>>>> AVCodecContext
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>  ----- Original Message ----- From: "Ronald S. Bultje" <
>>>>>>>>
>>>>>>>>  rsbultje at gmail.com>
>>>>>>>>> To: "FFmpeg development discussions and patches" <
>>>>>>>>> ffmpeg-devel at ffmpeg.org>
>>>>>>>>> Sent: Wednesday, January 01, 2014 11:14 AM
>>>>>>>>> Subject: Re: [FFmpeg-devel] Allow interrupt callback for
>>>>>>>>> AVCodecContext
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  Hi,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  On Mon, Dec 16, 2013 at 1:30 AM, Don Moir <donmoir at comcast.net>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>  ----- Original Message ----- From: "Ronald S. Bultje" <
>>>>>>>>>>
>>>>>>>>>>  rsbultje at gmail.com>
>>>>>>>>>>> To: "FFmpeg development discussions and patches" <
>>>>>>>>>>> ffmpeg-devel at ffmpeg.org>
>>>>>>>>>>> Sent: Wednesday, January 01, 2014 10:52 AM
>>>>>>>>>>> Subject: Re: [FFmpeg-devel] Allow interrupt callback for
>>>>>>>>>>> AVCodecContext
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>  Hi,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>  On Mon, Dec 16, 2013 at 2:07 AM, Don Moir <donmoir at comcast.net>
>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>  For now just seeing what you think about this. This is about
>>>>>>>>>>>> thread
>>>>>>>>>>>> based
>>>>>>>>>>>>
>>>>>>>>>>>>  apps where this makes sense.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> When attempting to do a new seek or waiting to close a video, I
>>>>>>>>>>>>> find
>>>>>>>>>>>>> that
>>>>>>>>>>>>> I am waiting on avcodec_decode_video2 to return before I can
>>>>>>>>>>>>> continue.
>>>>>>>>>>>>> Depending on machine and video, this wait time can be up to
>>>>>>>>>>>>> about
>>>>>>>>>>>>> 50ms
>>>>>>>>>>>>> but
>>>>>>>>>>>>> normally wait time about 5 to 20 ms or so.
>>>>>>>>>>>>>
>>>>>>>>>>>>> You might say 'so what' and I would agree for a simple player
>>>>>>>>>>>>> app
>>>>>>>>>>>>> it
>>>>>>>>>>>>> does
>>>>>>>>>>>>> not make that much difference.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If you are trying to stay on a timeline, or in case of
>>>>>>>>>>>>> scrubbing,
>>>>>>>>>>>>> or
>>>>>>>>>>>>> for
>>>>>>>>>>>>> editing apps, this wait time does make a difference. That is,
>>>>>>>>>>>>> you
>>>>>>>>>>>>> can not
>>>>>>>>>>>>> move on until avcodec_decode_video2 has returned.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I can pretty much seek instantly to zero for any file except
>>>>>>>>>>>>> when I
>>>>>>>>>>>>> have
>>>>>>>>>>>>> to wait on avcodec_decode_video2 if that be the case.
>>>>>>>>>>>>>
>>>>>>>>>>>>> For me, it's normal for any intense process like decoding to be
>>>>>>>>>>>>> interruptible but this is not the case for AVCodecContext in
>>>>>>>>>>>>> ffmpeg.
>>>>>>>>>>>>> This
>>>>>>>>>>>>> is strange, don't you think?
>>>>>>>>>>>>>
>>>>>>>>>>>>> For AVFormatContext you have:
>>>>>>>>>>>>>
>>>>>>>>>>>>> typedef struct AVIOInterruptCB {
>>>>>>>>>>>>>     int (*callback)(void*);
>>>>>>>>>>>>>     void *opaque;
>>>>>>>>>>>>> } AVIOInterruptCB;
>>>>>>>>>>>>>
>>>>>>>>>>>>> I would use this model for AVCodecContext but change naming to:
>>>>>>>>>>>>>
>>>>>>>>>>>>> typedef struct AVInterruptCB {
>>>>>>>>>>>>>     int (*callback)(void*);
>>>>>>>>>>>>>     void *opaque;
>>>>>>>>>>>>> } AVInterruptCB;
>>>>>>>>>>>>>
>>>>>>>>>>>>> Then make name changes to whereever and add to AVCodecContext.
>>>>>>>>>>>>>
>>>>>>>>>>>>> This callback could be implemented piecemeal whereever needed
>>>>>>>>>>>>> over
>>>>>>>>>>>>> time,
>>>>>>>>>>>>> hitting the more intense processes first.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>  Just open a (potentially pre-cached) new AVCodecContext, it'll
>>>>>>>>>>>>> be
>>>>>>>>>>>>>
>>>>>>>>>>>>>  even
>>>>>>>>>>>>>
>>>>>>>>>>>> faster than your solution.
>>>>>>>>>>>>
>>>>>>>>>>>> Ronald
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>  hmmm. That's a thought for seeking I suppose but does not apply
>>>>>>>>>>>> to
>>>>>>>>>>>>
>>>>>>>>>>>>  waiting
>>>>>>>>>>> to close. Why do I care about close time ? Because another video
>>>>>>>>>>> has
>>>>>>>>>>> come
>>>>>>>>>>> in to replace it or variations of it. This can happen rapidly.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>  This is where more evolved languages have the concept of garbage
>>>>>>>>>> collection. For your purpose, you simply have a queue where you
>>>>>>>>>> push
>>>>>>>>>> "AVCodecContexts I don't need anymore" into, and while the
>>>>>>>>>> application
>>>>>>>>>> is
>>>>>>>>>> in the idle loop, you pop it and destroy items left in it.
>>>>>>>>>>
>>>>>>>>>> Really, I understand your use case, but you don't want to add all
>>>>>>>>>> kind
>>>>>>>>>> of
>>>>>>>>>> clever hacks in AVCodecContext to get this kind of stuff done.
>>>>>>>>>> You're
>>>>>>>>>> not
>>>>>>>>>> using a shared I/O resource that may be protected by a cookie or
>>>>>>>>>> worse
>>>>>>>>>> for
>>>>>>>>>> pay-per-view video, and you're not in any sort of kernel wait, so
>>>>>>>>>> there's
>>>>>>>>>> no reason to add these kind of hacks. It's a logical thought, but
>>>>>>>>>> this
>>>>>>>>>> problem has been solved already and there's better, easier and
>>>>>>>>>> faster
>>>>>>>>>> solutions out there that do not involve adding hacks in every
>>>>>>>>>> single
>>>>>>>>>> FFmpeg
>>>>>>>>>> decoder to actually support this.
>>>>>>>>>>
>>>>>>>>>> Ronald
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>  Yeah really did not like the notion of changing decoders and I
>>>>>>>>>>
>>>>>>>>> don't
>>>>>>>>> like adding anything that might not be needed, but I will see what
>>>>>>>>> I
>>>>>>>>> can do
>>>>>>>>> with your suggestions. I never know what ffmpeg can tolerate. I had
>>>>>>>>> asked
>>>>>>>>> in libav-user but get the same old BS there when asking about
>>>>>>>>> things
>>>>>>>>> like
>>>>>>>>> this.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  Ok tried some test code allocating new context and that worked
>>>>>>>>>
>>>>>>>> pretty
>>>>>>>> well.
>>>>>>>>
>>>>>>>> I had to do this to get consistent results:
>>>>>>>>
>>>>>>>> AVCodec *codec ... already have it
>>>>>>>>
>>>>>>>> AVCodecContext *new_context = avcodec_alloc_context3 (NULL);
>>>>>>>> avcodec_copy_context (new_context,old_context);
>>>>>>>> avcodec_open2 (new_context,codec,NULL);
>>>>>>>>
>>>>>>>> The following worked for at least one file but for failed for others
>>>>>>>> like
>>>>>>>> Theora etc.
>>>>>>>>
>>>>>>>> AVCodecContext *new_context = avcodec_alloc_context3 (codec);
>>>>>>>> avcodec_open2 (new_context,codec,NULL);
>>>>>>>>
>>>>>>>> For Theora it failed in avcodec_open2 saying 'missing side data' or
>>>>>>>> similiar.
>>>>>>>>
>>>>>>>> Using a cached context the wait time is zero. Executing the 3
>>>>>>>> statements
>>>>>>>> above on slower machine is about 1 to 4 ms. It's also not like it
>>>>>>>> always
>>>>>>>> stuck in avcodec_decode_video2 either. In this case I don't need a
>>>>>>>> new
>>>>>>>> context and wait time is zero.
>>>>>>>>
>>>>>>>> Thanks Ronald.
>>>>>>>>
>>>>>>>>
>>>>>>>>  Ronald says:
>>>>>>>>
>>>>>>>
>>>>>>>  Just open a (potentially pre-cached) new AVCodecContext, it'll be
>>>>>>> even
>>>>>>>
>>>>>>>  faster than your solution.
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>>   and
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>  Really, I understand your use case, but you don't want to add all
>>>>>>> kind
>>>>>>> of
>>>>>>>
>>>>>>>  clever hacks in AVCodecContext to get this kind of stuff done.
>>>>>>> You're
>>>>>>>
>>>>>>>>
>>>>>>>>  not
>>>>>>>>>
>>>>>>>>>> using a shared I/O resource that may be protected by a cookie or
>>>>>>>>>> worse
>>>>>>>>>> for
>>>>>>>>>> pay-per-view video, and you're not in any sort of kernel wait, so
>>>>>>>>>> there's
>>>>>>>>>> no reason to add these kind of hacks. It's a logical thought, but
>>>>>>>>>> this
>>>>>>>>>> problem has been solved already and there's better, easier and
>>>>>>>>>> faster
>>>>>>>>>> solutions out there that do not involve adding hacks in every
>>>>>>>>>> single
>>>>>>>>>> FFmpeg
>>>>>>>>>> decoder to actually support this.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>   Using a cached and open AVCodecContext does work but it's like
>>>>>>>>>>
>>>>>>>>> trying
>>>>>>>>>
>>>>>>>>>  to
>>>>>>>>
>>>>>>> kill an ant with a sledgehammer. Using a cached and not opened
>>>>>>> context
>>>>>>> helps some but you still loose time when opening it. So best results
>>>>>>> are
>>>>>>> when using a pre-cached open context.
>>>>>>>
>>>>>>> This means you will have allocated resources and most likely opened
>>>>>>> threads in your cached context that are not doing anything.
>>>>>>>
>>>>>>> So you have to ask what the real hack is. Keeping an opened cached
>>>>>>> context
>>>>>>> or having an interrupt callback. An interrupt callback does not use
>>>>>>> any
>>>>>>> additional resources but then it has to be implemented for every
>>>>>>> decoder.
>>>>>>> An opened context works now, but uses addtional resources and mostly
>>>>>>> opened
>>>>>>> threads that are more or less dormant. Having an interruptible
>>>>>>> intense
>>>>>>> process is normal and not a hack and you should not have to beat it
>>>>>>> to
>>>>>>> death to get it to work.
>>>>>>>
>>>>>>> I restrict the context to have at most 2 threads. Yes I could limit
>>>>>>> it
>>>>>>> to
>>>>>>> no new threads, but I get better results with 2. So you have to be
>>>>>>> careful.
>>>>>>> If the thread_count is zero, which is the default, then it will
>>>>>>> choose
>>>>>>> the
>>>>>>> number of threads based on the cpu count. You will have this number
>>>>>>> of
>>>>>>> opened threads in a cached opened context. On one of my machines this
>>>>>>> would
>>>>>>> be 8 opened and unused threads for cached open context, but I set
>>>>>>> thread_count to 2, getting diminishing returns on greater number of
>>>>>>> threads.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> I'll bite. Please do define expensive, as Reimar also said. Do you
>>>>>> mean
>>>>>> "cpu usage"? Or "memory usage"? Or something else?
>>>>>>
>>>>>>
>>>>>>  I was definning expensive as allocating threads and memory that do
>>>>> nothing
>>>>> for what should be a simple process.
>>>>>
>>>>>
>>>>
>>>> But isn't that a completely academic concern?
>>>>
>>>>
>>> Don't think so as something is not good somewhere. Did some more checking
>>> and tried to figure the cost of this more accurately. It varies on codec
>>> and some other things so trying to get a handle on it.
>>>
>>> With H264 file, 2 threads, 2 opened context, one in use and one cached.
>>>
>>> Initially all is good. Then when I seek, I swap the context. The 'in use'
>>> context which might still be in avcode_decode_video2 is flushed later but
>>> for test I waited on video and flushed both cached and original context.
>>>
>>> When the seek is complete, I see about double the amount of memory being
>>> used for the file. This a large video file and goes from about 30mb to
>>> 60mb
>>> after seek give or take.
>>>
>>
>>
>> Right, because you didn't flush the old context yet. You need to flush and
>> reset it for the reference frames to be released. Until then, it's as
>> alive
>> as it could be, consuming any bit as much memory as the running one.
>>
>>
> I guess you missed reading this:
>
>
>  The 'in use' context which might still be in avcode_decode_video2 is
>>> flushed later but
>>> for test I waited on video and flushed both cached and original context.
>>>
>>
> For test I waiting on video and flushed both the old context and new
> context. Normally the old context is flushed later (very soon later), but
> for testing I flushed both to be sure. In either case it did not make any
> difference.


Did you avcodec_close() it?

Ronald