[FFmpeg-devel] Allow interrupt callback for AVCodecContext

[Ronald S. Bultje]

Hi,

On Mon, Jan 6, 2014 at 3:56 AM, Don Moir <donmoir at comcast.net> wrote:

>
> ----- Original Message ----- From: "Ronald S. Bultje" <rsbultje at gmail.com>
> To: "FFmpeg development discussions and patches" <ffmpeg-devel at ffmpeg.org>
> Sent: Monday, January 06, 2014 8:44 AM
>
> Subject: Re: [FFmpeg-devel] Allow interrupt callback for AVCodecContext
>
>
>  Hi,
>>
>> On Mon, Dec 16, 2013 at 1:21 AM, Don Moir <donmoir at comcast.net> wrote:
>>
>>  ----- Original Message ----- From: "Don Moir" <donmoir at comcast.net>
>>>
>>>> To: "FFmpeg development discussions and patches" <
>>>> ffmpeg-devel at ffmpeg.org
>>>> >
>>>> Sent: Monday, December 16, 2013 2:03 AM
>>>> Subject: Re: [FFmpeg-devel] Allow interrupt callback for AVCodecContext
>>>>
>>>>
>>>>
>>>>  ----- Original Message ----- From: "Ronald S. Bultje" <
>>>>> rsbultje at gmail.com>
>>>>> To: "FFmpeg development discussions and patches" <
>>>>> ffmpeg-devel at ffmpeg.org>
>>>>> Sent: Wednesday, January 01, 2014 11:14 AM
>>>>> Subject: Re: [FFmpeg-devel] Allow interrupt callback for AVCodecContext
>>>>>
>>>>>
>>>>>  Hi,
>>>>>
>>>>>>
>>>>>> On Mon, Dec 16, 2013 at 1:30 AM, Don Moir <donmoir at comcast.net>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>>  ----- Original Message ----- From: "Ronald S. Bultje" <
>>>>>>> rsbultje at gmail.com>
>>>>>>> To: "FFmpeg development discussions and patches" <
>>>>>>> ffmpeg-devel at ffmpeg.org>
>>>>>>> Sent: Wednesday, January 01, 2014 10:52 AM
>>>>>>> Subject: Re: [FFmpeg-devel] Allow interrupt callback for
>>>>>>> AVCodecContext
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>  Hi,
>>>>>>>
>>>>>>>
>>>>>>>> On Mon, Dec 16, 2013 at 2:07 AM, Don Moir <donmoir at comcast.net>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>  For now just seeing what you think about this. This is about thread
>>>>>>>> based
>>>>>>>>
>>>>>>>>  apps where this makes sense.
>>>>>>>>>
>>>>>>>>> When attempting to do a new seek or waiting to close a video, I
>>>>>>>>> find
>>>>>>>>> that
>>>>>>>>> I am waiting on avcodec_decode_video2 to return before I can
>>>>>>>>> continue.
>>>>>>>>> Depending on machine and video, this wait time can be up to about
>>>>>>>>> 50ms
>>>>>>>>> but
>>>>>>>>> normally wait time about 5 to 20 ms or so.
>>>>>>>>>
>>>>>>>>> You might say 'so what' and I would agree for a simple player app
>>>>>>>>> it
>>>>>>>>> does
>>>>>>>>> not make that much difference.
>>>>>>>>>
>>>>>>>>> If you are trying to stay on a timeline, or in case of scrubbing,
>>>>>>>>> or
>>>>>>>>> for
>>>>>>>>> editing apps, this wait time does make a difference. That is, you
>>>>>>>>> can not
>>>>>>>>> move on until avcodec_decode_video2 has returned.
>>>>>>>>>
>>>>>>>>> I can pretty much seek instantly to zero for any file except when I
>>>>>>>>> have
>>>>>>>>> to wait on avcodec_decode_video2 if that be the case.
>>>>>>>>>
>>>>>>>>> For me, it's normal for any intense process like decoding to be
>>>>>>>>> interruptible but this is not the case for AVCodecContext in
>>>>>>>>> ffmpeg.
>>>>>>>>> This
>>>>>>>>> is strange, don't you think?
>>>>>>>>>
>>>>>>>>> For AVFormatContext you have:
>>>>>>>>>
>>>>>>>>> typedef struct AVIOInterruptCB {
>>>>>>>>>     int (*callback)(void*);
>>>>>>>>>     void *opaque;
>>>>>>>>> } AVIOInterruptCB;
>>>>>>>>>
>>>>>>>>> I would use this model for AVCodecContext but change naming to:
>>>>>>>>>
>>>>>>>>> typedef struct AVInterruptCB {
>>>>>>>>>     int (*callback)(void*);
>>>>>>>>>     void *opaque;
>>>>>>>>> } AVInterruptCB;
>>>>>>>>>
>>>>>>>>> Then make name changes to whereever and add to AVCodecContext.
>>>>>>>>>
>>>>>>>>> This callback could be implemented piecemeal whereever needed over
>>>>>>>>> time,
>>>>>>>>> hitting the more intense processes first.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  Just open a (potentially pre-cached) new AVCodecContext, it'll be
>>>>>>>>>
>>>>>>>> even
>>>>>>>> faster than your solution.
>>>>>>>>
>>>>>>>> Ronald
>>>>>>>>
>>>>>>>>
>>>>>>>>  hmmm. That's a thought for seeking I suppose but does not apply to
>>>>>>> waiting
>>>>>>> to close. Why do I care about close time ? Because another video has
>>>>>>> come
>>>>>>> in to replace it or variations of it. This can happen rapidly.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> This is where more evolved languages have the concept of garbage
>>>>>> collection. For your purpose, you simply have a queue where you push
>>>>>> "AVCodecContexts I don't need anymore" into, and while the application
>>>>>> is
>>>>>> in the idle loop, you pop it and destroy items left in it.
>>>>>>
>>>>>> Really, I understand your use case, but you don't want to add all kind
>>>>>> of
>>>>>> clever hacks in AVCodecContext to get this kind of stuff done. You're
>>>>>> not
>>>>>> using a shared I/O resource that may be protected by a cookie or worse
>>>>>> for
>>>>>> pay-per-view video, and you're not in any sort of kernel wait, so
>>>>>> there's
>>>>>> no reason to add these kind of hacks. It's a logical thought, but this
>>>>>> problem has been solved already and there's better, easier and faster
>>>>>> solutions out there that do not involve adding hacks in every single
>>>>>> FFmpeg
>>>>>> decoder to actually support this.
>>>>>>
>>>>>> Ronald
>>>>>>
>>>>>>
>>>>> Yeah really did not like the notion of changing decoders and I don't
>>>>> like adding anything that might not be needed, but I will see what I
>>>>> can do
>>>>> with your suggestions. I never know what ffmpeg can tolerate. I had
>>>>> asked
>>>>> in libav-user but get the same old BS there when asking about things
>>>>> like
>>>>> this.
>>>>>
>>>>>
>>>> Ok tried some test code allocating new context and that worked pretty
>>>> well.
>>>>
>>>> I had to do this to get consistent results:
>>>>
>>>> AVCodec *codec ... already have it
>>>>
>>>> AVCodecContext *new_context = avcodec_alloc_context3 (NULL);
>>>> avcodec_copy_context (new_context,old_context);
>>>> avcodec_open2 (new_context,codec,NULL);
>>>>
>>>> The following worked for at least one file but for failed for others
>>>> like
>>>> Theora etc.
>>>>
>>>> AVCodecContext *new_context = avcodec_alloc_context3 (codec);
>>>> avcodec_open2 (new_context,codec,NULL);
>>>>
>>>> For Theora it failed in avcodec_open2 saying 'missing side data' or
>>>> similiar.
>>>>
>>>> Using a cached context the wait time is zero. Executing the 3 statements
>>>> above on slower machine is about 1 to 4 ms. It's also not like it always
>>>> stuck in avcodec_decode_video2 either. In this case I don't need a new
>>>> context and wait time is zero.
>>>>
>>>> Thanks Ronald.
>>>>
>>>>
>>> Ronald says:
>>>
>>>  Just open a (potentially pre-cached) new AVCodecContext, it'll be even
>>>
>>>> faster than your solution.
>>>>>>>>
>>>>>>>>
>>>>>>>  and
>>>
>>>
>>>  Really, I understand your use case, but you don't want to add all kind
>>> of
>>>
>>>> clever hacks in AVCodecContext to get this kind of stuff done. You're
>>>>>> not
>>>>>> using a shared I/O resource that may be protected by a cookie or worse
>>>>>> for
>>>>>> pay-per-view video, and you're not in any sort of kernel wait, so
>>>>>> there's
>>>>>> no reason to add these kind of hacks. It's a logical thought, but this
>>>>>> problem has been solved already and there's better, easier and faster
>>>>>> solutions out there that do not involve adding hacks in every single
>>>>>> FFmpeg
>>>>>> decoder to actually support this.
>>>>>>
>>>>>>
>>>>>  Using a cached and open AVCodecContext does work but it's like trying
>>> to
>>> kill an ant with a sledgehammer. Using a cached and not opened context
>>> helps some but you still loose time when opening it. So best results are
>>> when using a pre-cached open context.
>>>
>>> This means you will have allocated resources and most likely opened
>>> threads in your cached context that are not doing anything.
>>>
>>> So you have to ask what the real hack is. Keeping an opened cached
>>> context
>>> or having an interrupt callback. An interrupt callback does not use any
>>> additional resources but then it has to be implemented for every decoder.
>>> An opened context works now, but uses addtional resources and mostly
>>> opened
>>> threads that are more or less dormant. Having an interruptible intense
>>> process is normal and not a hack and you should not have to beat it to
>>> death to get it to work.
>>>
>>> I restrict the context to have at most 2 threads. Yes I could limit it to
>>> no new threads, but I get better results with 2. So you have to be
>>> careful.
>>> If the thread_count is zero, which is the default, then it will choose
>>> the
>>> number of threads based on the cpu count. You will have this number of
>>> opened threads in a cached opened context. On one of my machines this
>>> would
>>> be 8 opened and unused threads for cached open context, but I set
>>> thread_count to 2, getting diminishing returns on greater number of
>>> threads.
>>>
>>
>>
>> I'll bite. Please do define expensive, as Reimar also said. Do you mean
>> "cpu usage"? Or "memory usage"? Or something else?
>>
>
> I was definning expensive as allocating threads and memory that do nothing
> for what should be a simple process.


But isn't that a completely academic concern?

I mean, it's indeed nice not to have to do it, but you're going to have to
choose between two evils here:
- no cpu, allocating extra resources - which ends up being a few kb + zero
latency in the post-seek switch (from the codec's point of view) + no
maintenance cost
versus
-  no cpu (yes the check adds a few cycles but that's irrelevant unless you
do it per-pixel or per-mb which I don't think anyone is suggesting) + no
extra resources (zero memory increase) + invasive code changes where the
end result's latency depends on the granularity of the checks +no memory).

We need to judge these evils on real concerns (memory, cpu) that are
objectively measurable and relevant for end users, not by academic concerns
that just lead to more bikeshedding and procrastination. I'm claiming that
the memory is irrelevant in the first solution (b/c it's a few kb versus
many mbs for ref buffers), so therefore the end cost is zero for a
zero-latency solution. The second solution has latency, higher maintenance
cost - all of that to save a few irrelevant kb of memory? I wish I could
save the few kb of memory also, but I can't think of a solution like that
just yet. I'd love to hear them, esp. if they don't add latency and
maintenance cost.

Ronald