[FFmpeg-devel] Buffers Threads and refcounts

[Hendrik Leppkes]

On Wed, Jan 9, 2013 at 3:11 PM, Michael Niedermayer <michaelni at gmx.at> wrote:
>>
>>
>> I won't go into detail on all the pros and cons, however i would like
>> to say that refcounted buffers for both AVPacket and AVFrames is
>> something that has been bugging me for years. Not knowing the real
>> life-time of the buffers in those structs without studying the code
>> very carefully seriously limits the flexibility and basically ensures
>> that you start memcpy'ing them, so for my use-case, i would save a lot
>
> so
> 1. Docs are crap
> 2. Users unneccessarily memcpy just to be sure as the docs dont
>    document when its actually needed
> 3. Isnt it logic to fix the docs here ?
>

Even if it was documented, from what i've found out so far, the
"logic" which determines when you can keep/use a buffer is not quite
sound in itself.
And even if it was documented to all extend, there are many cases
where you still need to memcpy which a proper refcounted buffer would
avoid (or i have to implement quite a bit of boilerplate logic around
it which is prone to error)

Refcounting is just the right way to approach this topic. Its a
well-known system for concepts like this.

>>
>> Another thing i never understood was the use of a separate frame
>> structure for lavfi, which slowly adopted all kinds of fields from
>> AVFrame, and if this would continue for much longer, would become
>> equally bloated as AVFrames - this change makes the whole system much
>> simpler (with only one bloated struct :p)
>
> If you use libavcodec together with libavfilter then the use of the
> same struct simplifies your application. If you use only one of them
> chances are it will get harder to setup the more generic struct.
> At least that was the case with ogg and vorbis libs which used the
> same structs.
> Depending on what you do and how its implemented either can be better.
> If well designed i agree a single structure should be better.
>
>

If its too hard to setup an AVFrame, then the AVFrame needs better
docs or a simpler design.
It should be easy to fill this central structure which is returned by
decoders, used/returned by lavfi, and used by encoders. Switching
struct in the middle of this process is a rather annoying step for
people which do actually use everything together.

>>
>> Personally i fail to see which use-cases actually get slower (possibly
>> except the really hacky ones), because the AVBuffer spec from Anton
>> i've seen is quite flexible, and could adapt to a lot of weird/hacky
>> internal usage.
>
> droped audio buffer pool, speedloss documented in git log
>
> Quoting Anton: "Missing frame buffer reuse in lavc. Preliminary tests
> show that this causes a measurable slowdown, so it should be fixed."

Since he himself says it needs fixing, and the patch is nowhere near
being merged yet, i fail to see how you can even use this as an
argument?
It'll be fixed, problem solved?

>
> overhead for thread sync is not negligible, I benchmarked the
> code i saw from libav, id have to redo it though if you want hard
> numbers

>From what i've seen, it greatly simplifies reference management in
threaded decoders. Considering the complex threaded decoders like
H.264 are not done yet (as stated by Anton), i would wait with any
proper benchmarks until then.

>
> If you interface with libavcodec/libavfilter and the buffers you have
> from whichever platform specific API (de)uxer, (decoder, display, ...
> arent refcounted but instead only valid till the next call or return.
> Then you do NOT need a copy now in many cases but will need one with
> the new API
> (at least thats how i understand it, please correct me
>  if i missed something)
>

It should be very much possible to use non-refcounted buffers still,
at least i saw this in one of the new lavu headers. If a
non-refcounted buffer is encountered and lavfi needs to store it for
later, it should copy it to a ref-counted buffer, like its done now
with buffers without the "keep" permission. At least i don't see how
the API would limit this by design.

>
>>
>> PS:
>> Access to internal frame data from outside ala postproc is a horrible
>
>
>> design to begin with, and exposing private frame decoding data just
>> for the purpose of statistics or the like also increase overall
>> complexity for very niche use-cases only, which is something that made
>> ffmpeg suffer greatly in simplicity over the years.
>
> If the existence of a field in AVFrame that you never read, write,
> init or other need to care about increases complexity for you.
> Then yes
>

You yourself said that AVFrame is too complicated/overloaded which is
why lavfi had its own frame struct.
And yes, i do consider a whole load of extra fields with next to zero
documentation very much added complexity.