[FFmpeg-devel] Evolution of lavfi's design and API

[Paul B Mahol]

Hi,

On Thu, Oct 30, 2014 at 1:59 PM, Michael Niedermayer <michaelni at gmx.at>
wrote:

> On Thu, Oct 30, 2014 at 11:50:46AM +0100, Stefano Sabatini wrote:
> > Sorry for the slow reply.
> >
> > On date Wednesday 2014-10-22 23:45:42 +0200, Nicolas George encoded:
> > > [ CCing Anton, as most that is written here also apply to libav too,
> and
> > > this would be a good occasion to try a cross-fork cooperation; if that
> is
> > > not wanted, please let us know so we can drop the cc. ]
> > >
> > > 1. Problems with the current design
> > >
> > >   1.1. Mixed input-/output-driven model
> > >
> > >     Currently, lavfi is designed to work in a mixed input-driven and
> > >     output-driven model. That means the application needs sometimes to
> add
> > >     input to buffersources and sometimes request output to
> buffersinks. This
> > >     is a bit of a nuisance, because it requires the application to do
> it
> > >     properly: adding input on the wrong input or requesting a frame on
> the
> > >     wrong output will cause extra memory consumption or latency.
> > >
> > >     With the libav API, it can not work at all since there is no
> mechanism
> > >     to determine which input needs a frame in order to proceed.
> > >
> > >     The libav API is clearly designed for a more output-driven
> > >     implementation, with FIFOs anywhere to prevent input-driven frames
> to
> > >     reach unready filters. Unfortunately, since it is impossible from
> the
> > >     outside to guess what output will get a frame next, that can cause
> > >     frames to accumulate anywhere in the filter graph, eating a lot of
> > >     memory unnecessarily.
> > >
> > >     FFmpeg's API has eliminated FIFOs in favour of queues in filters
> that
> > >     need it, but these queues can not be controlled for unusual filter
> > >     graphs with extreme needs. Also, there still is an implicit FIFO
> inside
> > >     buffersink.
> > >
> > >   1.2. Recursive implementation
> > >
> > >     All work in a filter graph is triggered by recursive invocations
> of the
> > >     filters' methods. It makes debugging harder. It also can lead to
> large
> > >     stack usage and makes frame- and filter-level multithreading
> harder to
> > >     implement. It also prevents some diagnosis from working reliably.
> > >
> > >   1.3. EOF handling
> > >
> > >     Currently, EOF is propagated only through the return value of the
> > >     request_frame() method. That means it only works in an
> output-driven
> > >     scheme. It also means that it has no timestamp attached to it;
> this is
> > >     an issue for filters where the duration of the last frame is
> relevant,
> > >     like vf_fps.
> > >
> > >   1.4. Latency
> > >
> > >     Some filters need to know the timestamp of the next frame in order
> to
> > >     know when the current frame will stop and be able to process it:
> > >     overlay, fps are two examples. These filters will introduce a
> latency of
> > >     one input frame that could otherwise be avoided.
> > >
> > >   1.5. Timestamps
> > >
> > >     Some filters do not care about timestamps at all. Some check and
> have a
> > >     proper handling of NOPTS values. Some filters just assume the
> frames
> > >     will have timestamps, and possibly make extra assumptions on that:
> > >     monotony, consistency, etc. That is an inconsistent mess.
> > >
> > >   1.6. Sparse streams
> > >
> > >     There is a more severe instance of the latency issue when the input
> > >     comes from an interleaved sparse stream: in that case, waiting for
> the
> > >     next frame in order to find the end of the current one may require
> > >     demuxing a large chunk of input, in turn provoking a lot of
> activity on
> > >     other inputs of the graph.
> >
> > Other issues.
> >
>
> > S1. the filtergraph can't properly readapt to mid-stream
> > changes involving assumed invariants (aspect ratio, size, timebase,
> > pixel format, sample_rate). Indeed the framework was designed as
> > though some of these properties (the ones defined by query_formats)
> > were not allowed to change.
>
> no, no and no :)
> the filtergraph should be able adapt to some changes like aspect,
> resolution and pixel / sample format. These are not invariants, some
> of this definitly worked when i tried it long ago
> i posted a (incomplete) patchset that fixes bugs in this relation
> long ago There are filters that can perfectly well handle changes in
> resolution, aspect, formats, ...
> and there are filters which are buggy but could when they are fixed
> also equally well support such changes
> and there are filters which fundamentally do not support some changes,
> these need to either be reinited and loose state/history or a
> scale/aresample be inserted before them when changes on their input
> occur
> for some filters reinit is not appropriate, examples are things that
> are intended to collect global statistics
> Also scale / aresample filters can serve as
> "parameter change barriers", filters afterwards do not need to deal
> with such changes
>
>
> >
> > S2. Another problem is that we initialize the filter before the
> > filtergraph, so for example the single filter can't readapt to the
> > filtergraph topology. For example it would be useful to have the split
> > filter to change the number of outputs depending on the number of
> > outputs specified, but this can't be easily achieved. (That's in my
> > opinion a minor problem though).
> >
>
> > S3. It is not possible to direct commands towards a specific
> > filter. For this we can add an ID to each filter instance. We could
> > have something has:
> > color:left_color=c=red   [left]
> > color:right_color=c=blue [right]
> >
> > then you can send commands (e.g. with zmq) with:
> > echo left_color c yellow | tools/zmqsend
>
> c sends to all filters
> C sends to a specific filter
> What problem is there with C ?
>
>
> [...]
> > > 2. Proposed API changes
> > >
> > >   To fix/enhance all these issues, I believe a complete rethink of the
> > >   scheduling design of the library is necessary. I propose the
> following
> > >   changes.
> > >
> > >   Note: some of these changes are not 100% related to the issues I
> raised,
> > >   but looked like a good idea while thinking on an API rework.
> > >
> > >   2.1. AVFrame.duration
> > >
> > >     Add a duration field to AVFrame; if set, it indicates the duration
> of
> > >     the frame. Thus, it becomes unnecessary to wait for the next frame
> to
> > >     know when the current frame stops, reducing the latency.
> > >
> > >     Another solution would be to add a dedicated function on buffersrc
> to
> > >     inject a timestamp for end or activity on a link. That would avoid
> the
> > >     need of adding a field to AVFrame.
> >
> > Currently we have pkt_duration in AVFrame. The main problem is that we
> > would need to rescale the duration accordingly in all filters when we
> > change the timebase.
> >
> > Alternatively we should add a timebase value to AVFrame. Currently
> > that information is stored externally, and this was never easy to
> > handle. For example the timestamp in a filter is currently interpreted
> > according to the input link time base. Unfortunately this will
> > introduce redundancy.
>
> yes, also theres a somewhat related patch:
> Benoit Fouet    (4.5K) [FFmpeg-devel] [PATCH] libavu: add pkt_timebase to
> AVFrame.
>
> also speaking of patches, comments for
> [FFmpeg-devel] [PATCH] avutil/avstring: support asymmetric escaping in
> av_get_token()
> are welcome
> i cant just apply it as its not 100% compatible with past formating
> though its 99.9% and mayb 100% for actual used strings
> so this patch really needs comments from people ...
>
> [...]
> --
> Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
>
> While the State exists there can be no freedom; when there is freedom there
> will be no State. -- Vladimir Lenin
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
>
Nicolas, what is status of this?

I'm currently interested in frame multithreading in lavfi.