[FFmpeg-devel] Evolution of lavfi's design and API

[Michael Niedermayer]

On Thu, Oct 30, 2014 at 11:50:46AM +0100, Stefano Sabatini wrote:
> Sorry for the slow reply.
> 
> On date Wednesday 2014-10-22 23:45:42 +0200, Nicolas George encoded: 
> > [ CCing Anton, as most that is written here also apply to libav too, and
> > this would be a good occasion to try a cross-fork cooperation; if that is
> > not wanted, please let us know so we can drop the cc. ]
> > 
> > 1. Problems with the current design
> > 
> >   1.1. Mixed input-/output-driven model
> > 
> >     Currently, lavfi is designed to work in a mixed input-driven and
> >     output-driven model. That means the application needs sometimes to add
> >     input to buffersources and sometimes request output to buffersinks. This
> >     is a bit of a nuisance, because it requires the application to do it
> >     properly: adding input on the wrong input or requesting a frame on the
> >     wrong output will cause extra memory consumption or latency.
> > 
> >     With the libav API, it can not work at all since there is no mechanism
> >     to determine which input needs a frame in order to proceed.
> > 
> >     The libav API is clearly designed for a more output-driven
> >     implementation, with FIFOs anywhere to prevent input-driven frames to
> >     reach unready filters. Unfortunately, since it is impossible from the
> >     outside to guess what output will get a frame next, that can cause
> >     frames to accumulate anywhere in the filter graph, eating a lot of
> >     memory unnecessarily.
> > 
> >     FFmpeg's API has eliminated FIFOs in favour of queues in filters that
> >     need it, but these queues can not be controlled for unusual filter
> >     graphs with extreme needs. Also, there still is an implicit FIFO inside
> >     buffersink.
> > 
> >   1.2. Recursive implementation
> > 
> >     All work in a filter graph is triggered by recursive invocations of the
> >     filters' methods. It makes debugging harder. It also can lead to large
> >     stack usage and makes frame- and filter-level multithreading harder to
> >     implement. It also prevents some diagnosis from working reliably.
> > 
> >   1.3. EOF handling
> > 
> >     Currently, EOF is propagated only through the return value of the
> >     request_frame() method. That means it only works in an output-driven
> >     scheme. It also means that it has no timestamp attached to it; this is
> >     an issue for filters where the duration of the last frame is relevant,
> >     like vf_fps.
> > 
> >   1.4. Latency
> > 
> >     Some filters need to know the timestamp of the next frame in order to
> >     know when the current frame will stop and be able to process it:
> >     overlay, fps are two examples. These filters will introduce a latency of
> >     one input frame that could otherwise be avoided.
> > 
> >   1.5. Timestamps
> > 
> >     Some filters do not care about timestamps at all. Some check and have a
> >     proper handling of NOPTS values. Some filters just assume the frames
> >     will have timestamps, and possibly make extra assumptions on that:
> >     monotony, consistency, etc. That is an inconsistent mess.
> > 
> >   1.6. Sparse streams
> > 
> >     There is a more severe instance of the latency issue when the input
> >     comes from an interleaved sparse stream: in that case, waiting for the
> >     next frame in order to find the end of the current one may require
> >     demuxing a large chunk of input, in turn provoking a lot of activity on
> >     other inputs of the graph.
> 
> Other issues.
> 

> S1. the filtergraph can't properly readapt to mid-stream
> changes involving assumed invariants (aspect ratio, size, timebase,
> pixel format, sample_rate). Indeed the framework was designed as
> though some of these properties (the ones defined by query_formats)
> were not allowed to change.

no, no and no :)
the filtergraph should be able adapt to some changes like aspect,
resolution and pixel / sample format. These are not invariants, some
of this definitly worked when i tried it long ago
i posted a (incomplete) patchset that fixes bugs in this relation
long ago There are filters that can perfectly well handle changes in
resolution, aspect, formats, ...
and there are filters which are buggy but could when they are fixed
also equally well support such changes
and there are filters which fundamentally do not support some changes,
these need to either be reinited and loose state/history or a
scale/aresample be inserted before them when changes on their input
occur
for some filters reinit is not appropriate, examples are things that
are intended to collect global statistics
Also scale / aresample filters can serve as
"parameter change barriers", filters afterwards do not need to deal
with such changes


> 
> S2. Another problem is that we initialize the filter before the
> filtergraph, so for example the single filter can't readapt to the
> filtergraph topology. For example it would be useful to have the split
> filter to change the number of outputs depending on the number of
> outputs specified, but this can't be easily achieved. (That's in my
> opinion a minor problem though).
> 

> S3. It is not possible to direct commands towards a specific
> filter. For this we can add an ID to each filter instance. We could
> have something has:
> color:left_color=c=red   [left]
> color:right_color=c=blue [right]
> 
> then you can send commands (e.g. with zmq) with:
> echo left_color c yellow | tools/zmqsend

c sends to all filters
C sends to a specific filter
What problem is there with C ?


[...]
> > 2. Proposed API changes
> > 
> >   To fix/enhance all these issues, I believe a complete rethink of the
> >   scheduling design of the library is necessary. I propose the following
> >   changes.
> > 
> >   Note: some of these changes are not 100% related to the issues I raised,
> >   but looked like a good idea while thinking on an API rework.
> > 
> >   2.1. AVFrame.duration
> > 
> >     Add a duration field to AVFrame; if set, it indicates the duration of
> >     the frame. Thus, it becomes unnecessary to wait for the next frame to
> >     know when the current frame stops, reducing the latency.
> > 
> >     Another solution would be to add a dedicated function on buffersrc to
> >     inject a timestamp for end or activity on a link. That would avoid the
> >     need of adding a field to AVFrame.
> 
> Currently we have pkt_duration in AVFrame. The main problem is that we
> would need to rescale the duration accordingly in all filters when we
> change the timebase.
> 
> Alternatively we should add a timebase value to AVFrame. Currently
> that information is stored externally, and this was never easy to
> handle. For example the timestamp in a filter is currently interpreted
> according to the input link time base. Unfortunately this will
> introduce redundancy.

yes, also theres a somewhat related patch:
Benoit Fouet    (4.5K) [FFmpeg-devel] [PATCH] libavu: add pkt_timebase to AVFrame.

also speaking of patches, comments for
[FFmpeg-devel] [PATCH] avutil/avstring: support asymmetric escaping in av_get_token()
are welcome
i cant just apply it as its not 100% compatible with past formating
though its 99.9% and mayb 100% for actual used strings
so this patch really needs comments from people ...

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

While the State exists there can be no freedom; when there is freedom there
will be no State. -- Vladimir Lenin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: Digital signature
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20141030/701e5cea/attachment.asc>