[FFmpeg-devel] lavfi: push / poll / flush / drain implementation

[Stefano Sabatini]

On date Thursday 2012-03-15 21:43:32 +0100, Nicolas George encoded:
[...]
> I believe we can do without poll_frame() completely. That would make a lot
> of filters much much simpler.

I'm not against this if it can be done and we won't lose features by
doing so. Another concern is if we should do it by retaining backward
API/ABI compatibility.

> The basic way is very simple: request_frame() returning 0 frame is a normal
> condition when no frame can be immediately produced. Reporting that
> condition with a specific return code (AVERROR(EAGAIN)) is an useful
> convenience.
> 
> The problem to eliminate poll_frame() is graphs that have several buffer
> sinks, possibly filling at different rates: poll_frame() is what tells how
> many frames we must extract from each sink so that they are not
> accumulating.

Note that lavdev/lavfi doesn't use this mechanism at all (it only
relies of request_frame(), called through the buffersink
interface).

[...]
> [deterministic] A filter with several inputs must behave the same way
> whatever the order of arrival of frames on its different inputs, within
> reasonable limits.
> 
> Rationale: that order is an implementation detail and can change without
> notice.

I'm uneasy about that "reasonable limits". In general this condition
seems very difficult to specify with precision. Also the order of arrival
may affect which internal buffers are full/empty, which may in turn
affect the behavior of the filter.

For example the overlay filter may re-use an old cached buffer when no
new ones are still available, so in this case the behavior depends on
the *sequence of events* (inputs and commands) operating on the
filter.

> 
> If the filter at the input is not enough to regulate the loop (possibly when
> EOF is reached on the external input), I believe this warrants an exception:
> a special non-"passing" filter to do the regulation.
>
> 

> How to notify EOF and flush.
> 
> All we need to ensure that filters that have internal buffers are flushed is
> a way of notifying the end of frames on a link is reached. I see two obvious
> ways of doing it without changing the methods: pushing a special EOF frame
> (maybe NULL can do the trick), or having request_frame() return AVERROR_EOF.
> 
> I believe the second solution is better, because it does not require any
> changes at all for filters that do not need flushing, such as simple 1-to-1
> filters: avfilter_request_frame() already forwards the return code.

Pushing a frame: this is not always possible. Suppose that you build a
player based on a movie source connected to an output device, in this
case you only have control on the sink, so you can only request frames.

Returning EOF: this was already proposed, check here if you missed the
thread:
http://thread.gmane.org/gmane.comp.video.ffmpeg.devel/136130

In FFmbc there is an optional flag in avfilter_poll_frame(), which is
used to force a flush from the pulling side.

> An additional issue.
> 
> Filters with several inputs sometimes follow the following reasoning: "if I
> get a frame on input #0, I need one on input #1 too, so request() it." But
> I believe this reasoning is flawed: if I (the filter) was requested a frame
> on my output, then my own request_frame() function will probably request on
> input #1. And if not, I have no reason to make efforts to produce output.
> 
> Therefore, I propose another guideline:
> 
> [directional] For filters, request_frame() on an output should result in
> only request_frame()s on inputs (that is already ensured by "passing", in
> fact), and start_frame() on an input should result in only start_frame() on
> outputs; the flow of requests/responses should not change direction.

This is possibly more complicate. Consider the case of overlay. When a
frame is requested, it makes the effort of returning a frame.

For keeping synchronization with the main input, it needs to cache two
overlay buffers, which may be available or not depending on the exact
past history of events and on the timestamps of the cached frames. So
in this case you only know if you need another frame when you're
already pushing frames.

Alternatively you may not push frames in this case, and delay it to
the following request_frame() call.

The problem here is to define how a filter should behave when a
request_frame() is called, in particular if it should try hard at
sending a frame or should simply notify the filterchain that it can't
issue a frame at the current time (EAGAIN may do), and if this
mechanism may have bad implications.

> Summary: practical proposal
> 
> * Ensure that all (most) filters are "passing", "deterministic" and
>   "directional", as defined above. With that, we can be sure that frames
>   will flow correctly in the graph.
 
> * Eliminate poll_frame(); instead, allow request_frame() to return no frame
>   at all; report that circumstance as AVERROR(EAGAIN) as a convenience.

OK if works.
 
> * To filter frames using buffer sinks:
>   - place frames at the sources;
>   - request a frame on one sink (cycling evenly);
>   - let the frames flow in the graph;
>   - reap all frames at all sinks.
> 
>   (Imagine the beginning of some pool game: I hit the white ball
>   (request_frame()), and a few seconds later a lot of coloured balls come
>   back to me (start_frame() on the sinks).
> 

> * To allow loops, ensure that request_frame() can not be recursively nested
>   on the same link, with a flag; instead, return an error. Also, ensure that
>   all loops are "regulated" by at least one filter (if necessary, one that
>   is not "passing").

Yes, also a POC filter for this may be useful, patches or even an idea
of a simple filter which may implement a loop are welcome.

> * To flush frames at EOF, get request_frame to return AVERROR_EOF.

Isn't this already implemented? (at least movie works like this)

> 
> I believe the work to implement that is actually quite small: some fixes in
> a few filters, especially vf_overlay, plus the patch series I posted.
-- 
FFmpeg = Formidable & Faithful Minimalistic Pitiful Erratic Gadget