[FFmpeg-devel] lavfi: push / poll / flush / drain implementation

[Nicolas George]

The purpose of this mail is to summarize the ideas that have arisen recently
on how the flow of frames in lavfi should work.

It's long, but there is a summary at the end.

First of all: I will use the words for video frames, but what I will write
should apply to any kind of filters.

Currently, the general API is defined, but the fine points of how things
should be done are still blurry. For that reason, some filters implement
things differently from the majority, and it sometimes results in strange
effects, such a filter working with -vf but not -f lavfi.

Establishing guidelines on how things should be done exactly should avoid
that, and hopefully ensure that all filters are usable even in unusual
circumstances.

Of course, guidelines are just that: any filter would be allowed to break
them, provided it has good reason to do so and it is clearly documented.


My first point will be, of course, poll_frame().

First, there is a basic flaw with poll_frame(): returning 0 may mean both
"there is currently no frame available but there may be later" or "there
never will be anything here". But this is not that big a problem, if it was
only that, it would be an easy fix.

My second problem with poll_frame() is that it is very hard to implement
with unusual filters. Consider vf_select: to implement it, it has to pump an
unbounded amount of frames from its input, and because of that must
implement a queue. The reason for that is that it often depends on
information not yet available to the filter.

My third problem with poll_frame() is that it does a job very similar to
request_frame(), but with a different code path, and keeping different code
paths in sync hard. For that reason, some filters will behave differently
whether poll_frame() has been called before request_frame().

I believe we can do without poll_frame() completely. That would make a lot
of filters much much simpler.

The basic way is very simple: request_frame() returning 0 frame is a normal
condition when no frame can be immediately produced. Reporting that
condition with a specific return code (AVERROR(EAGAIN)) is an useful
convenience.

The problem to eliminate poll_frame() is graphs that have several buffer
sinks, possibly filling at different rates: poll_frame() is what tells how
many frames we must extract from each sink so that they are not
accumulating.

The solution I propose for that issue requires this guideline:

[passing] When a filter has had enough input to produce a frame, it should
push it to its output without waiting request_frame().

Note that this does not apply to sources.

Also note that to achieve that, filters with several inputs need to
implement some kind of buffering of their own. A common set of utility
functions to do that properly would be in order.

If all filters in a graph respect that guideline, then frames will never
accumulate in the middle of the filter, except when it is logically
necessary, they will directly reach the sinks. Then, how many frames must be
read on a buffer sink is: exactly as much as currently buffered.

The process of filtering frames becomes then: place frames at input buffers
if necessary; request a frame on one of the buffer sinks: the request
recurses through the graph to all the sources, who produce one step of
frames if possible, these frames traverse the graph and reach the sinks;
then reap all available frames in all buffer sinks.

There is a flaw with that solution: filters that duplicate frames plus
computationally heavy filters may cause an irregular output, unsuited for
real-time applications. But I believe the solution is easy: if this is an
issue, use a different kind of sinks.


About loops.

The possibility of make loops in the filter graph is a prerequisite, but
loops can cause several problems.

Currently, with most filters, a loop will cause an infinite recursion both
on poll_frame() and request_frame().

The solution I see to that problem is to add a flag / state field to all
links, and reject, with as specific error code, nested calls on the same
link.

Now, if all filters in the loop are "passing", then the loop may start
issuing an infinite number of frames before request_frame() terminates. A
loop needs to be "regulated".

Fortunately, a loop will almost always have an input: a filter with several
input pads, and at least one of them coming from out of the loop. Hopefully,
that filter will serve to regulate the loop, by waiting for frames on its
external input.

To ensure that it will happen, the following guideline may be useful:

[deterministic] A filter with several inputs must behave the same way
whatever the order of arrival of frames on its different inputs, within
reasonable limits.

Rationale: that order is an implementation detail and can change without
notice.

If the filter at the input is not enough to regulate the loop (possibly when
EOF is reached on the external input), I believe this warrants an exception:
a special non-"passing" filter to do the regulation.


How to notify EOF and flush.

All we need to ensure that filters that have internal buffers are flushed is
a way of notifying the end of frames on a link is reached. I see two obvious
ways of doing it without changing the methods: pushing a special EOF frame
(maybe NULL can do the trick), or having request_frame() return AVERROR_EOF.

I believe the second solution is better, because it does not require any
changes at all for filters that do not need flushing, such as simple 1-to-1
filters: avfilter_request_frame() already forwards the return code.


An additional issue.

Filters with several inputs sometimes follow the following reasoning: "if I
get a frame on input #0, I need one on input #1 too, so request() it." But
I believe this reasoning is flawed: if I (the filter) was requested a frame
on my output, then my own request_frame() function will probably request on
input #1. And if not, I have no reason to make efforts to produce output.

Therefore, I propose another guideline:

[directional] For filters, request_frame() on an output should result in
only request_frame()s on inputs (that is already ensured by "passing", in
fact), and start_frame() on an input should result in only start_frame() on
outputs; the flow of requests/responses should not change direction.


Summary: practical proposal

* Ensure that all (most) filters are "passing", "deterministic" and
  "directional", as defined above. With that, we can be sure that frames
  will flow correctly in the graph.

* Eliminate poll_frame(); instead, allow request_frame() to return no frame
  at all; report that circumstance as AVERROR(EAGAIN) as a convenience.

* To filter frames using buffer sinks:
  - place frames at the sources;
  - request a frame on one sink (cycling evenly);
  - let the frames flow in the graph;
  - reap all frames at all sinks.

  (Imagine the beginning of some pool game: I hit the white ball
  (request_frame()), and a few seconds later a lot of coloured balls come
  back to me (start_frame() on the sinks).

* To allow loops, ensure that request_frame() can not be recursively nested
  on the same link, with a flag; instead, return an error. Also, ensure that
  all loops are "regulated" by at least one filter (if necessary, one that
  is not "passing").

* To flush frames at EOF, get request_frame to return AVERROR_EOF.

I believe the work to implement that is actually quite small: some fixes in
a few filters, especially vf_overlay, plus the patch series I posted.

Regards,

-- 
  Nicolas George
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20120315/b6ad46b0/attachment.asc>