[FFmpeg-devel] lavfi: push / poll / flush / drain implementation

[Michael Niedermayer]

On Thu, Mar 15, 2012 at 09:43:32PM +0100, Nicolas George wrote:
> The purpose of this mail is to summarize the ideas that have arisen recently
> on how the flow of frames in lavfi should work.
> 
> It's long, but there is a summary at the end.
> 
> First of all: I will use the words for video frames, but what I will write
> should apply to any kind of filters.
> 
> Currently, the general API is defined, but the fine points of how things
> should be done are still blurry. For that reason, some filters implement
> things differently from the majority, and it sometimes results in strange
> effects, such a filter working with -vf but not -f lavfi.
> 
> Establishing guidelines on how things should be done exactly should avoid
> that, and hopefully ensure that all filters are usable even in unusual
> circumstances.
> 
> Of course, guidelines are just that: any filter would be allowed to break
> them, provided it has good reason to do so and it is clearly documented.
> 
> 
> My first point will be, of course, poll_frame().
> 
> First, there is a basic flaw with poll_frame(): returning 0 may mean both
> "there is currently no frame available but there may be later" or "there
> never will be anything here". But this is not that big a problem, if it was
> only that, it would be an easy fix.
> 
> My second problem with poll_frame() is that it is very hard to implement
> with unusual filters. Consider vf_select: to implement it, it has to pump an
> unbounded amount of frames from its input, and because of that must
> implement a queue. The reason for that is that it often depends on
> information not yet available to the filter.
> 
> My third problem with poll_frame() is that it does a job very similar to
> request_frame(), but with a different code path, and keeping different code
> paths in sync hard. For that reason, some filters will behave differently
> whether poll_frame() has been called before request_frame().
> 
> I believe we can do without poll_frame() completely. That would make a lot
> of filters much much simpler.
> 
> The basic way is very simple: request_frame() returning 0 frame is a normal
> condition when no frame can be immediately produced. Reporting that
> condition with a specific return code (AVERROR(EAGAIN)) is an useful
> convenience.
> 
> The problem to eliminate poll_frame() is graphs that have several buffer
> sinks, possibly filling at different rates: poll_frame() is what tells how
> many frames we must extract from each sink so that they are not
> accumulating.
> 
> The solution I propose for that issue requires this guideline:
> 

> [passing] When a filter has had enough input to produce a frame, it should
> push it to its output without waiting request_frame().

I suggest to change this to:
When a filter has output-able frame(s) for an output link, it should
output them before buffering more output-able frames for the same
output link.

As example, if yadif gets a frame, it turns it in 2 fields and
outputs one and buffers the second.
If now it gets a request_frame call it outputs the 2nd and returns
to its initial state.
If OTOH it gets another push it returns (at least) all fields it has
buffered and stores the new fields for request_frame() to pull.

This should reduce CPU usage spikes due to push multiplication


> 
> Note that this does not apply to sources.
> 
> Also note that to achieve that, filters with several inputs need to
> implement some kind of buffering of their own. A common set of utility
> functions to do that properly would be in order.

as an alternative this could be implemented using existing fifo filters
that ping the following filter when new frames become available and
allow the following filter to inspect their whole que.
I do not know if its better to worse to your suggestion, actually it
looks pretty much identical from the outside.


[...]
> 
> About loops.
> 
> The possibility of make loops in the filter graph is a prerequisite, but
> loops can cause several problems.
> 
> Currently, with most filters, a loop will cause an infinite recursion both
> on poll_frame() and request_frame().
> 
> The solution I see to that problem is to add a flag / state field to all
> links, and reject, with as specific error code, nested calls on the same
> link.
> 
> Now, if all filters in the loop are "passing", then the loop may start
> issuing an infinite number of frames before request_frame() terminates. A
> loop needs to be "regulated".
> 
> Fortunately, a loop will almost always have an input: a filter with several
> input pads, and at least one of them coming from out of the loop. Hopefully,
> that filter will serve to regulate the loop, by waiting for frames on its
> external input.
> 
> To ensure that it will happen, the following guideline may be useful:
>

> [deterministic] A filter with several inputs must behave the same way
> whatever the order of arrival of frames on its different inputs, within
> reasonable limits.

This rule will be broken by filters that attempt to achive realtime
communication. An example would be a multiway teleconference where
3 people each see a split screen of 2 other people.
Here a combine filter would likely mix what it has and not wait long
for the other side if for whatever reason no data is received.
But i guess such filter in this configuration would not be used in a
loop
maybe some flag(s) should be added to filters so the core can know if
they are loop safe or not. This isnt really important ATM though

I agree with the remainder of your mail

Thanks

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Into a blind darkness they enter who follow after the Ignorance,
they as if into a greater darkness enter who devote themselves
to the Knowledge alone. -- Isha Upanishad
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20120317/150c4da1/attachment.asc>