[FFmpeg-devel] [RFC] Flush API for libavfilter

[Stefano Sabatini]

On date Monday 2011-10-24 19:12:57 +0200, Michael Niedermayer encoded:
> On Mon, Oct 24, 2011 at 06:49:24PM +0200, Stefano Sabatini wrote:
> > On date Thursday 2011-10-20 17:37:18 +0200, Michael Niedermayer encoded:
> > > On Thu, Oct 20, 2011 at 05:06:21PM +0200, Stefano Sabatini wrote:
> > > > On date Wednesday 2011-03-23 06:25:15 +0200, Mina Nagy Zaki encoded:
> > > > > On Tuesday 22 March 2011 17:35:39 Stefano Sabatini wrote:
> > > > [...]
> > > > > > Check: cmdutils.c:get_filtered_audio_buffer(), it implements a pull
> > > > > > model, correct me if I'm wrong.
> > > > > 
> > > > > I have already checked that, which is what lead me to the question in the first 
> > > > > place. I can't really read AVERROR_EOF from request_frame() because audio 
> > > > > filters don't(/shouldn't?) use it. get_filtered_audio_buffer() does a 
> > > > > request_frame() on aout which propagates all the way to asrc (without passing 
> > > > > through intermediate filters, because they don't/shouldn't use it) which then 
> > > > > does a filter_samples that propagates all the way to aout! If I try to 
> > > > > request_frame from the previous buffer it will reach asrc then eventually lead 
> > > > > to calling my own effect's filter_samples, so now do I filter in filter_samples or 
> > > > > request_frame :D  And in any case, asrc never returns AVERROR_EOF anyway.
> > > > 
> > > > Hi,
> > > > 
> > > > I was trying to get the af_sox.c filter in shape when I stumbled
> > > > across this.
> > > > 
> > > > Resuming, the problem is that when a filterchain source returns
> > > > AVERROR_EOF (e.g. becase the generated media has terminated) it is not
> > > > possible to notify the intermediate filters, which may keep cached
> > > > data which is never released.
> > > > 
> > > > For example some sox filters cache data, which is supposed to be
> > > > flushed through the drain API when *all* the input has been already
> > > > provided (so they won't return data *at all* if such a mechanism is
> > > > not implemented).
> > > > 
> > > > This can't be done with the current API (neither with audio nor with
> > > > video).
> > > 
> > > I dont understand the problem for video
> > > 
> > > At EOF
> > > A filter currently can, when it receives a request_frame()
> > > push the next cached frame out via
> > > avfilter_start_frame + avfilter_draw_slice + avfilter_end_frame
> > > vf_yadif does this currently for different reasons
> > > 
> > 
> > > And a filter knows its at EOF when poll & request_frame() signal so
> > > that is via 0 and AVERROR_EOF
> > 
> > You need to implement poll() in the filter for applying this logic,
> > and poll() may return 0 even when no end of file was reached
> > (e.g. when the buffer source was emptied).
> > 
> > > 
> > > The pull system is needed or more complex networks of filters will
> > > have all kinds of issues.
> > > one example is a graph with 2 inputs and 2 outputs, we assume
> > > one wants to mux these 2 outputs together in the same file, the inputs
> > > are from seperate files. With a pull system we know from which file
> > > to read next with a push system we dont and could end up having to
> > > buffer the whole decoded file in memory. Not to mention the delay
> > > So i think audio should use request_frame() or a equivalent too
> > 
> > I'm not proposing to drop request_frame() (which is currently
> > implemented for audio).
> > 
> > But in the case of sox a filter may release data all at once when the
> > end of file is reached, and for doing this you need to know when the
> > input data is terminated (sox API is meant for batch processing, so
> > yes this will not be usable for real-time filtering).
> 
> iam still not understanding the problem. Filters dont run in an
> isolated environment, they get calls from outside
> 
> for example data can be pushed into a filter, then it has data and
> thus not EOF
> or data can be pulled out of a filter via request_frame() and for
> that a filter would calls its predecessors request_frame() and this
> either would produce data or a AVERROR_EOF
> 
> from an outside view when EOF is reached on the input nothing special
> should happen, the pulling from the end would just continue until
> a EOF error is pulled out.
> 
> where is the flaw in my logic ?

Sorry for the slow reply.

With sox API, data is cached in the filter, and released only when a
particular API is called (the so called drain API).

Problem is, in order to flush this data you need to know that the
input ended, or in other words you need to signal the sox filter that
the input source reached EOF, and give a chance to the filter to flush
all the cached data.

Is this currently possible? No, what happened now is that
request_frame() is recursively called on the current filter input. The
filtergraph source will eventually return EOF, the sink filter is
notified that the input reached EOF but the data cached in the
intermediate filters is never released.

I see two mechanisms for dealing with this:

1. a filter may request frames and *force* flushing
(avfilter_request_frame() + FLUSH flag). In this case all the filters
are forced to flush cached data (via the drain API in case of libsox).

2. we may implement an avfilter_flush(), to be called on the filter
output link, to force flushing the cached frames. This could be called
by a source on all its output links when it reaches EOF.

I'll try to come up with a proof of concept soon.
-- 
FFmpeg = Faithful and Funny Multimedia Ponderous Experimenting Governor