[FFmpeg-devel] [RFC] Flush API for libavfilter
Michael Niedermayer
michaelni at gmx.at
Wed Nov 30 17:29:31 CET 2011
On Wed, Nov 30, 2011 at 04:12:27PM +0100, Michael Niedermayer wrote:
> On Tue, Nov 29, 2011 at 08:09:39PM +0100, Michael Niedermayer wrote:
> > On Tue, Nov 29, 2011 at 07:13:33PM +0100, Stefano Sabatini wrote:
> > > On date Monday 2011-10-24 19:12:57 +0200, Michael Niedermayer encoded:
> > > > On Mon, Oct 24, 2011 at 06:49:24PM +0200, Stefano Sabatini wrote:
> > > > > On date Thursday 2011-10-20 17:37:18 +0200, Michael Niedermayer encoded:
> > > > > > On Thu, Oct 20, 2011 at 05:06:21PM +0200, Stefano Sabatini wrote:
> > > > > > > On date Wednesday 2011-03-23 06:25:15 +0200, Mina Nagy Zaki encoded:
> > > > > > > > On Tuesday 22 March 2011 17:35:39 Stefano Sabatini wrote:
> > > > > > > [...]
> > > > > > > > > Check: cmdutils.c:get_filtered_audio_buffer(), it implements a pull
> > > > > > > > > model, correct me if I'm wrong.
> > > > > > > >
> > > > > > > > I have already checked that, which is what lead me to the question in the first
> > > > > > > > place. I can't really read AVERROR_EOF from request_frame() because audio
> > > > > > > > filters don't(/shouldn't?) use it. get_filtered_audio_buffer() does a
> > > > > > > > request_frame() on aout which propagates all the way to asrc (without passing
> > > > > > > > through intermediate filters, because they don't/shouldn't use it) which then
> > > > > > > > does a filter_samples that propagates all the way to aout! If I try to
> > > > > > > > request_frame from the previous buffer it will reach asrc then eventually lead
> > > > > > > > to calling my own effect's filter_samples, so now do I filter in filter_samples or
> > > > > > > > request_frame :D And in any case, asrc never returns AVERROR_EOF anyway.
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I was trying to get the af_sox.c filter in shape when I stumbled
> > > > > > > across this.
> > > > > > >
> > > > > > > Resuming, the problem is that when a filterchain source returns
> > > > > > > AVERROR_EOF (e.g. becase the generated media has terminated) it is not
> > > > > > > possible to notify the intermediate filters, which may keep cached
> > > > > > > data which is never released.
> > > > > > >
> > > > > > > For example some sox filters cache data, which is supposed to be
> > > > > > > flushed through the drain API when *all* the input has been already
> > > > > > > provided (so they won't return data *at all* if such a mechanism is
> > > > > > > not implemented).
> > > > > > >
> > > > > > > This can't be done with the current API (neither with audio nor with
> > > > > > > video).
> > > > > >
> > > > > > I dont understand the problem for video
> > > > > >
> > > > > > At EOF
> > > > > > A filter currently can, when it receives a request_frame()
> > > > > > push the next cached frame out via
> > > > > > avfilter_start_frame + avfilter_draw_slice + avfilter_end_frame
> > > > > > vf_yadif does this currently for different reasons
> > > > > >
> > > > >
> > > > > > And a filter knows its at EOF when poll & request_frame() signal so
> > > > > > that is via 0 and AVERROR_EOF
> > > > >
> > > > > You need to implement poll() in the filter for applying this logic,
> > > > > and poll() may return 0 even when no end of file was reached
> > > > > (e.g. when the buffer source was emptied).
> > > > >
> > > > > >
> > > > > > The pull system is needed or more complex networks of filters will
> > > > > > have all kinds of issues.
> > > > > > one example is a graph with 2 inputs and 2 outputs, we assume
> > > > > > one wants to mux these 2 outputs together in the same file, the inputs
> > > > > > are from seperate files. With a pull system we know from which file
> > > > > > to read next with a push system we dont and could end up having to
> > > > > > buffer the whole decoded file in memory. Not to mention the delay
> > > > > > So i think audio should use request_frame() or a equivalent too
> > > > >
> > > > > I'm not proposing to drop request_frame() (which is currently
> > > > > implemented for audio).
> > > > >
> > > > > But in the case of sox a filter may release data all at once when the
> > > > > end of file is reached, and for doing this you need to know when the
> > > > > input data is terminated (sox API is meant for batch processing, so
> > > > > yes this will not be usable for real-time filtering).
> > > >
> > > > iam still not understanding the problem. Filters dont run in an
> > > > isolated environment, they get calls from outside
> > > >
> > > > for example data can be pushed into a filter, then it has data and
> > > > thus not EOF
> > > > or data can be pulled out of a filter via request_frame() and for
> > > > that a filter would calls its predecessors request_frame() and this
> > > > either would produce data or a AVERROR_EOF
> > > >
> > > > from an outside view when EOF is reached on the input nothing special
> > > > should happen, the pulling from the end would just continue until
> > > > a EOF error is pulled out.
> > > >
> > > > where is the flaw in my logic ?
> > >
> > > Sorry for the slow reply.
> > >
> > > With sox API, data is cached in the filter, and released only when a
> > > particular API is called (the so called drain API).
> > >
> > > Problem is, in order to flush this data you need to know that the
> > > input ended, or in other words you need to signal the sox filter that
> > > the input source reached EOF, and give a chance to the filter to flush
> > > all the cached data.
> > >
> > > Is this currently possible? No, what happened now is that
> > > request_frame() is recursively called on the current filter input. The
> > > filtergraph source will eventually return EOF, the sink filter is
> > > notified that the input reached EOF but the data cached in the
> > > intermediate filters is never released.
> >
> > static int request_frame(AVFilterLink *link)
> > {
> > AVFilterContext *ctx = link->src;
> > soxctx->did_we_output_a_frame= 0;
> >
> > do {
> > int ret;
> >
> > if ((ret = avfilter_request_frame(link->src->inputs[0]))){
> > if(ret == EOF){
> > sox_drain()
> > if(soxctx->did_we_output_a_frame)
> > return 0;
> > }
> > return ret;
> > }
> > } while (!soxctx->did_we_output_a_frame);
> >
> > return 0;
> > }
> >
> > filter_samples(){
> >
> > if we output something here
> > soxctx->did_we_output_a_frame=1;
> > }
> >
> >
> >
> > >
> > > I see two mechanisms for dealing with this:
> > >
> > > 1. a filter may request frames and *force* flushing
> > > (avfilter_request_frame() + FLUSH flag). In this case all the filters
> > > are forced to flush cached data (via the drain API in case of libsox).
> > >
> > > 2. we may implement an avfilter_flush(), to be called on the filter
> > > output link, to force flushing the cached frames. This could be called
> > > by a source on all its output links when it reaches EOF.
> > >
> > > I'll try to come up with a proof of concept soon.
> >
> > with:
> > soxfilter1 -> soxfilter2 ->out
> >
> > you need to flush soxfilter1 first then normally filter with 2
> > and only once it runs dry flush it.
>
> also with filters that mix video and audio links there may be problems
> with mixing different APIs for flushing.
> I mean for example a filter that has video output and audio input
> wont recive audio specific calls on its output side.
>
> another problem i see is that a filter that has 2 inputs
> and one output, the output cannot know when to flush.
> depending on the kind of filter flush may be needed when any inputs
> are EOF or it may be needed only when all inputs are EOF
> Only the filter itself knows when it hits EOF ...
>
> another example would be
>
> in->truncate->sox->out
>
> in is a life stream without EOF
> the truncate filter would reach EOF at 10 second (maybe it stops based
> on detecting commercials or maybe it detects silence as a stop
> criterion)
>
> when would you call flush here?
>
> with request_frame, out would call request_frame of sox which in
> turn would call truncates request_frame which would say EOF and
> sox would then call drain(), return its last samples.
> and on the next run the same would happen but as drain would produce
> nothing sox would return EOF too, ending playback
The only alternative i can see to the request_frame() based system
would be to push in EOF (size=0) packets into filter_samples from
the input to indicate EOF and start drain.
This from a quick look would be probably slightly less likely for the
programmer to mis-implement in complex filters because the information
would be with the affected input stream.
But it would require the video filters to be changed to this system
too
instead of size=0 input, a seperate flush function could be used but
this would be on the input side of a filter and not the output.
[...]
--
Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
No snowflake in an avalanche ever feels responsible. -- Voltaire
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20111130/1e4a681c/attachment.asc>
More information about the ffmpeg-devel
mailing list