[FFmpeg-soc] Libavfilter RFC: problem with ffmpeg.c/ffplay.c integration

[Michael Niedermayer]

On Mon, Jan 07, 2008 at 11:56:32PM -0500, Bobby Bingham wrote:
> On Mon, 7 Jan 2008 23:07:19 +0100
> Michael Niedermayer <michaelni at gmx.at> wrote:
> 
> > On Wed, Jan 02, 2008 at 10:00:26AM -0500, Bobby Bingham wrote:
> > > On Wed, 02 Jan 2008 12:49:24 +0100
> > > Vitor Sessak <vitor1001 at gmail.com> wrote:
> > > 
> > > > Hi all,
> > > > 
> > > > I've spotted a problem with the ffmpeg.c integration using the
> > > > command
> > > > 
> > > > ffmpeg -vfilters fps=1 -i input.avi output.avi
> > > > 
> > > > In the way the integration is done in ffmpeg.c (my fault, I know),
> > > > the filter infrastructure expects one input frame for every output
> > > > frame. But that is not right for all filters. FFplay is also
> > > > affected by this problem (although it's harder to reproduce, I
> > > > didn't found a test case), it has a video queue and a filter will
> > > > not work properly if the queue is empty.
> > > > 
> > > > The root of the problem is that in the way libavfilter is
> > > > designed, libavfilter must ask ffmpeg.c/ffplay.c every time it
> > > > wants a frame, no matter how (if any) frame was outputted by the
> > > > filter chain.
> > > > 
> > > > I've thought in a few solutions:
> > > > 
> > > > Solution 1 (the closest to the actual code):
> > > > 
> > > > In the patch in soc svn, a input filter is defined in ffmpeg.c. A 
> > > > solution would be to move the call to av_read_frame and 
> > > > avcodec_decode_video to the request_frame method of the input
> > > > filter, so it'll decode a video frame from the file each time the
> > > > filter asks for one.
> > > > 
> > > > Problem: Messy, ugly to handle audio and -vcodec copy, makes
> > > > ffmpeg.c even more complex than it is already, scare people away
> > > > of using libavfilter in other applications.
> > > > 
> > > > Solution 2 (threading and queuing):
> > > > 
> > > > Make ffmpeg.c have a thread for audio decoding, another one for
> > > > video and another for filtering. The video thread will add frames
> > > > to a video queue and the filtering thread will sleep and wait for
> > > > frames is the video queue is empty.
> > > > 
> > > > Problem: May introduce bugs, big patch, gives the impression that
> > > > it is impossible to use libavfilter in a single-threaded
> > > > application. There may also be corner cases where the queues gets
> > > > too big and uses gigs of ram.
> > > > 
> > > > Solution 3 (change libavfilter):
> > > > 
> > > > Allow avfilter_request_frame to give a positive value if it
> > > > outputs a frame, a negative one if error or end of video and zero
> > > > if it has no frame to output be may have one latter. Then make a
> > > > small video queue and just make the request_frame of the input
> > > > filter return 0 if the queue is empty. The advantage is that
> > > > it'll be easier to use it as a library and will not increase too
> > > > much the complexity of ffmpeg.c/ffplay.c.
> > > > 
> > > > Problem: Every video filter will have to handle the case where
> > > > the previous filter do not have a frame now but may have latter...
> > > > 
> > > > What is your opinion on that?
> > > 
> > > I'll take a closer look at this tonight, but I think solution 3
> > > sounds reasonable.
> > 
> > As nothing has been commited yet to solve this IIRC ...
> > 
> > what about
> > 
> >     /**
> >      * Frame poll callback.  This returns the number of immedeately
> > available
> >      * frames. That is if it returns 1 or larger than the next
> > request_frame()
> >      * is guranteed to return one frame (with no delay)
> >      *
> >      * Output video pads only.
> >      */
> >     int (*poll_frame)(AVFilterLink *link);
> > 
> > with that we could just check each output stream (there could be more
> > than 1) if a frame could be output. And if yes run a request_frame()
> > on that. If no stream has a >0 return for poll_frame() than another
> > input packet could be demuxed and decoded (from a file which caused a
> > poll()==0) and send it to one of the input vf_filo
> > 
> > poll_frame() could have a trivial default implementation which just
> > polls its input filters and returns that, this would be sufficient
> > for 90% of filters.
> > 
> > [...]
> 
> Consider a filter which needs to check the timestamp or such of the
> next frame to determine whether or not to output it.  This will require
> it to request the frame in poll_frame and either drop it or buffer it
> until it's requested by the next filter, so you can't do slice
> rendering.

hmm, what about simply letting poll_frame() return the timestamps
(or even more) of the available frames as well? This should work IMHO
with the use case of having a vf_fifo on every input and an application
pushing frames into them.


> 
> The idea to use the return value of request_frame avoids this.

no, think of a filter needing more than 1 frame where the first succeeds
but the second fails requesting. This could be 2 frames from the same source
or 2 frames from different sources, like in a "picture in picture" filter
mixing 2 videos. This would also need some buffering in the filter ...
Ironically poll_frame() would avoid that problem.

Also theres another possible issue without poll_frame() but maybe iam missing
something ...
If you have several outputs, from which do you request a frame?
Each in sequence? like 0,1,0,1,0,1, ... ?
Imagine a single input, a simple split filter where each input frame from the
single input is send to each of the 2 outputs, and then a frame drop filter
droping every 2nd frame on one of these outputs.
The naive 0,1,0,1, ... requesting will cause a problem in terms of buffer
requirements. If you would instead use the timestamps, to decide, a filter
changing timestamps (like to produce a slow motion effect) instead of
droping frames would break that as well.


[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

I know you won't believe me, but the highest form of Human Excellence is
to question oneself and others. -- Socrates
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-soc/attachments/20080108/843b5ae4/attachment.pgp>