[FFmpeg-devel] [PATCH 6/6] lavfi: make AVFilterLink opaque in two major bumps.

[Nicolas George]

Le primidi 1er nivôse, an CCXXV, Michael Niedermayer a écrit :
> The framework could monitor filters to determine their apparent
> behavior. This would of course not give exact prediction of future
> behavior.
> I cant say how useful it would be to use this of course ...

I do not see how to integrate that in something reliable, but somebody
may.

> well, in the original design a filter graph can either be used in a
> pull based application in which primarly data is requested from its
> outputs and the requests recursivly move to its inputs triggering
> data read through callbacks from some source filter. [applications
> could implement their own source filter as there was a public API]
> 
> Or in a push based application each source would have a fifo,
> if its empty the application needs to push data into the fifo, data
> again is returned by requesting from the sink(s).
> 
> Which sink to pull data from could be determied by first pulling
> from ones that had data when polled and then it would be up to the
> application to decide, your lowest timestamp choice would have been
> a possibility, keeping track of apparent in-out relations would
> be another. (this was either way application side and not lavfis
> choice)

I am not sure I can easily keep up the discussion: we are going back to
the basics of the scheduling, I worked on it in spring-summer 2012.
Since then, I remember a lot of "that does not work" but not all the
"because" that come after. I can give a few examples.

I think pull-only mode may work if the application is aware of all the
sinks and makes assumptions about their synchronization (i.e.
split -> normal speed + slow motion would not work), which seems
reasonable. Unfortunately, pull-only is the least useful way of using
that kind of library.

Push-only mode does not work with several inputs (or sources), because
you can not know which one needs a frame (the actual need may be far
downstream), and assumptions about synchronization are really not
acceptable in input.

Local-mixed pull-push (i.e. filters can issue a pull in reaction to a
push and reciprocally) solves these issues, but can result in infinite
loops: split pushes, first on out0 connected to overlay in0, overlays
pulls on in1 connected to split out1, split pushes, Redo From Start,
stack overflow. That was what we had before 2012.

Global-mixed pull-push (i.e. the application pushes on inputs and drives
by pulling on outputs, but filters are not allowed to make U-turns)
works, with the same caveats as pull-only. That is what we have since
2012.

And on top of that, you have to consider the case of inputs versus
sources that are always ready like testsrc. They bring their own
complications.

Of course, any of these issues can be solved using flags or something,
but the hard part is to solve all of them at the same time. We no longer
are at the simple original design, but a complicated set of workarounds
on top of the original design.

And the new non-recursive design is not more complex than the recursive
one, the one that works. It is the same, plus the code for the FIFO. If
it was done like that in the first place, it would have worked fine. The
complex part in it is the compatibility layer: use filters designed for
the recursive version unchanged in the non-recursive version. Hopefully,
some of that complexity will go away as difficult filters are adapted.

Plus, unlike the recursive design that mandates a depth-first order of
processing, the non-recursive design can work in any order. I suspect it
will give us the local-mixed pull-push mode for almost free. But I have
yet to test.

> differences in corner cases yes, i didnt mean to imply that its
> purely and 100% cosmetic. More that its basically a cosmetic change
> replacing how the more or less same code is triggered and that maybe
> some of this could be done by some gsoc student or other volunteer.
> Aka at least part of this seems time consuming but not highly complex
> work.

Indeed. And I must say it makes a nice change after a year spent
understanding why corner cases in the non-recursive design did not work.

> i dont think not knowing the duration is a problem.
> you need replicated frames possibly elsewere already. Like for
> subtitles, its not much different to duplicating the last frame with
> the remainining to EOF duration to be added to the last 1 or 0 duration
> but i didnt think deeply about this now so i might miss details
> the issue is also not specific to subtitles, audio tracks with "holes"
> in them exist too so do video slidshows. At least in some usecases
> limiting the distance between frames is needed. (for example to
> ensure random access as in keyframes. The issue can to some extend
> be pushed into the container format i guess but for truely streamed
> formats if you dont repeat your video frame and subtitles which
> are currently disaplyed it just wont get displayed if you start viewing
> around that point)
> so to me it seems there are a lot of issues that all can be dealt with
> by some support to replicate frames in long stretches of no frames
> and later drop them if they arent needed, the last EOF duration
> containing frame could then be just another such case
> but again i didnt think deeply about this

That may be true, but I really think that relying only on timestamps
instead of mixing duration (optional) and timestamps (almost mandatory)
will be simpler.

> > > > - Allow to set a callback on buffersinks and buffersrcs to be notified
> > > >   when a frame arrives or is needed. It is much more convenient than
> > > >   walking the buffers to check one by one.
> > You could have said it: cosmetic :)
> i should be more verbose, i said cosmetic but i meant more than just
> what cosmetic means litteraly :)

No, no, I meant: you really could have said it, this change is really
just syntactic sugar.

Regards,

-- 
  Nicolas George