[FFmpeg-devel] Evolution of lavfi's design and API

[wm4]

On Fri, 24 Oct 2014 13:07:22 +0200
Nicolas George <george at nsup.org> wrote:

> Le duodi 2 brumaire, an CCXXIII, Clement Boesch a écrit :
> > I'd be curious to hear about how VapourSynth & friends handle that
> > problem, because AFAIK it's only one way. It's likely they don't have to
> > deal with the same problems we have though (the usage is more limited, no
> > audio typically); typically because they don't seem stream but file based
> > (so easy to index and exact seek etc.).
> 
> I am not sure what you mean here. This was about recursive design. It is not
> very difficult to turn the recursive design into an iterative one: just
> handle the call stack manually. In this particular case, most of the filters
> use only tail-recursion ("return ff_filter_frame(outlink, frame);"), so
> there is even no need to handle a stack: just replace the function calls by
> message passing.
> 
> People here seem impressed by VapourSynth's "nice" python syntax to build
> scripts, but it does not look extraordinary to me. I am convinced this is
> "just" a syntax to build the graph. For example, where lavfi requires to
> write "[v][s]overlay=x=42:y=12[vs]", VapourSynth would require something
> like "vs = v.overlay(s, 42, 12)". Just a syntactic matter. The actual work
> certainly happens when the graph is connected to its output.

But guess which syntax is nicer. And this is not only about syntax.
VapourSynth is truly more flexible, and there's a damn good reason why
avisynth is still alive (despite being a pile of crap code-wise), and
why VapourSynth bases its design on it (hopefully only the good and
sane parts).

Also note that Python is not required for VapourSynth. I could hack up
a Lua scripting frontend in 2 hours or so.

Last but not least, the nicest part about VapourSynth is that you can
use external filters. Nobody who writes a very sophisticated video
filter wants to deal with FFmpeg development practices and the
monolithic repo (except FFmpeg devs).

> I believe it would be rather easy to have the same kind of interface for
> lavfi; all the filters are already introspectable, and that is the most
> major point. Unfortunately, two issues would make it of little use. First,
> lavfi handles only frames, not packets, so all the encoding and muxing would
> have to be handled separately.

That has nothing to do with VapourSynth, though.

> Second, lavfi is designed to make a lot of
> format decisions when the graph is complete, so the scripting capabilities
> would not have a lot of relevant information to work on. I wonder how
> VapourSynth handles that, but I am afraid to learn that it just do automatic
> conversions on the spot without globally optimizing them.

Yes, lavfi refuses to work unless you "configure" the graph, which
makes the whole thing very inflexible.

As far as I know, VapourSynth has no automatic conversions. If it really
matters, the script can manually insert conversion filters where it
matters. Lavfi's format negotiation is a monster: how often did even
FFmpeg devs wonder why the hell lavfi is converting to a specific
format at a certain point? After all, the complexity tradeoff might not
be worth the return of the "global optimization" (which often makes bad
decisions too).

VapourSynth's approach in this case is making all filters support a
small set of sane formats. So you don't have to care about conversions,
and in fact you don't need to convert at all, unless input or output
uses fringe pixel formats.