[FFmpeg-devel] On libavfilter: A Summary of Issues

[Stefano Sabatini]

On date Sunday 2012-10-07 14:18:15 +0200, Michael Niedermayer encoded:
> Hi
> 
> On Sat, Oct 06, 2012 at 07:38:36PM -0400, Derek Buitenhuis wrote:
> [...]
> 
> > 
> > 
> > It's a Monolithic Library
> > -------------------------
> 
> is it ?
> all filters are registered through avfilter_register()
> one could load a filter by whichever means and register it, it would
> work.
> A patch actually implementing such dynamic loading is welcome!
> but see below also, there are other options ...

The problem is that an external C filter can't access all the
internal utilities in libavfilter, which *are* required for a sane
implementation, the alternative is to copycat all the utilities in the
userspace code, which kinda sucks.

Note that this was an issue since the internal interface was made
"private". There are a few reasons for that (many considered the API
still incomplete and non-functional, and keeping backward
compatibility was not regarded feasible). Of course we could/should
re-enable the internal API when the interface can be considered
stable/mature enough. This may require a bit of coordination with
libav, since most API changes come from their branch (and of course,
they're free to do whatever they want with their fork, but again this
fork situation is doing more harm than good to the overall economy).

> 
> 
> [...]
> 
> > Ideally there would be a -stable- and -long term- API and ABI for
> > user written plugins. This, and I cannot stress this enough, is
> > -essential-. If I write a plugin in 2012, and distro a binary under
> > a commercial license, or simply just a freeware license, I should
> > be able to load it in 2015. This is a something FOSS in general
> > is VERY bad at all, and FFmpeg in particular. One of the reasons
> > Avisynth has done so well to this way is that:
> > 
> > 1) Users may write closed source plugins.
> > 2) They still work with AviSynth today. You can even still load 2.0
> >    plugins in 2.5 and 2.6, if you are so inclined.
> 
> Now i dont agree with all the details of above paragraph but in the
> general sense sure we arent disagreeing.
> 
> a stable API for libavfilter will be hard as long as we merge from
> libav and try to stay compatible with them.
> 
> But skiping that problem with the libavfilter API. I would suggest
> something else, write a libavfilter filter that dynamically loads
> and wraps a *Synth filter.
> This in one fell swoop fixes the problem of monolithsm and you also
> have a stable API (if you belive *Synth is stable).

And another option, already implemented, is using Frei0r API which
achieves exactly those objectives. You just need to contribute a
plugin in Frei0r (or write your closed source stuff), and you can load
into libavfilter through frei0r.

Of course frei0r API is not as flexible as the one supported by FFmpeg
(e.g. it doesn't allow all the formats supported by FFmpeg, and not
YUV and force frame-copy, no audio and not transcoding), *nor* is
scriptable the way I think avisynth is.

> 
> 
> [...]
> > There's No Real Scripting (No, Really)
> > --------------------------------------
> > 
> > Don't kid ourselves. The "scripting" in libavfilter is not really scripting.
> > There is no real language, or syntax, just a graph and a lackluster eval. On
> > top of that you must specify it all in the cli arguments rather than script
> > files, and the syntax is absolutely atrocious. To quote our own docs:
> > 
> >     ffmpeg -i opening.mkv -i episode.mkv -i ending.mkv -filter_complex \
> >       '[0:0] [0:1] [0:2] [1:0] [1:1] [1:2] [2:0] [2:1] [2:2]
> >        concat=n=3:v=1:a=2 [v] [a1] [a2]' \
> >       -map '[v]' -map '[a1]' -map '[a2]' output.mkv
> > 
> > Yeah. No.
> 
> can you show how doing the same would look in *Synth ?

Definitively, there is no scripting in libavfilter, and nobody claimed
otherwise. The syntax only allows to describe a filtergraph (with a
somewhat fixed structure. eval is just a nicety which makes possible
to have dynamic behavior in filters).

Note that you could have a sort of scripting engine which builds a
graph from a more highlevel description, this could be implemented
like a pre-processing step (externally to FFmpeg). Also an external
graph editor may be nice. I wanted to write such a thing for VLC,
maybe I'll do it soon or later.

> 
> [...]
> > The Public API is Overly Complex (Some Even Say Terrible)
> > ---------------------------------------------------------
> > 
> > I'm going to compare again to Avisynth here. Its API for using it, and for
> > writing plugins for it is super simple. It has a very low bar for entry.
> > Get a buffer and metadata. Do stuff with it. Return a buffer, or skip. It's not
> > nearly this simple in libavformat. Just look at what should be an
> > extremely simple filter of no more than a few lines: vf_select.c. 443 lines.
> 
> libavfilter is designed to be efficient, fast and flexible. The API
> you describe for AviSynth is clearly none of that.

Yes. If you can show a framework with the same features of libavfilter
(generic and automatic colorspace handling, memcpyless, with the
possibility to handle A/V and transcoding), I would be very interested.

Also if people has suggestions about how to improve the API, that
would also be very welcome. I'm aware that we should document better
what we have, now we have more doxy, some examples and a
libavfilter-design document in doc, what we lack is a complete
tutorial.

> There where plans to write a slow/simple API on top of the not so
> simple one, but as you know this did not happen yet.
> 
> The good news though is if you write that *Synth wraper filter you get
> its API and its API docs for free too. Which would make writing our
> own (yet another) simple API maybe unneeded.
> 

> [...]
> > It Offers Nothing Of Value; Filters or Otherwise
> > ------------------------------------------------
> > 
> > Every filter in libavfilter is available in some other, better framework,
> > and as been for a very long time. Usually in an easier-to-use capacity. Most
> > of libavfilter consists of ancient mplayer plugins, or similar. They're no
> > longer useful in 2012, either.

Do you have specific examples? MPlayer filters (which are not anymore
the most part) were useful in MEncoder, so why shouldn't they be
useful in FFmpeg?

Can you show specific filters which you don't consider useful? What
are we lacking? Wrapping is a possibility. Keep in mind that filtering
is not the main capability/focus of FFmpeg, so for example there are a
relative small number of developers which have interest/focus on the
filtering capabilities, so most external developers interested into
contributing will likely contribute code to some more dedicated
project.

So you may ask why libavfilter was conceived and implemented in the
first place?

Well there are several reasons: one reason was to reduce the
dependency on external projects (an ffmpeg which needs to link against
avisynth or another framework for cropping or padding would suck
hard), we wanted/needed to simplify application code (resampling,
padding, rescaling with as few hacks as possible in the ff*
tools/application code), we wanted to provide a filtering *and* high
level *internal* library (for having a consistent framework, and
leverage the cababilities already included in the codebase, e.g. DSP
utils).

> Almost every single plugin in libavfilter has
> > better working alternatives in other frameworks.
> To name some of the obvious
> > ones:
> > 
> > - mvtools (No, exposing the motion estimation in libavcodec is not a valid
> >   alternaive.)
> > - Anything by tritical (TIVTC, EEDI3, etc)
> > - FieldHints
> > - SeparateFields
> > - SelectEvery
> > - AddGrain
> > - QTGMC (Yes it's script. A good one. See an earlier point.)
> > - DitherTools
> > - Non-ancient copies of gradfun, and pals.
> > - Many, many more. See Doom9.
>
> these too would be fixed in one fell swoop with a *Synth wraper or
> other filter wrapers.
> 
> once such wraper(s) are written they should be made to get enabled
> seamlessly when a filter is requested that is not natively available
> but in these frameworks

Of course, being filtering not the main focus of FFmpeg, and being
libavfilter relatively new with regards to this application area, it
may compare disfavorably with other more dedicated frameworks.

I disagree with the fact that libavfilter is not useful at all, there
are a lot of users which rely on libavfilter filtering capabilities,
for doing stuff from simple padding/cropping to video/audio
composition through overlay/amerge/amix etc.

Of course this would be achievable by integrating ffmpeg with other
frameworks, but this is not always possible for several reasons
(dependencies, portability, etc. etc.) and you would achieve a lesser
level of integration with the existing framework. Of course you're
free to use whatever suits you best, which may easily *not* be
libavfilter.

[...]

Thanks for the time you spent elaborating and expressing your
concerns.
-- 
FFmpeg = Friendly Forgiving Murdering Perfectionist Elaborated Game