[FFmpeg-devel] On libavfilter: A Summary of Issues

[Derek Buitenhuis]

On 07/10/2012 9:44 AM, Stefano Sabatini wrote:
> The problem is that an external C filter can't access all the
> internal utilities in libavfilter, which *are* required for a sane
> implementation, the alternative is to copycat all the utilities in the
> userspace code, which kinda sucks.

This would seem to be a fundamental design flaw...

> Note that this was an issue since the internal interface was made
> "private". There are a few reasons for that (many considered the API
> still incomplete and non-functional, and keeping backward
> compatibility was not regarded feasible). Of course we could/should
> re-enable the internal API when the interface can be considered
> stable/mature enough. This may require a bit of coordination with
> libav, since most API changes come from their branch (and of course,
> they're free to do whatever they want with their fork, but again this
> fork situation is doing more harm than good to the overall economy).

Not touching the fork business with a 10-foot pole.

> And another option, already implemented, is using Frei0r API which
> achieves exactly those objectives. You just need to contribute a
> plugin in Frei0r (or write your closed source stuff), and you can load
> into libavfilter through frei0r.
> 
> Of course frei0r API is not as flexible as the one supported by FFmpeg
> (e.g. it doesn't allow all the formats supported by FFmpeg, and not
> YUV and force frame-copy, no audio and not transcoding), *nor* is
> scriptable the way I think avisynth is.

I was not aware of frei0r before this outside of "it has a funny name".
Its website doesn't really explain well if it is cross-platorm.

> Definitively, there is no scripting in libavfilter, and nobody claimed
> otherwise. The syntax only allows to describe a filtergraph (with a
> somewhat fixed structure. eval is just a nicety which makes possible
> to have dynamic behavior in filters).

I've had a few people, who shall remain nameless. ;)

> Note that you could have a sort of scripting engine which builds a
> graph from a more highlevel description, this could be implemented
> like a pre-processing step (externally to FFmpeg). Also an external
> graph editor may be nice. I wanted to write such a thing for VLC,
> maybe I'll do it soon or later.

[...]

>> libavfilter is designed to be efficient, fast and flexible. The API
>> you describe for AviSynth is clearly none of that.
> 
> Yes. If you can show a framework with the same features of libavfilter
> (generic and automatic colorspace handling, memcpyless, with the
> possibility to handle A/V and transcoding), I would be very interested.

My problem with libav* in generic is that there is this obsession with
generic-ism flexibility at all costs, which generally means at the cost
of usability. KISS. Users prefer simple to mega-flexible-but-a-pain-to-use.

> Also if people has suggestions about how to improve the API, that
> would also be very welcome. I'm aware that we should document better
> what we have, now we have more doxy, some examples and a
> libavfilter-design document in doc, what we lack is a complete
> tutorial.

I can try and collect some.

One thing that i find really annoying is that every plugin has to implement
its own sort of getopt-like parser, as far as I can tell. This is needless
boilerplate code and downright dangerous in some cases.

As for a tutorial, here's Avisynth's:

http://neuron2.net/www.math.berkeley.edu/benrg/avisynth-extensions.html

Countless people have used this with success, so I think it is
a good reference.

>> There where plans to write a slow/simple API on top of the not so
>> simple one, but as you know this did not happen yet.
>>
>> The good news though is if you write that *Synth wraper filter you get
>> its API and its API docs for free too. Which would make writing our
>> own (yet another) simple API maybe unneeded.

There really is no point in writing one. ffmpeg can take av input as it is,
and why on earth would you want to use it through libavfilter's C api?

> Do you have specific examples? MPlayer filters (which are not anymore
> the most part) were useful in MEncoder, so why shouldn't they be
> useful in FFmpeg?

You're falling victim to a fallacy here. They are most certainly not useful
in MEncoder either, in 2012. They're all from ancient times and of dubious
use in 2012. In fact, MEncoder isn't useful anymore in 2012.

> Can you show specific filters which you don't consider useful? What
> are we lacking? Wrapping is a possibility. Keep in mind that filtering
> is not the main capability/focus of FFmpeg, so for example there are a
> relative small number of developers which have interest/focus on the
> filtering capabilities, so most external developers interested into
> contributing will likely contribute code to some more dedicated
> project.

The list is extremely long... I listed a few below.

> So you may ask why libavfilter was conceived and implemented in the
> first place?
> 
> Well there are several reasons: one reason was to reduce the
> dependency on external projects (an ffmpeg which needs to link against
> avisynth or another framework for cropping or padding would suck
> hard), we wanted/needed to simplify application code (resampling,
> padding, rescaling with as few hacks as possible in the ff*
> tools/application code), we wanted to provide a filtering *and* high
> level *internal* library (for having a consistent framework, and
> leverage the cababilities already included in the codebase, e.g. DSP
> utils).

I could understand it as "needed for ffmpeg.c to simplify things", but
I think pushing it as a generic purpose filtering lib is pretty silly,
as well as some of its plugins.

>>> - mvtools (No, exposing the motion estimation in libavcodec is not a valid
>>>   alternaive.)
>>> - Anything by tritical (TIVTC, EEDI3, etc)
>>> - FieldHints
>>> - SeparateFields
>>> - SelectEvery
>>> - AddGrain
>>> - QTGMC (Yes it's script. A good one. See an earlier point.)
>>> - DitherTools
>>> - Non-ancient copies of gradfun, and pals.
>>> - Many, many more. See Doom9.
>>
>> these too would be fixed in one fell swoop with a *Synth wraper or
>> other filter wrapers.
>>
>> once such wraper(s) are written they should be made to get enabled
>> seamlessly when a filter is requested that is not natively available
>> but in these frameworks
> 
> Of course, being filtering not the main focus of FFmpeg, and being
> libavfilter relatively new with regards to this application area, it
> may compare disfavorably with other more dedicated frameworks.

VapourSynth is FAR newer and has already surpassed lavfi in usefulness.
Hell, its almost passed Avisynth.

> I disagree with the fact that libavfilter is not useful at all, there
> are a lot of users which rely on libavfilter filtering capabilities,
> for doing stuff from simple padding/cropping to video/audio
> composition through overlay/amerge/amix etc.

I admit the audio-related stuff is useful. I've mostly used SoX for this
in the past. I most am focusing on video-related things here.

- Derek

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 895 bytes
Desc: OpenPGP digital signature
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20121007/e4f4dfdc/attachment.asc>