[FFmpeg-devel] [RFC] 5 year plan & Inovation

[Andrew Sayers]

On Tue, Apr 23, 2024 at 02:20:51AM +0200, Michael Niedermayer wrote:
> On Thu, Apr 18, 2024 at 10:46:35AM +0200, Stefano Sabatini wrote:
> > On date Wednesday 2024-04-17 15:58:32 +0200, Michael Niedermayer wrote:
> > > Hi all
> > > 
> > > The pace of inovation in FFmpeg has been slowing down.
> > > Most work is concentarted nowadays on code refactoring, and adding
> > > support for new codecs and formats.
> > > 
> > > Should we
> > > * make a list of longer term goals
> > > * vote on them
> > > * and then together work towards implementing them
> > > ?
> > > 
> > > (The idea here is to increase the success of larger efforts
> > >  than adding codecs and refactoring code)
> > > It would then also not be possible for individuals to object
> > > to a previously agreed goal.
> > > And it would add ideas for which we can try to get funding/grants for
> > > 
> > > (larger scale changes need consensus first that we as a whole want
> > >  them before we would be able to ask for funding/grants for them)
> > > 
> > > Some ideas and why they would help FFmpeg:
> > > 
> > [...]
> > > * client side / in browser support
> > >     (expand towards webapps, webpages using ffmpeg client side in the browser)
> > >     bring in more users and developers, and it will be costly for us
> > >     if we let others take this area as its important and significant
> > 
> > There are already several projects on github, the most prominent one:
> > https://github.com/ffmpegwasm/ffmpeg.wasm/
> > 
> > In general it would be useful to provide libav* bindings to other
> > languages, for example:
> > https://github.com/PyAV-Org/PyAV
> > https://github.com/zmwangx/rust-ffmpeg
> > 
> > Not sure these should be really moved to FFmpeg though.
> 
> From a user PoV it would be nice if there was a official
> python, rust and wasm binding
> 
> It also would draw in more developers and users to FFmpeg.
> test coverage might also improve
> 
> I think the 2 questions are.
>  1. is there a binding for some language that wants to become the official
>     FFmpeg binding for that language ?
>  2. does the FFmpeg community want that too ?
> 
> thx

I've thought about this a lot while trying to learn FFmpeg.
IMHO there are two big hurdles to good other-language bindings:

First, FFmpeg's interface is full of C idioms that are unintuitive to
programmers from other languages.  For example, Stefano Sabatini is
patiently explaining to me in anoher thread how contexts are a central
concept in FFmpeg's design.  Even where I understood the code on a
mechanical level, I had drastically underestimated their importance
because I didn't have a mental model to understand them.  Binding
FFmpeg functionality in another language is only half the problem -
the interface needs to be explained in terms they can understand,
or rewritten in terms they already know.

Second, the interface is full of special cases that make translation
to other languages burdensome.  For example, C errors are based on
returning a value and requiring the caller to check it explicitly;
whereas most other languages throw an error and allow the caller to
catch it or not.  A translator needs to convert every one of those,
but FFmpeg functions don't have a standard mechanism to signal the
correct behaviour for a given function.  Even the documentation isn't
reliably helpful, sometimes saying a variant of "returns an AVERROR",
sometimes "returns a negative number", and sometimes it just
returns an int and expects the reader to dig through the source.
That eats up a huge amount of programmer time, and has to be done for
every language that wants a binding.

Solving those problems would make it far more practical for translators
to make bindings in other languages, and for new people to learn FFmpeg
even in C.  For example, creating an `enum AVERROR` and rewriting
functions to return it would make the code easier to read and drastically
cut translator time.

	- Andrew Sayers