[FFmpeg-devel] [RFC] 5 year plan & Inovation

Fri Apr 19 02:19:29 EEST 2024

On Thu, Apr 18, 2024 at 06:13:40PM -0300, James Almer wrote:
> On 4/18/2024 5:53 PM, Michael Niedermayer wrote:
> > On Thu, Apr 18, 2024 at 04:02:07PM +0200, Niklas Haas wrote:
> > > On Wed, 17 Apr 2024 15:58:32 +0200 Michael Niedermayer <michael at niedermayer.cc> wrote:
> > > > Hi all
> > > > 
> > > > The pace of inovation in FFmpeg has been slowing down.
> > > > Most work is concentarted nowadays on code refactoring, and adding
> > > > support for new codecs and formats.
> > > > 
> > > > Should we
> > > > * make a list of longer term goals
> > > > * vote on them
> > > > * and then together work towards implementing them
> > > > ?
> > > > 
> > > > (The idea here is to increase the success of larger efforts
> > > >   than adding codecs and refactoring code)
> > > > It would then also not be possible for individuals to object
> > > > to a previously agreed goal.
> > > > And it would add ideas for which we can try to get funding/grants for
> > > > 
> > > > (larger scale changes need consensus first that we as a whole want
> > > >   them before we would be able to ask for funding/grants for them)
> > > > 
> > > > Some ideas and why they would help FFmpeg:
> > > > 
> > > > * Switch to a plugin architecture
> > > >      (Increase the number of developers willing to contribute and reduce
> > > >       friction as the team and community grows)
> > > 
> > > This would almost surely hurt productivity
> > 
> > i dont agree, ill elaborae below
> > 
> > 
> > > as it will require exposing,
> > > versioning, documenting and maintaining a vast number of internal APIs
> > 
> > yes to some extend that is needed. It can be more or less depending
> > on how things are implemented
> > 
> > 
> > > which we are currently at the liberty of modifying freely.
> > 
> > we are modifying these APIs for decades. That modification of APIs
> > their documentation and all code using them costs time.
> 
> The AVCodec hooks being internal allowed us to add autoinserted bsfs and to
> painlessly rewrite the decouple I/O callbacks to work as a pure pull based
> interface. Were said internals public, that wouldn't have been possible.

A decoder API needs packet in, frame out.
AVPacket and AVFrame are public.
(and a bunch of key-value data like width / height / timebase / pixelformat / aspect / ...
 teh struct for that is also public since a long time)
I dont see the problem.

you have the decoder API facing the user, that causes no problems,
i dont agree that having a decoder API (or encoder, muxer, demuxer, ...)
facing a plugin would be a bigger problem than the APIs we already have
public
sure there are details, sure there are things that need one to think about
and sleep over and all that but these are from a high level point of
view simple and also the same interfaces to what we already have public

> 
> > 
> > More so we have issues with patch-management. And i claim this is
> > not the mailing list but a lack of time and in some cases a lack of
> > interrest in many areas.
> > 
> > A plugin system moves this patch-management to people who actually
> > care, that is the authors of the codecs and (de)muxers.
> > 
> > Our productivity as is, is not good, many patches are ignored.
> 
> A lot of patches fall through the cracks rather than being ignored.
> There are people that send patchsets unthreaded (Because of misconfigured
> git send-email i suppose), and it makes browsing your mailbox hard.

well i can say that i dont review many patches anymore because i just dont have
the time, its not that i cant keep track of what i wanted to review.

either i make a note in a TODO list or i keep the patch marked as NEW
in my mail user agent.

trac in some sense or patchwork are just more public TODO lists
that can be shared between people so if one doesnt do it another
developer sees it and can do it.

> 
> > The people caring about these patches are their Authors and yet they
> > are powerless as they must sometimes wait many months for reviews
> > Besides that, developers are leaving for various reasons and they
> > are forced to setup full forks not being able to maintain their
> > code in any other way.
> > IMO A plugin system would improve productivity as everyone could work
> > on their own terms.
> 
> You say the ML is not the problem, but it sort of is. An MR based
> development would greatly improve this problem.

People historically where very opposed to merge requests

But, having a public git repo (that people already have)
asking for it to be merged. You get a merge commit and someone will probably
feel offended by that. (thats not what you meant i guess)
but i would 100% support doing git merge requests.
(there are good arguments from people much smarter than me why
merging is better than rebasing)

> 
> > No week or month long delays and weekly pinging patches
> > No risk about patches being rejected or ignored
> > No need to read every other discussion on the ML. One can just
> > simply work on their own plugin looking just at the API documentation
> 
> I don't personally have a strong opinion one way or another on this subject
> at this moment, but any such approach would need to be carefully thought and
> designed, to prevent all the potential problems people have expressed so
> far.

of course this would require carefull design as any public API
would.

[...]

> > > 
> > > > * AI / neural network filters and codecs
> > > >      The future seems to be AI based. Future Filters and Codecs will use
> > > >      neural networks. FFmpeg can be at the forefront, developing these
> > > 
> > > We already have TensorFlow support, no? (vf_sr etc.) What more is
> > > needed?
> > 
> > more of that AND more convenience
> > 
> > lets pick a comparission
> > to run fate
> > you write "make fate"
> > if you want to do it with the samples its
> > make fate-rsync ; make fate
> > 
> > if you want to use vf_sr, its reading the docs, looking at some scripts reading their docs
> > and i presume selecting a training set ? creating a model ? ....
> 
> By no means we could ever ship a custom AI model for the sake of a "git
> fetch, compile, and run" scenario. This was already a problem when
> tensorflow was first committed. So this can't be avoided.

I am not sure i understand you, but training a model on lets say the
fate samples or some public domain images. Why would we not be able
to ship that in a similar way to the fate samples ?

or why not ship a standarized script to build such a model from
user data ?
(i mean by standarized, the same for every filter, like ffmpeg is the same
 for every format not link to different papers and random off site scripts)

> 
> > 
> > how many people do that ?
> 
> Every external dependency has its documented requirements...

vf_sr doesnt have a example one can copy and paste that would
work on a fresh checkout of ffmpeg. That alone is a fail IMHO

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Concerning the gods, I have no means of knowing whether they exist or not
or of what sort they may be, because of the obscurity of the subject, and
the brevity of human life -- Protagoras
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20240419/57982d14/attachment.sig>