[FFmpeg-devel] [RFC] 5 year plan & Inovation

[Paul B Mahol]

On Fri, Apr 19, 2024 at 1:19 AM Michael Niedermayer <michael at niedermayer.cc>
wrote:

> On Thu, Apr 18, 2024 at 06:13:40PM -0300, James Almer wrote:
> > On 4/18/2024 5:53 PM, Michael Niedermayer wrote:
> > > On Thu, Apr 18, 2024 at 04:02:07PM +0200, Niklas Haas wrote:
> > > > On Wed, 17 Apr 2024 15:58:32 +0200 Michael Niedermayer <
> michael at niedermayer.cc> wrote:
> > > > > Hi all
> > > > >
> > > > > The pace of inovation in FFmpeg has been slowing down.
> > > > > Most work is concentarted nowadays on code refactoring, and adding
> > > > > support for new codecs and formats.
> > > > >
> > > > > Should we
> > > > > * make a list of longer term goals
> > > > > * vote on them
> > > > > * and then together work towards implementing them
> > > > > ?
> > > > >
> > > > > (The idea here is to increase the success of larger efforts
> > > > >   than adding codecs and refactoring code)
> > > > > It would then also not be possible for individuals to object
> > > > > to a previously agreed goal.
> > > > > And it would add ideas for which we can try to get funding/grants
> for
> > > > >
> > > > > (larger scale changes need consensus first that we as a whole want
> > > > >   them before we would be able to ask for funding/grants for them)
> > > > >
> > > > > Some ideas and why they would help FFmpeg:
> > > > >
> > > > > * Switch to a plugin architecture
> > > > >      (Increase the number of developers willing to contribute and
> reduce
> > > > >       friction as the team and community grows)
> > > >
> > > > This would almost surely hurt productivity
> > >
> > > i dont agree, ill elaborae below
> > >
> > >
> > > > as it will require exposing,
> > > > versioning, documenting and maintaining a vast number of internal
> APIs
> > >
> > > yes to some extend that is needed. It can be more or less depending
> > > on how things are implemented
> > >
> > >
> > > > which we are currently at the liberty of modifying freely.
> > >
> > > we are modifying these APIs for decades. That modification of APIs
> > > their documentation and all code using them costs time.
> >
> > The AVCodec hooks being internal allowed us to add autoinserted bsfs and
> to
> > painlessly rewrite the decouple I/O callbacks to work as a pure pull
> based
> > interface. Were said internals public, that wouldn't have been possible.
>
> A decoder API needs packet in, frame out.
> AVPacket and AVFrame are public.
> (and a bunch of key-value data like width / height / timebase /
> pixelformat / aspect / ...
>  teh struct for that is also public since a long time)
> I dont see the problem.
>
> you have the decoder API facing the user, that causes no problems,
> i dont agree that having a decoder API (or encoder, muxer, demuxer, ...)
> facing a plugin would be a bigger problem than the APIs we already have
> public
> sure there are details, sure there are things that need one to think about
> and sleep over and all that but these are from a high level point of
> view simple and also the same interfaces to what we already have public
>
>
> >
> > >
> > > More so we have issues with patch-management. And i claim this is
> > > not the mailing list but a lack of time and in some cases a lack of
> > > interrest in many areas.
> > >
> > > A plugin system moves this patch-management to people who actually
> > > care, that is the authors of the codecs and (de)muxers.
> > >
> > > Our productivity as is, is not good, many patches are ignored.
> >
> > A lot of patches fall through the cracks rather than being ignored.
> > There are people that send patchsets unthreaded (Because of misconfigured
> > git send-email i suppose), and it makes browsing your mailbox hard.
>
> well i can say that i dont review many patches anymore because i just dont
> have
> the time, its not that i cant keep track of what i wanted to review.
>
> either i make a note in a TODO list or i keep the patch marked as NEW
> in my mail user agent.
>
> trac in some sense or patchwork are just more public TODO lists
> that can be shared between people so if one doesnt do it another
> developer sees it and can do it.
>
>
> >
> > > The people caring about these patches are their Authors and yet they
> > > are powerless as they must sometimes wait many months for reviews
> > > Besides that, developers are leaving for various reasons and they
> > > are forced to setup full forks not being able to maintain their
> > > code in any other way.
> > > IMO A plugin system would improve productivity as everyone could work
> > > on their own terms.
> >
> > You say the ML is not the problem, but it sort of is. An MR based
> > development would greatly improve this problem.
>
> People historically where very opposed to merge requests
>
> But, having a public git repo (that people already have)
> asking for it to be merged. You get a merge commit and someone will
> probably
> feel offended by that. (thats not what you meant i guess)
> but i would 100% support doing git merge requests.
> (there are good arguments from people much smarter than me why
> merging is better than rebasing)
>
>
> >
> > > No week or month long delays and weekly pinging patches
> > > No risk about patches being rejected or ignored
> > > No need to read every other discussion on the ML. One can just
> > > simply work on their own plugin looking just at the API documentation
> >
> > I don't personally have a strong opinion one way or another on this
> subject
> > at this moment, but any such approach would need to be carefully thought
> and
> > designed, to prevent all the potential problems people have expressed so
> > far.
>
> of course this would require carefull design as any public API
> would.
>
>
> [...]
>
> > > >
> > > > > * AI / neural network filters and codecs
> > > > >      The future seems to be AI based. Future Filters and Codecs
> will use
> > > > >      neural networks. FFmpeg can be at the forefront, developing
> these
> > > >
> > > > We already have TensorFlow support, no? (vf_sr etc.) What more is
> > > > needed?
> > >
> > > more of that AND more convenience
> > >
> > > lets pick a comparission
> > > to run fate
> > > you write "make fate"
> > > if you want to do it with the samples its
> > > make fate-rsync ; make fate
> > >
> > > if you want to use vf_sr, its reading the docs, looking at some
> scripts reading their docs
> > > and i presume selecting a training set ? creating a model ? ....
> >
> > By no means we could ever ship a custom AI model for the sake of a "git
> > fetch, compile, and run" scenario. This was already a problem when
> > tensorflow was first committed. So this can't be avoided.
>
> I am not sure i understand you, but training a model on lets say the
> fate samples or some public domain images. Why would we not be able
> to ship that in a similar way to the fate samples ?
>
> or why not ship a standarized script to build such a model from
> user data ?
> (i mean by standarized, the same for every filter, like ffmpeg is the same
>  for every format not link to different papers and random off site scripts)
>
>
> >
> > >
> > > how many people do that ?
> >
> > Every external dependency has its documented requirements...
>
> vf_sr doesnt have a example one can copy and paste that would
> work on a fresh checkout of ffmpeg. That alone is a fail IMHO
>
>
Current AI tech is just inflated hype, for both audio and video processing.


> thx
>
> [...]
> --
> Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
>
> Concerning the gods, I have no means of knowing whether they exist or not
> or of what sort they may be, because of the obscurity of the subject, and
> the brevity of human life -- Protagoras
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
>