[FFmpeg-devel] [RFC] 5 year plan & Inovation

[James Almer]

On 4/18/2024 5:53 PM, Michael Niedermayer wrote:
> On Thu, Apr 18, 2024 at 04:02:07PM +0200, Niklas Haas wrote:
>> On Wed, 17 Apr 2024 15:58:32 +0200 Michael Niedermayer <michael at niedermayer.cc> wrote:
>>> Hi all
>>>
>>> The pace of inovation in FFmpeg has been slowing down.
>>> Most work is concentarted nowadays on code refactoring, and adding
>>> support for new codecs and formats.
>>>
>>> Should we
>>> * make a list of longer term goals
>>> * vote on them
>>> * and then together work towards implementing them
>>> ?
>>>
>>> (The idea here is to increase the success of larger efforts
>>>   than adding codecs and refactoring code)
>>> It would then also not be possible for individuals to object
>>> to a previously agreed goal.
>>> And it would add ideas for which we can try to get funding/grants for
>>>
>>> (larger scale changes need consensus first that we as a whole want
>>>   them before we would be able to ask for funding/grants for them)
>>>
>>> Some ideas and why they would help FFmpeg:
>>>
>>> * Switch to a plugin architecture
>>>      (Increase the number of developers willing to contribute and reduce
>>>       friction as the team and community grows)
>>
>> This would almost surely hurt productivity
> 
> i dont agree, ill elaborae below
> 
> 
>> as it will require exposing,
>> versioning, documenting and maintaining a vast number of internal APIs
> 
> yes to some extend that is needed. It can be more or less depending
> on how things are implemented
> 
> 
>> which we are currently at the liberty of modifying freely.
> 
> we are modifying these APIs for decades. That modification of APIs
> their documentation and all code using them costs time.

The AVCodec hooks being internal allowed us to add autoinserted bsfs and 
to painlessly rewrite the decouple I/O callbacks to work as a pure pull 
based interface. Were said internals public, that wouldn't have been 
possible.

> 
> More so we have issues with patch-management. And i claim this is
> not the mailing list but a lack of time and in some cases a lack of
> interrest in many areas.
> 
> A plugin system moves this patch-management to people who actually
> care, that is the authors of the codecs and (de)muxers.
> 
> Our productivity as is, is not good, many patches are ignored.

A lot of patches fall through the cracks rather than being ignored.
There are people that send patchsets unthreaded (Because of 
misconfigured git send-email i suppose), and it makes browsing your 
mailbox hard.

> The people caring about these patches are their Authors and yet they
> are powerless as they must sometimes wait many months for reviews
> Besides that, developers are leaving for various reasons and they
> are forced to setup full forks not being able to maintain their
> code in any other way.
> IMO A plugin system would improve productivity as everyone could work
> on their own terms.

You say the ML is not the problem, but it sort of is. An MR based 
development would greatly improve this problem.

> No week or month long delays and weekly pinging patches
> No risk about patches being rejected or ignored
> No need to read every other discussion on the ML. One can just
> simply work on their own plugin looking just at the API documentation

I don't personally have a strong opinion one way or another on this 
subject at this moment, but any such approach would need to be carefully 
thought and designed, to prevent all the potential problems people have 
expressed so far.

> ...
> 
> 
> 
>>
>>> * ffchat
>>>      (expand into realtime chat / zoom) this would
>>>      bring in more users and developers, and we basically have almost
>>>      all parts for it already but some people where against it
>>
>> This seems like a user application on top of FFmpeg, not something that
>> should be part of FFmpeg core. Can you explain what modifications in
>> FFmpeg would be necessary for something like this?
> 
> ffmpeg, ffplay, ffprobe are also user applications.
> 
> 
>>
>>> * client side / in browser support
>>>      (expand towards webapps, webpages using ffmpeg client side in the browser)
>>>      bring in more users and developers, and it will be costly for us
>>>      if we let others take this area as its important and significant
>>
>> I don't understand this point - don't all major browsers already vendor
>> FFmpeg for decoding?
> 
> FFmpeg does more than decoding.
> 
> 
>>
>>> * AI / neural network filters and codecs
>>>      The future seems to be AI based. Future Filters and Codecs will use
>>>      neural networks. FFmpeg can be at the forefront, developing these
>>
>> We already have TensorFlow support, no? (vf_sr etc.) What more is
>> needed?
> 
> more of that AND more convenience
> 
> lets pick a comparission
> to run fate
> you write "make fate"
> if you want to do it with the samples its
> make fate-rsync ; make fate
> 
> if you want to use vf_sr, its reading the docs, looking at some scripts reading their docs
> and i presume selecting a training set ? creating a model ? ....

By no means we could ever ship a custom AI model for the sake of a "git 
fetch, compile, and run" scenario. This was already a problem when 
tensorflow was first committed. So this can't be avoided.

> 
> how many people do that ?

Every external dependency has its documented requirements...

> 
> thx
> 
> [...]
> 
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".