[FFmpeg-devel] Plans for libavfilter

[Michael Niedermayer]

On Thu, Sep 16, 2021 at 01:55:17PM +0200, Nicolas George wrote:
> Since it is not healthy to keep everything for myself, here is a summary
> of the projects I have in mind for the enhancement of libavfilter.
> 
> The thing is, there are a lot of dependencies between these projects.
> Working on them in a proper order would make achieving the goals easier.
> On the other hand, if task B depends on task A, then working B without
> addressing A involves a little more work for B, but also a lot more work
> for A, and a lot more work for everything else that depends on A.
> 
> This is one reason I am writing this mail: to make clear what needs to
> be done, and how tasks depend on each-other and fit together.
> 
> The earliest tasks are not very sexy. They are about making the code
> clearer or more robust and generic rather than adding features. It is
> always boring. But they are necessary, because parts of the process is
> quite shaky, and we cannot afford to build on shaky code.
> 
> If you want to help, please consider this and you will be welcome.
> 
> Now that is established the specifics:
> 
> - Multi-stage negotiation.
> 
>   That means negotiating properties that depend on other properties. I
>   think the colorspace stuff is in that case, it cannot be negotiated
>   until the pixel format is known.
> 

> - Media type negotiation.
> 
>   That means that filters that do not care about the contents of what
>   they are filtering, filters that only act on the frame information
>   fields, can also not care if they are filtering audio or video or
>   other. There are a lot of these filters: pts manipulation, concat,
>   split, selection, etc.

+1
the current need to duplicate filters per media type is ugly.
Also "negotiation" is a strong term for this. Filters in general
have either one media type on their pads or pass the type from
input to output and support any. In that sense this is not hard
to solve, it should not be dependant on anything else


[...]

> 
> - Partial graph configuration.
> 
>   Right now, the negotiation happens in steps, first query_formats then
>   merge, then pick. But pick can be done on some links as soon as the
>   formats are merged to a singleton while other filters are still
>   delaying their query_formats, and once pick is done, the graph can
>   start running. That would allow filters with constraints more complex
>   than what AVFilterFormats can express. (We have a mechanism for that,
>   used by amerge and one or two other filters; I introduced it early and
>   it was a mistake, it is much too fragile and only converges in the
>   simplest cases.)
> 
> - Partial graph re-configuration.
> 
>   If we can run a graph when not all filters are configured, then we can
>   de-configure and re-configure part of a graph. That would bring us the
>   support for changes in pixel format.

Maybe some orthogonal direction could be considered too
completely seperating graph configuration and using some more generic
algorithm to solve it.
What we have basically are a bunch of constraints and then we need to
color the links between vertexes so as to minimize a weighted average
of computational complexity and quality loss. And each link can have
also different colors on both ends which would correspond to an inserted
convert filter. The "color" here would be a vector of all elements that
need configuration.

This view if it works would also allow very easy changes to the things
needing configuration. For example in this framework one could replace
a colorspace enum by a AxB element matrix and nothing would change on
the solver it would just need a different quality estimation function.

That means all the heuristics would be isolated in the function which
is to be minimized. While the solver would just deal with generic 
values of some kind like ints


[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

The smallest minority on earth is the individual. Those who deny 
individual rights cannot claim to be defenders of minorities. - Ayn Rand
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20210916/25ebdf15/attachment.sig>