[FFmpeg-devel] [RFC] FFmpeg Execution Graph Visualization

[Michael Niedermayer]

Hi Nicolas

On Fri, Apr 11, 2025 at 07:09:09PM +0200, Nicolas George wrote:
> Michael Niedermayer (HE12025-03-29):
> > can you repost the AVWriter patches ?
> 
> Sure, but I need to find the latest branch, and it is proving a cause of
> procrastination for me. In the meantime, let me explain anew the
> principles behind it, after a few years I might be able to express them
> better.
> 

> (I hope it means you are considering re-asserting your authority to
> approve it on principle pending review of the code. I will not invest
> more coding into it unless I know nobody has the authority to put it to
> waste with “it does not belong”.)

FFmpeg belongs to the People, to the Community. Its our project not just
my project.

This doesnt mean anyone should be able to block anything, this worked
when the Community was smaller.

In this specific case, I think the people who previously objected
are no longer active in FFmpeg. So simply reposting the patchset
seemed like a good idea to me.
Maybe noone has objections, maybe someone has valid and good suggestions
maybe someone has valid objections, that require technical changes.
Maybe we will have an objection by one person but a clear consensus
that most people are in favor of this being added to FFmpeg.

OTOH, if there are objections from multiple active developers who would
be affected by the patchset, then this is something we need to accept
and then the patchset would not be able to be applied as long as these
objections are standing.
And even if there is just one active developer objecting, we need to
very carefully listen and understand what he is saying before concluding
that theres a consensus with one objection. Because maybe we can learn
from the objection and improve the code for example


>
> 
> 1. Why we need it
> 
> We use a lot of text for small things. As a command-line tool, we use
> text to communicate with users. Even as a library, when GUIs do not have
> the proper widget they will use text, and we need to provide the
> necessary support functions.
> 
> We should have a rule that for every type we add, we must have a
> corresponding to-text API function. Furthermore, these functions should
> all have the same type (up to the type of the opaque pointer), so that
> we can store them in a structure that describes the type along with
> other information (from string, ref/unref, etc.).
> 
> We also need text for inter-process communication, like exporting
> statistics from filters.
> 
> Some of these to-text conversions happen once per frame. That means they
> should happen without dynamic allocation. Quite complex code is
> warranted to prevent dynamic allocations once per frame: see the frame
> and buffer pools too. But some of these to-text conversion have an
> unbounded output. That means the API needs to be able to handle long
> output.
> 
> BPrint meets the criteria: BPrint is as fast as a buffer on the stack
> but can handle strings of arbitrary length.
> 
> Unfortunately, BPrint is ugly. The name is ugly, and more importantly,
> it is inflexible, it works only with flat buffers in memory.
> 
> 
> 2. How AVWriter does it
> 
> AVWriter is an attempt — successful, in my opinion — at keeping what is
> good in BPrint and fixing what is ugly, in particular by giving it
> features similar to AVIO.
> 
> The design principle is the same as AVIO: a structure with callbacks
> that perform the low-level operations.
> 
> The difference with AVIO is that effort were made to keep the structures
> small and allocatable by the caller.
> 
> Also, the back-end is allowed to provide only a few methods: printf-like
> or write-like or get_buffer-like, and the framework will make sure to
> use one that is available. That means quite a bit of code to handle
> doing a printf() on a back-end that only has get_buffer, but it is code
> isolated within a single file and with ample FATE coverage.
> 
> One of the tricks I used to avoid dynamic allocations is to store the
> structure of methods and the structure with an instance as a structure
> with two pointer elements passed by value.
> 
> Another trick I used to avoid dynamic allocations is to store the size
> of a structure in it. That way, if a caller is compiled with an old
> version of the library, it stores a smaller size than the current one,
> and the new version of the library can test and avoid accessing the new
> fields. That lets the caller allocate the structure while keeping the
> ability to add fields to the structure.
> 
> Apart from that, is is rather straightforward. The default AVWriter is a
> simple wrapper around BPrint, but there are also quite a few other
> built-in ones: wrapper around stdio and av_log (without storing the
> whole string), wrapper around a fixed-size buffer. AVwriter can also act
> as a filter: get an AVWriter, you can create a new one that will encode
> to base64 or HTML entities on the fly.
> 
> 
> 3. Where it will go
> 
> The trick passing a struct of methods and the object as a structure with
> two members passed by value is meant to become a very light-weight
> object system. Thus if ctx is a pointer to a structure of type T, we can
> define serialize_t as a structure with functions and consider
> { serialize_t, ctx } as an object of the class “serializable”. But we
> can also make it an object of the class “refcounted” by pairing it with
> another structure.
> 
> The object system, including casts from one class to another, can be
> de-centralized, new classes ca be defined anywhere in the code without a
> central registry. That will be useful to enhance several parts of our
> code:
> 
> - Side data would not need to be added in libavutil. A pair of
>   demuxer-muxer in libavformat can define a new type of side data by
>   defining the methods to handle it.
> 
> - Filters with unusual parsing needs can define a new type of option and
>   plug it into the standard options parsing system. It is extremely
>   useful if the needs of the filter are too specific to add a type in
>   libavutil but too far from something existing to be able to use it
>   conveniently.
> 
> - Pluging new types into the options system will automatically fix our
>   “escaping hell” issue where users sometimes need 42 levels of \ in
>   front of a : to achieve their goals.
> 
> - We can implement a full-featured av_printf() function that serializes
>   on the fly:
> 
>   av_printf(out, "Stream %d at %d Hz, %@ with layout %@\n",
>       st->id, st->sample_rate,
>       avany_sample_fmt(st->sample_fmt),
>       avany_layout(st->channel_layout));
> 
> With this, it becomes possible to implement new features that rely a lot
> on text. That includes:
> 
> - Better error reporting. Instead the error message going to the log,
>   where nobody will see it if it is a GUI, instead of the application
>   having to install a log handler, with the responsibility to assemble
>   lines together and filter out unrelated messages that just happened to
>   arrive at the same time, the error message is stored in the relevant
>   context and can be retrieved cleanly later.
> 
> - Built-in comprehensive documentation. You can do
>   av_get_documentation(ctx, …) and get explanations on how to use it.
>   Depending on the flags given, the returned documentation can be terse
>   and suitable for a tooltip or comprehensive including hyperlinks and
>   reference for the syntax of the options.
> 
> - Serialization of high-level data structures into various standard
>   formats: JSON, XML, etc., precisely what started this thread. That
>   allows to factor the writers in ffprobe and that gives us means to
>   exfiltrate statistics from filters.

what you describe, really sounds cool

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Into a blind darkness they enter who follow after the Ignorance,
they as if into a greater darkness enter who devote themselves
to the Knowledge alone. -- Isha Upanishad
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20250413/75816435/attachment.sig>