[FFmpeg-user] personal project - need tech advice

[Cyril Comparon]

Hello ffmpeg gurus!

I'd like to start a music-visualization-like project and I could really use
some hints about the best approach and architecture to take.

Users should be able to dynamically submit audio-video files to the
application. In near-real time, the application will decode some or all of
the input files audio and video data and adaptatively produce a single
output audio/video stream to be displayed live and saved to file. The
compositor needs access to all input audio and video data with some amount
of look-ahead room to make compositing decisions on the fly. To achieve the
required performance, I need to minimize the number of video frame copies
and conversions, including conversion to hardware textures if achievable.

I'm eager to hear any technical/architectural advice you may give. At this
point, my main question is about the choice between options A and B below:

[A] "ffmpeg tool based". I've grown quite comfortable with the ffmpeg tool
itself, including setting up encoder params and writing filter graphs.
Would it make sense to start ffmpeg child processes and tap into them? If
so, is there an existing filter that allows me to hook some sort of custom
logic (in this case, exporting audio/video frames for another process to
consume on the fly) or do I need to write a full blown new filter?

[B] "libavxxx based". If not relying on the ffmpeg tool, the alternative is
to rely on libavxxx directly. This scares me a bit in terms of complexity
as I feel I'd be missing out on tons of built-in abstraction and
corner cases taken care of by the ffmpeg tool, not to mention crash
protection. On the plus side, there would be no filters or hooks to write,
and no cross-process boundary to deal with.

Ideally I'll use C++/Qt6 or TS/node for this project. Obviously I'll use
plain C/libavxxx if I need to write a libav filter or to patch ffmpeg.

Thanks a ton!
Cyril