[FFmpeg-devel] How to support audio data placed within video data

[Manuel Lauss]

On Thu, Nov 28, 2024 at 3:19 PM Anton Khirnov <anton at khirnov.net> wrote:
>
> Quoting Manuel Lauss (2024-11-26 15:25:30)
> > Hello,
> >
> > I'd like to add some audio support for the old libavformat/smush
> > formats (mainly the "ANIM" variants; the "SANM" variant already has
> > audio decoding support).
> >
> > The audio data (16bit stereo PCM) however is placed at (more or less)
> > random places within all the video data, also with no relation to the
> > actual video frame it is embedded into (i.e. most files place a few
> > hundred ms of audio in the first video frame, while the rest are
> > roughly the length of a video frame).
> >
> > What is the best way to support this scenario?
>
> Meaning you have to parse the coded bytestream to get the audio? Is
> there at least some signalling that audio is present at al?

No, audio and video are distinct chunks which are again contained
in a super chunk. This super chunk (a "FRME" since it encodes exactly
one video frame) is passed by smush.c as a video packet to sanm.c.
It make some sense since besides pure video/audio data it also
contains (delta-)palette data, instructions to store/restore a frame,
subtitles for this video frame, ... and also a few kB of audio which
is not tied (on the timeline) to the video frame also in this super chunk,
and it's not even a full standalone audio packet either, it may depend
on data from the previous FRME and also provide a few bytes of data
for the following one.
(i think this was designed for streaming from slow cdroms).

The end of one super chunk then signals to display the video/audio data
worked on so far.

> The options I can think of are:
> * parse the bytestream in the demuxer

So pass the individual chunks of the super chunk off to the
video/audio codec as packets?  Can I invoke the codecs
decode function multiple times per frame?

Thanks!
     Manuel