[FFmpeg-devel] [PATCH] attachments support in matroska demuxer

[Uoti Urpala]

On Wed, 2008-01-02 at 11:21 +0100, Michael Niedermayer wrote:
> On Wed, Jan 02, 2008 at 12:51:46AM -0500, Rich Felker wrote:
> > On Wed, Jan 02, 2008 at 06:49:28AM +0200, Uoti Urpala wrote:

> > > Libavformat should provide access to the contents of
> > > Matroska attachments. 
> 
> Libavformat first needs a generic API for this, though if there is no
> interrest in a clean solution iam prepared to reject attachment support
> completely.
> Ive no interrest in a "lets quickly just export attachments" to make ASS/SSA
> in mplayer work with lavf.

Do you not want to provide access to the exact contents stored in the
container through lavf? For Matroska attachments that most likely
requires some container-specific functionality. Or do you intend lavf to
only allow access to some common-denominator subset of container
functionality that is easy to remux in any of them?

MPlayer's native Matroska demuxer is still preferable to the lavf one so
there's no real hurry to make the lavf one fully functional. Even so I
think that insisting on a fully generic and clean implementation as the
first step is a bad choice. Especially when there are no would-be users
of the more generic functionality, it's not even clear what that
functionality should be, and a simple "let's just quickly export the
attachments" interface would not be particularly messy or hard to change
later.

> lavc doesnt even have a ASS/SSA decoder, so from my point of view attachments
> are not needed currently. Especially not in a half thought through way.

Why would lavf functionality be "not needed" unless lavc has
corresponding functionality too? Do you consider ffmpeg.c to be the only
worthwhile program using the libraries? Or think that all video-related
functionality must be added to FFmpeg and other programs shouldn't use
SSA/ASS decoding without adding it to lavc? 

> > > What other things should the same API be
> > > generalized to do? Hypothetical attachments in other containers?
> 
> Lets say if the system doesnt work with avi/mov/nut its rejected.

Work to give what result? What are you supposed to get when you try to
remux an mkv with attachments to avi?


> > > Duplicating the fonts can be expensive if there are lots of subtitle
> > > tracks (which there can be as they're otherwise small). In addition to
> > > the memory directly used by the font files there are also whatever data
> > > structures parsing and rendering them requires - duplicating the fonts
> > > in the extradata of each track would force a decoder to do extra work
> > > comparing them to go back to the shared representation if it wants to
> > 
> > This makes sense.
> 
> As already said this doesnt make attachments/fonts special, extradata or
> parts of extradata are also commonly identical.

There are (as already said...) reasons why memory usage by fonts is
different from video tracks. Video extradata may commonly happen to be
identical, but subtitles refer to "the same font" in a stricter sense.
The memory usage associated with fonts can be significant while the
subtitle track itself is small. It's very rare to have a huge number of
video tracks as that would make the file size much larger than versions
with separate tracks, but adding a large number of subtitle tracks
causes no such problems.

> The solution here (attachments) is quite specific ignoring the general
> problem of redundancy in extradata. That as well can be redundancy relative
> to some default large vector quantizer tables.

It's much more of a problem for fonts than for video or audio extradata.
And memory use is not the only reason to use attachments. I already
explained that whether to attach the fonts or assume the decoder has
them is a decision which can be separate from the contents of the
subtitle track (unlike video), and that a natural format for demuxing
a .mkv into separate files is to have .ass files and separate font files
(unlike video).

> Also sharing any non constant data generated from fonts or extradata 
> between several decoder instances will require thread synchronization so half
> of the "sensible" comment above really only applies for decoders peppered with
> mutexes fromyour favorite thread API. (and its unlikely that a decoder would
> be written like that ...)

It requires either a decoder using mutexes for shared caches (which is
not particularly unlikely even if you claim otherwise), or a decoder
that has a per-played-file context shared by all subtitle streams from
that file (and of course only one thread may be decoding a stream using
that context at once).

> Well, iam waiting to see some actually working solution from you and uoti
> which is better
> Minimum requirements are
> * efficiently remuxing a subset of streams

Again, what exactly is the behavior you want? You cannot choose the
minimum used set of font files referred to from a subset of streams in
current mkv automatically without parsing those streams (and you might
not always want to attach the fonts to the output file even if the
streams did refer to them). So what should the system do? Actually parse
the streams to find out which fonts might be used (not very practical)?
Have functionality which would allow picking only required fonts in case
it was used with some hypothetical container which did have dependency
information (adding code which at the moment would be totally useless)?

> * mov and avi support (the fonts can even be in seperate files but id like
>   to see a working patch here not some general sketch of how it might work)

Again, mov and avi "support" in what sense? AFAIK there is no spec on
how to store SSA in mov or avi. So what kind of mov/avi files would you
want to create?