[FFmpeg-devel] [PATCH] attachments support in matroska demuxer

[Uoti Urpala]

On Tue, 2008-01-01 at 13:27 +0100, Michael Niedermayer wrote:
> On Tue, Jan 01, 2008 at 01:17:57PM +0300, Evgeniy Stepanov wrote:
> > On 12/31/07, Michael Niedermayer <michaelni at gmx.at> wrote:
> > > such data belongs in AVCodecContext.extradata/extradata_size
> > 
> > They are not related to any stream. 
> 
> Then no stream needs them per definition, and they can be droped.

A decoder may need them to show the subtitles accurately if it does not
already have the required fonts. However the resources are not
associated with any particular stream on the container level.
Libavformat can guess (but NOT know for sure) that attached files which
have a font type are used by decoders of subtitle streams of type SSA or
ASS and not by other decoders. The same font may be used by multiple
subtitle streams.

> > Should I add a fake stream with
> > attachments as extradata ? 
> 
> this does not sound reasonable ...

I think this would be the least bad fit if you want to force the extra
resources into existing libavformat data structures.

> as far as i understand matroska attachments, they are a mere duplication of
> extradata. It seems everyone thinks the codec or container they designed has
> special needs outside what already exists. Completely missing that the problem
> if one exists at all really applies to all codecs. And then goes on designing
> a new fancy solution not really solving anything ... other codecs store their
> huffman tables, vector quantizer tables, quantization tables, ... in extradata
> or before every keyframe. Fonts for subtitles are not different from other
> tables needed for decoding streams.

There are differences from extradata used by video/audio codecs. The
fonts are often shared by multiple subtitle streams. The decision
whether to attach the fonts or assume the decoder already has them is
mostly independent of the subtitle stream (in contrast to video/audio
extradata which is almost always an essential part of the stream
itself).

> The only way to see fonts as special relative to global data (=extradata)
> would be if one argues that a subtitle decoder outputs UTF-8 + position
> + font name + color + size + ...

You can't make an exhaustive list of properties easily. Position cannot
be expressed with simple coordinates for some fixed part of rendered
text unless the decoder knows the exact rendered size in advance. There
can be rotated text, different kinds of border, and various special
effects (more of which can appear in the future).

> anyway comments welcome!
> but most important is that we need generic solutions, not something which
> can only be used with mkv-ass (= no hacks)

No other container can properly handle SSA/ASS with fonts AFAIK, so this
requirement is rather vague. In practice no way of handling fonts (or
other extra resource files) can be used with other containers before
there's a spec on how to store such data in those containers. I think
trying to make an overly general solution is not a good idea as long as
the only real use will be with mkv+SSA/ASS. Better make something which
works for that particular use and worry about a more general solution
later when there's a need for it. Trying to guess now what kind of extra
generality will be needed later is unlikely to produce a good result.