[FFmpeg-devel] [RFC] Talk about subtitles

[Reimar Döffinger]

On Thu, Nov 24, 2011 at 02:24:13AM +0100, Clément Bœsch wrote:
> On Tue, Nov 22, 2011 at 08:52:14PM +0100, Reimar Döffinger wrote:
> > >  - Jacosub: this is somewhat an ancestor of ASS/SSA, and it's also a crazy
> > >    format. If we do things properly, we might want to support it (so mplayer
> > >    could for instance use FFmpeg internals to replace its own old subtitles
> > >    support, which includes a partial jacosub support). Note that this format
> > >    supports the *include* of external *images*. This is also something we need
> > >    to care about in the internal struct. I already wrote a demuxer a while ago,
> > >    and it could be used.
> > 
> > To my knowledge the structs already allow combining text, ass and image
> > rects, since each rect has a separate type.
> > You just need to be able to "normalize" them. E.g. all-bitmap for
> > rendering on video, if you want to be crazy all-text (even from bitmaps)
> > for some other uses...
> > 
> 
> Yes we always need a "normalized" bitmap we can blit (if that's what you
> meant); I think Nicolas mentioned that (I'll reply to everyone in order).

Well, is someone wants subtitles on a text-only display (e.g. braille)
they will want to convert them all to plain text.

> > >  - SAMI subtitles: this is HTML-only-valid-with-iexplorer5 crap, with CSS and
> > >    such. Converting them to ASS might be difficult but possible. Though, parsing
> > >    HTML is a pain: since it's not the only existing XML-like subtitles, we might
> > >    need to have subtitles with a dependency to a random HTML/XML library...
> > 
> > I don't really see the bother. We could add an extra type, but just
> > throw away (most) of the markup. Haven't seen many people complain about
> > it. Doesn't hurt to give people a reason to avoid inventing yet another
> > format in the case of subtitles.
> > 
> 
> I'm not sure if there is much benefit in having two engines like there is
> in MPlayer: one that strip most of the markup (because the old OSD is not
> able to render something complicated) and the one using libass for fancy
> layouts. But I indeed mentioned an extra "purified" text version of the
> subtitle (that could go along the normalized bitmap version in the
> AVSubtitle), but I think the simple form should be a "bonus", not a
> requirement.

Sure, if someone wants to convert CSS with HTML to ASS that's fine.
But if you want to force people to write a converter from whatever
strange format to properly formatted ASS they'll just give up on FFmpeg
and put their support somewhere else.

> > > This is all I can think about ATM. So if we want to support this properly, I can
> > > only think of this: libavsubtitles (with a dependency to libass to render a lot
> > > of crazy markup in subtitles).
> > > 
> > > This library will be used to demux/decode/etc (for conversion purpose too) and
> > > render the subtitles. And more importantly, we would be able to use it to burn
> > > them through the libavfilter with something like vf_subtitles.
> > 
> > I don't see why you would need or even want to have demux/decode in a
> > separate lib instead of where it is now.
> > You will want something to convert between the different AVRect types,
> > but whether that justifies a separate lib, especially if it's mostly
> > going to be a libass wrapper seems questionable to me.
> > 
> 
> I just though the way it is now is a bit confusing, and having most of the
> subtitles code in one place could help making things clear. Of course,
> having the main engine in a dedicated lavf (and lavc?) file would do the
> trick I guess.

Well, it depends on what code exactly you think about.

>  - hardsub: I would see a -vf subtitles=f=inputsub.srt:... just like the
>    proposed vf_ass. That filter would actually decode any kind of
>    subtitles (SRT included) and burn them with the appropriate settings.
>    To decode the subtitles and get the bitmap it would use libass, but
>    through the API.

Unless it is a source filter a filter has no business messing with files
directly. And I don't see why you should need or want any other filter
than src_movie, which already will have to handle subtitles anyway.
You then have AVSubtitle structs you can pass around - and they will
only have one of bitmap, ASS or plaintext - having a whole library
just for converting between those seems a bit like overkill.
By making the reading, decoding and hardsubbing a single filter you
destroy any possibility of doing any kind of processing without having
to implement it twice, once for soft- and once for hardsub.
E.g. adjusting the aspect of the subs, delaying it, adjusting
contrast/brightness, coloring them, ...

> In order to write the filter, we need to have an API that decode almost
> every subtitles and get the bitmap (and also deal with the fonts, that is
> an issue I forgot to mention earlier). How would that filter work? I guess
> it could call avformat/avcodec open routines, and decode it just like
> ffmpeg.c does, then just update the frame when the pts match.

src_movie should handle the opening and decoding, hopefully with not
much additional code.