[FFmpeg-devel] [RFC] Talk about subtitles

[Clément Bœsch]

Hi everyone,

[TL;DR: I want to improve/rewrite a real subtitles support in FFmpeg and need
some hints.]

Today again someone asked how to burn subtitles in a video. This is a recurrent
question which I think need to be solved now: some people still stick to the
unmaintained mencoder, try some pipes with MPlayer, or some lost code in
mplayer2, etc. And still nothing user friendly for this. Considering this, I
really wish to improve the situation with FFmpeg.

But before we make the same mistakes we already did, I'd like to discuss the
matter here, and share various issues with subtitles we have to considerate
before starting.

So first, what are the "required" features?
 - decode subtitles: we need a bitmap we can blit on a surface at a given
   position, and eventually a purified text version (so it could be printed to
   stdout) for a given pts.
 - convert subtitles: ATM we have mixed demuxer/decoder/encoder/muxer for
   SubRip, MicroDVD (and ASS right?). The question about how we will keep or
   make use of this will be discussed later.
 - we need to be able to support subtitles from muxed files and external files.

What could go wrong? Actually, a lot of things. Let me enumerate a few issues we
can have (and already have):

 - ASS/SSA: AFAIK we use internally a slightly modified representation of ASS to
   store all the different subtitles tracks. While now ASS is used to render
   almost everything, it might not be the perfect approach because it can't
   store the information of all the subtitles format (such as pts precision).
   See next points for various related headaches.

 - VOBSUB: this is basically a stream of bitmap subtitles + another file for
   indexes; we need to be able to support multiple inputs. Also note such
   subtitles are sometimes store in an archive...

 - Jacosub: this is somewhat an ancestor of ASS/SSA, and it's also a crazy
   format. If we do things properly, we might want to support it (so mplayer
   could for instance use FFmpeg internals to replace its own old subtitles
   support, which includes a partial jacosub support). Note that this format
   supports the *include* of external *images*. This is also something we need
   to care about in the internal struct. I already wrote a demuxer a while ago,
   and it could be used.

 - SubRip and MicroDVD: the *text* of these formats (excluding time
   representation then) can easily be converted to ASS markup. This has already
   been done by Aurelien in FFmpeg, and I also did it a while ago for MPlayer.
   This can be re-used, but writing them raises an issue: some crazy people mix
   the format markup with the ASS markup (you have ASS tags in the middle of
   SubRip specific tags for instance) because some players just pass them to the
   ASS library without any escaping. The current behaviour in mplayer2 is to
   escape those ASS tags while converting so they are displayed verbatim, but
   this might need to be configurable. So this is another thing to care about:
   we need to be able to pass options to the new API.

 - SAMI subtitles: this is HTML-only-valid-with-iexplorer5 crap, with CSS and
   such. Converting them to ASS might be difficult but possible. Though, parsing
   HTML is a pain: since it's not the only existing XML-like subtitles, we might
   need to have subtitles with a dependency to a random HTML/XML library...

 - Closed caption and teletext: I don't know them enough, but we also need to
   support them. If anyone can comment on this...

This is all I can think about ATM. So if we want to support this properly, I can
only think of this: libavsubtitles (with a dependency to libass to render a lot
of crazy markup in subtitles).

This library will be used to demux/decode/etc (for conversion purpose too) and
render the subtitles. And more importantly, we would be able to use it to burn
them through the libavfilter with something like vf_subtitles.

There is ATM a pending patch (vf_ass) that can workaround the situation for most
of the people: a lot of software allow the convert from srt to ass, so it won't
be needed to duplicate the srt to ass convert code yet another time for the
filter. Users will be satisfied enough for a while I think with a simple ASS
burning filter. It will give us some time to think about a proper way to write
and integrate libavsubtitles.

I already see people asking why we should support subtitles burning, so before
there are some flames about this:
 - some old divx players doesn't support them
 - some old config can not render ASS fast enough
 - most of the config can not render ASS fast enough with complicated karaoke
 - a lot of portable device don't have a subtitle support good enough
Those situations are common and need to be solved.

If anyone wants to work on this, please raise your hand. Also, if people working
on video players (Uoti? Reimar?) could comment on the way they would like to use
the API, please speak up! Nothing is really started ATM, and I'd like to work on
something where everyone could agree.

Thanks,

-- 
Clément B.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 490 bytes
Desc: not available
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20111122/48bc8d38/attachment.asc>