[FFmpeg-devel] [RFC] Talk about subtitles

[Reimar Döffinger]

On Tue, Nov 29, 2011 at 01:49:48PM +0100, Nicolas George wrote:
> Le quintidi 5 frimaire, an CCXX, Reimar Döffinger a écrit :
> > Well, is someone wants subtitles on a text-only display (e.g. braille)
> > they will want to convert them all to plain text.
> 
> Do you want to consider including an OCR in the encoding process?

In a perfect world yes.

> (And this is not a completely stupid idea: most bitmap subtitles are
> rendered in a way that makes it possible to do a perfect OCR without too
> much effort.)

Perfect is unlikely, but at least "good enough" could be nice.
But it might be quite some effort to do and get right.
However I'd consider that even more reason to avoid a design that
makes this even more painful, because I think it would be a really
nice feature.

> > Unless it is a source filter a filter has no business messing with files
> > directly. And I don't see why you should need or want any other filter
> > than src_movie
> 
> I do not agree with that. src_movie has a basic limitation that comes from
> libavfilter's design: all filters/sources/sinks must have a fixed number of
> input and output pads. Therefore, you can not just write "movie=file.mkv"
> and get all streams in the file, you need to write "movie=file.mkv" for the
> video and then "smovie=file.mkv" for the subtitles, and possibly also
> "amovie=file.mkv" if we want to filter the audio too.

Maybe we misunderstand each other, but I don't see how this is a
counter-point.
If you'd have the subtitle filter read the file you'd still have to
specify it multiple times and you either have to reimplement the movie
filter or you can (like vf_ass) only handle a very, very limited set of
files/streams.
What you say are I guess arguments for
1) supporting FFmpeg directly connecting to a subtitle filter pad
2) fixing the design to handle movie sources better (is this really a design
   issue or more a current implementation issue?)
and one point I forgot. But not really for having the subtitle filter
read from a file by itself?

> The other side of this problem is that for ffmpeg (the command-line tool),
> filters are a per-input thing. We can not write something like that:
> 
> ffmpeg \
>   -i movie.mpg \
>   -iname logo -i logo.png \
>   -vf '[video][logo]overlay' \
>   -acodec copy movie.avi
> 
> And for the same reason, we will not be able to write something like that:
> 
> ffmpeg -i movie.mkv -vf '[video] [sub-fr] hardsub' -acodec copy movie.avi
> 
> Until that problem is solved, any possible solution will be awkward and
> basically a hack.

A hack with a smovie filter however would result in a filter that is
usable as-is once these are fixed.

> And I did not even speak about embedded fonts...

I would expect data stream pads should be hackable for that purpose
somehow. Admittedly we don't have that yet I guess.
If using fontconfig I think that FFmpeg could just register all embedded
fonts with fontconfig and any subtitle renderer using fontconfig can
then just use them.
Would however break with multiple files with a different font but the same
name/attributes.