[FFmpeg-devel] [RFC] Implementation of closed caption support

[Reimar Döffinger]

On Mon, Aug 04, 2014 at 08:19:38PM +0200, Hendrik Leppkes wrote:
> Am 04.08.2014 19:59 schrieb "Gisle Sælensminde" <gisle at snirklasjon.no>:
> >
> > I'm trying to add support for closed captions in ffmpeg, namely cea608 and
> > cea708. Unlike normal subtitles, these are embedded in the video frames
> > themselves rather than a separate track or file. As a first step I try to
> write
> > bitstreams filters for extraction and inserting closed captions into
> h.264 and
> > mpeg2 videos. For cea608, I use .scc files to represent the cea608 data.
> > Later an interpreter and producer of scc files can be made as a subtitle
> > codec.
> >
> > I have already written a cea 608 tools for my employer, but that is in
> > python, so not suitable for inclusion in ffmpeg, so this will be a
> > new implementation.
> >
> > I'm currently writing the extraction filter and I had hoped that I could
> use
> > it as a filter with parameters as follows.
> >
> > ffmpeg -i vid_with_cc.ts -acodec copy -vcodec copy -bsf
> cea680_extract?scc=out.scc -f null /dev/null
> >
> > This is not possible, since bitstream filters don't have parameters. The
> main
> > problems can be summarized as follows:
> >
> > - The lack of parameters means that I don't have a way to specify where
> to store
> >   the .scc file. How can I do that?
> >
> > - I can't find a way to get out the timestamps (pts and dts) in a
> bitstream
> >   filter. The cea 608 and 708 data is stored in pts order in the frames,
> so
> >   I need to reorder the data before writing them to file. Also, the scc
> files
> >   have timestamps, so I need timestamps for that too. I can only find the
> >   timebase in AVCodecContext, and the AVFormatContext or AVPacket is not
> passed
> >   in to the bitstream filter. The AVPacket is a deprecated field, but it
> is
> >   NULL, so of no use. Is there something I have overlooked here?
> >
> > I may be missing something, and it is of cause possible that the
> bitstream filter
> > approch is misguided, but as far as I can tell, that is the best option.
> Does anyone
> > have a clear idea of how this should be done?
> >
> Its probably a better idea to export the CC data during video decoding as
> side data, like the mpeg2 decoder already does today. That way you get it
> reordered and with timestamps, but of course it means you need to perform
> decoding.

It's also fairly silly, since it means you have to decode the video just
to get the subtitle stream! And as far as I can tell if you want to remux
but with separate subtitle stream that would even mean that you have to
re-encode the video for no good reason.
A _good_ (but complex/quite some effort) solution would be for the
AVParser to extract it (optionally even removing it from the video frames?)
and create a proper subtitle stream.