[FFmpeg-devel] [RFC] Implementation of closed caption support

[wm4]

On Mon, 4 Aug 2014 20:47:37 +0200
Reimar Döffinger <Reimar.Doeffinger at gmx.de> wrote:

> On Mon, Aug 04, 2014 at 08:19:38PM +0200, Hendrik Leppkes wrote:
> > Am 04.08.2014 19:59 schrieb "Gisle Sælensminde" <gisle at snirklasjon.no>:
> > >
> > > I'm trying to add support for closed captions in ffmpeg, namely cea608 and
> > > cea708. Unlike normal subtitles, these are embedded in the video frames
> > > themselves rather than a separate track or file. As a first step I try to
> > write
> > > bitstreams filters for extraction and inserting closed captions into
> > h.264 and
> > > mpeg2 videos. For cea608, I use .scc files to represent the cea608 data.
> > > Later an interpreter and producer of scc files can be made as a subtitle
> > > codec.
> > >
> > > I have already written a cea 608 tools for my employer, but that is in
> > > python, so not suitable for inclusion in ffmpeg, so this will be a
> > > new implementation.
> > >
> > > I'm currently writing the extraction filter and I had hoped that I could
> > use
> > > it as a filter with parameters as follows.
> > >
> > > ffmpeg -i vid_with_cc.ts -acodec copy -vcodec copy -bsf
> > cea680_extract?scc=out.scc -f null /dev/null
> > >
> > > This is not possible, since bitstream filters don't have parameters. The
> > main
> > > problems can be summarized as follows:
> > >
> > > - The lack of parameters means that I don't have a way to specify where
> > to store
> > >   the .scc file. How can I do that?
> > >
> > > - I can't find a way to get out the timestamps (pts and dts) in a
> > bitstream
> > >   filter. The cea 608 and 708 data is stored in pts order in the frames,
> > so
> > >   I need to reorder the data before writing them to file. Also, the scc
> > files
> > >   have timestamps, so I need timestamps for that too. I can only find the
> > >   timebase in AVCodecContext, and the AVFormatContext or AVPacket is not
> > passed
> > >   in to the bitstream filter. The AVPacket is a deprecated field, but it
> > is
> > >   NULL, so of no use. Is there something I have overlooked here?
> > >
> > > I may be missing something, and it is of cause possible that the
> > bitstream filter
> > > approch is misguided, but as far as I can tell, that is the best option.
> > Does anyone
> > > have a clear idea of how this should be done?
> > >
> > Its probably a better idea to export the CC data during video decoding as
> > side data, like the mpeg2 decoder already does today. That way you get it
> > reordered and with timestamps, but of course it means you need to perform
> > decoding.
> 
> It's also fairly silly, since it means you have to decode the video just
> to get the subtitle stream! And as far as I can tell if you want to remux
> but with separate subtitle stream that would even mean that you have to
> re-encode the video for no good reason.
> A _good_ (but complex/quite some effort) solution would be for the
> AVParser to extract it (optionally even removing it from the video frames?)
> and create a proper subtitle stream.

Isn't the problem with this that CC data needs to be reordered like the
video frames? That's AFAIK why it was made video frame side-data,
instead of extracting it with a parser.