[FFmpeg-devel] [RFC] Implementation of closed caption support

[Reimar Döffinger]

On Mon, Aug 04, 2014 at 08:51:57PM +0200, wm4 wrote:
> On Mon, 4 Aug 2014 20:47:37 +0200
> Reimar Döffinger <Reimar.Doeffinger at gmx.de> wrote:
> 
> > On Mon, Aug 04, 2014 at 08:19:38PM +0200, Hendrik Leppkes wrote:
> > > Am 04.08.2014 19:59 schrieb "Gisle Sælensminde" <gisle at snirklasjon.no>:
> > > >
> > > > I'm trying to add support for closed captions in ffmpeg, namely cea608 and
> > > > cea708. Unlike normal subtitles, these are embedded in the video frames
> > > > themselves rather than a separate track or file. As a first step I try to
> > > write
> > > > bitstreams filters for extraction and inserting closed captions into
> > > h.264 and
> > > > mpeg2 videos. For cea608, I use .scc files to represent the cea608 data.
> > > > Later an interpreter and producer of scc files can be made as a subtitle
> > > > codec.
> > > >
> > > > I have already written a cea 608 tools for my employer, but that is in
> > > > python, so not suitable for inclusion in ffmpeg, so this will be a
> > > > new implementation.
> > > >
> > > > I'm currently writing the extraction filter and I had hoped that I could
> > > use
> > > > it as a filter with parameters as follows.
> > > >
> > > > ffmpeg -i vid_with_cc.ts -acodec copy -vcodec copy -bsf
> > > cea680_extract?scc=out.scc -f null /dev/null
> > > >
> > > > This is not possible, since bitstream filters don't have parameters. The
> > > main
> > > > problems can be summarized as follows:
> > > >
> > > > - The lack of parameters means that I don't have a way to specify where
> > > to store
> > > >   the .scc file. How can I do that?
> > > >
> > > > - I can't find a way to get out the timestamps (pts and dts) in a
> > > bitstream
> > > >   filter. The cea 608 and 708 data is stored in pts order in the frames,
> > > so
> > > >   I need to reorder the data before writing them to file. Also, the scc
> > > files
> > > >   have timestamps, so I need timestamps for that too. I can only find the
> > > >   timebase in AVCodecContext, and the AVFormatContext or AVPacket is not
> > > passed
> > > >   in to the bitstream filter. The AVPacket is a deprecated field, but it
> > > is
> > > >   NULL, so of no use. Is there something I have overlooked here?
> > > >
> > > > I may be missing something, and it is of cause possible that the
> > > bitstream filter
> > > > approch is misguided, but as far as I can tell, that is the best option.
> > > Does anyone
> > > > have a clear idea of how this should be done?
> > > >
> > > Its probably a better idea to export the CC data during video decoding as
> > > side data, like the mpeg2 decoder already does today. That way you get it
> > > reordered and with timestamps, but of course it means you need to perform
> > > decoding.
> > 
> > It's also fairly silly, since it means you have to decode the video just
> > to get the subtitle stream! And as far as I can tell if you want to remux
> > but with separate subtitle stream that would even mean that you have to
> > re-encode the video for no good reason.
> > A _good_ (but complex/quite some effort) solution would be for the
> > AVParser to extract it (optionally even removing it from the video frames?)
> > and create a proper subtitle stream.
> 
> Isn't the problem with this that CC data needs to be reordered like the
> video frames? That's AFAIK why it was made video frame side-data,
> instead of extracting it with a parser.

Yes, that's one of the things that make it complex to implement.
But the parser certainly has all the information to do the reordering.
It still is the only proper solution, its simplicity is the only
thing that makes the current hack acceptable.
And I can't see how it could be called anything but a hack.