[FFmpeg-devel] Discussion: How to extend the AVFrame to hold caption data?

[Jonathan Campbell]

While I'm finishing up the CNG patch, I'd like to start developing another feature that I think FFMPEG should be able to at least expose through the API.

I'm aware that FFMPEG supports subtitle tracks, but as far as I know, doesn't support caption data embedded in the video stream itself.

Some scenarios especially in broadcast TV transmit MPEG-2 or H.264 video with caption data embedded in the encoded frame. I would like to write a patch that can read the caption data embedded in MPEG-2 user packets and expose it in the AVFrame.

There are two common forms I work with in software: One carries EIA 608 closed caption packets per keyframe (typically seen on DVDs here in America), the other carries CEA 708 closed caption data (Caption Description Packets) every frame (or every I/P frame) with a side channel for 608 (typically seen in terrestrial HDTV broadcast here in America). How could I modify the MPEG-2 decoder and/or bitstream parser to read and expose this data through the API for software that needs it? I'm also interested in modding the MPEG-2 encoder to take this data through the API and encode it into the frame. If done right, the extension could allow FFMPEG to transcode DVD or HDTV broadcasts while keeping the caption data intact.

Similar standards exist to carry caption data packets in H.264 (as SEI packets) and DV video (as DIF packets).

Jonathan Campbell