[FFmpeg-devel] Discussion: How to extend the AVFrame to hold caption data?

Jonathan Campbell jonathan at impactstudiopro.com
Sun Sep 4 09:34:24 EEST 2016

On 09/03/2016 11:26 PM, Andrey Turkin wrote:
> AVFrame already has some support for embedded 608/708 subtitles in a form
> of side data (AV_FRAME_DATA_A53_CC tag), and some decoders/encoders already
> can handle that.
> Main issue with this approach is that video frames can be modified, removed
> or duplicated (mainly by video filters but also by ffmpeg's video sync
> code).
> 2016-09-04 6:14 GMT+03:00 Jonathan Campbell <jonathan at impactstudiopro.com>:
>> While I'm finishing up the CNG patch, I'd like to start developing another
>> feature that I think FFMPEG should be able to at least expose through the
>> API.
>> I'm aware that FFMPEG supports subtitle tracks, but as far as I know,
>> doesn't support caption data embedded in the video stream itself.
>> Some scenarios especially in broadcast TV transmit MPEG-2 or H.264 video
>> with caption data embedded in the encoded frame. I would like to write a
>> patch that can read the caption data embedded in MPEG-2 user packets and
>> expose it in the AVFrame.
>> There are two common forms I work with in software: One carries EIA 608
>> closed caption packets per keyframe (typically seen on DVDs here in
>> America), the other carries CEA 708 closed caption data (Caption
>> Description Packets) every frame (or every I/P frame) with a side channel
>> for 608 (typically seen in terrestrial HDTV broadcast here in America). How
>> could I modify the MPEG-2 decoder and/or bitstream parser to read and
>> expose this data through the API for software that needs it? I'm also
>> interested in modding the MPEG-2 encoder to take this data through the API
>> and encode it into the frame. If done right, the extension could allow
>> FFMPEG to transcode DVD or HDTV broadcasts while keeping the caption data
>> intact.
>> Similar standards exist to carry caption data packets in H.264 (as SEI
>> packets) and DV video (as DIF packets).
>> Jonathan Campbell
>> _______________________________________________
>> ffmpeg-devel mailing list
>> ffmpeg-devel at ffmpeg.org
>> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
Cool, that's why I asked. I see it now in mpeg12dec.c

It looks like mpeg12enc.c though is lacking the code to read the A53_CC side data and put the caption data into the encoded frame. Mind if I write a patch to do that?

Jonathan Campbell

More information about the ffmpeg-devel mailing list