[FFmpeg-user] A53 Captions Garbled If Frame Duplication Occurs

[Dennis Mungai]

On Tue, 7 May 2024 at 18:21, Devin Heitmueller <
devin.heitmueller at ltnglobal.com> wrote:

> Hi Shane,
>
> On Fri, May 3, 2024 at 5:48 PM Shane Warren <shanew at innovsys.com> wrote:
> >
> > I ingest some multicast video that has a53 captions embedded in h264
> video, the video claims to have a framerate of 29.97 but it must not quite
> be that because ffmpeg duplicates frames when transcoding this stream. The
> output stream has a53 captions, but they appear to be somewhat garbled,
> like some letters are duplicated, sometimes.
> >
> > I decided to git bisect since I knew this worked in a much older ffmpeg
> binary I had. I finally found the commit that broke this:
> >
> > a11ab647304307d0340fd052e04cf638b6bef309 : fftools/ffmpeg: share the
> code encoding a single frame between video and audio
> >
> > Call do_video_stats() for every video packet produced by the encoder,
> > rather than for every frame sent to the encoder.
> >
> > ------------------------------------------------------------------------
> >
> > The key part that was missed in this commit was calling this directly
> after avcodec_send_frame:
> >
> > // Make sure Closed Captions will not be duplicated
> > av_frame_remove_side_data(in_picture, AV_FRAME_DATA_A53_CC);
> >
> > I fixed this in that commit, and captions worked again. I tried to fix
> it the same way in a the current ffmpeg source by changing:
> >
> > ffmpeg_enc.c encode_frame:
> >
> >     ret = avcodec_send_frame(enc, frame);
> >     if (ret < 0 && !(ret == AVERROR_EOF && !frame)) {
> >         av_log(ost, AV_LOG_ERROR, "Error submitting %s frame to the
> encoder\n",
> >                type_desc);
> >         return ret;
> >     }
> >
> >     +if (frame != NULL)
> >     +{
> >     +   av_frame_remove_side_data(frame, AV_FRAME_DATA_A53_CC);
> >     +}
> >
> > But that didn't fix the problem, I suspect this is needed when a frame
> is duplicated, but I can't figure out the place to put that.
> >
> > Anyone have any ideas what I need to change to get captions working in
> the current ffmpeg source?
> > _______________________________________________
> > ffmpeg-user mailing list
> > ffmpeg-user at ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-user
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-user-request at ffmpeg.org with subject "unsubscribe".
>
> This was discussed last May in the following thread:
>
> https://ffmpeg.org/pipermail/ffmpeg-devel/2023-June/310534.html
>
> The easiest thing for you to do is use the "fps" filter (i.e. -vf
> fps=60000/1001) rather than using "-r".  That makes use of the ccfifo
> framework to ensure that the data is properly preserved.  The hack to
> do it with "-r" actually doesn't work correctly in a number of cases
> (e.g. reducing the framerate), and in fact even with upconverting to
> 59.94 the data doesn't actually conform to the spec, although it works
> in some decoders (specifically you should have every frame with a
> cc_count of 10, when in fact you have it alternating between 20 and
> being absent).
>
> Devin
>
> --
> Devin Heitmueller, Senior Software Engineer
> LTN Global Communications
> o: +1 (301) 363-1001
> w: https://ltnglobal.com  e: devin.heitmueller at ltnglobal.com
>

Interesting.

Last time I tried to use the fps filter(s) with NVENC and OneVPL, both of
these platforms misbehaved.
For some reason, both seem to require a hint to the "frame length/duration"
, and omitting -r here and switching to the fps filter with these two
implementations wrecks rate-control.
The only work-around to this in FFmpeg with either NVENC or QSV/OneVPL is
tapping into libplacebo OR qsv_vpp filters respectively.
I'm not sure how these two filters handle A53CC emission with frame-rate
upconversion. Will definitely re-test over the course of the week.

BR,

Dennis.