[Libav-user] Repeating frame while encoding using ffmpeg api

[Bradley O'Hearne]

> On Sep 26, 2014, at 4:54 AM, Pradeep Karosiya <praks411 at gmail.com> wrote:
> 
> Hi,
> 
> I'm trying to create small screen capture application using ffmpeg encoding
> api.
> The application mostly working fine when I encode each frame as per
> specified frame rate.
> However I would like to do some changes in above approach. If there is no
> change on screen (frames are exactly same) then I don't want to encode the
> duplicate frames and want ffmpeg to keep the last displayed frame till
> something changes on screen. I'm doing following steps.
> 1. Check difference between previous and current frame, if difference == 0
> then skip frame else add frame to encoder.
> 2. In add_frame_encoder I'm encoding the previous screen with previous time
> stamp and setting extra_delay between current time stamp and previous time
> stamp in repeat_pict field of the encode frame.
> frame->repeact_pict = 2*fps*extra_delay
> where extra_delay is the duration for ffmpeg to display the frame. 
> 
> However the above approach is not working fine. Even though video duration
> is correct. When I'm playing the video the frame rate is reduced and I don't
> see repeat_pict option is working. 
> For example I'm encoding a video for 30 seconds at 10 fps containing many
> duplicate frames. The final 
> number of frames matches the frame which are added for encoding but I don't
> see any frame repeated by ffmpeg and final fps is reduced to match the video
> duration.
> Please suggest what could be going wrong.

Pradeep — I’m not the authority on this, I can only tell you about my personal experience. I am capturing live video and encoding it in real-time using FFmpeg. Over months of testing, the behavior I have seen is that whatever the time_base.den value is set on your codec, that number of frames per second absolutely has to be fed to the encoder, or else playback of the video will be out of sync. What this means is that if the capture mechanism produces frames at a different frame rate than the encoder is expecting, you have to either encode frames more than once if the capture frame rate drops below time_base.den (which in my case was constantly) or drop the frame if the capture frame rate exceeds time_base.den (which never happened). 

In my experience and testing over the course of months, changing pts, dts, and duration could not accommodate for this problem. As an example, no matter what the pts, dts, and duration were set at, feeding 15fps to an encoder with a time_base.den of 30 (30fps) produced a video that played back at twice the intended speed. Other folks posted to this list saying otherwise, but I have never seen a code example which could prove this, nor could I ever make it work in my own code, and neither could anyone else when I posted my code to this mailing list. I also received a number of offline responses to the list discussion from other developers who had arrived at the same conclusions from their own testing. So if there is a way to make it work, as far as I’m concerned it will remain legend until code emerges which demonstrates this can be done. 

Because of this fact, my conclusion — which is my own, I mean no harm to anyone else by stating this — is that FFmpeg’s design is particularly suited to non-realtime scenarios, where the whole of a video and its characteristics are known at the time of the creation of the codec and do not fluctuate during the encoding process. In other words, things are pretty straightforward if the video already exists, and is a known entity where you have control of the input to the encoder. But if the video is being captured in realtime, where a frame rate theoretically can fluctuate, or in my case commonly was received at half the time_base.den but with double duration (15fps vs. a time_base.den of 30fps), the only workable solution I found was to make the video look like it was static to the encoder — i.e. encode the same frame multiple times or drop the frame, depending on which was needed to keep the frame rate constant to time_base.den. It makes for ugly code…but it is working code. 

No one will be more thrilled than I if working code ever does emerge proving that merely setting pts, dts, and duration (or some other parameter) on the frame will cause the encoder to be able to properly encode video received at a frame rate different from time_base.den. But until that time, this is the only guidance I can offer. 

I hope you get things worked out, and if in your efforts you find a way to feed an encoder frames at a different frame rate than indicated by time_base.den, please post it. 

Good luck, 

Brad