[FFmpeg-user] How to dynamically drop frames as required to keep transcoding real time?

[Adrian Cable]

Hi,
I have a question that I thought should have an ‘easy’ answer – I have spent a lot of time searching and I cannot find it, so thought I would ask here.

My application sounds simple. I have an input RTSP stream, which I need to transcode and output as an RTP stream. Command line is very simple, something like:

./ffmpeg -i rtsp://… -vf scale=1024:-1 -c:v libx264 -f rtp rtp://…

Now, ffmpeg is running on a CPU-constrained platform, and it may be that, in order for the transcoding to ‘keep up’ with the input, the output framerate needs to be less than the input framerate, which is totally fine. The problem/question is: how do I get ffmpeg to drop frames as required to keep the pipeline running real time?

I can’t simply ‘guess’ a sustainable output framerate and set using -r XXX or -vf fps=fps=XXX, because the properties of the input can change at any time, so any XXX won’t be constant. Besides, if the ‘guess’ is too high for the CPU power I have available, the encoder won’t keep up with the input frames and so will ‘run away’ (i.e. the time difference between each input frame and its corresponding output frame will get bigger and bigger), which is the problem I am trying to solve. On the other hand, if the ‘guess’ is too low, I will end up with real-time output but I will also be potentially transcoding to a lower output frame rate than I have the CPU power to do, which I don’t want either.

What I need is some kind of dynamic framerate filter that doesn’t take a number at all (or maybe takes simply a maximum), and drops frames if and only if the encoder can’t keep up … so the pipeline produces as high a frame rate output as possible, while still being real time.

I cannot figure out how to do this. Surely this wish can’t be super uncommon as you would imagine it would be quite important in all sorts of streaming applications where unlimited transcoding power is not guaranteed to be available. Can anyone help?

-Adrian