[Libav-user] Video and audio timing / syncing

[Alex Cohn]

On Sun, Mar 31, 2013 at 10:48 AM, Brad O'Hearne
<brado at bighillsoftware.com> wrote:
> On Mar 31, 2013, at 12:03 AM, Alex Cohn <alexcohn at netvision.net.il> wrote:
>> The trick is that not all containers and codecs support variable frame rate.
>
> Thanks for the reply, Alex. I'm somewhat torn with labeling this a case of a "variable frame rate" -- while being able to handle a variable frame rate would certainly solve the problem, the truth is that the video frames I'm receiving don't appear to actually be variable. The frame rate actually appears to be consistent the entire video (albeit not the frame rate I was initially expecting). To be completely specific about the nature of the problem, it appears the problem is that the frame rate isn't actually known until frames are received, but at the same time the time_base.den which is based on the frame rate needs to be configured prior to receiving captured frames.

If your use case allows to buffer enough frames to determine the
"actual, fixed but not as requested" frame rate, you can try to set
time_base accordingly and create the container accordingly. Otherwise,
it is equivalent to variable frame rate in my eyes. Unfortunately, not
all containers (avformat) and streams (avcodec) support arbitrary
frame rates. Most of these support variable frame rate (i.e. pts may
increase not by 1).

> This has me wondering about the nature of the AVPacket duration property. This is another value whose use is rather vague and inconsistent across example code and documentation. On one hand, the documentation says:
>
> "Duration of this packet in AVStream->time_base units, 0 if unknown. Equals next_pts - this_pts in presentation order."
>
> Virtually all examples I've read don't set the duration at all, in fact it isn't even referenced. So I'm not exactly sure how duration is used if it can be set and left at zero without a problem, and to what degree there is a relationship between it and pts. Given that QT capture delivers duration time with each sample buffer, then perhaps this can be used to manage the frame rate difference. For example, suppose that the codec context's time_base.den value is set to 30 (for a 30fps). However let's say capture produces and delivers only 15fps. That's essentially the same scenario I'm dealing with (though I'm looking for 24fps, not 30, but 30 makes a good round number for this example). Anyway, suppose that I receive only 15fps from capture, or to restate, half the number of frames as expected by time_base.den, what happens if the duration is set to 2 * 1/frame rate -- meaning that each frame had two frames worth of duration at the time_base frame rate.
>
> Will setting a longer packet duration compensate for receiving fewer frames?

I am not sure when "duration" is taken into account, but you could
simply set current->pts = prev->pts+2. Note that this was my original
proposal.

BR,
Alex