[Libav-user] Video and audio timing / syncing

[Brad O'Hearne]

Onward! It sure feels like I'm on the verge of knocking this use case out that I've been working on. With a little more help, I think a win is not far off...

So coming in off of QTKit capture and being processed using FFmpeg, I've now got perfect (sounding) audio. It would also appear that I have proper video too. However, the audio and video are out of sync -- great for a dubbed karate movie, not so great for customer usage (though I recommend they all learn karate). My suspicions are the following: 

1. Either it has something to do with the AVPacket pts and duration settings, which I am setting using time_base adjusted values for presentationTime and duration coming from the QTSampleBuffer. 

2. That the codec is not properly queuing / arranging the video frames and audio sample buffers in proper order. I'm not familiar with the internals of the adpcm_swf codec, or most video codecs in general, but I have had a question about timing. QTKit delivers captured audio samples and video frames on different callbacks, so technically, the time and frequency each are received are arbitrary. I'm processing each when received, so I believe in theory it might be possible for video frames and audio samples to be received out of order. Is the codec smart enough to arrange these in proper order? Is there a setting on the codec which governs or affects whatever internal queue or buffer time the codec waits for another packet with an earlier presentation time before outputting the buffer last sent to it? 

I would think the codec would have to be smart enough to handle this in encoding. However, after reading the source in this example: 

http://ffmpeg.org/doxygen/0.6/output-example_8c-source.html

it appears that the sample is managing the order in which video and audio frames are written. Any insight into the proper approach? Other than that one example, I've seen no other examples which worry about sorting audio and video packets based on pts, they all appear to let that be something the codec worries about. 

If this isn't the source of the problem, can someone enlighten me as to what the usual suspects are when you have out of sync audio and video? 

Thanks, 

Brad