[Libav-user] Video and audio timing / syncing

[Kalileo]

On Mar 30, 2013, at 04:49 , Brad O'Hearne wrote:

> On Mar 29, 2013, at 2:12 PM, Kalileo <kalileo at universalx.net> wrote:
> 
> All of the below really helps my understanding...I think a few more things I need to know to fill in the gaps:
>>> To the best of what I've been able to determine from mailing list responses, doc, and my testing, it would appear that these settings for audio don't have any material effect on settings for video and vice versa,
>> 
>> Correct, except that they are used for syncing.
> 
> Ok, is this a logical syncing (how synced video and audio appear to the user when played, i.e. an independent audio player just following audio its

Why?

If you play audio only, you do not need to sync with anything, DTS/PTS is not really needed, the length of an audio packet is determined already by its content. When one packet has finished playing the next is played.

> while an independent video player following video pts) or a literal syncing -- i.e. the player is doing a direct comparison of the video pts and audio pts values to determine sequencing. 
> 
>> Depends on your Player. In the case you describe the audio is the "master",
> 
>> Not correct. You can take the video timing as the master
> 
> What determines whether audio or video is the "master"? Is this something I need to specify in output format context, codec, or stream configuration? 

The player can do as it pleases = whoever coded the player decided how to handle that. There might be options to tell the player what you prefer.

> 
>>> 
>>> - If I completely turn off the writing of all audio frames, there is absolutely no change in video rendering -- it still renders video frames at twice the speed.
>> 
>> What Player are you using, what player shows that behavior?
> 
> My output format is an FLV file. Once I run my app and output an FLV file to my desktop, I've tried in both VLC and Wondershare Video Converter Ultimate -- same result. 

You give the player wrong info. You apparently missed what Alex told you earlier:

On Mar 30, 2013, at 03:43 , Alex Cohn wrote:
> This may sound oversimplified, but maybe it will be OK to set the PTS to 2× the value you tried today.

What I think you are still missing is the fact that audio packets have a fixed length. For every audio packet you can calculate how long it is (using audio sample rate, channels count, size of the (decoded) data). So an audio packet contains audio for exactly x ms. 

Video does not have that info built in that strongly, You can show a the image is correct wether displayed 1 ms or 100 ms. To decide how long to show it the player can use fps, which is much less precise than audio sample rate. And it might be not only fps, but also time base and ticks to consider.

If you play audio and video together, you will find that - despite playing each correctly - they drift apart after a while, and you can correct that drift by checking the DTS/PTS of each stream, and slowing down one or speeding up the other accordingly. Up tp the player to decide which one to correct.

Please do, as already suggested, some reading on the basics:

On Mar 30, 2013, at 04:12 , Kalileo wrote:
> You might want to study some examples based on the old Dranger's tutorial, where that stuff is explained in much better words than mine.


It's too long ago that I read them, so I cannot give you a link, but they helped me a lot to get started. Google for Dranger. You wouldn't need to ask all these questions if you already read that stuff.

Another thing you can do is to take a stream which plays correctly, and analyze the dts values of audio and video used there. This might show you right away where you're different.