[MPlayer-dev-eng] A better explanation of the A/V sync problem

[Ross Finlayson]

I was able to figure out more details about what was causing the A/V sync 
problem.  Basically, the problem is caused by the fact that UDP streams - 
unlike TCP - have no control backchannel, so incoming UDP packets keep 
arriving (to be buffered within the OS) at roughly the same rate, 
regardless of the rate at which MPlayer reads them.  Because MPlayer 
current does A/V synchronization primarily by adjusting the rate at which 
it reads audio and/or video (rather than by dropping audio or video 
frames), it has difficulty maintaining synchronization for such streams.

This problem will occur for any audio/video session in which at least one 
of the demuxers uses UDP.  I was able to add a fix (described below) to my 
RTP demuxing code to overcome the problem, but in the long term, I think it 
would be better if the general MPlayer A/V sync code were to do this instead.

Once again, 'pausing' provides a good illustration of the problem.  Suppose 
that the user pauses an audio/video RTP session.  Although MPlayer is not 
reading any packets during the pause, packets continue to stream in over 
the network, to be buffered by the OS's UDP socket buffer.  (The size of 
this OS buffer is configurable; I have set it to be quite large: 2 Mbytes 
for the video socket, and 100 kBytes for the audio socket.)  Nonetheless, 
during a pause, these buffers will eventually fill up, after which any 
subsequent arriving packets get 'dropped on the floor'.  Usually, the video 
socket's buffer fills up first.  For illustration, suppose that, during the 
pause, 10 seconds worth of video data got lost, but no audio data got lost.

After the pause, MPlayer continues reading audio and video packets once 
again.  Soon, it will reach the point at which video data was lost, at 
which point it will see the video packets' RTP timestamps (and thus 
presentation times) jump by 10 seconds.  At this point, video is 10 seconds 
ahead of audio!  MPlayer tries to recover from this by reading from the 
video demuxer at a slower rate than from the audio demuxer - but this just 
has the effect of causing the OS's video socket buffer to fill up once 
again!  Although MPlayer begins to get back in sync, once the OS video 
buffer fills up (and video packets get lost again), MPlayer will see 
another large gap in the video data, which will cause it to get way out of 
sync yet again.  This cycle can continue indefinitely.

(I have also seen a similar situation occur - although less repeatably - 
when video starts dropping behind audio.)

The obvious solution to this problem is to notice when the presentation 
times on the two incoming streams start differing by more than a 
threshold.  When this happens, start discarding incoming packets from the 
stream that is running behind, until its presentation times catch up to the 
other stream.  This is what I've now done in my RTP demuxer code.  In the 
'pausing' example described above, when the code notices that video has 
jumped way ahead of audio, it will read and discard all following audio 
packets until audio catches up again.  I.e., if incoming data for one 
stream gets dropped (by the OS) during the pause, then a corresponding 
amount of data for the other stream will also get dropped (by MPlayer's RTP 
demuxer) to compensate.

Within a day or so I will be submitting a patch to my RTP code that 
includes this fix (plus a lot more goodies, such as MPEG-4 
support).  However, I think that in the longer term, it might be better if 
the general MPlayer A/V sync code did this instead.

	Ross.