[MPlayer-users] Bug: av sync, bframes, dwStart issues

[John Stebbins]

Thanks for the feedback and links to additional background.  I see your
rational and pretty much
agree with you (grudgingly).

Corey Hickey wrote:
> John Stebbins wrote:
>   
>> I've scanned over the mail list archives and found activity in this area
>> back in March 06 by John Koleszar and Corey Hickey.  Some fixes were
>> applied, but I believe the real problem is being masked, or worked
>> around rather than truly fixed.
>>
>> The symptom is audio leading video by 1 or 2 video frame times
>> (depending on bframe settings) at start of playback.  Currently,
>> mencoder masks this by applying an audio delay through the use of
>> dwStart in the audio stream header of the avi file. I believe this to be
>> an incorrect fix because bframes are not guaranteed to be generated even
>> though you have enabled them in your encoding.  Some video sequences may
>> be generated that contain no bframes, so your av sync will jitter by as
>> much as 2 video frame periods between sequences that contain bframes and
>> sequences that do not.
>>     
>
> Are you sure about this? I can't test right now, so that's not a 
> rhetorical question; I didn't think that was the case, though.
>   
I don't know specifically what mencoder will generate.  But I do know
that commercial encoders
are highly configurable and can generate such sequences.  This jitter
problem forces the av sync
threshold to be fairly sloppy.  I'm guessing that AV sync can't be kept
any tighter than about 45ms
without creating oscillations. Typical commercial decoders maintain AV
sync within about 25ms.
>   
>> This masking becomes more problematic when using encoder front-ends that
>> encode the audio and video separately and then mux them in a final step
>> using tools such as mkvmerge or ogmmerge.  OgmRip is a good example of
>> such a front-end. Because the streams are encoded separately and the
>> front-end knows little (nothing) about the delays that could be caused
>> by the various decoding processes, the front-end does not intelligently
>> insert compensating stream delays. And it shouldn't need to.
>>
>> I believe a correct fix belongs in mplayer and would delay the
>> presentation of initial audio and video until the decoder has queued
>> enough decoded video frames to handle its worst case video delay due to
>> bframe processing.  This would be done entirely by the player with no
>> hints from the encoder (e.g. dwStart fudge factor).  The player knows if
>> it supports bframe processing and should be able to delay initial
>> presentation until sufficient buffering of decoded frames has occurred
>> to handle its worst case delay that can be caused by bframe processing.
>>     
>
> As for the rest of your argument, I won't disagree with you. There was 
> no way to do it that was clearly right; while making mplayer compensate 
> for decoding delay seems correct, ideally, it ended up being more 
> practical to use dwStart because other players (at least the ones 
> tested) did not handle the delay either. If you want to implement a 
> different approach, feel free to do so, but be sure to read some of the 
> rationale behind the original decision.
>
> Here's where I confer with Michael about how to handle decoder delay. Be 
> sure to read the quoted material:
> http://lists.mplayerhq.hu/pipermail/mplayer-dev-eng/2006-January/040088.html
> http://lists.mplayerhq.hu/pipermail/mplayer-dev-eng/2006-January/040098.html
>
> Here's where Rich objects to dropping frames and Michael suggests 
> delaying other streams:
> http://lists.mplayerhq.hu/pipermail/mplayer-dev-eng/2006-January/040115.html
>
> Here's where I propose using dwStart to delay other streams:
> http://lists.mplayerhq.hu/pipermail/mplayer-dev-eng/2006-January/040126.html
>
>
> There's other information in the rest of the thread. Beware that I 
> didn't initially know what I was doing; that was a very educational 
> process for me.
>   
I can understand how my proposal could be difficult to wedge into
mplayer. I haven't tried to figure
out where specifically the code would need to be modified, but if it was
not initially designed to support
this kind of decoder management, it could be quite difficult to add.

I also see your point that most (if not all) the commonly used players
also have this problem.  I verified
totem-gstreamer and xine just this morning. Gstreamer is especially
broken as it doesn't even pay attention to dwStart.

The thing that really bugs me about this is that all encoders (including
many encoder front ends) need to be
aware of this ad-hoc requirement and "do the right thing".  In the case
of front-ends that encode audio and video
completely separately and then mux them into alternative containers,
they have to be aware of the decoding requirements
of the chosen codec and options selected for the codec. It also appears
that the ogg container format may not have the
equivalent of dwStart.  The method ogmmerge uses to delay audio is to
duplicate audio packets at the beginning
of the stream (ugh!).

I agree that it is very unclear what the right thing to do is.  In an
ideal world, all players would be fixed to handle
this sync issue transparently.  But I'm not prepared to lead a crazy
crusade to try to make that happen. So,
we get the most bang for the buck by doing what has been done in
mencoder.  I'll just have to see what can be
worked out with the authors of mkvmerge, ogmmerge, and ogmrip to make my
personal favorite choice of
tools work better.

Has anyone pinged the transcode people on this or tested what transcode
does?  Its been a while since I tried avimerge,
but it doesn't even have an option for setting a delay on a stream.

> -Corey
> _______________________________________________
> MPlayer-users mailing list
> MPlayer-users at mplayerhq.hu
> http://lists.mplayerhq.hu/mailman/listinfo/mplayer-users
>