[FFmpeg-devel] Audio Video Sync issue with dvr-ms files

[Michael Niedermayer]

Hi

On Tue, May 01, 2007 at 11:02:15PM -0400, Jonathon Elwood wrote:
> Hi
> 
> On 5/1/07, Michael Niedermayer <michaelni at gmx.at> wrote:
> >Hi
> >
> >On Tue, May 01, 2007 at 02:47:31PM -0400, Jonathon Elwood wrote:
> >> On 4/3/07, Jonathon Elwood <jaelwood at gmail.com> wrote:
> >> >
> >> >
> >> >There is an AV sync issue with the latest SVN (8618).   I've uploaded a
> >> >sample file that demonstrates this issue to:
> >> >
> >> >ftp://upload.mplayerhq.hu/MPlayer/incoming/Backpackers -
> >> >_23_03_2007_23_00_03_comms.dvr-ms
> >> >
> >> >Michael Niedermayer recommended that I test with previous versions of
> >> >ffmpeg
> >> >and find the patch that broke the AV sync.   I traced the problem to 
> >patch
> >> >#7905.  ffmpeg 7904 worked fine, but 7905 and above demonstrate this AV
> >> >sync
> >> >issue.   I modified the latest SVN to reverse the changes made in patch
> >> >7905, and the AV sync looks good.  However, this obviously isn't a 
> >proper
> >> >fix.   Also without the changes in patch 7905 I have other dvr-ms shows
> >> >where the fps is detected incorrectly, so this patch is obviously 
> >needed.
> >> >Could you please look at this?   I'm not sure what the code in patch 
> >7905
> >> >is
> >> >doing, so I won't be able to create the appropriate patch.
> >> >
> >>
> >> I have been working with John Donaghy trying to solve the problems I
> >> reported above (thanks John for all your help).  I've made some
> >> progress fixing the timestamps (the video isn't "freezing" and the fps
> >> is detected correctly on all shows now); however, the video frames
> >> before the first keyframe are being discarded.  This is leading to a
> >> bad AV sync issue because the audio starts from the beginning of the
> >> stream, but the video starts from the first Key Frame.
> >> I've attached two patches (one to asf.c and the other to asf.h)
> >> showing the changes I've made so far.  These patches obviously aren't
> >> ready for submission so please be kind, this is obviously still a work
> >> in progress.  I've also attached a debug log file showing the output
> >> of my debug statements.
> >> My questions are:
> >> 1) If you look at the debug.txt file you will see the timestamps I'm
> >> now allocating to each segment.  However, I'm not sure what to do with
> >> the video segments before the first Key Frame.  My guess is I need to
> >> do something special to these timestamps so the frames aren't
> >> discarded, but what?  Right now I'm just setting them to
> >> AV_NOPTS_VALUE,
> >
> >AV_NOPTS_VALUE is correct if you dont know their timestamp
> >
> >
> >> however, they are being discarded.  I tried setting a
> >> value on the new segments, however, there was a big jump in the
> >> timestamp values once I got to the first Key Frame and they were
> >> discarded anyway.  So, I need to know what to do with these sequences.
> >> John suggested I just discard both the audio and video sequences
> >> until I get to the Video Key Frame.  I don't know how to do this, is
> >> there a mechanism in place to make this easy?
> >
> >a demuxer should not discard frames at random, if this causes a problem
> >then the bug is not in the demuxer
> >
> >
> >> 2) If you look at the segments marked "audio timestamp" you will
> >> notice that the audio is split into two segments.  I'm assigning the
> >> timestamp to the first segment, and setting the second segment to
> >> AV_NOPTS_VALUE.  Is this correct, or should the second segment also
> >> have a timestamp?  If the second segment has to have a timestamp, then
> >> what timestamp do I use (if I read the timestamp from the file, it is
> >> the same timestamp as the first segment)?
> >
> >i dont know that, but i dont think setting the timestamp for both is
> >correct, either first or second ... but iam just guessing, it depends on
> >what dvr-ms really does ...
> >
> >
> >> 3) Should both audio segments have the Key Frame flag set, or should I
> >> remove the Key Frame flag from the second segment?
> >>
> >> If there is anything else you see wrong regarding the timestamps I've
> >> allocated please let me know.  I've learned a lot, but all of this is
> >> still new to me.
> >
> >yes i will NOT accept a patch with some odd interpolation code without
> >fully understanding why that is needed
> >that is if your only explanation is like "it doesnt work without" then
> >the patch is rejected
> >
> >also all float/double calculations must be removed they are unacceptable
> >due to breaking binary reproduceable output between different architectures
> >
> >the debug output you provide below is fairly hard to read its not obvious
> >from it what timestamps where in the file, what would be wrong with them or
> >what better timestamps you set
> >
> >also without having looked at the files, did you try to simply set the
> >frame rate correctly if it was wrong? the internal timestamp code uses
> >the frame rate and will not work if its wrong ...
> 
> Given your input, I decided to start simple.  I've removed all the
> interpolation code and only set the timestamps on the first segment of
> a multi-segment payload.  I know my debug is a little hard to read,
> but I did modify it a little so hopefully you will be able to see
> what's happing with the timestamps.  Also, I'm setting the Key Frame
> based on the flag in the Replic Data (before this change every segment
> was marked as a Key Frame).  This fixes the AV sync issues on 60% of
> my samples (a big improvement).  Now I only two of my three origional
> outstanding issues.  First, I have some shows where the fps isn't
> being detected correctly.  I get the message "Seems stream 2 codec
> frame rate differs from container frame rate: 29.97 (30000/1001) ->
> 30.33 (91/3)".  I know I can fix this with all the interpolation code,
> but I'm going to take your advice and try setting the correct frame
> rate.
> 
> Sorry for asking such a basic question, but I'm not sure what
> variables I should look at to set the frame rate from the demuxer.
> After searching I found the variables below.  Which set would be
> appropriate for the demuxer?
> frame_rate and frame_rate_base
> r_frame_rate.num and r_frame_rate.den
> rfps and rfps_base
> time_base.den and time_base.num

try setting AVCodecContext.time_base and AVStream.r_frame_rate
that said this is for experimentation and might not be the final
solution ...


> 
> Lastly, I have a problem where the first frame is "frozen" until the
> video gets to the first Key Frame (of course after that the AV is
> completly out of sync).  I can fix this problem with the interpolation
> code as well, so maybe once I set the frame rate this problem will go
> away...
> 
> >> +            audio_video_type = get_byte(pb); //This value appears to 
> >identify the type of audio and video.
> >
> >can this be used to identify ac3/mp3 properly?
> On 99% of my samples yes.  However, I do have one sample where this
> value is very strange.  Having said that, I'm not convinced this value
> is unreliable because something else is going on with this sample.
> The timestamps are all messed up as well (for example: the stamps will
> be 192467, 189164, 193264).  Plus the Key Frame markers arn't there.
> I haven't look into this yet, but my guess is values have shifted in
> the header, so we are just looking in the wrong spot.  Once I figure
> out what's going on with this sample I'll be able to say whether this
> value will reliably identify the audio and video.  However, John
> Donaghy's code to probe for the audio type is working well for me so
> I'm not sure we even need this.

i dont like the current probe hack ...
if we can detected the the audio codec from some field in the header this
would be greatly preferred


[...]

> +			if (s->streams[asf->stream_index]->codec->codec_type == CODEC_TYPE_AUDIO) {
> +			   //only assign the timestamp to the first segment in a multiple segment payload
> +			   if (ts0 != -1) {
> +				   if (asf->last_aud_seq == asf->packet_seq)
> +					   asf->packet_frag_timestamp = AV_NOPTS_VALUE;
> +				   else
> +				       asf->packet_frag_timestamp= ts0/10000;

isnt a check based on packet_frag_offset possible and simpler?

also tabs are forbidden in svn

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

It is dangerous to be right in matters on which the established authorities
are wrong. -- Voltaire
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20070502/2ac16631/attachment.pgp>