[FFmpeg-devel] Audio Video Sync issue with dvr-ms files

Jonathon Elwood jaelwood
Thu May 10 23:28:52 CEST 2007


Hi

On 5/2/07, Michael Niedermayer <michaelni at gmx.at> wrote:
> Hi
>
> On Tue, May 01, 2007 at 11:02:15PM -0400, Jonathon Elwood wrote:
> > Hi
> >
> > On 5/1/07, Michael Niedermayer <michaelni at gmx.at> wrote:
> > >Hi
> > >
> > >On Tue, May 01, 2007 at 02:47:31PM -0400, Jonathon Elwood wrote:
> > >> On 4/3/07, Jonathon Elwood <jaelwood at gmail.com> wrote:
> > >> >
> > >> >
> > >> >There is an AV sync issue with the latest SVN (8618).   I've uploaded a
> > >> >sample file that demonstrates this issue to:
> > >> >
> > >> >ftp://upload.mplayerhq.hu/MPlayer/incoming/Backpackers -
> > >> >_23_03_2007_23_00_03_comms.dvr-ms
> > >> >
> > >> >Michael Niedermayer recommended that I test with previous versions of
> > >> >ffmpeg
> > >> >and find the patch that broke the AV sync.   I traced the problem to
> > >patch
> > >> >#7905.  ffmpeg 7904 worked fine, but 7905 and above demonstrate this AV
> > >> >sync
> > >> >issue.   I modified the latest SVN to reverse the changes made in patch
> > >> >7905, and the AV sync looks good.  However, this obviously isn't a
> > >proper
> > >> >fix.   Also without the changes in patch 7905 I have other dvr-ms shows
> > >> >where the fps is detected incorrectly, so this patch is obviously
> > >needed.
> > >> >Could you please look at this?   I'm not sure what the code in patch
> > >7905
> > >> >is
> > >> >doing, so I won't be able to create the appropriate patch.
> > >> >
> > >>
> > >> I have been working with John Donaghy trying to solve the problems I
> > >> reported above (thanks John for all your help).  I've made some
> > >> progress fixing the timestamps (the video isn't "freezing" and the fps
> > >> is detected correctly on all shows now); however, the video frames
> > >> before the first keyframe are being discarded.  This is leading to a
> > >> bad AV sync issue because the audio starts from the beginning of the
> > >> stream, but the video starts from the first Key Frame.
> > >> I've attached two patches (one to asf.c and the other to asf.h)
> > >> showing the changes I've made so far.  These patches obviously aren't
> > >> ready for submission so please be kind, this is obviously still a work
> > >> in progress.  I've also attached a debug log file showing the output
> > >> of my debug statements.
> > >> My questions are:
> > >> 1) If you look at the debug.txt file you will see the timestamps I'm
> > >> now allocating to each segment.  However, I'm not sure what to do with
> > >> the video segments before the first Key Frame.  My guess is I need to
> > >> do something special to these timestamps so the frames aren't
> > >> discarded, but what?  Right now I'm just setting them to
> > >> AV_NOPTS_VALUE,
> > >
> > >AV_NOPTS_VALUE is correct if you dont know their timestamp
> > >
> > >
> > >> however, they are being discarded.  I tried setting a
> > >> value on the new segments, however, there was a big jump in the
> > >> timestamp values once I got to the first Key Frame and they were
> > >> discarded anyway.  So, I need to know what to do with these sequences.
> > >> John suggested I just discard both the audio and video sequences
> > >> until I get to the Video Key Frame.  I don't know how to do this, is
> > >> there a mechanism in place to make this easy?
> > >
> > >a demuxer should not discard frames at random, if this causes a problem
> > >then the bug is not in the demuxer
> > >
> > >
> > >> 2) If you look at the segments marked "audio timestamp" you will
> > >> notice that the audio is split into two segments.  I'm assigning the
> > >> timestamp to the first segment, and setting the second segment to
> > >> AV_NOPTS_VALUE.  Is this correct, or should the second segment also
> > >> have a timestamp?  If the second segment has to have a timestamp, then
> > >> what timestamp do I use (if I read the timestamp from the file, it is
> > >> the same timestamp as the first segment)?
> > >
> > >i dont know that, but i dont think setting the timestamp for both is
> > >correct, either first or second ... but iam just guessing, it depends on
> > >what dvr-ms really does ...
> > >
> > >
> > >> 3) Should both audio segments have the Key Frame flag set, or should I
> > >> remove the Key Frame flag from the second segment?
> > >>
> > >> If there is anything else you see wrong regarding the timestamps I've
> > >> allocated please let me know.  I've learned a lot, but all of this is
> > >> still new to me.
> > >
> > >yes i will NOT accept a patch with some odd interpolation code without
> > >fully understanding why that is needed
> > >that is if your only explanation is like "it doesnt work without" then
> > >the patch is rejected
> > >
> > >also all float/double calculations must be removed they are unacceptable
> > >due to breaking binary reproduceable output between different architectures
> > >
> > >the debug output you provide below is fairly hard to read its not obvious
> > >from it what timestamps where in the file, what would be wrong with them or
> > >what better timestamps you set
> > >
> > >also without having looked at the files, did you try to simply set the
> > >frame rate correctly if it was wrong? the internal timestamp code uses
> > >the frame rate and will not work if its wrong ...
> >
> > Given your input, I decided to start simple.  I've removed all the
> > interpolation code and only set the timestamps on the first segment of
> > a multi-segment payload.  I know my debug is a little hard to read,
> > but I did modify it a little so hopefully you will be able to see
> > what's happing with the timestamps.  Also, I'm setting the Key Frame
> > based on the flag in the Replic Data (before this change every segment
> > was marked as a Key Frame).  This fixes the AV sync issues on 60% of
> > my samples (a big improvement).  Now I only two of my three origional
> > outstanding issues.  First, I have some shows where the fps isn't
> > being detected correctly.  I get the message "Seems stream 2 codec
> > frame rate differs from container frame rate: 29.97 (30000/1001) ->
> > 30.33 (91/3)".  I know I can fix this with all the interpolation code,
> > but I'm going to take your advice and try setting the correct frame
> > rate.
> >
> > Sorry for asking such a basic question, but I'm not sure what
> > variables I should look at to set the frame rate from the demuxer.
> > After searching I found the variables below.  Which set would be
> > appropriate for the demuxer?
> > frame_rate and frame_rate_base
> > r_frame_rate.num and r_frame_rate.den
> > rfps and rfps_base
> > time_base.den and time_base.num
>
> try setting AVCodecContext.time_base and AVStream.r_frame_rate
> that said this is for experimentation and might not be the final
> solution ...

The Good news is I didn't have to set the frame rate.  I found out the
segment_marker isn't in the same place for all samples.  The attached
patch accounts for this and now the fps is detected correctly for all
dvr-ms files.

>
>
> >
> > Lastly, I have a problem where the first frame is "frozen" until the
> > video gets to the first Key Frame (of course after that the AV is
> > completly out of sync).  I can fix this problem with the interpolation
> > code as well, so maybe once I set the frame rate this problem will go
> > away...
> >
> > >> +            audio_video_type = get_byte(pb); //This value appears to
> > >identify the type of audio and video.
> > >
> > >can this be used to identify ac3/mp3 properly?
> > On 99% of my samples yes.  However, I do have one sample where this
> > value is very strange.  Having said that, I'm not convinced this value
> > is unreliable because something else is going on with this sample.
> > The timestamps are all messed up as well (for example: the stamps will
> > be 192467, 189164, 193264).  Plus the Key Frame markers arn't there.
> > I haven't look into this yet, but my guess is values have shifted in
> > the header, so we are just looking in the wrong spot.  Once I figure
> > out what's going on with this sample I'll be able to say whether this
> > value will reliably identify the audio and video.  However, John
> > Donaghy's code to probe for the audio type is working well for me so
> > I'm not sure we even need this.
>
> i dont like the current probe hack ...
> if we can detected the the audio codec from some field in the header this
> would be greatly preferred
>
>
> [...]
>
> > +                     if (s->streams[asf->stream_index]->codec->codec_type == CODEC_TYPE_AUDIO) {
> > +                        //only assign the timestamp to the first segment in a multiple segment payload
> > +                        if (ts0 != -1) {
> > +                                if (asf->last_aud_seq == asf->packet_seq)
> > +                                        asf->packet_frag_timestamp = AV_NOPTS_VALUE;
> > +                                else
> > +                                    asf->packet_frag_timestamp= ts0/10000;
>
> isnt a check based on packet_frag_offset possible and simpler?
>
> also tabs are forbidden in svn
>
Unfortunately, I'm stuck?  With the attached patch the fps is now
detected correctly on all samples.  Also the AV sync is correct on 80%
of my samples.  This is a nice improvement, but I still have the
problem where the video is "stuck" on the first frame (this accounts
for the 20% where the AV sync is incorrect).  Unfortunately, I don't
know where to look.  Any help or guidance would be greatly
appreciated.
If anyone would like to see a sample that exhibits the "stuck frame"
problem, I uploaded the sample on April 3rd that demonstrated this
issue.   ftp://upload.mplayerhq.hu/MPlayer/incoming/ NBA
Basketball_WPTA-DT_28_01_2007_14_59_45.dvr-ms.  If this sample doesn't
exist on the ftp site any longer, please let me know and I'll upload
it again.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: asf.c.patch
Type: application/octet-stream
Size: 3218 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20070510/2910785b/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: asf.h.patch
Type: application/octet-stream
Size: 377 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20070510/2910785b/attachment-0001.obj>



More information about the ffmpeg-devel mailing list