[Libav-user] Frame order from TS demux - mpeg2video vs h264

[jettoblack]

Hello,
I have a question about determining the display order of picture frames when
demuxing a MPEG-2 transport stream that contains either MPEG-2 video or
H.264 video.

I'm using ffmpeg git head latest as of the past couple days.  My code uses
libavformat to demux the stream, add some metadata which must be in picture
display order, then remux the stream.  I have a loop to get packets from the
source, modify them if it's a video packet, and write packets back out to
the output muxer.  This is working fine for MPEG-2 video streams but I'm
having some issues getting the proper frame order when demuxing TS with
H.264 video.  I tried decoding the codec to a picture and reading
picture.coded_picture_number but this doesn't always give the right display
order either.  Is there a better way?

Both test files were created with ffmpeg latest and from the same source,
only difference is one is MPEG-2 video/audio and the other is H.264/AAC.

Here is the basic loop, edited to be concise (full compilable source code
included in link below):

    while (1) {
        av_init_packet(&pkt);
        r = av_read_frame(in, &pkt);

        if (pkt.stream_index == videoStream) {
            avcodec_decode_video2(in->streams[videoStream]->codec, &picture,
&got_picture, &pkt);
            if (got_picture) {
                printf("picture_packet %d, coded_picture_number %d, pts
%"PRId64"\n", picture_packets, picture.coded_picture_number, pkt.pts);
                
                // very simple test to insert a byte that should increase in
displayed frame order
                // note: I know this is not a proper way to insert extra
data, this is for simple testing only
                av_grow_packet(&pkt, 1);
                pkt.data[pkt.size - 1] =
(uint8_t)(picture.coded_picture_number & 0xff);
                
                picture_packets++;
            }
        }
        
        // write frame
        r = av_interleaved_write_frame(out, &pkt);
        av_free_packet(&pkt);        
    }

When the source file has MPEG-2 video, the frames come out of the demuxer in
frame display order:
picture_packet 0, coded_picture_number 0, pts 129003
picture_packet 1, coded_picture_number 1, pts 132006
picture_packet 2, coded_picture_number 2, pts 135009
picture_packet 3, coded_picture_number 3, pts 138012

So my code is working fine for MPEG-2 video streams.  I can read the output
stream and when demuxed back into picture display order, all of my metadata
is in the proper order.

However when the source file contains H.264 video, the packets are not
demuxing in picture display order, nor is the pattern consistent:
picture_packet 0, coded_picture_number 0, pts 156000
picture_packet 1, coded_picture_number 2, pts 153150
picture_packet 2, coded_picture_number 1, pts 162150
picture_packet 3, coded_picture_number 4, pts 159150
...
picture_packet 146, coded_picture_number 145, pts 595200
picture_packet 147, coded_picture_number 147, pts 592350
picture_packet 148, coded_picture_number 148, pts 601350
picture_packet 149, coded_picture_number 150, pts 598350
picture_packet 150, coded_picture_number 149, pts 607350


If I make my algorithm depend on the coded_picture_number, the results are
right most of the time, but every 150 frames or so, I end up getting 2
frames swapped in order, as if the coded_picture_number is getting swapped
once in a while.  E.g. when decoding the resulting output stream, my
metadata ends up with the order "144 145 146 148 147 149 150..." instead of
being in order.  Notice above that the pts for coded_picture_number 150 is
less than the pts for coded_picture_number 148 or 149.

Any ideas?  What is the best way to reliably tell get the frames' picture
display order when demuxing a MPEG-2 TS w/ H.264 video?  Is there a way to
convert PTS back into a display-order frame number?

Source code: http://dl.dropbox.com/u/7730988/main.c.zip
Two sample videos (3MB): http://dl.dropbox.com/u/7730988/sotu11.zip

Thanks!


--
View this message in context: http://libav-users.943685.n4.nabble.com/Frame-order-from-TS-demux-mpeg2video-vs-h264-tp4597204p4597204.html
Sent from the libav-users mailing list archive at Nabble.com.