[Ffmpeg-devel] Problems with output picture reordering inH.264 decoder

[Loren Merritt]

On Thu, 10 Aug 2006, Santa Cruz, Diego (GE Indust, Security) wrote:

> I guess the best solution would be to limit the amount of reordering by
> using the num_reorder_frames if present and use the dpb_output_delay in
> the SEI picture timing to know when to output each picture and the POC
> to actually reorder the pictures. I think that is the way it is supposed
> to work in H.264. Without the dpb_output_delay a decoder can only
> guarantee output order conformance but not *timing*, so it would not be
> possible to guarantee a "no jerk" operation if dpb_output_delay is not
> used.

No SEIs are necessary. Picture timing SEIs are useful only if you're 
storing them in an elementary stream or other container that doesn't keep 
adequate timestamps; in any real container they're just redundant. 
(Well, picture timing SEI can also be used for soft-telecine, but that's 
not really relevent to this discussion.)

All that is needed to guarantee perfect "no jerk" operation is to have a 
correct value of num_reorder_frames, and disable ffh264's autodetection 
thereof.
If you have that, then the correct algorithm is:
   Keep a buffer of size=(num_reorder_frames+1).
   Put each newly decoded frame into the buffer.
   Whenever the buffer is full, remove the frame with the lowest {idr_pic_id,poc}
   (where idr_pic_id is a monotonically increasing value, not literally
    the variable idr_pic_id from the standard).
   This gives you the pictures in display order, but with no timestamps.
   The demuxer should provide a PTS of each frame; use those instead of
   any timing SEI from the h264 stream.

> BTW, if I understood well the reordering algorithm currently in h264.c
> an IDR, P, B, B stream ... (with POCs 0, 6, 2, 4) that is decoded frame
> by frame will output the IDR right away and then output no picture
> (since it detects the POC gap from 0 to 6) and then output the pictures
> normally. This already creates a jerk (although small), but correct me
> if I misunderstood.

Yes, that's what it does now if the stream does not specify 
num_reorder_frames.
If it does so with num_reorder_frames too, that would be a bug in the 
implementation (failing to disable autodetection), not a limitation of 
the algorithm.

> Would relying on the dpb_output_delay be reasonable? Does x264 output
> them?

I don't remember ever seeing an h264 stream that had picture timing SEIs. 
Not that I would have noticed if I were just playing the video, lavc 
silently drops SEIs it doesn't understand. But I haven't found any in the 
streams I have analysed for debugging either.

IMHO, it is reasonable to rely on num_reorder_frames if the codec wants to 
do any funny reordering, i.e. anything other than conventional B-frames.

--Loren Merritt