[FFmpeg-devel] [PATCH] H.264 timestamps in h264_parser - complete set

Tue Feb 17 21:57:32 CET 2009

Hi *,

so, now it's complete. Proper H.264 timestamp support can be added to 
FFmpeg by attached set of patches.

First some trivial patches as prerequisites:

#1 (h264_timing_1_init_cpb_del_len): Parse 
initial_cpb_removal_delay_length (needed for SEI buffering period).
#2 (h264_timing_2_output_delay): Parse dpb_output_delay in SEI picture 
structure (needed for timestamp calculation).
#3 (h264_timing_3_cpb_removal_delay): Parse cbp_removal_delay in SEI 
picture structure (needed for timestamp calculation; depends on #2).
#4 (h264_timing_4_cpb_cnt): Store CPB count in the context (needed for 
SEI buffering period).
#5 (h264_timing_5_field_flags): Parse field_pic_flag and 
bottom_field_flag (needed for timestamp calculation).
#6 (h264_default_sps): Parsing SPS doesn't assign the just-parsed SPS as 
current SPS on the context. Thus, decoding of SEIs (especially SEI 
buffering period) depending on current SPS wouldn't work. Assign 
just-parsed SPS as current SPS for the context.

To properly generate H.264 timestamps, we need to get some information 
from SEI buffering period. Until now, it was not parsed. Nontrivial 
patch to add it:

#7 (h264_timing_6_buffering_period): Parse SEI buffering period (depends 
on #1-#4 and #6).

The actual timing code is implemented in the parser. First of all, we 
need to have the parser to actually parse frame header to get relevant 
information. Following patches add parsing of header NAL units to the 
parser:

#8 (h264_parser_1_funcs): Make some relevant h264.c functions visible to 
h264_parser.c. Already OK-ed by Michael as prereq for the rest of 
parsing code (after it is OK-ed).
#9 (h264_parser_2_parser): Add parsing code to parse basic information 
about the frame (depends on #8). Already in review in another thread.

Now, key frames are handled differently in H.264 than in MPEG-2. An 
I-frame is not necessarily a key frame. To address that, an additional 
flag must be passed to lavf so it knows about key frames:

#10 (avformat_1_keyframe): Add two new flags on the parser context. One 
to tell lavf to handle key frames properly (i.e., independent of 
pict_type) and one to actually signalize key frame. If the first flag is 
not present, old (workaround) method of detecting key frames via 
I-frames will be used. When all parsers are migrated to the new flag, 
key frame detection via I-frames can be dropped completely. Nice thing 
is, this change is forward and backward binary compatible.
#11 (h264_parser_3_keyframe): Actually set the key frame flag for H.264 
for IDR frames and for frames having SEI recovery point (initialization 
routine to set key frame flag presence flag must be added here as well; 
depends on #8-#10).

H.264 timestamp computation also needs field_pic_flag:

#12 (h264_parser_4_field_flags): Add parsing of field flags (depends on 
#8-#11).

And finally:

#13 (h264_parser_5_timestamps): Correctly compute H.264 timestamps based 
on cpb_removal_delay, dpb_output_delay and various other bits from 
#1-#12.  Unfortunately, H.264 doesn't have notion about "timestamps" as 
such, only about delays. So the timestamps from container format (e.g., 
MPEG-TS) are taken as the basis to compute initial timestamp for first 
frame in buffering period. Only if DTS is not available from the format, 
initial delay from H.264 is used to start the timestamp generator as a 
fallback (of course, in this case, after a seek the timestamps will 
diverge, as they will start from arbitrary small timestamp).

This code generates correct timestamps not only for frame-based, but 
also for field-based H.264 movies.

So please review it.

I recommend applying the patches in the order as they were mentioned 
here due to interdependencies.

Regards,

Ivan

-------------- next part --------------
A non-text attachment was scrubbed...
Name: h264_timing_1_init_cpb_del_len.patch
Type: text/x-patch
Size: 1247 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20090217/3c95ff0d/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: h264_timing_2_output_delay.patch
Type: text/x-patch
Size: 1794 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20090217/3c95ff0d/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: h264_timing_3_cpb_removal_delay.patch
Type: text/x-patch
Size: 1822 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20090217/3c95ff0d/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: h264_timing_4_cpb_cnt.patch
Type: text/x-patch
Size: 854 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20090217/3c95ff0d/attachment-0003.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: h264_timing_5_field_flags.patch
Type: text/x-patch
Size: 1490 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20090217/3c95ff0d/attachment-0004.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: h264_default_sps.patch
Type: text/x-patch
Size: 330 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20090217/3c95ff0d/attachment-0005.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: h264_timing_6_buffering_period.patch
Type: text/x-patch
Size: 3551 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20090217/3c95ff0d/attachment-0006.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: h264_parser_1_funcs.patch
Type: text/x-patch
Size: 4962 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20090217/3c95ff0d/attachment-0007.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: h264_parser_2_parser.patch
Type: text/x-patch
Size: 3160 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20090217/3c95ff0d/attachment-0008.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: avformat_1_keyframe.patch
Type: text/x-patch
Size: 1475 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20090217/3c95ff0d/attachment-0009.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: h264_parser_3_keyframe.patch
Type: text/x-patch
Size: 1614 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20090217/3c95ff0d/attachment-0010.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: h264_parser_4_field_flags.patch
Type: text/x-patch
Size: 1535 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20090217/3c95ff0d/attachment-0011.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: h264_parser_5_timestamps.patch
Type: text/x-patch
Size: 6427 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20090217/3c95ff0d/attachment-0012.bin>