[FFmpeg-devel] [PATCH] H.264 timestamps

[Ivan Schreter]

Ivan Schreter wrote:
> Ivan Schreter wrote:
>> attached three patches to get/compute some timestamp-relevant 
>> information from H.264 streams as a prerequisite for correct H.264 
>> timestamp computation (comes later).
>>
> I've now finished the code for computation of PTS/DTS for H.264 
> correctly (according to H.264 standard). I'll post the timestamp code 
> later (it is not as complex as I thought), as it requires parser 
> patch, which has to be cleaned up and reviewed first (there are still 
> some deficiencies).

Here the first version of H.264 timestamp computation routine in 
h264_parser.c for pre-review (depends on parsing patches being reviewed 
in another ML thread and patches #1-#7 posted in last post in this thread):

+/**
+ * Compute timestamp of currently parsed frame.
+ *
+ * @param s parser context
+ */
+static inline void h264_compute_timestamp(AVCodecParserContext *s)
+{
+    H264Context *h = s->priv_data;
+    int cpb_removal_delay = h->sei_cpb_removal_delay;
+    int duration = 1;   // in ticks
+    int64_t pts;
+    int64_t in_dts = s->dts, in_pts = s->pts;
+
+    /*
+     * H.264 C.1.2 Timing of coded picture removal (equivalent to DTS):
+     *   Tr,n(0) = initial_cpb_removal_delay[ SchedSelIdx ] / 90000
+     *   Tr,n(n) = Tr,n(nb) + Tc * cpb_removal_delay(n)
+     * where
+     *   Tc = num_units_in_tick / time_scale
+     */
+
+    if (h->sps.timing_info_present_flag) {
+        // update clock Tc
+        h->ts_tc = (AVRational) {h->sps.num_units_in_tick, 
h->sps.time_scale};
+        if(h->x264_build > 0 && h->x264_build < 44)
+            h->ts_tc.den *= 2;
+    }
+
+    if (h->ts_tc.den == 0) {
+        // cannot compute timestamps as no timing info present
+        return;
+    }
+
+    if (h->ts_trn_nb < 0) {
+        // set initial timestamp
+        if (h->sps.nal_hrd_parameters_present_flag || 
h->sps.vcl_hrd_parameters_present_flag) {
+            h->ts_trn_nb = h->initial_cpb_removal_delay[h->sched_sel_idx];
+        } else {
+            // no initial TS, use 0
+            h->ts_trn_nb = 0;
+        }
+        cpb_removal_delay = 0;
+    }
+
+    if (s->dts != AV_NOPTS_VALUE) {
+        /*
+         * Each H.264 buffering period starts own sequence of timestamps.
+         * These are then computed contiguously for following buffering
+         * periods using given formulas above. When decoding from the
+         * beginning, the timestamps actually match those of the container
+         * (e.g., MPEG-TS) almost perfectly (with only small offset of 
~1ms).
+         *
+         * However, after a seek, when we start decoding in the middle 
of the
+         * stream, we don't have a timestamp history. So H.264 would start
+         * generating different (offset) timestamps.
+         *
+         * We want to actually generate timestamps in sync with the
+         * container format. Therefore, if the container does provide a DTS
+         * for the frame, adjust Trn(nb) timestamp to match timing of the
+         * container. Timestamp deltas are then computed using above 
mentioned
+         * H.264 formula.
+         */
+        if (h->sei_buffering_period_present_flag) {
+            cpb_removal_delay = 0;
+            h->ts_trn_nb = s->dts;
+        } else if (cpb_removal_delay >= 0) {
+            h->ts_trn_nb = s->dts -
+                cpb_removal_delay * h->ts_tc.num * 90000 / h->ts_tc.den;
+        }
+    }
+
+    if (h->sps.pic_struct_present_flag) {
+        switch (h->sei_pic_struct) {
+            case SEI_PIC_STRUCT_TOP_FIELD:
+            case SEI_PIC_STRUCT_BOTTOM_FIELD:
+                duration = 1;
+                break;
+            case SEI_PIC_STRUCT_FRAME:
+            case SEI_PIC_STRUCT_TOP_BOTTOM:
+            case SEI_PIC_STRUCT_BOTTOM_TOP:
+                duration = 2;
+                break;
+            case SEI_PIC_STRUCT_TOP_BOTTOM_TOP:
+            case SEI_PIC_STRUCT_BOTTOM_TOP_BOTTOM:
+                duration = 3;
+                break;
+            case SEI_PIC_STRUCT_FRAME_DOUBLING:
+                duration = 4;
+                break;
+            case SEI_PIC_STRUCT_FRAME_TRIPLING:
+                duration = 6;
+                break;
+        }
+    } else {
+        duration = h->field_pic_flag ? 1 : 2;
+    }
+
+    if (cpb_removal_delay >= 0) {
+        h->cur_dts = h->ts_trn_nb +
+                cpb_removal_delay * h->ts_tc.num * 90000 / h->ts_tc.den;
+    } else {
+        // no removal delay specified, use best guess (add prev frame 
duration)
+        h->cur_dts += h->cur_duration;
+    }
+    pts = h->cur_dts +
+            h->sei_dpb_output_delay * h->ts_tc.num * 90000 / h->ts_tc.den;
+    h->cur_duration = duration * h->ts_tc.num * 90000 / h->ts_tc.den;
+
+    if (h->sei_buffering_period_present_flag) {
+        // set reference for next buffering period, _after_ computing the
+        // timestamp for current frame (TS of current frame is computed 
against
+        // initial timestamp of previous buffering period).
+        h->ts_trn_nb = h->cur_dts;
+    }
+
+    s->dts = h->cur_dts;
+    s->pts = pts;
+    // TODO: handle wrap bits and pass duration to lavf
+}

Michael: I'd like to hear especially your comments...

Regards,

Ivan