[FFmpeg-user] DTSs & PTSs in MPEG2-TS streams

[Devin Heitmueller]

On Thu, Feb 1, 2024 at 1:55 PM Mark Filipak <markfilipak.imdb at gmail.com> wrote:
>
> Hey Devin,
>
> On 01/02/2024 09.54, Devin Heitmueller wrote:
> > Hi Mark,
> >
> > On Thu, Feb 1, 2024 at 1:15 AM Mark Filipak <markfilipak.imdb at gmail.com> wrote:
> >>
> >> FFmpeg assigns DTSs & PTSs to P-frames and B-frames in MPEG2-TS streams. How does it determine those
> >> DTSs & PTSs?
> >>
> >> Since DTSs & PTSs don't actually exist for P-/B-frames in MPEG2-TS streams, FFmpeg must have a way
> >> that's unknown to me.
> >
> > P/B frames can certainly have PTS values.
>
> Well, they _can_ as in they're not prohibited, but I can't recall ever seeing a P-/B-frame in
> MPEG2-TS supplying a DTS or a PTS. I've parsed quite a few MPEG2-TSs.

If you want to share a sample with me privately, I can take a look and
possibly offer you a better explanation.

P/B frames generally have PTS values.  B frames *must* have both PTS
and DTS since they are different values.  If you're not seeing them
then either your sample is broken or your parser is.

> >  In fact, you have to have
> > PTS values in B frames since they differ from the DTS (since with B
> > frames the presentation order differs from the decoding order).
>
> I don't see DTS or PTS in the PES header. What I do see is 'PES_extension_flag'=='0' and
> 'PTS_DTS_flags'=='00', so no PES ext and no dts or pts. That's why I asked my question.

The PES extension flag is mutually exclusive of whether the
PTS_DTS_flags field is set.  It's very common for there to
PTS_DTS_flags to be non-zero while PES_extension_flag is false.

> Every once in awhile I'll see 'PTS_DTS_flags'=='10' but that's unusual. Mind you, this is in
> MPEG2-TSs. I don't how to parse anything else.

That shouldn't be unusual.  It means there is a PTS but no DTS in that
PES.  In fact, specifying the PTS without the DTS is generally what
you're supposed to do if they are the same value (which is the norm
for audio streams and video streams without B frames).

> > There are potentially cases with some streams where the PTS and/or DTS
> > is not specified for every frame,
>
> I'd say "usually", not "potentially".
>
> > in which case libavformat will
> > interpolate those values based on the frame duration and the prior
> > packet.
>
> Of course. But why? They're not needed. Once the TS packets are put in correct sequence via SCR, the
> PS packets transported by those TS packets are in presentation order. The process to reorder the PS
> in order to decode B-frames and then put the decoded pictures back into the original order is
> straight forward. It's not "tricky" at all.

Um, I'm not sure what to say here.  The DTS is what defines the order
that the packets are announced to the decoder.  The PTS defines the
order in which they are presented.  The SCR will constantly increment.
I think perhaps you have a fundamental misunderstanding of how PTS/DTS
works.

> It has become "tricky". That's what I'm trying to investigate, but I can't read 'C' code.
>
> >> I'm seeing some strange things. They can't be as strange as they look because they look disastrous.
> >> At issue is the cause of non-monotonous DTSs and obviously bogus DTSs for P-frames such as
> >> '-9223372036854775808'.
> >
> > 9223372036854775808 is a magic number.  It corresponds to the #define
> > for AV_NOPTS_VALUE.  If you see that value it means that the packet
> > did not have a PTS or DTS value assigned to it and thus you may have
> > to take special measures.
>
> Oh, dear. I guessed that.
>
> Magic number. Fixup. There's a million fixups. I don't see why they're needed. That's why my
> preliminary finding is that FFmpeg is creating its own faults through fixups.

This is essentially how all demuxers/parsers work.  The PES packet in
the stream doesn't contain a PTS or DTS, the API tells you that, and
thus the application has to use a heuristic to assign a value.  Some
implementations may do the interpolation for you, but libavformat
basically tells you there is no PTS and leaves it to the application
to decide what to do.

Devin

-- 
Devin Heitmueller, Senior Software Engineer
LTN Global Communications
o: +1 (301) 363-1001
w: https://ltnglobal.com  e: devin.heitmueller at ltnglobal.com