[FFmpeg-user] DTSs & PTSs in MPEG2-TS streams

[Mark Filipak]

On 01/02/2024 16.09, Devin Heitmueller wrote:
> On Thu, Feb 1, 2024 at 1:55 PM Mark Filipak <markfilipak.imdb at gmail.com> wrote:
>> On 01/02/2024 09.54, Devin Heitmueller wrote:
>>> On Thu, Feb 1, 2024 at 1:15 AM Mark Filipak <markfilipak.imdb at gmail.com> wrote:
>>>>
>>>> FFmpeg assigns DTSs & PTSs to P-frames and B-frames in MPEG2-TS streams. How does it determine those
>>>> DTSs & PTSs?
>>>>
>>>> Since DTSs & PTSs don't actually exist for P-/B-frames in MPEG2-TS streams, FFmpeg must have a way
>>>> that's unknown to me.
>>>
>>> P/B frames can certainly have PTS values.
>>
>> Well, they _can_ as in they're not prohibited, but I can't recall ever seeing a P-/B-frame in
>> MPEG2-TS supplying a DTS or a PTS. I've parsed quite a few MPEG2-TSs.
> 
> If you want to share a sample with me privately, I can take a look and
> possibly offer you a better explanation.
> 
> P/B frames generally have PTS values.  B frames *must* have both PTS
> and DTS since they are different values.  If you're not seeing them
> then either your sample is broken or your parser is.

There is no PES extension, so there is no place for DTS & PTS to reside. I realize that FFmpeg is 
creating DTSs & PTSs. That appears to be part of the problem.

I've seen many reports of DTS out of sequence when the original frame, a P-/B-frame, had no DTS. It 
appears that FFmpeg is reporting an issue that it created.

>>>   In fact, you have to have
>>> PTS values in B frames since they differ from the DTS (since with B
>>> frames the presentation order differs from the decoding order).
>>
>> I don't see DTS or PTS in the PES header. What I do see is 'PES_extension_flag'=='0' and
>> 'PTS_DTS_flags'=='00', so no PES ext and no dts or pts. That's why I asked my question.
> 
> The PES extension flag is mutually exclusive of whether the
> PTS_DTS_flags field is set.  It's very common for there to
> PTS_DTS_flags to be non-zero while PES_extension_flag is false.

Okay. If DTS & PTS are not in the PES extension because there is no PES extension, then where are 
they? There are only a handful of bits in an MPEG2-TS/-PS I haven't figured out. You may know more 
than I do.

>> Every once in awhile I'll see 'PTS_DTS_flags'=='10' but that's unusual. Mind you, this is in
>> MPEG2-TSs. I don't how to parse anything else.
> 
> That shouldn't be unusual.  It means there is a PTS but no DTS in that
> PES.  In fact, specifying the PTS without the DTS is generally what
> you're supposed to do if they are the same value (which is the norm
> for audio streams and video streams without B frames).
> 
>>> There are potentially cases with some streams where the PTS and/or DTS
>>> is not specified for every frame,
>>
>> I'd say "usually", not "potentially".
>>
>>> in which case libavformat will
>>> interpolate those values based on the frame duration and the prior
>>> packet.
>>
>> Of course. But why? They're not needed. Once the TS packets are put in correct sequence via SCR, the
>> PS packets transported by those TS packets are in presentation order. The process to reorder the PS
>> in order to decode B-frames and then put the decoded pictures back into the original order is
>> straight forward. It's not "tricky" at all.
> 
> Um, I'm not sure what to say here.  The DTS is what defines the order
> that the packets are announced to the decoder.  The PTS defines the
> order in which they are presented.  The SCR will constantly increment.
> I think perhaps you have a fundamental misunderstanding of how PTS/DTS
> works.

Yes, the SCR (in the TS) will constantly increment. It's when the _receiver_ of the TS sees SCRs 
that are out of order that it knows when packets are being received out of order and how to put them 
into proper order. That's why TSs are called "fault tolerant".

FFmpeg seems to think that DTS has that function for every frame but where is the DTS if there's no 
PES_ext? I think that DTS & PTS applies to GOPs, not individual frames. I think that FFmpeg is 
misusing DTS.

But I confess that I don't know why an occasional P-frame has a PES_ext with a PTS while P-/B-frames 
generally don't have PES_ext or DTSs or PTSs.

>> It has become "tricky". That's what I'm trying to investigate, but I can't read 'C' code.
>>
>>>> I'm seeing some strange things. They can't be as strange as they look because they look disastrous.
>>>> At issue is the cause of non-monotonous DTSs and obviously bogus DTSs for P-frames such as
>>>> '-9223372036854775808'.
>>>
>>> 9223372036854775808 is a magic number.  It corresponds to the #define
>>> for AV_NOPTS_VALUE.  If you see that value it means that the packet
>>> did not have a PTS or DTS value assigned to it and thus you may have
>>> to take special measures.
>>
>> Oh, dear. I guessed that.
>>
>> Magic number. Fixup. There's a million fixups. I don't see why they're needed. That's why my
>> preliminary finding is that FFmpeg is creating its own faults through fixups.
> 
> This is essentially how all demuxers/parsers work.  The PES packet in
> the stream doesn't contain a PTS or DTS, the API tells you that, and
> thus the application has to use a heuristic to assign a value.

Okay. So the decoder _is_ expecting a 'DTS' & 'PTS' with every frame -- I've put them in quotes to 
indicate they're aliases (because the names "DTS" & "PTS" are already taken) -- is that correct? 
Have I read your reply correctly? That FFmpeg decoders expect 'DTS' & 'PTS' for every frame?

Thank you so much, Devin.

--Mark.