[FFmpeg-devel] How to correctly handle Matroska timestamps

[MattKC]

Follow up: I appear to have solved this and have some further 
information which may be useful.

TL;DR: Setting AVStream's avg_frame_rate, which corresponds to mkvinfo's 
"Default duration" field, appears to have fixed this.

 From my investigation, it appears VLC will use a few different sources 
of information to determine the timecode. The first and foremost is the 
aforementioned "default duration", corresponding to avg_frame_rate. 
However if it's missing, it appears to use the MKV cluster size 
(corresponding to the codec's gop_size). I noticed from mkvinfo that 
each cluster started at the times VLC's playhead would appear to jump to 
(0:00, 0:03, 0:07, etc.) and that changing the gop_size would shift 
these around.

Following from this, it was somewhat inaccurate to say all of the 
examples exhibit this issue. The "muxing" example, which sets the 
gop_size to 12, creates an MKV that plays reasonably in VLC. With far 
more frequent MKV clusters, VLC's timecode doesn't hang for seconds at a 
time. However the downside of decreasing the gop_size is that naturally 
it makes the file larger due to more intra frames and it presumably 
means VLC's playhead is still only accurate to the nearest 12th frame. 
When FFmpeg remuxes, it obviously doesn't change the gop_size, but it 
does set the "default duration"/avg_frame_rate. Output from my program, 
with avg_frame_rate set, works as expected without the file size 
increases that result from changing the gop_size, so that appears to be 
the solution.

As for the "remuxing" and "transcoding" examples:

  * The output from "transcoding" will always exhibit this issue since
    it doesn't set gop_size nor avg_frame_rate.
  * The "remuxing" example depends on the input. If the MKV has a low
    gop_size, it will obviously preserve that. However, it doesn't set
    nor preserve "avg_frame_rate", so the output will exhibit this issue
    with larger gop_size inputs (including non-MKV files such as MP4s).
  * Disregard my comment about "encode_video", I forgot it didn't mux
    anything.

It may be worth updating the relevant tests to fix this, but it depends 
on whether this is considered a priority or not.

PS Andreas: You were correct about my subtitle packet duration being set 
wrong, I have fixed this now.

Matt

On 8/7/2021 10:52 PM, MattKC wrote:
> Andreas,
>
>     (A quick look showed that the transcoding example does not use
>     pkt_timebase for decoding, as does your code;
>
> This is certainly the first time I've heard of using pkt_timebase. 
> Almost all examples and tutorials I've seen use AVStream::time_base. 
> What's the usage of this though? Documentation says "encoding unused" 
> and it seems to be null (0/1) when I tried using it just now.
>
>     furthermore, your code
>     sets the duration of subtitle packets wrong, as the end_display_time is
>     in a different timebase than the subtitle's pts.)
>
> Is that incorrect? The AVSubtitle struct definition says 
> start_display_time and end_display_time are in ms not in time base 
> units. Also, the subtitles, as encoded by my program, appear to be 
> correct when played back.
>
> I can open an issue on trac at some point, but the issue can be fairly 
> easily recreated just by using "remuxing <input-file> output.mkv" or 
> "encode-video output.mkv libx264" or "transcoding <input-file> 
> output.mkv". It would seem most if not all of the examples produce a 
> broken file when outputting to MKV.
>
> I've attached the output of "mkvinfo -a" on both a broken MKV exported 
> from my program and a remuxed MKV from FFmpeg. They're largely 
> similar; the "Default duration" section in the remux and slightly 
> different order of packets are the only parts that stands out to me. 
> Let me know if you can get any meaningful information out of them.
>
> Matt
>
> On 8/7/2021 9:58 PM, Andreas Rheinhardt wrote:
>> MattKC:
>>> ==Summary==
>>>
>>> I've been implementing ffmpeg/libav into an application, and have run
>>> into timing-related issues when playing back Matroska/MKV videos created
>>> by my application in VLC. I tried asking about this on the libav-user
>>> mailing list and received no response so I thought I'd try here. While
>>> it's true that there may be issues with VLC's implementation too, I've
>>> noticed specifically that videos generated by FFmpeg do not cause these
>>> issues, while my application and the examples in "doc/examples" (which
>>> my code is modeled from) do, which makes me think this code is missing
>>> something crucial.
>>>
>>> ==Symptoms==
>>>
>>> During playback, VLC's playback time will appear to stay frozen at 0:00
>>> while the video and audio plays normally. It will then jump only in
>>> increments of a few seconds (0:03, then 0:07, then 0:11, etc.) This
>>> behavior occurs with videos produced both by my application and by the
>>> FFmpeg examples programs "transcoding.c" and "remuxing.c".
>>>
>>> The fact that this occurs with "remuxing.c" is particularly interesting
>>> since that would seem to imply it's an issue with the format/container
>>> writing specifically. If "remuxing.c" is given a correct MKV file (i.e.
>>> one that does not exhibit this issue) produced by FFmpeg as its input,
>>> and outputs to a second MKV file, that second MKV will demonstrate this
>>> issue even though the first one didn't, indicating that something is
>>> breaking during the remuxing process despite no decoding/encoding
>>> occurring.
>>>
>>> The worst demonstration of this issue that I've run into is if my
>>> application writes any audio packets before writing the first video
>>> packet (with "av_interleaved_write_frame"), the first few seconds of
>>> video (until that first 0:03) will be considered completely blank by
>>> VLC. The fact that it will pick up after 0:03, the same time the timer
>>> will make its first update on other MKV files, leads me to believe it's
>>> the exact same root cause, just a different presentation of it.
>>>
>>> This issue only occurs on MKV files. MP4/MOV appear to have no issues
>>> whatsoever.
>>>
>>> ==Conclusion==
>>>
>>> Considering FFmpeg produces correct MKV files, I don't believe this to
>>> be a bug in libav (though it could be considered a bug in the examples).
>>> I'm assuming the examples and my program are missing something or doing
>>> something incorrectly.
>>>
>>> My code can be seen here, however the examples will almost certainly be
>>> easier to read and test:
>>> https://github.com/olive-editor/olive/blob/master/app/codec/ffmpeg/ffmpegencoder.cpp
>>>
>>>
>>> Thanks in advance for any help, I'm a little confused about this whole
>>> issue.
>>>
>> If it happens with our examples, too (as you said), then you should open
>> an issue on trac (for the examples). Be sure to add samples that allow
>> to reproduce it.
>> It would probably be enlightening to use mkvinfo to look at the
>> generated files. The files are probably badly interleaved (your
>> description makes me suspect that the generated files have lots of audio
>> packets at the beginning before the first video packet).
>> (A quick look showed that the transcoding example does not use
>> pkt_timebase for decoding, as does your code; furthermore, your code
>> sets the duration of subtitle packets wrong, as the end_display_time is
>> in a different timebase than the subtitle's pts.)
>>
>> - Andreas
>> _______________________________________________
>> ffmpeg-devel mailing list
>> ffmpeg-devel at ffmpeg.org
>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>>
>> To unsubscribe, visit link above, or email
>> ffmpeg-devel-request at ffmpeg.org  with subject "unsubscribe".