[FFmpeg-user] Subtitle Timing: Clock Timestamps vs. Frame Timestamps

[jeremie bergeron]

Hi,

This is not strictly an FFmpeg question but rather a general video
processing question regarding when subtitles should be rendered.

In this discussion, we are debating whether subtitles should be displayed
based on *clock timestamps* or *frame timestamps (PTS)*.
Let's say I have a video with those timestamps:

   -

   Frame 0: 0 ms
   -

   Frame 1: 42 ms
   -

   Frame 2: 83 ms
   -

   Frame 3: 125 ms
   -

   Frame 4: 167 ms
   -

   Frame 5: 209 ms
   -

   Frame 6: 250 ms
   -

   Frame 7: 292 ms
   -

   Frame 8: 334 ms
   -

   Frame 9: 375 ms
   -

   Frame 10: 417 ms
   -

   Frame 11: 459 ms
   -

   Frame 12: 500 ms
   -

   Frame 13: 542 ms
   -

   Frame 14: 584 ms
   -

   Frame 15: 626 ms
   -

   Frame 16: 667 ms
   -

   Frame 17: 709 ms
   -

   Frame 18: 751 ms
   -

   Frame 19: 792 ms
   -

   Frame 20: 834 ms
   -

   Frame 21: 876 ms

Now, let's consider a subtitle that starts at *400 ms* and ends at *860 ms*.

   -

   Using frame timestamps (PTS):
   The subtitle would align with Frame 10 (417 ms) to Frame 20 (834 ms).
   -

   Using clock timestamps:
   The subtitle would attempt to display exactly at 400 ms. However, since
   Frame 9 covers the interval [375 ms, 417 ms[, the subtitle would
   effectively appear in the middle of Frame 9 rather than at the start of a
   frame.

Given this discrepancy, which method should be used for accurate subtitle
rendering? Should we prioritize aligning with frame timestamps (PTS), or
should subtitles be placed at the exact clock timestamps, even if that
means appearing mid-frame?

Also, does it depend on the subtitle format?

I need this information to try to convince the developer of Media3 in this
issue <https://github.com/androidx/media/issues/2289>.