[FFmpeg-devel] [PATCH v3] ffmpeg: add option -isync

[Gyan Doshi]

On 2022-07-11 12:21 am, Anton Khirnov wrote:
> Quoting Gyan Doshi (2022-07-10 20:02:38)
>>
>> On 2022-07-10 10:46 pm, Anton Khirnov wrote:
>>> Quoting Gyan Doshi (2022-07-08 05:56:21)
>>>> On 2022-07-07 03:11 pm, Anton Khirnov wrote:
>>>>> Quoting Gyan Doshi (2022-07-04 18:29:12)
>>>>>> This is a per-file input option that adjusts an input's timestamps
>>>>>> with reference to another input, so that emitted packet timestamps
>>>>>> account for the difference between the start times of the two inputs.
>>>>>>
>>>>>> Typical use case is to sync two or more live inputs such as from capture
>>>>>> devices. Both the target and reference input source timestamps should be
>>>>>> based on the same clock source.
>>>>>>
>>>>>> If not all inputs have timestamps, the wallclock times at the time of
>>>>>> reception of inputs shall be used. FFmpeg must have been compiled with
>>>>>> thread support for this last case.
>>>>> I'm wondering if simply using the other input's InputFile.ts_offset
>>>>> wouldn't achieve the same effect with much less complexity.
>>>> That's what I initially did. But since the code can also use two other
>>>> sources for start times (start_time_realtime, first_pkt_wallclock),
>>>> those intervals may not exactly match the difference between
>>>> fmctx->start_times so I use a generic calculation.
>>> In what cases is it better to use either of those two other sources?
>>>
>>> As per the commit message, the timestamps of both inputs are supposed to
>>> come from the same clock. Then it seems to me that offsetting each of
>>> those streams by different amounts would break synchronization rather
>>> than improve it.
>> The first preference, when available, stores the epoch time closest to
>> time of capture. That would eliminate some jitter.
>> The 2nd preference is the fmctx->start_time. The 3rd is the reception
>> wallclock. It is a fallback. It will likely lead to the worst sync.
> You did not answer my question.
> If both streams use the same clock, then how is offsetting them by
> different amounts improve sync?

Because the clocks can be different at different stages of stream 
conveyance  i.e. capture -> encode -> network relay -> ffmpeg reception.
As long as both use the same clock at a given stage, they represent the 
same sync relation but with some jitter in the mix added with each stage.
The semantics of start_time_realtime is "pts=0 in the stream was 
captured at this real world time" (unix epoch).
The fmctx start will usually be system timestamps at encode or mux. We 
should prefer the earliest stage available, which is what the patch does.

Regards,
Gyan