[FFmpeg-devel] How to synchronize device sources

[Noah Bergbauer]

> Devices that do live capture should, at least as an option, use the

> absolute clock as pts. With that information, syncing becomes easy.

Capturing with DXGI essentially *must* be done on a separate thread, as it is
extremely time sensitive and starts dropping frames at the slightest delays.
Dropping and duplicating frames also happens on that thread. This is why
my DXGI device implements pts as a simple frame counter - I want ffmpeg
to reproduce exactly the frames I return without ever duplicating or dropping
anything by itself.

The goal it to use all of this directly from ffmpeg.exe, not only as a library.
And from what I can see ffmpeg always calculates pts relative to the first
value it sees, which would nullify the effect of using wallclock timestamps,
even if I were to switch both devices to those.

Or is there a way explain to ffmpeg that yes, I actually want you to correlate
the pts of these two devices with each other?

Thanks,
Noah Bergbauer



________________________________
Von: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> im Auftrag von Nicolas George <george at nsup.org>
Gesendet: Sonntag, 26. Januar 2020 21:36:57
An: FFmpeg development discussions and patches
Betreff: Re: [FFmpeg-devel] How to synchronize device sources

Noah Bergbauer (12020-01-26):
> I am currently working on a patch that implements DXGI desktop
> capturing and WASAPI audio capturing for Windows.
>
> For this, I implemented two new devices in libavdevice.
>
> The basics work, but now I'm not sure how to synchronize the audio
> stream with the video stream. Specifically, the two devices
> obviously maintain independent pts, starting from the point in time
> when they were first read from. So while in most cases there
> is no human-noticeable offset, this does occasionally - with some bad
> luck - produce a rather large error.
>
> So is there anything I'm missing here? How can I make sure the two
> streams sync up (other than hardcoding some kind of communication
> between them)?

Devices that do live capture should, at least as an option, use the
absolute clock as pts. With that information, syncing becomes easy.

Regards,

--
  Nicolas George