[FFmpeg-user] Unable to sync audio and video

[Nicolas George]

Livio Tenze (12021-02-16):
> At the moment I have only one thread (the main one, as in doc/examples).
> The process I implemented is the following:
> 0) Initialize PTS output frame (one PTS for video and another for audio)
> 1) check PTS difference between PTS video and audio
> 2) if ptsvideo>ptsaudio then decode one video frame, and encode it in the
> output stream
>     otherwise decode audio frame, and it in the output stream.
> 3) go to point 1

That should work, provided you checked that your timestamps relate to
the same origin. If some timestamps relate to the system boot and some
to 1970-01-01, you will get a desync.

Plus, if the capture did not start at the same time, you will get extra
frames at the beginning of a stream, and it is possible that some
players will not catch up or catch up slowly. It would probably be more
reliable to discard frames captured before the first frame of the other
stream.

> > Do the timestamps between audio and video match before encoding?
> Yes, please check above. I use two PTS values: one for audio and one for
> video. The PTS from the input stream differ from the ones of the output
> stream. I check the distance between the first packet.pts and the current:
> here I saw that strange behaviour where the audio elapsed time is very high
> with respect to the video input stream.

All you talk here is timestamps consistency within each stream. I was
asking about timestamps consistency between the streams.

> Do you think that this behaviour is due to missing multithread? I found the

Yes. Without parallelism for the capture, the codec initialization could
take so much time as to cause a buffer overrun in one of the capture
drivers.

> above flowchart in doc/examples where no threads are used.

It is entirely possible this flowchart is not up to date or chose to
gloss over implementations details that are not deemed relevant for the
information expressed. The command line tool ffmpeg uses threads for
inputs as soon as there are more than one.

Regards,

-- 
  Nicolas George
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-user/attachments/20210216/7047263e/attachment.sig>