[FFmpeg-user] How to automatically synchronize different video- and audio-streams

[neumeise1 at mail.de]

Hello,
I have a small question about syncing a video- and an audio-stream(Laptop camera and microphone) on linuxlite(based on ubuntu).
I HAVE USED THE FOLLOWING COMMAND: date +%s; \ stime=0; \  ffmpeg -f pulse -i alsa_input.pci-0000_00_1b.0.analog-stereo -f \  video4linux2 -s 320x240 -itsoffset +0.6 -i /dev/video0 -f matroska \ -filter_complex "[0]aresample=async=1000[a1]; \
  [1]setpts=PTS+$stime/TB[v1]" \  -map "[a1]" -map "[v1]" /home/linuxlite/Downloads/test.mkv

AND GET THE FOLLOWING OUTPUT(NOTE: START OF THE AUDIO STREAM MATCHES EXACTLY THE LINUX-EPOCH-TIME): 1633698020
ffmpeg version 4.2.4-1ubuntu0.1 Copyright (c) 2000-2020 the FFmpeg developers 
 built with gcc 9 (Ubuntu 9.3.0-10ubuntu2) 
 configuration: --prefix=/usr --extra-version=1ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --
incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-fil
ter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --
enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enabl
e-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libm
ysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubbe
rband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --e
nable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enabl
e-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-open
al --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nv
enc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared 
 libavutil 56. 31.100 / 56. 31.100 
 libavcodec 58. 54.100 / 58. 54.100 
 libavformat 58. 29.100 / 58. 29.100 
 libavdevice 58. 8.100 / 58. 8.100 
 libavfilter 7. 57.100 / 7. 57.100 
 libavresample 4. 0. 0 / 4. 0. 0 
 libswscale 5. 5.100 / 5. 5.100 
 libswresample 3. 5.100 / 3. 5.100 
 libpostproc 55. 5.100 / 55. 5.100 
Guessed Channel Layout for Input Stream #0.0 : stereo 
Input #0, pulse, from 'alsa_input.pci-0000_00_1b.0.analog-stereo': 

 Duration: N/A, start: 1633698020.739039, bitrate: 1536 kb/s 
 Stream #0:0: Audio: pcm_s16le, 48000 Hz, stereo, s16, 1536 kb/s 
Input #1, video4linux2,v4l2, from '/dev/video0': 
 Duration: N/A, start: 8881.575774, bitrate: 36864 kb/s 
 Stream #1:0: Video: rawvideo (YUY2 / 0x32595559), yuyv422, 320x240, 36864 kb/s, 30 fps, 30 tbr, 1000k tbn, 1000k
tbc 
File '/home/linuxlite/Downloads/test.mkv' already exists. Overwrite ? [y/N] y 
Stream mapping: 
 Stream #0:0 (pcm_s16le) -> aresample 
 Stream #1:0 (rawvideo) -> setpts 
 aresample -> Stream #0:0 (libvorbis) 
 setpts -> Stream #0:1 (libx264) 
Press [q] to stop, [?] for help 
[video4linux2,v4l2 @ 0x556d42eb7fc0] Thread message queue blocking; consider raising the thread_queue_size option (c
urrent value: 8) 
[libx264 @ 0x556d42ebef80] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 

[libx264 @ 0x556d42ebef80] profile High 4:2:2, level 1.3, 4:2:2 8-bit 
[libx264 @ 0x556d42ebef80] 264 - core 155 r2917 0a84d98 - H.264/MPEG-4 AVC codec - Copyleft 2003-2018 - http://www.v
ideolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00
mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 thread
s=3 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 
b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_re
fresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00 
Output #0, matroska, to '/home/linuxlite/Downloads/test.mkv': 
 Metadata: 
 encoder : Lavf58.29.100 
 Stream #0:0: Audio: vorbis (libvorbis) (oV[0][0] / 0x566F), 48000 Hz, stereo, fltp (default) 
 Metadata: 
 encoder : Lavc58.54.100 libvorbis 
 Stream #0:1: Video: h264 (libx264) (H264 / 0x34363248), yuv422p, 320x240, q=-1--1, 30 fps, 1k tbn, 30 tbc (defau
lt) 
 Metadata: 
 encoder : Lavc58.54.100 libx264 
 Side data: 
 cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1 
[pulse @ 0x556d42e9d940] Thread message queue blocking; consider raising the thread_queue_size option (current value
: 8) 
[libvorbis @ 0x556d42ebd5c0] Queue input is backward in time  
frame=108728 fps= 30 q=-1.0 Lsize= 52908kB time=01:00:29.75 bitrate= 119.4kbits/s dup=0 drop=639 speed= 1x   
video:16098kB audio:34859kB subtitle:0kB other streams:0kB global headers:4kB muxing overhead: 3.829125% 
[libx264 @ 0x556d42ebef80] frame I:2437 Avg QP:19.54 size: 984 
[libx264 @ 0x556d42ebef80] frame P:60740 Avg QP:22.53 size: 177 
[libx264 @ 0x556d42ebef80] frame B:45551 Avg QP:23.33 size: 73 
[libx264 @ 0x556d42ebef80] consecutive B-frames: 41.9% 5.1% 4.7% 48.3% 
[libx264 @ 0x556d42ebef80] mb I I16..4: 53.8% 45.0% 1.2% 
[libx264 @ 0x556d42ebef80] mb P I16..4: 1.8% 0.1% 0.0% P16..4: 23.4% 0.4% 2.0% 0.0% 0.0% skip:72.3% 
[libx264 @ 0x556d42ebef80] mb B I16..4: 0.0% 0.0% 0.0% B16..8: 12.5% 0.1% 0.0% direct: 0.3% skip:87.2% L0:
34.0% L1:65.6% BI: 0.3% 
[libx264 @ 0x556d42ebef80] 8x8 transform intra:31.8% inter:88.6% 
[libx264 @ 0x556d42ebef80] coded y,uvDC,uvAC intra: 10.8% 54.1% 8.9% inter: 0.1% 12.0% 0.0% 
[libx264 @ 0x556d42ebef80] i16 v,h,dc,p: 12% 1% 2% 85% 
[libx264 @ 0x556d42ebef80] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 45% 3% 28% 1% 7% 11% 3% 1% 0% 
[libx264 @ 0x556d42ebef80] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 45% 8% 25% 2% 7% 6% 4% 1% 2% 
[libx264 @ 0x556d42ebef80] i8c dc,h,v,p: 63% 16% 17% 3% 
[libx264 @ 0x556d42ebef80] Weighted P-Frames: Y:0.9% UV:0.2% 
[libx264 @ 0x556d42ebef80] ref P L0: 63.2% 0.4% 26.2% 10.0% 0.2% 
[libx264 @ 0x556d42ebef80] ref B L0: 84.6% 13.7% 1.8% 
[libx264 @ 0x556d42ebef80] ref B L1: 95.1% 4.9% 
[libx264 @ 0x556d42ebef80] kb/s:36.34

THE PROBLEM: the start times of the different streams are way off. The start time of the audio stream is exactly the Linux-epoch-time(matches the date +%s command). The start-time of the video-stream on the other hand seams to be the running-time of the system. Between both numbers is a huge difference but the actual deviation in the video is about 0.6seconds. I found that out by recording a video and using the "clapping technique". I basically loaded that file in a video editing tool and counted the frames between my hands hitting each other and the actual clapping sound.

I used that aquired deviation-time with the -itsoffset flag. The video-output pretty much fits it right now, even after 1hour recording time there is no deviation!   MY QUESTION: That time seems to be quite constant, therefore there probably should be a way to calculate it. How can I synchronize different video- and audio-streams without doing it by hand? My goal is to later add more video- and audio-streams and stream them synchronized at the same time.   I found the following post on stackexchange-superuser: https://superuser.com/questions/1584945/ffmpeg-audio-start-time-way-off   In my case that solution doesn't work and I don't understand what the single parts do. I also didn't found much resources on the internet explaining what PTS and TB actually do. Can someone please point out a good ressource to read about such things?       Thanks in advance for your help!   Sebastian 

-------------------------------------------------------------------------------------------------
FreeMail powered by mail.de - MEHR SICHERHEIT, SERIOSITÄT UND KOMFORT