[FFmpeg-user] audio video sync and drift
John Smith
localdevjs at gmail.com
Wed May 2 23:19:55 EEST 2018
First, as this is my first email to the list, let me say that ffmpeg is a
true work of art! I recently discovered this software for a project of
mine, and I am very impressed with what this can all do.
My project takes various audio and video sources and starts a "mixer"; some
video sources are merged and written to /dev/video0 (using v4l2 kernel
module). Audio is written to a local socket.
Now at this point I want to stream video, and record video from my mixer.
ffmpeg reads the video from /dev/video1, and the audio from
tcp://localhost:55555.
This works almost perfectly; but the problem is that over time, the video
and audio drift apart from eachother. Initially (first minute), the audio
and video is perfectly in sync. But after 80 minutes, the audio may be out
of sync with the video by as much as half a second or more. That is the
problem in a nutshell. The longer the video (or stream) is, the more the
drift expands; so it's not a linear constant, it gets progressively worse
through time.
I have tried a bunch of different things mainly relating to frame rate and
audio resampling, but I must be missing something since I can't prevent the
drift. It has been much worse at the beginning of this project, and through
my tweaking the drift has gotten less; but it's still there now. I've run
out of ideas and fear this is going beyond my capability to debug and
figure out on my own.
Here is the command I use for recording a video file + an audio file at the
same time:
ffmpeg -y -f video4linux2 -pix_fmt yuv420p -framerate 25 -i /dev/video1 \
-f s16le -ar 44100 -ac 2 -i tcp://localhost:55555 \
-ar 44100 -af "aresample=async=1000" -c:v libx264 -vf
"fps=25,format=yuv420p" -g 60 -crf 23 -preset superfast
/data/13202a7ec918dd43e280a9aeb7063f01.mp4 \
-ar 44100 /data/13202a7ec918dd43e280a9aeb7063f01.mp3
I run a second "ffmpeg" command, at the same time, to stream, but it looks
identical except for the output stuff; it follows the same idea.
The v4l2 loopback device has a framerate set to 25 as well.
At the bottom of this email you can see ffmpeg output. I snipped a bunch of
lines out of the middle for brevity, but I find it interesting how the
"speed" value at the end of each line start at > 1, then eventually falls
to 1. Same with FPS, which starts > 25 but then over a bit of time, drops
to the target of 25. Would this have anything to do with it?
So my question, again; how do I get ffmpeg to set the video/audio
timestamps such that they are set equal to the MS they arrive in
/dev/video1 and in tcp:localhost:55555.
Thanks in advance for any tips/tricks/suggestions/verbal abuse etc!
ffmpeg version 3.4.2-2 Copyright (c) 2000-2018 the FFmpeg developers
built with gcc 7 (Ubuntu 7.3.0-16ubuntu2)
configuration: --prefix=/usr --extra-version=2 --toolchain=hardened
--libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu
--enable-gpl --disable-stripping --enable-avresample --enable-avisynth
--enable-gnutls --enable-ladspa --enable-libass --enable-libbluray
--enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite
--enable-libfontconfig --enable-libfreetype --enable-libfribidi
--enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa
--enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse
--enable-librubberband --enable-librsvg --enable-libshine
--enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh
--enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx
--enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2
--enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx
--enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394
--enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r
--enable-libopencv --enable-libx264 --enable-shared
libavutil 55. 78.100 / 55. 78.100
libavcodec 57.107.100 / 57.107.100
libavformat 57. 83.100 / 57. 83.100
libavdevice 57. 10.100 / 57. 10.100
libavfilter 6.107.100 / 6.107.100
libavresample 3. 7. 0 / 3. 7. 0
libswscale 4. 8.100 / 4. 8.100
libswresample 2. 9.100 / 2. 9.100
libpostproc 54. 7.100 / 54. 7.100
[video4linux2,v4l2 @ 0x55e2e68e0c80] The driver does not permit changing
the time per frame
Input #0, video4linux2,v4l2, from '/dev/video1':
Duration: N/A, start: 1525290092.292895, bitrate: 61632 kb/s
Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 428x240,
61632 kb/s, 25 fps, 25 tbr, 1000k tbn, 1000k tbc
Guessed Channel Layout for Input Stream #1.0 : stereo
Input #1, s16le, from 'tcp://localhost:55555':
Duration: N/A, bitrate: 1411 kb/s
Stream #1:0: Audio: pcm_s16le, 44100 Hz, stereo, s16, 1411 kb/s
Stream mapping:
Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
Stream #1:0 -> #0:1 (pcm_s16le (native) -> aac (native))
Stream #1:0 -> #1:0 (pcm_s16le (native) -> mp3 (libmp3lame))
Press [q] to stop, [?] for help
[libx264 @ 0x55e2e692be40] using cpu capabilities: MMX2 SSE2Fast SSSE3
SSE4.2 AVX
[libx264 @ 0x55e2e692be40] profile High, level 2.1
[libx264 @ 0x55e2e692be40] 264 - core 152 r2854 e9a5903 - H.264/MPEG-4 AVC
codec - Copyleft 2003-2017 - http://www.videolan.org/x264.html - options:
cabac=1 ref=1 deblock=1:0:0 analyse=0x3:0x3 me=dia subme=1 psy=1
psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=1
cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=7
lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0
bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1
b_bias=0 direct=1 weightb=1 open_gop=0 weightp=1 keyint=60 keyint_min=6
scenecut=40 intra_refresh=0 rc=crf mbtree=0 crf=23.0 qcomp=0.60 qpmin=0
qpmax=69 qpstep=4 ip_ratio=1.40 pb_ratio=1.30 aq=1:1.00
[s16le @ 0x55e2e68e42e0] Thread message queue blocking; consider raising
the thread_queue_size option (current value: 8)
Output #0, mp4, to '/data/13202a7ec918dd43e280a9aeb7063f01.mp4':
Metadata:
encoder : Lavf57.83.100
Stream #0:0: Video: h264 (libx264) (avc1 / 0x31637661), yuv420p,
428x240, q=-1--1, 25 fps, 12800 tbn, 25 tbc
Metadata:
encoder : Lavc57.107.100 libx264
Side data:
cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
Stream #0:1: Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo,
fltp, 128 kb/s
Metadata:
encoder : Lavc57.107.100 aac
Output #1, mp3, to '/data/13202a7ec918dd43e280a9aeb7063f01.mp3':
Metadata:
TSSE : Lavf57.83.100
Stream #1:0: Audio: mp3 (libmp3lame), 44100 Hz, stereo, s16p
Metadata:
encoder : Lavc57.107.100 libmp3lame
[swscaler @ 0x55e2e693ca00] Warning: data is not aligned! This can lead to
a speed loss
frame= 42 fps=0.0 q=17.0 size= 0kB time=00:00:01.06 bitrate=
0.4kbits/s speed=2.11x
frame= 54 fps= 53 q=20.0 size= 0kB time=00:00:01.57 bitrate=
0.2kbits/s speed=1.56x
frame= 67 fps= 44 q=22.0 size= 0kB time=00:00:02.04 bitrate=
0.2kbits/s speed=1.34x
frame= 79 fps= 39 q=24.0 size= 0kB time=00:00:02.57 bitrate=
0.1kbits/s speed=1.27x
frame= 92 fps= 36 q=22.0 size= 0kB time=00:00:03.06 bitrate=
0.1kbits/s speed=1.21x
frame= 105 fps= 35 q=23.0 size= 256kB time=00:00:03.62 bitrate=
579.1kbits/s speed=1.19x
frame= 118 fps= 33 q=18.0 size= 256kB time=00:00:04.10 bitrate=
510.4kbits/s speed=1.16x
frame= 130 fps= 32 q=23.0 size= 256kB time=00:00:04.59 bitrate=
456.2kbits/s speed=1.14x
frame= 142 fps= 31 q=16.0 size= 256kB time=00:00:05.10 bitrate=
410.6kbits/s speed=1.12x
frame= 155 fps= 31 q=19.0 size= 256kB time=00:00:05.64 bitrate=
371.7kbits/s speed=1.12x
frame= 167 fps= 30 q=17.0 size= 512kB time=00:00:06.10 bitrate=
686.9kbits/s speed= 1.1x
frame= 180 fps= 30 q=20.0 size= 512kB time=00:00:06.59 bitrate=
636.1kbits/s speed=1.09x
frame= 193 fps= 29 q=26.0 size= 512kB time=00:00:07.15 bitrate=
586.5kbits/s speed=1.09x
frame= 205 fps= 29 q=24.0 size= 512kB time=00:00:07.61 bitrate=
550.8kbits/s speed=1.08x
frame= 218 fps= 29 q=22.0 size= 512kB time=00:00:08.15 bitrate=
514.7kbits/s speed=1.08x
frame= 231 fps= 29 q=20.0 size= 512kB time=00:00:08.63 bitrate=
485.6kbits/s speed=1.07x
frame= 243 fps= 28 q=19.0 size= 768kB time=00:00:09.14 bitrate=
687.7kbits/s speed=1.07x
frame= 256 fps= 28 q=19.0 size= 768kB time=00:00:09.65 bitrate=
651.4kbits/s speed=1.06x
frame= 269 fps= 28 q=16.0 size= 768kB time=00:00:10.14 bitrate=
620.1kbits/s speed=1.06x
frame= 281 fps= 28 q=20.0 size= 768kB time=00:00:10.68 bitrate=
589.1kbits/s speed=1.06x
frame= 294 fps= 28 q=18.0 size= 768kB time=00:00:11.19 bitrate=
562.2kbits/s speed=1.05x
frame= 307 fps= 28 q=17.0 size= 1024kB time=00:00:11.65 bitrate=
719.7kbits/s speed=1.05x
frame= 319 fps= 27 q=17.0 size= 1024kB time=00:00:12.19 bitrate=
688.2kbits/s speed=1.05x
frame= 332 fps= 27 q=16.0 size= 1024kB time=00:00:12.70 bitrate=
660.5kbits/s speed=1.05x
frame= 344 fps= 27 q=16.0 size= 1024kB time=00:00:13.18 bitrate=
636.1kbits/s speed=1.04x
frame= 357 fps= 27 q=19.0 size= 1024kB time=00:00:13.69 bitrate=
612.3kbits/s speed=1.04x
frame= 370 fps= 27 q=23.0 size= 1024kB time=00:00:14.23 bitrate=
589.4kbits/s speed=1.04x
frame= 383 fps= 27 q=17.0 size= 1280kB time=00:00:14.74 bitrate=
711.2kbits/s speed=1.04x
frame= 395 fps= 27 q=17.0 size= 1280kB time=00:00:15.20 bitrate=
689.5kbits/s speed=1.04x
frame= 408 fps= 27 q=18.0 size= 1280kB time=00:00:15.71 bitrate=
667.1kbits/s speed=1.04x
frame= 418 fps= 27 q=-1.0 size= 1536kB time=00:00:16.60 bitrate=
757.9kbits/s speed=1.06x
... snip ...
frame= 2869 fps= 25 q=21.0 size= 11008kB time=00:01:54.19 bitrate=
789.7kbits/s speed=1.01x
frame= 2882 fps= 25 q=23.0 size= 11008kB time=00:01:54.68 bitrate=
786.3kbits/s speed= 1x
frame= 2894 fps= 25 q=23.0 size= 11008kB time=00:01:55.19 bitrate=
782.8kbits/s speed= 1x
frame= 2907 fps= 25 q=20.0 size= 11008kB time=00:01:55.70 bitrate=
779.4kbits/s speed= 1x
frame= 2919 fps= 25 q=21.0 size= 11264kB time=00:01:56.19 bitrate=
794.2kbits/s speed= 1x
frame= 2932 fps= 25 q=17.0 size= 11264kB time=00:01:56.70 bitrate=
790.7kbits/s speed= 1x
More information about the ffmpeg-user
mailing list