[FFmpeg-user] Use source wallclock (NTP) in RTSP/RTCP streams to setpts for inter-media synchronization

David Gessel gessel at blackrosetech.com
Mon Jan 18 00:38:02 EET 2021



-------- Original Message --------
Subject: Re: [FFmpeg-user] Use source wallclock (NTP) in RTSP/RTCP streams to setpts for inter-media synchronization
From: Carl Eugen Hoyos <ceffmpeg at gmail.com>
To: FFmpeg user questions <ffmpeg-user at ffmpeg.org>
Date: 2021-01-18 00:24+0300

> Am Sa., 16. Jan. 2021 um 23:49 Uhr schrieb David Gessel
> <gessel at blackrosetech.com>:
> 
>> ffmpeg  -max_delay 500000 -reorder_queue_size 10000 \
>>            -fflags nobuffer -re -rtsp_transport udp -an -flags low_delay -strict experimental\
>>            -fflags nobuffer -re -thread_queue_size 1024 -i rtsp://192.168.100.34:554/stream/profile0\
>>            -fflags nobuffer -re -thread_queue_size 1024 -i rtsp://192.168.100.57:554/stream/profile0\
>>            -fflags nobuffer -re -thread_queue_size 1024 -i rtsp://192.168.100.48:554/stream/profile0\
>>            -fflags nobuffer -re -thread_queue_size 1024 -i rtsp://192.168.100.34:554/stream/profile1\
>>            -fflags nobuffer -re -thread_queue_size 1024 -i rtsp://192.168.100.57:554/stream/profile1\
>>            -fflags nobuffer -re -thread_queue_size 1024 -i rtsp://192.168.100.48:554/stream/profile1\
>>           -filter_complex "
>>           nullsrc=size=3554x480 [base];
>>           [0:v] setpts=N/(8.33*TB) [CAM1];
>>           [1:v] setpts=N/(8.33*TB) [CAM2];
>>           [2:v] setpts=N/(8.33*TB) [CAM3];
>>           [3:v] setpts=N/(8.33*TB) [CAM4];
>>           [4:v] setpts=N/(8.33*TB) [CAM5];
>>           [5:v] setpts=N/(8.33*TB) [CAM6];
>>           [base][CAM1] overlay=x=0 [tmp1];
>>           [tmp1][CAM2] overlay=x=583 [tmp2];
>>           [tmp2][CAM3] overlay=x=1165 [tmp3];
>>           [tmp3][CAM4] overlay=x=1748 [tmp4];
>>           [tmp4][CAM5] overlay=x=2331 [tmp5];
>>           [tmp5][CAM6] overlay=x=2914 " \
>>           -c:v libx264 -tune zerolatency -an -preset ultrafast -crf 22 -f matroska - |\
>>       ffplay -framedrop -sync ext -probesize 32 -
> 
> You should really test with file output and complete, uncut console
> output missing.

Thanks for the help Carl,

The console output is attached below

> 
> As long as you are using the setpts filter, you disable all
> synchronisation and you
> cannot get the output you want.

Makes sense, thanks for the hint.  A link to a screen cap of the command without setpts attached below.  The time code is burned in at the sources.  I let it run for a few minutes and the streams did not sync.

https://www.brt.llc/insecure/ffmpeg_sync_issue.webp

(Babri the kitty says hi)

Console output follows:



$ ffmpeg  -max_delay 500000 -reorder_queue_size 10000 \
>          -fflags nobuffer -re -rtsp_transport udp -an -flags low_delay -strict experimental\
>          -fflags nobuffer -re -thread_queue_size 1024 -i rtsp://192.168.100.34:554/stream/profile0\
>          -fflags nobuffer -re -thread_queue_size 1024 -i rtsp://192.168.100.57:554/stream/profile0\
>          -fflags nobuffer -re -thread_queue_size 1024 -i rtsp://192.168.100.48:554/stream/profile0\
>          -fflags nobuffer -re -thread_queue_size 1024 -i rtsp://192.168.100.34:554/stream/profile1\
>          -fflags nobuffer -re -thread_queue_size 1024 -i rtsp://192.168.100.57:554/stream/profile1\
>          -fflags nobuffer -re -thread_queue_size 1024 -i rtsp://192.168.100.48:554/stream/profile1\
>         -filter_complex "
>         nullsrc=size=3554x480 [base];
>         [base][0:v] overlay=x=0 [tmp1];
>         [tmp1][1:v] overlay=x=583 [tmp2];
>         [tmp2][2:v] overlay=x=1165 [tmp3];
>         [tmp3][3:v] overlay=x=1748 [tmp4];
>         [tmp4][4:v] overlay=x=2331 [tmp5];
>         [tmp5][5:v] overlay=x=2914 " \
>         -c:v libx264 -tune zerolatency -an -preset ultrafast -crf 22 -f matroska - |\
>     ffplay -framedrop -sync ext -probesize 32 -
ffmpeg version N-100596-geacad34 Copyright (c) 2000-2021 the FFmpeg developers
   built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
   configuration: --prefix=/home/gessel/ffmpeg_build --pkg-config-flags=--static --extra-cflags=-I/home/gessel/ffmpeg_build/include --extra-ldflags=-L/home/gessel/ffmpeg_build/lib --extra-libs='-lpthread -lm' --bindir=/home/gessel/bin --enable-gpl --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-nonfree
   libavutil      56. 63.100 / 56. 63.100
   libavcodec     58.115.102 / 58.115.102
   libavformat    58. 65.101 / 58. 65.101
   libavdevice    58. 11.103 / 58. 11.103
   libavfilter     7. 95.100 /  7. 95.100
   libswscale      5.  8.100 /  5.  8.100
   libswresample   3.  8.100 /  3.  8.100
   libpostproc    55.  8.100 / 55.  8.100
ffplay version 4.2.4-1ubuntu0.1 Copyright (c) 2003-2020 the FFmpeg developers
   built with gcc 9 (Ubuntu 9.3.0-10ubuntu2)
   configuration: --prefix=/usr --extra-version=1ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
   libavutil      56. 31.100 / 56. 31.100
   libavcodec     58. 54.100 / 58. 54.100
   libavformat    58. 29.100 / 58. 29.100
   libavdevice    58.  8.100 / 58.  8.100
   libavfilter     7. 57.100 /  7. 57.100
   libavresample   4.  0.  0 /  4.  0.  0
   libswscale      5.  5.100 /  5.  5.100
   libswresample   3.  5.100 /  3.  5.100
   libpostproc    55.  5.100 / 55.  5.100
Input #0, rtsp, from 'rtsp://192.168.100.34:554/stream/profile0':
   Metadata:
     title           : Mobotix IP-Camera
     comment         : Profile profile0
   Duration: N/A, start: 0.000000, bitrate: N/A
     Stream #0:0: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/unknown), 640x480 [SAR 1:1 DAR 4:3], 8.33 fps, 8.33 tbr, 90k tbn, 90k tbc
Input #1, rtsp, from 'rtsp://192.168.100.57:554/stream/profile0':
   Metadata:
     title           : Mobotix IP-Camera
     comment         : Profile profile0
   Duration: N/A, start: 0.000000, bitrate: N/A
     Stream #1:0: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/unknown), 640x480 [SAR 1:1 DAR 4:3], 8.33 fps, 8.33 tbr, 90k tbn, 90k tbc
Input #2, rtsp, from 'rtsp://192.168.100.48:554/stream/profile0':
   Metadata:
     title           : Mobotix IP-Camera
     comment         : Profile profile0
   Duration: N/A, start: 0.000000, bitrate: N/A
     Stream #2:0: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/unknown), 640x480 [SAR 1:1 DAR 4:3], 8.33 fps, 8.33 tbr, 90k tbn, 90k tbc
Input #3, rtsp, from 'rtsp://192.168.100.34:554/stream/profile1':
   Metadata:
     title           : Mobotix IP-Camera
     comment         : Profile profile1
   Duration: N/A, start: 0.000000, bitrate: N/A
     Stream #3:0: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/unknown), 640x480 [SAR 1:1 DAR 4:3], 8.33 fps, 8.33 tbr, 90k tbn, 90k tbc
Input #4, rtsp, from 'rtsp://192.168.100.57:554/stream/profile1':
   Metadata:
     title           : Mobotix IP-Camera
     comment         : Profile profile1
   Duration: N/A, start: 0.000000, bitrate: N/A
     Stream #4:0: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/unknown), 640x480 [SAR 1:1 DAR 4:3], 8.33 fps, 8.33 tbr, 90k tbn, 90k tbc
Input #5, rtsp, from 'rtsp://192.168.100.48:554/stream/profile1':
   Metadata:
     title           : Mobotix IP-Camera
     comment         : Profile profile1
   Duration: N/A, start: 0.000000, bitrate: N/A
     Stream #5:0: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/unknown), 640x480 [SAR 1:1 DAR 4:3], 8.33 fps, 8.33 tbr, 90k tbn, 90k tbc
Stream mapping:
   Stream #0:0 (mjpeg) -> overlay:overlay
   Stream #1:0 (mjpeg) -> overlay:overlay
   Stream #2:0 (mjpeg) -> overlay:overlay
   Stream #3:0 (mjpeg) -> overlay:overlay
   Stream #4:0 (mjpeg) -> overlay:overlay
   Stream #5:0 (mjpeg) -> overlay:overlay
   overlay -> Stream #0:0 (libx264)
Press [q] to stop, [?] for help
[swscaler @ 0x5615e8adc0c0] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x5615e8afce40] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x5615e8b1e340] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x5615e8b3f9c0] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x5615e8b61240] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x5615e8b82b80] deprecated pixel format used, make sure you did set range correctly
[rtsp @ 0x5615e87a56c0] max delay reached. need to consume packet
[rtsp @ 0x5615e87a56c0] RTP: missed 2152 packets
[rtsp @ 0x5615e875c400] max delay reached. need to consume packet
[rtsp @ 0x5615e875c400] RTP: missed 5058 packets
[rtsp @ 0x5615e875c400] RTP: PT=1a: bad cseq bbf9 expected=a837
[rtsp @ 0x5615e875c400] max delay reached. need to consume packet
[rtsp @ 0x5615e875c400] RTP: missed 5059 packets
[rtsp @ 0x5615e875c400] RTP timestamps don't match.
[rtsp @ 0x5615e875c400] Received packet without a start chunk; dropping frame.
     Last message repeated 10 times
[rtsp @ 0x5615e87e54c0] max delay reached. need to consume packet
[rtsp @ 0x5615e87e54c0] RTP: missed 1058 packets
[rtsp @ 0x5615e875c400] Received packet without a start chunk; dropping frame.
     Last message repeated 52 times 0KB vq=    0KB sq=    0B f=0/0
[libx264 @ 0x5615e88a4300] using SAR=1/1
[libx264 @ 0x5615e88a4300] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 0x5615e88a4300] profile Constrained Baseline, level 4.0
[libx264 @ 0x5615e88a4300] 264 - core 155 r2917 0a84d98 - H.264/MPEG-4 AVC codec - Copyleft 2003-2018 - http://www.videolan.org/x264.html - options: cabac=0 ref=1 deblock=0:0:0 analyse=0:0 me=dia subme=0 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=1 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=7 lookahead_threads=7 sliced_threads=1 slices=7 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=0 keyint=250 keyint_min=25 scenecut=0 intra_refresh=0 rc=crf mbtree=0 crf=22.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=0
Output #0, matroska, to 'pipe:':
   Metadata:
     title           : Mobotix IP-Camera
     comment         : Profile profile0
     encoder         : Lavf58.65.101
     Stream #0:0: Video: h264 (H264 / 0x34363248), yuv420p(progressive), 3554x480 [SAR 1:1 DAR 1777:240], q=2-31, 25 fps, 1k tbn (default)
     Metadata:
       encoder         : Lavc58.115.102 libx264
     Side data:
       cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
[rtsp @ 0x5615e8761400] max delay reached. need to consume packet   4856.0kbits/s speed=0.00769x
[rtsp @ 0x5615e8761400] RTP: missed 1424 packets
[matroska,webm @ 0x7f5a40000bc0] decoding for stream 0 failed=0/0
Input #0, matroska,webm, from 'pipe:':
   Metadata:
     title           : Mobotix IP-Camera
     COMMENT         : Profile profile0
     ENCODER         : Lavf58.65.101
   Duration: N/A, bitrate: N/A
     Stream #0:0: Video: h264 (Constrained Baseline), yuv420p(progressive), 3554x480 [SAR 1:1 DAR 1777:240], 25 fps, 25 tbr, 1k tbn, 50 tbc (default)
     Metadata:
       ENCODER         : Lavc58.115.102 libx264
[rtsp @ 0x5615e8730080] max delay reached. need to consume packet
[rtsp @ 0x5615e8730080] RTP: missed 5791 packets
[rtsp @ 0x5615e8730080] RTP: PT=1a: bad cseq 32e9 expected=1c4a
[rtsp @ 0x5615e8730080] max delay reached. need to consume packet
[rtsp @ 0x5615e8730080] RTP: missed 5792 packets
[rtsp @ 0x5615e8730080] RTP timestamps don't match.
[rtsp @ 0x5615e8730080] Received packet without a start chunk; dropping frame.
     Last message repeated 58 times 0KB vq=    5KB sq=    0B f=0/0
Error writing trailer of pipe:: Broken pipe   0KB sq=    0B f=0/0   18908.6kbits/s speed=0.998x
frame=  877 fps= 25 q=17.0 Lsize=   80881kB time=00:00:35.04 bitrate=18908.6kbits/s speed=0.983x
video:80935kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[libx264 @ 0x5615e88a4300] frame I:4     Avg QP:16.75  size:532250
[libx264 @ 0x5615e88a4300] frame P:873   Avg QP:18.00  size: 92495
[libx264 @ 0x5615e88a4300] mb I  I16..4: 100.0%  0.0%  0.0%
[libx264 @ 0x5615e88a4300] mb P  I16..4:  0.8%  0.0%  0.0%  P16..4: 62.2%  0.0%  0.0%  0.0%  0.0%    skip:37.0%
[libx264 @ 0x5615e88a4300] coded y,uvDC,uvAC intra: 60.5% 89.4% 86.4% inter: 41.4% 57.7% 38.1%
[libx264 @ 0x5615e88a4300] i16 v,h,dc,p: 31% 18% 38% 13%
[libx264 @ 0x5615e88a4300] i8c dc,h,v,p: 44% 20% 22% 14%
[libx264 @ 0x5615e88a4300] kb/s:18900.13
Exiting normally, received signal 2.



More information about the ffmpeg-user mailing list