[FFmpeg-user] Trying to stream to rtmp from my rsp4 and a GoPro

[Mike Soultanian]

On 2/13/2021 5:36 PM, Carl Eugen Hoyos wrote:
>> It is not possible to synchronize these two streams due
>> to the different start times, test the ts option:
>> https://ffmpeg.org/ffmpeg-devices.html#video4linux2_002c-v4l2
> Where do I use the -ts option (audio, video, output, all)?  And are 
> you recommending using default, abs, or mono2abs?
>
Just fyi, I tried all of the following and none of them really worked:

1. ffmpeg -f v4l2 -input_format yuyv422 -framerate 30 -i /dev/video0 -f 
alsa -ac 2 -i hw:CARD=capture,DEV=0 -c:v libx264 -vf format=yuv420p -y 
-ts default Desktop/test.mp4

2. ffmpeg -f v4l2 -input_format yuyv422 -framerate 30 -i /dev/video0 -f 
alsa -ac 2 -i hw:CARD=capture,DEV=0 -c:v libx264 -vf format=yuv420p -y 
-ts abs Desktop/test.mp4

3. ffmpeg -f v4l2 -input_format yuyv422 -framerate 30 -i /dev/video0 -f 
alsa -ac 2 -i hw:CARD=capture,DEV=0 -c:v libx264 -vf format=yuv420p -y 
-ts mono2abs Desktop/test.mp4

So 2 seems to be the closest to working, but with really inconsistent 
behavior it's a stretch to say working - on the first run (only after a 
reboot) I get glitchy video that freezes every now and then with little 
blips of audio every 5 seconds or so, and if I run the command again the 
fps drops down super low and no video is recorded, but strangely audio 
is recorded.  Here is the output:

*pi at raspberrypi:~ $ v4l2-ctl --set-fmt-video=width=1280,height=720 *
*pi at raspberrypi:~ $ ffmpeg -f v4l2 -input_format yuyv422 -framerate 30 
-i /dev/video0 -f alsa -ac 2 -i hw:CARD=capture,DEV=0 -c:v libx264 -vf 
format=yuv420p -y -ts abs Desktop/test.mp4*
ffmpeg version N-101069-g000b250 Copyright (c) 2000-2021 the FFmpeg 
developers
   built with gcc 8 (Raspbian 8.3.0-6+rpi1)
   configuration: --prefix=/home/pi/ffmpeg_build 
--pkg-config-flags=--static 
--extra-cflags=-I/home/pi/ffmpeg_build/include 
--extra-ldflags=-L/home/pi/ffmpeg_build/lib --extra-libs='-lpthread -lm' 
--bindir=/home/pi/bin --enable-gpl --enable-gnutls --enable-libass 
--enable-libfdk-aac --enable-libfreetype --enable-libmp3lame 
--enable-libopus --enable-libsvtav1 --enable-libdav1d --enable-libvorbis 
--enable-libvpx --enable-libx264 --enable-libx265 --enable-omx 
--enable-omx-rpi --enable-nonfree
   libavutil      56. 65.100 / 56. 65.100
   libavcodec     58.122.100 / 58.122.100
   libavformat    58. 67.100 / 58. 67.100
   libavdevice    58. 11.103 / 58. 11.103
   libavfilter     7.106.100 /  7.106.100
   libswscale      5.  8.100 /  5.  8.100
   libswresample   3.  8.100 /  3.  8.100
   libpostproc    55.  8.100 / 55.  8.100
Input #0, video4linux2,v4l2, from '/dev/video0':
   Duration: N/A, start: 306.070836, bitrate: 442368 kb/s
   Stream #0:0: Video: rawvideo (YUY2 / 0x32595559), yuyv422, 1280x720, 
442368 kb/s, 30 fps, 30 tbr, 1000k tbn, 1000k tbc
Guessed Channel Layout for Input Stream #1.0 : stereo
Input #1, alsa, from 'hw:CARD=capture,DEV=0':
   Duration: N/A, start: 1613290013.175295, bitrate: 1536 kb/s
   Stream #1:0: Audio: pcm_s16le, 48000 Hz, stereo, s16, 1536 kb/s
Stream mapping:
   Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
   Stream #1:0 -> #0:1 (pcm_s16le (native) -> aac (native))
Press [q] to stop, [?] for help
[libx264 @ 0x35387a0] using cpu capabilities: ARMv6 NEON
[libx264 @ 0x35387a0] profile High, level 3.1
[libx264 @ 0x35387a0] 264 - core 146 - H.264/MPEG-4 AVC codec - Copyleft 
2003-2015 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 
deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 
mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 
deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 
lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 
bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 
b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 
keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf 
mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 
aq=1:1.00
[alsa @ 0x35262c0] Thread message queue blocking; consider raising the 
thread_queue_size option (current value: 8)
Output #0, mp4, to 'Desktop/test.mp4':
   Metadata:
     encoder         : Lavf58.67.100
   Stream #0:0: Video: h264 (avc1 / 0x31637661), yuv420p(tv, 
progressive), 1280x720, q=2-31, 30 fps, 15360 tbn
     Metadata:
       encoder         : Lavc58.122.100 libx264
     Side data:
       cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
   Stream #0:1: Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, 
fltp, 128 kb/s
     Metadata:
       encoder         : Lavc58.122.100 aac
[video4linux2,v4l2 @ 0x35233e0] Thread message queue blocking; consider 
raising the thread_queue_size option (current value: 8)
*[alsa @ 0x35262c0] ALSA buffer xrun.   0kB time=00:00:00.36 bitrate=   
1.0kbits/s dup=1 drop=0 speed=0.121x **
**[alsa @ 0x35262c0] ALSA buffer xrun.1024kB time=00:00:03.35 
bitrate=2503.1kbits/s dup=21 drop=0 speed=0.351x **
**[alsa @ 0x35262c0] ALSA buffer xrun.2304kB time=00:00:10.05 
bitrate=1877.4kbits/s dup=168 drop=0 speed=0.586x **
**[alsa @ 0x35262c0] ALSA buffer xrun.4096kB time=00:00:20.19 
bitrate=1661.3kbits/s dup=413 drop=0 speed=0.732x **
**[alsa @ 0x35262c0] ALSA buffer xrun.5376kB time=00:00:28.33 
bitrate=1554.5kbits/s dup=637 drop=0 speed=0.753x **
**frame= 1111 fps= 24 q=-1.0 Lsize=    6536kB time=00:00:38.64 
bitrate=1385.4kbits/s dup=816 drop=0 speed=0.832x *
video:6504kB audio:17kB subtitle:0kB other streams:0kB global 
headers:0kB muxing overhead: 0.225371%
[libx264 @ 0x35387a0] frame I:5     Avg QP:18.71  size:137251
[libx264 @ 0x35387a0] frame P:461   Avg QP:21.46  size: 11990
[libx264 @ 0x35387a0] frame B:645   Avg QP:29.15  size:   690
[libx264 @ 0x35387a0] consecutive B-frames: 19.2%  4.9% 16.2% 59.8%
[libx264 @ 0x35387a0] mb I  I16..4: 22.8% 38.1% 39.1%
[libx264 @ 0x35387a0] mb P  I16..4:  0.7%  3.1%  1.5%  P16..4: 5.3%  
4.1%  2.8%  0.0%  0.0%    skip:82.4%
[libx264 @ 0x35387a0] mb B  I16..4:  0.0%  0.0%  0.0%  B16..8: 4.0%  
0.8%  0.2%  direct: 0.2%  skip:94.8%  L0:32.0% L1:51.3% BI:16.7%
[libx264 @ 0x35387a0] 8x8 transform intra:55.3% inter:49.9%
[libx264 @ 0x35387a0] coded y,uvDC,uvAC intra: 66.3% 62.9% 26.2% inter: 
3.6% 2.3% 0.1%
[libx264 @ 0x35387a0] i16 v,h,dc,p: 31% 44%  4% 21%
[libx264 @ 0x35387a0] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 17% 15% 23% 7%  7%  
7%  8%  7% 10%
[libx264 @ 0x35387a0] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 23% 20% 19% 6%  6%  
6%  7%  6%  7%
[libx264 @ 0x35387a0] i8c dc,h,v,p: 51% 24% 18%  7%
[libx264 @ 0x35387a0] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x35387a0] ref P L0: 70.2% 21.2%  6.6%  2.0%
[libx264 @ 0x35387a0] ref B L0: 92.0%  7.2%  0.8%
[libx264 @ 0x35387a0] ref B L1: 93.1%  6.9%
[libx264 @ 0x35387a0] kb/s:1438.51
[aac @ 0x353e590] Qavg: 152.630