[FFmpeg-user] HW Acceleration 101? 2-Up Streaming from RTSP-->ffmpeg-->YouTube

Mon Nov 8 00:40:13 EET 2021

> On Jan 18, 2021, at 10:42 PM, Carl Eugen Hoyos <ceffmpeg at gmail.com> wrote:
> 
> Am Mo., 18. Jan. 2021 um 23:34 Uhr schrieb Steven Kan <steven at kan.org>:
>> 
>>> On Jan 18, 2021, at 12:50 PM, Michael Koch <astroelectronic at t-online.de> wrote:
> 
>>>> C:\Program Files\ffmpeg\bin> .\ffmpeg.exe -re -thread_queue_size 1024 -i rtsp://anonymous:password@192.168.1.47:554 -i rtsp://anonymous:password@192.168.1.50:554 -vcodec h264_amf -acodec copy -t 01:47:02 -filter_complex hstack=inputs=2 -f flv out.flv
>>>> 
>>>> [snip
>> 
>>>> Input #0, rtsp, from 'rtsp://anonymous:password@192.168.1.47:554':
>>>> Metadata:
>>>>   title           : Media Server
>>>> Duration: N/A, start: 0.080000, bitrate: N/A
>>>>   Stream #0:0: Video: h264 (High), yuvj420p(pc, bt709, progressive), 1920x1080, 25 fps, 25 tbr, 90k tbn, 180k tbc
>>>> Input #1, rtsp, from 'rtsp://anonymous:password@192.168.1.50:554':
>>>> Metadata:
>>>>   title           : Media Server
>>>> Duration: N/A, start: 0.100000, bitrate: N/A
>>>>   Stream #1:0: Video: h264 (Main), yuv420p(progressive), 1920x1080, 100 tbr, 90k tbn, 180k tbc
>>> 
>>> I see that the two streams have different pixel formats yuvj420p and yuv420p. You could try to bring them to the same pixel format before using hstack.
>>> [0]format=yuv420p[a];[a][1]hstack
>>> 
>>> It's only a wild guess, I'm not sure.
>> 
>> Do I put this into the filter_complex argument, e.g. -filter_complex "[0]format=yuv420p[a];[a][1] hstack=inputs=2”
>> 
>> That still results in the "Conversion failed!” error.
> 
> There is a "scale" missing behind format iirc but for performance reasons you
> want to overwrite the pix_fmt instead, not sure if this is possible.

Some progress, here. I’m now attempting this on an M1-powered Mac mini:

./ffmpeg -thread_queue_size 1024 -hwaccel videotoolbox -i rtsp://anonymous:password@192.168.1.47:554 -hwaccel videotoolbox -i rtsp://anonymous:password@192.168.1.50:554 -vcodec h264_videotoolbox -acodec copy -t 01:00:00 -filter_complex hstack=inputs=2 -f flv "rtmp://a.rtmp.youtube.com/live2/<my-youtube-streaming-key”

which results in:

ffmpeg version 4.4 Copyright (c) 2000-2021 the FFmpeg developers
  built with Apple clang version 12.0.0 (clang-1200.0.32.27)
  configuration: --prefix=/Volumes/tempdisk/sw --extra-cflags=-fno-stack-check --arch=arm64 --cc=/usr/bin/clang --enable-gpl --enable-videotoolbox --enable-libopenjpeg --enable-libopus --enable-libmp3lame --enable-libx264 --enable-libx265 --enable-libvpx --enable-libwebp --enable-libass --enable-libfreetype --enable-libtheora --enable-libvorbis --enable-libsnappy --enable-libaom --enable-libvidstab --enable-libzimg --enable-libsvtav1 --enable-version3 --pkg-config-flags=--static --disable-ffplay --enable-postproc --enable-nonfree --enable-neon --enable-runtime-cpudetect --disable-indev=qtkit --disable-indev=x11grab_xcb
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
Input #0, rtsp, from 'rtsp://anonymous:password@192.168.1.47:554':
  Metadata:
    title           : Media Server
  Duration: N/A, start: 0.080000, bitrate: N/A
  Stream #0:0: Video: h264 (High), yuvj420p(pc, bt709, progressive), 1920x1080, 25 fps, 25 tbr, 90k tbn, 180k tbc
Input #1, rtsp, from 'rtsp://anonymous:password@192.168.1.50:554':
  Metadata:
    title           : Media Server
  Duration: N/A, start: 0.060000, bitrate: N/A
  Stream #1:0: Video: h264 (Main), yuv420p(progressive), 1920x1080, 100 tbr, 90k tbn, 180k tbc
  Stream #1:1: Audio: aac (LC), 8000 Hz, mono, fltp
Stream mapping:
  Stream #0:0 (h264) -> hstack:input0
  Stream #1:0 (h264) -> hstack:input1
  hstack -> Stream #0:0 (h264_videotoolbox)
  Stream #1:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[rtsp @ 0x14a00ca00] max delay reached. need to consume packet
[rtsp @ 0x14a00ca00] RTP: missed 146 packets
[rtsp @ 0x139013a00] Thread message queue blocking; consider raising the thread_queue_size option (current value: 8)
[rtsp @ 0x14a00ca00] max delay reached. need to consume packet
[rtsp @ 0x14a00ca00] RTP: missed 80 packets
Output #0, flv, to 'rtmp://a.rtmp.youtube.com/live2/q9rg-sqaq-f0mg-yj1c-e42f':
  Metadata:
    title           : Media Server
    encoder         : Lavf58.76.100
  Stream #0:0: Video: h264 ([7][0][0][0] / 0x0007), nv12(progressive), 3840x1080, q=2-31, 200 kb/s, 1k tbn (default)
    Metadata:
      encoder         : Lavc58.134.100 h264_videotoolbox
  Stream #0:1: Audio: aac (LC) ([10][0][0][0] / 0x000A), 8000 Hz, mono, fltp
[h264_videotoolbox @ 0x14b01b800] Color range not set for nv12. Using MPEG range.
[rtsp @ 0x139013a00] max delay reached. need to consume packettrate=6235.2kbits/s dup=0 drop=1 speed=0.794x    
[rtsp @ 0x139013a00] RTP: missed 29 packets
[rtsp @ 0x139013a00] max delay reached. need to consume packet
[rtsp @ 0x139013a00] RTP: missed 78 packets
[rtsp @ 0x139013a00] max delay reached. need to consume packet
[rtsp @ 0x139013a00] RTP: missed 132 packets
[rtsp @ 0x139013a00] max delay reached. need to consume packet
[rtsp @ 0x139013a00] RTP: missed 49 packets
[rtsp @ 0x139013a00] max delay reached. need to consume packet
[rtsp @ 0x139013a00] RTP: missed 12 packets
[h264 @ 0x139074600] hardware accelerator failed to decode picture
[h264 @ 0x139074c00] hardware accelerator failed to decode picture
[h264 @ 0x139075200] hardware accelerator failed to decode picture    
[h264 @ 0x139070a00] hardware accelerator failed to decode picture
[h264 @ 0x139072800] hardware accelerator failed to decode picture
[h264 @ 0x139072e00] hardware accelerator failed to decode picture
[h264 @ 0x139073400] hardware accelerator failed to decode picture
[h264 @ 0x139073a00] hardware accelerator failed to decode picture
Error while decoding stream #1:0: Unknown error occurred
[h264 @ 0x139074000] hardware accelerator failed to decode picture
Error while decoding stream #1:0: Unknown error occurred

After a few seconds those errors go away, and then the status line just reads:

frame=20249 fps= 55 q=-0.0 size=   67873kB time=00:06:12.99 bitrate=1490.7kbits/s dup=0 drop=2 speed=   1x    

I don’t know why the errors occur, nor why they resolve themselves, but eventually it works. ¯\_(ツ)_/¯

top shows ffmpeg consuming about ~30% of the CPU, which is more than I had hoped for, but far better than I’d seen on my previous i5- and AMD-powered boxes. 

So I should be able to stream 4 cameras to 2 YT channels without completely cooking my mini.

I tried using hevc_videotoolbox, but I get the error, "Video codec hevc not compatible with flv"

I am hopeful that further optimization of ffmpeg for AS is coming!