[FFmpeg-user] Continue audio even when RTSP video stutters?

[Steven Kan]

Hi! For the last 7 years I've been streaming RTSP from an IP camera to one of my 4 YouTube channels:

https://www.youtube.com/@IAmTheWaterbug/live

I am adding a series of royalty-free MP3 to the stream because the camera lacks audio, but YouTube Stream Now requires an audio channel, and this was better (IMNSHO) than adding an empty audio track. I also like the music :-). This has worked very well for the last 7 years. 

About 6 months ago the camera started malfunctioning, and the RTSP stream now stutters every few seconds. It's definitely a problem with the stream from this particular camera because my other 3 streams from different IP cameras, on the same network, and from the same streaming computer, are working fine at minimal CPU load. 

If I paste the URL for the problem camera into VLC, the video stutters just like the YT stream. ffmpeg reports:

vt decoder cb: output image buffer is null

repeatedly (see command string, below). 

As you can see on the stream, the video stuttering is annoying, but not crippling, but the audio interruptions are intolerable. 

I am still looking to replace the camera, but while I'm looking, is there a way to modify my processing chain so that, when the RTSP video stutters, the audio does _not_ pause? The audio is being sourced directly from MP3s on the local HD of the computer running ffmpeg, so there's no issue with that data being accessible in a timely manner. When the video stream stutters I'd like it to continue doing what it does now, e.g. just display the last available frame until a new one comes along. 

If I output to DeleteMe.flv, then the FLV video is a mess, while the audio continues to play normally. If I open that FLV in VLC the video will just stall completely. If I open it in IINA I can skip around in the timeline, and it will play a few frames, but it won't play through:

https://www.kan.org/download/DeleteMe.flv

I don't care about latency, within reason, because it's already at least 10 seconds behind real-time, based on the timestamp. 

Thanks!

./ffmpeg1 -thread_queue_size 2048 -hwaccel videotoolbox -i "rtsp://anonymous:password@192.168.1.11:554" -f concat -safe 0 -i playlist.txt -vf drawtext="fontfile=/Users/steven/Library/Fonts/Nouveau_IBM.ttf: text='%{localtime\:%Y-%m-%d\ %H\\\\\:%M\\\\\:%S}': fontcolor=white: fontsize=66: x=(w-text_w)*0.01: y=(h-text_h)*0.08" -vcodec h264_videotoolbox -b:v 5000k -acodec copy -t 00:1:00 -f flv DeleteMe.flv 
ffmpeg version N-109676-g40512dbd96-tessus  https://evermeet.cx/ffmpeg/  Copyright (c) 2000-2023 the FFmpeg developers
  built with Apple clang version 11.0.0 (clang-1100.0.33.17)
  configuration: --cc=/usr/bin/clang --prefix=/opt/ffmpeg --extra-version=tessus --enable-avisynth --enable-fontconfig --enable-gpl --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libfreetype --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libmysofa --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvmaf --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-version3 --pkg-config-flags=--static --disable-ffplay
  libavutil      57. 44.100 / 57. 44.100
  libavcodec     59. 57.100 / 59. 57.100
  libavformat    59. 36.100 / 59. 36.100
  libavdevice    59.  8.101 / 59.  8.101
  libavfilter     8. 54.100 /  8. 54.100
  libswscale      6.  8.112 /  6.  8.112
  libswresample   4.  9.100 /  4.  9.100
  libpostproc    56.  7.100 / 56.  7.100
Input #0, rtsp, from 'rtsp://anonymous:password@192.168.1.11:554':
  Metadata:
    title           : Session streamed by "preview"
    comment         : 
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: h264 (High), yuv420p(progressive), 3072x1728, 20 fps, 20 tbr, 90k tbn
  Stream #0:1: Audio: aac (LC), 16000 Hz, mono, fltp
[mp3 @ 0x7fc0f9804c80] Estimating duration from bitrate, this may be inaccurate
Input #1, concat, from 'playlist.txt':
  Duration: N/A, start: 0.000000, bitrate: 320 kb/s
  Stream #1:0: Audio: mp3, 44100 Hz, stereo, fltp, 320 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_videotoolbox))
  Stream #1:0 -> #0:1 (copy)
Press [q] to stop, [?] for help
Output #0, flv, to 'DeleteMe.flv':
  Metadata:
    title           : Session streamed by "preview"
    comment         : 
    encoder         : Lavf59.36.100
  Stream #0:0: Video: h264 ([7][0][0][0] / 0x0007), nv12(progressive), 3072x1728, q=2-31, 5000 kb/s, 20 fps, 1k tbn
    Metadata:
      encoder         : Lavc59.57.100 h264_videotoolbox
  Stream #0:1: Audio: mp3 ([2][0][0][0] / 0x0002), 44100 Hz, stereo, fltp, 320 kb/s
[h264_videotoolbox @ 0x7fc0f8f0e5c0] Color range not set for nv12. Using MPEG range.
[h264 @ 0x7fc0f8f10ac0] vt decoder cb: output image buffer is null: -12909kbits/s speed=0.44x     
[h264 @ 0x7fc0f8f1b680] hardware accelerator failed to decode picture
[h264 @ 0x7fc0f8f10ac0] vt decoder cb: output image buffer is null: -12909kbits/s speed=0.414x    
[h264 @ 0x7fc0f8f26040] hardware accelerator failed to decode picture
Error while decoding stream #0:0: Unknown error occurred.29 bitrate=5329.5kbits/s speed=0.41x    
    Last message repeated 1 times
[h264 @ 0x7fc0f8f10ac0] vt decoder cb: output image buffer is null: -12909kbits/s speed=0.742x    
[h264 @ 0x7fc0e90043c0] hardware accelerator failed to decode picture
Error while decoding stream #0:0: Unknown error occurred.84 bitrate=4727.5kbits/s speed=0.729x    
frame=  315 fps=7.9 q=-0.0 size=   19712kB time=00:00:39.96 bitrate=4040.4kbits/s speed=0.996x