[FFmpeg-user] overlay_qsv with hardware decode and encode
Greg Oliver
oliver.greg at gmail.com
Fri Feb 28 01:36:36 EET 2025
On Thu, Feb 27, 2025 at 11:25 AM Vladimir Mishonov via ffmpeg-user <
ffmpeg-user at ffmpeg.org> wrote:
> So, I've managed to figure out that this may be related to input streams
> having different resolutions.
>
https://ffmpeg.org/pipermail/ffmpeg-user/2024-August/058638.html
> In the meantime, I've also switched to VA-API because it seems more
> stable on my system, and uses less CPU. (No idea why it uses CPU at all,
> shouldn't it use GPU? The gputop utility does confirm the latter to be
> the case, at the very least.)
>
> In any way. The following command-line will work if both inputs have the
> same resolution:
>
> ffmpeg \
> -hwaccel vaapi -hwaccel_output_format vaapi -i rtmp://localhost/stream1
> \
> -hwaccel vaapi -hwaccel_output_format vaapi -i rtmp://localhost/stream2
> \
> -filter_complex "[0:v][1:v]overlay_vaapi=w=500" \
> -c:v h264_vaapi -global_quality 25 -g 50 -r 25 -an -f null -
>
> However, it fails with the same error as before ("Impossible to
> convert..") if resolutions do not match.
>
> Running with -loglevel verbose reveals some details:
>
> [graph 0 input from stream 0:1 @ 0x7f3fc4005b40] w:2560 h:1440
> pixfmt:vaapi tb:1/1000 fr:25/1 sar:0/1 csp:bt709 range:pc
> [graph 0 input from stream 1:1 @ 0x7f3fc4006600] w:2688 h:1520
> pixfmt:vaapi tb:1/1000 fr:26/1 sar:1/1 csp:unknown range:pc
> [auto_scale_0 @ 0x7f3fc40082c0] w:iw h:ih flags:'' interl:0
> [Parsed_overlay_vaapi_0 @ 0x7f3fc4005500] auto-inserting filter
> 'auto_scale_0' between the filter 'graph 0 input from stream 1:1' and
> the filter 'Parsed_overlay_vaapi_0'
> Impossible to convert between the formats supported by the filter 'graph
> 0 input from stream 1:1' and the filter 'auto_scale_0'
> [fc#0 @ 0x56490d7a1ec0] Error reinitializing filters!
> [fc#0 @ 0x56490d7a1ec0] Task finished with error code: -38 (Function not
> implemented)
> [fc#0 @ 0x56490d7a1ec0] Terminating thread with return code -38
> (Function not implemented)
>
> Note this part: "...auto-inserting filter 'auto_scale_0' between the
> filter 'graph 0 input from stream 0:1' and the filter
> 'Parsed_overlay_vaapi_0". This is obviously not going to work because
> those are hardware frames, not software!
>
> I've found an option called -noauto_conversion_filters, but it does not
> appear to do any better:
>
> ffmpeg -loglevel verbose -noauto_conversion_filters \
> -hwaccel vaapi -hwaccel_output_format vaapi -i rtmp://localhost/stream1
> \
> -hwaccel vaapi -hwaccel_output_format vaapi -i rtmp://localhost/stream2
> \
> -filter_complex "[0:v][1:v]overlay_vaapi=w=500" \
> -c:v h264_vaapi -global_quality 25 -g 50 -r 25 -an test.h264
>
> It says: "The filters 'graph 0 input from stream 1:1' and
> 'Parsed_overlay_vaapi_0' do not have a common format and automatic
> conversion is disabled."
>
> Is this a bug that I should report?? It certainly does look like one to
> me...
>
> ---
> Kind regards,
> Vladimir
>
> On 2025-02-21 10:12, Vladimir Mishonov via ffmpeg-user wrote:
> > Greetings everyone.
> >
> > I am attempting to overlay 2 video streams in real-time with QuickSync
> > hardware acceleration. I want to use both hardware decode and encode to
> > make the most use out of the GPU and avoid excessive CPU usage.
> >
> > I've found an example command-line here:
> > https://github.com/intel/vaapi-fits/issues/173 and adapted it:
> >
> > ./ffmpeg \
> > -hwaccel qsv -c:v h264_qsv -i rtmp://localhost/stream1 \
> > -hwaccel qsv -c:v h264_qsv -i rtmp://localhost/stream2 \
> > -filter_complex "[0:v][1:v]overlay_qsv=w=500" \
> > -c:v h264_qsv -global_quality 25 -g 50 -r 25 -an test.h264
> >
> > However, this gives me the following error:
> >
> > ffmpeg version N-118510-gbc1a3bfd2c-20250220 Copyright (c) 2000-2025
> > the FFmpeg developers
> > built with gcc 14.2.0 (crosstool-NG 1.26.0.120_4d36f27)
> > configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static
> > --pkg-config=pkg-config --cross-prefix=x86_64-ffbuild-linux-gnu-
> > --arch=x86_64 --target-os=linux --enable-gpl --enable-version3
> > --disable-debug --enable-iconv --enable-zlib --enable-libfreetype
> > --enable-libfribidi --enable-gmp --enable-libxml2 --enable-openssl
> > --enable-lzma --enable-fontconfig --enable-libharfbuzz
> > --enable-libvorbis --enable-opencl --enable-libpulse --enable-libvmaf
> > --enable-libxcb --enable-xlib --enable-amf --enable-libaom
> > --enable-libaribb24 --enable-avisynth --enable-chromaprint
> > --enable-libdav1d --enable-libdavs2 --enable-libdvdread
> > --enable-libdvdnav --disable-libfdk-aac --enable-ffnvcodec
> > --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libkvazaar
> > --enable-libaribcaption --enable-libass --enable-libbluray
> > --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librist
> > --enable-libssh --enable-libtheora --enable-libvpx --enable-libwebp
> > --enable-libzmq --enable-lv2 --enable-libvpl --enable-openal
> > --enable-libopencore-amrnb --enable-libopencore-amrwb
> > --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt
> > --enable-librav1e --enable-librubberband --disable-schannel
> > --enable-sdl2 --enable-libsnappy --enable-libsoxr --enable-libsrt
> > --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d
> > --enable-libdrm --enable-vaapi --enable-libvidstab --enable-vulkan
> > --enable-libshaderc --enable-libplacebo --disable-libvvenc
> > --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid
> > --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC
> > --extra-cxxflags= --extra-libs='-ldl -lgomp' --extra-ldflags=-pthread
> > --extra-ldexeflags=-pie --cc=x86_64-ffbuild-linux-gnu-gcc
> > --cxx=x86_64-ffbuild-linux-gnu-g++ --ar=x86_64-ffbuild-linux-gnu-gcc-ar
> > --ranlib=x86_64-ffbuild-linux-gnu-gcc-ranlib
> > --nm=x86_64-ffbuild-linux-gnu-gcc-nm --extra-version=20250220
> > libavutil 59. 57.100 / 59. 57.100
> > libavcodec 61. 33.102 / 61. 33.102
> > libavformat 61. 9.107 / 61. 9.107
> > libavdevice 61. 4.100 / 61. 4.100
> > libavfilter 10. 9.100 / 10. 9.100
> > libswscale 8. 13.100 / 8. 13.100
> > libswresample 5. 4.100 / 5. 4.100
> > libpostproc 58. 4.100 / 58. 4.100
> > [vist#0:1/h264 @ 0x562c70e48780] WARNING: defaulting
> > hwaccel_output_format to qsv for compatibility with old commandlines.
> > This behaviour is DEPRECATED and will be removed in the future. Please
> > explicitly set "-hwaccel_output_format qsv".
> > Input #0, flv, from 'rtmp://localhost/stream1':
> > Metadata:
> > |RtmpSampleAccess: true
> > Server : NGINX RTMP (github.com/arut/nginx-rtmp-module)
> > displayWidth : 2560
> > displayHeight : 1440
> > fps : 25
> > profile :
> > level :
> > Duration: 00:00:00.00, start: 138266.736000, bitrate: N/A
> > Stream #0:0: Audio: aac (LC), 48000 Hz, stereo, fltp, 128 kb/s
> > Stream #0:1: Video: h264 (Main), yuvj420p(pc, bt709, progressive),
> > 2560x1440, 25 fps, 25 tbr, 1k tbn
> > [vist#1:1/h264 @ 0x562c70ef3b00] WARNING: defaulting
> > hwaccel_output_format to qsv for compatibility with old commandlines.
> > This behaviour is DEPRECATED and will be removed in the future. Please
> > explicitly set "-hwaccel_output_format qsv".
> > Input #1, flv, from 'rtmp://localhost/stream2':
> > Metadata:
> > |RtmpSampleAccess: true
> > Server : NGINX RTMP (github.com/arut/nginx-rtmp-module)
> > displayWidth : 2688
> > displayHeight : 1520
> > fps : 26
> > profile :
> > level :
> > Duration: 00:00:00.00, start: 138268.361000, bitrate: N/A
> > Stream #1:0: Audio: aac (LC), 48000 Hz, stereo, fltp, 128 kb/s
> > Stream #1:1: Video: h264 (High), yuvj420p(pc, progressive), 2688x1520
> > [SAR 1:1 DAR 168:95], 26 fps, 26 tbr, 1k tbn
> > Stream mapping:
> > Stream #0:1 (h264_qsv) -> overlay_qsv
> > Stream #1:1 (h264_qsv) -> overlay_qsv
> > overlay_qsv:default -> Stream #0:0 (h264_qsv)
> > Press [q] to stop, [?] for help
> > Impossible to convert between the formats supported by the filter
> > 'graph 0 input from stream 1:1' and the filter 'auto_scale_0'
> > [fc#0 @ 0x562c70e30a40] Error reinitializing filters!
> > [fc#0 @ 0x562c70e30a40] Task finished with error code: -38 (Function
> > not implemented)
> > [fc#0 @ 0x562c70e30a40] Terminating thread with return code -38
> > (Function not implemented)
> > [vost#0:0/h264_qsv @ 0x562c70ede940] [enc:h264_qsv @ 0x562c70edec40]
> > Could not open encoder before EOF
> > [vost#0:0/h264_qsv @ 0x562c70ede940] Task finished with error code: -22
> > (Invalid argument)
> > [vost#0:0/h264_qsv @ 0x562c70ede940] Terminating thread with return
> > code -22 (Invalid argument)
> > [out#0/h264 @ 0x562c70e31240] Nothing was written into output file,
> > because at least one of its streams received no packets.
> > frame= 0 fps=0.0 q=0.0 Lsize= 0KiB time=N/A bitrate=N/A
> > speed=N/A
> > Conversion failed!
> >
> >
> > If I remove the filter_complex and use just one input stream, it works
> > without issue:
> >
> > ./ffmpeg \
> > -hwaccel qsv -c:v h264_qsv -i rtmp://localhost/stream1 \
> > -c:v h264_qsv -global_quality 25 -g 50 -r 25 -an test.h264
> >
> > Could anyone please tell me what I'm doing wrong?
> >
> > Thank you very much.
> >
> > --
> > Kind regards,
> > Vladimir
> > _______________________________________________
> > ffmpeg-user mailing list
> > ffmpeg-user at ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-user
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-user-request at ffmpeg.org with subject "unsubscribe".
> _______________________________________________
> ffmpeg-user mailing list
> ffmpeg-user at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
> To unsubscribe, visit link above, or email
> ffmpeg-user-request at ffmpeg.org with subject "unsubscribe".
>
More information about the ffmpeg-user
mailing list