[FFmpeg-user] overlay_qsv with hardware decode and encode
Vladimir Mishonov
me at player701.net
Thu Feb 27 19:25:20 EET 2025
So, I've managed to figure out that this may be related to input streams
having different resolutions.
In the meantime, I've also switched to VA-API because it seems more
stable on my system, and uses less CPU. (No idea why it uses CPU at all,
shouldn't it use GPU? The gputop utility does confirm the latter to be
the case, at the very least.)
In any way. The following command-line will work if both inputs have the
same resolution:
ffmpeg \
-hwaccel vaapi -hwaccel_output_format vaapi -i rtmp://localhost/stream1
\
-hwaccel vaapi -hwaccel_output_format vaapi -i rtmp://localhost/stream2
\
-filter_complex "[0:v][1:v]overlay_vaapi=w=500" \
-c:v h264_vaapi -global_quality 25 -g 50 -r 25 -an -f null -
However, it fails with the same error as before ("Impossible to
convert..") if resolutions do not match.
Running with -loglevel verbose reveals some details:
[graph 0 input from stream 0:1 @ 0x7f3fc4005b40] w:2560 h:1440
pixfmt:vaapi tb:1/1000 fr:25/1 sar:0/1 csp:bt709 range:pc
[graph 0 input from stream 1:1 @ 0x7f3fc4006600] w:2688 h:1520
pixfmt:vaapi tb:1/1000 fr:26/1 sar:1/1 csp:unknown range:pc
[auto_scale_0 @ 0x7f3fc40082c0] w:iw h:ih flags:'' interl:0
[Parsed_overlay_vaapi_0 @ 0x7f3fc4005500] auto-inserting filter
'auto_scale_0' between the filter 'graph 0 input from stream 1:1' and
the filter 'Parsed_overlay_vaapi_0'
Impossible to convert between the formats supported by the filter 'graph
0 input from stream 1:1' and the filter 'auto_scale_0'
[fc#0 @ 0x56490d7a1ec0] Error reinitializing filters!
[fc#0 @ 0x56490d7a1ec0] Task finished with error code: -38 (Function not
implemented)
[fc#0 @ 0x56490d7a1ec0] Terminating thread with return code -38
(Function not implemented)
Note this part: "...auto-inserting filter 'auto_scale_0' between the
filter 'graph 0 input from stream 0:1' and the filter
'Parsed_overlay_vaapi_0". This is obviously not going to work because
those are hardware frames, not software!
I've found an option called -noauto_conversion_filters, but it does not
appear to do any better:
ffmpeg -loglevel verbose -noauto_conversion_filters \
-hwaccel vaapi -hwaccel_output_format vaapi -i rtmp://localhost/stream1
\
-hwaccel vaapi -hwaccel_output_format vaapi -i rtmp://localhost/stream2
\
-filter_complex "[0:v][1:v]overlay_vaapi=w=500" \
-c:v h264_vaapi -global_quality 25 -g 50 -r 25 -an test.h264
It says: "The filters 'graph 0 input from stream 1:1' and
'Parsed_overlay_vaapi_0' do not have a common format and automatic
conversion is disabled."
Is this a bug that I should report?? It certainly does look like one to
me...
---
Kind regards,
Vladimir
On 2025-02-21 10:12, Vladimir Mishonov via ffmpeg-user wrote:
> Greetings everyone.
>
> I am attempting to overlay 2 video streams in real-time with QuickSync
> hardware acceleration. I want to use both hardware decode and encode to
> make the most use out of the GPU and avoid excessive CPU usage.
>
> I've found an example command-line here:
> https://github.com/intel/vaapi-fits/issues/173 and adapted it:
>
> ./ffmpeg \
> -hwaccel qsv -c:v h264_qsv -i rtmp://localhost/stream1 \
> -hwaccel qsv -c:v h264_qsv -i rtmp://localhost/stream2 \
> -filter_complex "[0:v][1:v]overlay_qsv=w=500" \
> -c:v h264_qsv -global_quality 25 -g 50 -r 25 -an test.h264
>
> However, this gives me the following error:
>
> ffmpeg version N-118510-gbc1a3bfd2c-20250220 Copyright (c) 2000-2025
> the FFmpeg developers
> built with gcc 14.2.0 (crosstool-NG 1.26.0.120_4d36f27)
> configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static
> --pkg-config=pkg-config --cross-prefix=x86_64-ffbuild-linux-gnu-
> --arch=x86_64 --target-os=linux --enable-gpl --enable-version3
> --disable-debug --enable-iconv --enable-zlib --enable-libfreetype
> --enable-libfribidi --enable-gmp --enable-libxml2 --enable-openssl
> --enable-lzma --enable-fontconfig --enable-libharfbuzz
> --enable-libvorbis --enable-opencl --enable-libpulse --enable-libvmaf
> --enable-libxcb --enable-xlib --enable-amf --enable-libaom
> --enable-libaribb24 --enable-avisynth --enable-chromaprint
> --enable-libdav1d --enable-libdavs2 --enable-libdvdread
> --enable-libdvdnav --disable-libfdk-aac --enable-ffnvcodec
> --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libkvazaar
> --enable-libaribcaption --enable-libass --enable-libbluray
> --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librist
> --enable-libssh --enable-libtheora --enable-libvpx --enable-libwebp
> --enable-libzmq --enable-lv2 --enable-libvpl --enable-openal
> --enable-libopencore-amrnb --enable-libopencore-amrwb
> --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt
> --enable-librav1e --enable-librubberband --disable-schannel
> --enable-sdl2 --enable-libsnappy --enable-libsoxr --enable-libsrt
> --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d
> --enable-libdrm --enable-vaapi --enable-libvidstab --enable-vulkan
> --enable-libshaderc --enable-libplacebo --disable-libvvenc
> --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid
> --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC
> --extra-cxxflags= --extra-libs='-ldl -lgomp' --extra-ldflags=-pthread
> --extra-ldexeflags=-pie --cc=x86_64-ffbuild-linux-gnu-gcc
> --cxx=x86_64-ffbuild-linux-gnu-g++ --ar=x86_64-ffbuild-linux-gnu-gcc-ar
> --ranlib=x86_64-ffbuild-linux-gnu-gcc-ranlib
> --nm=x86_64-ffbuild-linux-gnu-gcc-nm --extra-version=20250220
> libavutil 59. 57.100 / 59. 57.100
> libavcodec 61. 33.102 / 61. 33.102
> libavformat 61. 9.107 / 61. 9.107
> libavdevice 61. 4.100 / 61. 4.100
> libavfilter 10. 9.100 / 10. 9.100
> libswscale 8. 13.100 / 8. 13.100
> libswresample 5. 4.100 / 5. 4.100
> libpostproc 58. 4.100 / 58. 4.100
> [vist#0:1/h264 @ 0x562c70e48780] WARNING: defaulting
> hwaccel_output_format to qsv for compatibility with old commandlines.
> This behaviour is DEPRECATED and will be removed in the future. Please
> explicitly set "-hwaccel_output_format qsv".
> Input #0, flv, from 'rtmp://localhost/stream1':
> Metadata:
> |RtmpSampleAccess: true
> Server : NGINX RTMP (github.com/arut/nginx-rtmp-module)
> displayWidth : 2560
> displayHeight : 1440
> fps : 25
> profile :
> level :
> Duration: 00:00:00.00, start: 138266.736000, bitrate: N/A
> Stream #0:0: Audio: aac (LC), 48000 Hz, stereo, fltp, 128 kb/s
> Stream #0:1: Video: h264 (Main), yuvj420p(pc, bt709, progressive),
> 2560x1440, 25 fps, 25 tbr, 1k tbn
> [vist#1:1/h264 @ 0x562c70ef3b00] WARNING: defaulting
> hwaccel_output_format to qsv for compatibility with old commandlines.
> This behaviour is DEPRECATED and will be removed in the future. Please
> explicitly set "-hwaccel_output_format qsv".
> Input #1, flv, from 'rtmp://localhost/stream2':
> Metadata:
> |RtmpSampleAccess: true
> Server : NGINX RTMP (github.com/arut/nginx-rtmp-module)
> displayWidth : 2688
> displayHeight : 1520
> fps : 26
> profile :
> level :
> Duration: 00:00:00.00, start: 138268.361000, bitrate: N/A
> Stream #1:0: Audio: aac (LC), 48000 Hz, stereo, fltp, 128 kb/s
> Stream #1:1: Video: h264 (High), yuvj420p(pc, progressive), 2688x1520
> [SAR 1:1 DAR 168:95], 26 fps, 26 tbr, 1k tbn
> Stream mapping:
> Stream #0:1 (h264_qsv) -> overlay_qsv
> Stream #1:1 (h264_qsv) -> overlay_qsv
> overlay_qsv:default -> Stream #0:0 (h264_qsv)
> Press [q] to stop, [?] for help
> Impossible to convert between the formats supported by the filter
> 'graph 0 input from stream 1:1' and the filter 'auto_scale_0'
> [fc#0 @ 0x562c70e30a40] Error reinitializing filters!
> [fc#0 @ 0x562c70e30a40] Task finished with error code: -38 (Function
> not implemented)
> [fc#0 @ 0x562c70e30a40] Terminating thread with return code -38
> (Function not implemented)
> [vost#0:0/h264_qsv @ 0x562c70ede940] [enc:h264_qsv @ 0x562c70edec40]
> Could not open encoder before EOF
> [vost#0:0/h264_qsv @ 0x562c70ede940] Task finished with error code: -22
> (Invalid argument)
> [vost#0:0/h264_qsv @ 0x562c70ede940] Terminating thread with return
> code -22 (Invalid argument)
> [out#0/h264 @ 0x562c70e31240] Nothing was written into output file,
> because at least one of its streams received no packets.
> frame= 0 fps=0.0 q=0.0 Lsize= 0KiB time=N/A bitrate=N/A
> speed=N/A
> Conversion failed!
>
>
> If I remove the filter_complex and use just one input stream, it works
> without issue:
>
> ./ffmpeg \
> -hwaccel qsv -c:v h264_qsv -i rtmp://localhost/stream1 \
> -c:v h264_qsv -global_quality 25 -g 50 -r 25 -an test.h264
>
> Could anyone please tell me what I'm doing wrong?
>
> Thank you very much.
>
> --
> Kind regards,
> Vladimir
> _______________________________________________
> ffmpeg-user mailing list
> ffmpeg-user at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
> To unsubscribe, visit link above, or email
> ffmpeg-user-request at ffmpeg.org with subject "unsubscribe".
More information about the ffmpeg-user
mailing list