[FFmpeg-user] overlay_qsv with hardware decode and encode
Vladimir Mishonov
me at player701.net
Sat Mar 1 09:35:37 EET 2025
On 2025-03-01 00:36, Vladimir Mishonov via ffmpeg-user wrote:
> On 2025-02-28 23:26, BloodMan wrote:
>> Hi Vladimir,
>>
>> I see on previous posts that resolutions are different, but also frame
>> rates are different and colour spaces (bt709 vs "") are different.
>>
>> Are you really sure the problem is the resolution?
>> (I have no way to check, I'm just thinking out)
>
> It is possible, but needs furher testing to say for sure.
BloodMan, it turns out you were right!
I've just tested with a spare RTSP IP camera that provides both a
primary and a backup stream. Both streams have the same characteristics
EXCEPT the resolution, of course.
And this command works:
ffmpeg -loglevel verbose \
-hwaccel vaapi -hwaccel_output_format vaapi -rtsp_transport tcp -i
rtsp://login:password@test.ip.camera/primary/stream \
-hwaccel vaapi -hwaccel_output_format vaapi -rtsp_transport tcp -i
rtsp://login:password@test.ip.camera/backup/stream \
-filter_complex "[0:v][1:v]overlay_vaapi=w=500" \
-c:v h264_vaapi -global_quality 25 -g 50 -r 25 -an test.mkv
The produced video file contains the expected output.
Of course I still need to figure out how to work around the case when
the streams are from different sources and their properties might not
match. If that is at all possible, of course.
Here is what MediaInfo says about the problematic streams (unimportant
bits omitted). The first one:
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.089
Default : No
Forced : No
Color range : Full
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709
And the other one:
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.013
Default : No
Forced : No
Color range : Full
Notice that only the first stream has "Color primaries", "Transfer
characteristics", and "Matrix coefficients"; for the second stream this
info is not present. Not sure what it means, maybe it's just missing
metadata and it could be injected somehow?
Also, FFmpeg log (warning: LONG):
ffmpeg version 7.1 Copyright (c) 2000-2024 the FFmpeg developers
built with gcc 14 (SUSE Linux)
configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64
--incdir=/usr/include/ffmpeg --extra-cflags='-O2 -Wall -U_FORTIFY_SOURCE
-D_FORTIFY_SOURCE=3 -fstack-protector-strong -funwind-tables
-fasynchronous-unwind-tables -fstack-clash-protection
-Werror=return-type -flto=auto -ffat-lto-objects -g' --optflags='-O2
-Wall -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -fstack-protector-strong
-funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection
-Werror=return-type -flto=auto -ffat-lto-objects -g' --disable-htmlpages
--enable-pic --disable-stripping --enable-shared --disable-static
--enable-gpl --enable-version3 --enable-libsmbclient --disable-openssl
--enable-gnutls --enable-ladspa --enable-libshaderc --enable-vulkan
--enable-libplacebo --enable-libaom --enable-libass --enable-libbluray
--enable-libbs2b --enable-libcdio --enable-libcodec2 --enable-libdav1d
--enable-libdc1394 --enable-libdrm --enable-libfontconfig
--enable-libfreetype --enable-libfribidi --enable-libgsm
--enable-libjack --enable-libjxl --enable-librist --enable-libmp3lame
--enable-libmysofa --enable-libopenjpeg --enable-libopenmpt
--enable-libopenh264-dlopen --enable-libopus --enable-libpulse
--enable-librav1e --enable-librubberband --enable-libsvtav1
--enable-libsoxr --enable-libspeex --enable-libssh --enable-libsrt
--enable-libtheora --enable-libtwolame --enable-libvidstab
--enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp
--enable-libxml2 --enable-libzimg --enable-libzmq --enable-libzvbi
--enable-lto --enable-lv2 --enable-libvpl --enable-vaapi --enable-vdpau
--enable-version3 --enable-libfdk-aac-dlopen --enable-nonfree
--enable-libopencore-amrnb --enable-libopencore-amrwb
--enable-libvo-amrwbenc --enable-libx264 --enable-libx265
--enable-libxvid
libavutil 59. 39.100 / 59. 39.100
libavcodec 61. 19.100 / 61. 19.100
libavformat 61. 7.100 / 61. 7.100
libavdevice 61. 3.100 / 61. 3.100
libavfilter 10. 4.100 / 10. 4.100
libswscale 8. 3.100 / 8. 3.100
libswresample 5. 3.100 / 5. 3.100
libpostproc 58. 3.100 / 58. 3.100
[tcp @ 0x55c953b39a40] Starting connection attempt to xx.xx.xx.xx port
554
[tcp @ 0x55c953b39a40] Successfully connected to xx.xx.xx.xx port 554
[rtsp @ 0x55c953b37140] SDP:
v=0
o=- 1740823419375162 1740823419375162 IN IP4 xx.xx.xx.xx
s=Media Presentation
e=NONE
b=AS:5100
t=0 0
a=control:rtsp://test.ip.camera:554/primary/stream
m=video 0 RTP/AVP 96
c=IN IP6 ::
b=AS:5000
a=recvonly
a=x-dimensions:1280,720
a=control:rtsp://test.ip.camera:554/primary/stream/trackID=1
a=rtpmap:96 H264/90000
a=fmtp:96 profile-level-id=420029; packetization-mode=1;
sprop-parameter-sets=Z01AII2NQCgC3/gLcBAQFAAAD6AAAw1DoYADz9AABMS0u8uNDAAefoAAJiWl3lwo,aO44gA==
m=audio 0 RTP/AVP 0
c=IN IP6 ::
b=AS:50
a=recvonly
a=control:rtsp://test.ip.camera:554/primary/stream/trackID=2
a=rtpmap:0 PCMU/8000
m=application 0 RTP/AVP 107
c=IN IP6 ::
b=AS:50
a=recvonly
a=control:rtsp://test.ip.camera:554/primary/stream/trackID=3
a=rtpmap:107 isapi.metadata/90000
a=Media_header:MEDIAINFO=494D4B48010300000400000110710110401F000000FA000000000000000000000000000000000000;
a=appversion:1.0
[rtsp @ 0x55c953b37140] setting jitter buffer size to 0
Last message repeated 2 times
[h264 @ 0x55c953b3aa00] Reinit context to 1280x720, pix_fmt: yuvj420p
Selecting decoder 'h264' because of requested hwaccel method vaapi
Input #0, rtsp, from
'rtsp://login:password@test.ip.camera/primary/stream':
Metadata:
title : Media Presentation
Duration: N/A, start: 0.000000, bitrate: N/A
Stream #0:0: Video: h264 (Main), 1 reference frame, yuvj420p(pc,
bt709, progressive, left), 1280x720 [SAR 1:1 DAR 16:9], 25 fps, 25 tbr,
90k tbn
Stream #0:1: Audio: pcm_mulaw, 8000 Hz, mono, s16, 64 kb/s
Stream #0:2: Data: none
[tcp @ 0x55c953b40d80] Starting connection attempt to xx.xx.xx.xx port
554
[tcp @ 0x55c953b40d80] Successfully connected to xx.xx.xx.xx port 554
[rtsp @ 0x55c953ed12c0] SDP:
v=0
o=- 1740823421379119 1740823421379119 IN IP4 xx.xx.xx.xx
s=Media Presentation
e=NONE
b=AS:5100
t=0 0
a=control:rtsp://test.ip.camera:554/backup/stream
m=video 0 RTP/AVP 96
c=IN IP6 ::
b=AS:5000
a=recvonly
a=x-dimensions:640,480
a=control:rtsp://test.ip.camera:554/backup/stream/trackID=1
a=rtpmap:96 H264/90000
a=fmtp:96 profile-level-id=420029; packetization-mode=1;
sprop-parameter-sets=Z01AHo2NQFAe//+AAgABtwEBAUAAAPoAADDUOhgB2UAAJiWrvLjQwA7KAAExLV3lwoA=,aO44gA==
m=audio 0 RTP/AVP 0
c=IN IP6 ::
b=AS:50
a=recvonly
a=control:rtsp://test.ip.camera:554/backup/stream/trackID=2
a=rtpmap:0 PCMU/8000
m=application 0 RTP/AVP 107
c=IN IP6 ::
b=AS:50
a=recvonly
a=control:rtsp://test.ip.camera:554/backup/stream/trackID=3
a=rtpmap:107 isapi.metadata/90000
a=Media_header:MEDIAINFO=494D4B48010300000400000110710110401F000000FA000000000000000000000000000000000000;
a=appversion:1.0
[rtsp @ 0x55c953ed12c0] setting jitter buffer size to 0
Last message repeated 2 times
[h264 @ 0x55c953b3e480] Reinit context to 640x480, pix_fmt: yuvj420p
Selecting decoder 'h264' because of requested hwaccel method vaapi
Input #1, rtsp, from
'rtsp://login:password@test.ip.camera/backup/stream':
Metadata:
title : Media Presentation
Duration: N/A, start: 0.000000, bitrate: N/A
Stream #1:0: Video: h264 (Main), 1 reference frame, yuvj420p(pc,
bt709, progressive, left), 640x480 [SAR 4:3 DAR 16:9], 25 fps, 24.92
tbr, 90k tbn
Stream #1:1: Audio: pcm_mulaw, 8000 Hz, mono, s16, 64 kb/s
Stream #1:2: Data: none
[out#0/matroska @ 0x55c953ef0340] Creating output stream from unlabeled
output of complex filtergraph 0. This overrides automatic video mapping.
[vost#0:0/h264_vaapi @ 0x55c953eca600] Created video stream from complex
filtergraph 0:[overlay_vaapi:default]
[vost#0:0/h264_vaapi @ 0x55c953eca600]
[out#0/matroska @ 0x55c953ef0340] No explicit maps, mapping streams
automatically...
[fc#0 @ 0x55c953b35380] Binding input with label '0:v' to input stream
0:0
[AVHWDeviceContext @ 0x55c953b63940] Trying to use DRM render node for
device 0.
[AVHWDeviceContext @ 0x55c953b63940] libva: VA-API version 1.22.0
[AVHWDeviceContext @ 0x55c953b63940] libva: Trying to open
/usr/lib64/dri/iHD_drv_video.so
[AVHWDeviceContext @ 0x55c953b63940] libva: Found init function
__vaDriverInit_1_22
[AVHWDeviceContext @ 0x55c953b63940] libva: va_openDriver() returns 0
[AVHWDeviceContext @ 0x55c953b63940] Initialised VAAPI connection:
version 1.22
[AVHWDeviceContext @ 0x55c953b63940] VAAPI driver: Intel iHD driver for
Intel(R) Gen Graphics - 25.1.1 ().
[AVHWDeviceContext @ 0x55c953b63940] Driver not found in known
nonstandard list, using standard behaviour.
[fc#0 @ 0x55c953b35380] Binding input with label '1:v' to input stream
1:0
Stream mapping:
Stream #0:0 (h264) -> overlay_vaapi
Stream #1:0 (h264) -> overlay_vaapi
overlay_vaapi:default -> Stream #0:0 (h264_vaapi)
[vost#0:0/h264_vaapi @ 0x55c953eca600] Starting thread...
[fc#0 @ 0x55c953b35380] Starting thread...
[vist#0:0/h264 @ 0x55c953ee5c00] [dec:h264 @ 0x55c953b3a040] Starting
thread...
[vist#1:0/h264 @ 0x55c953b6d0c0] [dec:h264 @ 0x55c954e6b480] Starting
thread...
[in#0/rtsp @ 0x55c953b36f40] Starting thread...
[in#1/rtsp @ 0x55c953ee2900] Starting thread...
Press [q] to stop, [?] for help
[h264 @ 0x55c953f73e40] Reinit context to 1280x720, pix_fmt: vaapi
[h264 @ 0x55c954f3e040] Reinit context to 640x480, pix_fmt: vaapi
[graph 0 input from stream 0:0 @ 0x7f5548005980] w:1280 h:720
pixfmt:vaapi tb:1/90000 fr:25/1 sar:1/1 csp:bt709 range:pc
[graph 0 input from stream 1:0 @ 0x7f5548006440] w:640 h:480
pixfmt:vaapi tb:1/90000 fr:299/12 sar:4/3 csp:bt709 range:pc
[Parsed_overlay_vaapi_0 @ 0x7f5548005340] [framesync @ 0x7f55480054d0]
Sync level 2
[graph 0 input from stream 0:0 @ 0x7f5548005980] video frame properties
congruent with link at pts_time: 0
[graph 0 input from stream 1:0 @ 0x7f5548006440] video frame properties
congruent with link at pts_time: 0
[h264_vaapi @ 0x55c953ed8140] Using input frames context (format vaapi)
with h264_vaapi encoder.
[h264_vaapi @ 0x55c953ed8140] Input surface format is nv12.
[h264_vaapi @ 0x55c953ed8140] Using VAAPI profile VAProfileH264High (7).
[h264_vaapi @ 0x55c953ed8140] Using VAAPI entrypoint
VAEntrypointEncSlice (6).
[h264_vaapi @ 0x55c953ed8140] Using VAAPI render target format YUV420
(0x1).
[h264_vaapi @ 0x55c953ed8140] RC mode: ICQ.
[h264_vaapi @ 0x55c953ed8140] Block Level bitrate control: OFF.
[h264_vaapi @ 0x55c953ed8140] RC quality: 25.
[h264_vaapi @ 0x55c953ed8140] RC framerate: 25/1 (25.00 fps).
[h264_vaapi @ 0x55c953ed8140] Driver does not report any additional
prediction constraints.
[h264_vaapi @ 0x55c953ed8140] Using intra, P- and B-frames (supported
references: 8 / 2).
[h264_vaapi @ 0x55c953ed8140] All wanted packed headers available
(wanted 0xd, found 0x1f).
[h264_vaapi @ 0x55c953ed8140] Using level 3.1.
Output #0, matroska, to 'test.mkv':
Metadata:
title : Media Presentation
encoder : Lavf61.7.100
Stream #0:0: Video: h264 (High), 1 reference frame (H264 /
0x34363248), vaapi(pc, bt709, progressive, left), 1280x720 [SAR 1:1 DAR
16:9], q=2-31, 25 fps, 1k tbn
Metadata:
encoder : Lavc61.19.100 h264_vaapi
[out#0/matroska @ 0x55c953ef0340] Starting thread...
frame= 22 fps=0.0 q=-0.0 size= 1KiB time=00:00:00.84 bitrate=
6.0kbits/s speed=1.68x
frame= 35 fps= 35 q=-0.0 size= 1KiB time=00:00:01.36 bitrate=
3.7kbits/s speed=1.36x
frame= 47 fps= 31 q=-0.0 size= 1KiB time=00:00:01.84 bitrate=
2.8kbits/s speed=1.23x
frame= 60 fps= 30 q=-0.0 size= 256KiB time=00:00:02.36 bitrate=
888.6kbits/s speed=1.18x
...
--
Kind regards,
Vladimir
More information about the ffmpeg-user
mailing list