[Libav-user] Vulkan accelerated filter to convert frame data to RGBA

[Böszörményi Zoltán]

Hi,

I have created a short program to exercise the FFmpeg 7.0+ API
using hardware accelerated decoding and pass the frame data
in shared memory to some consumer:

https://github.com/zboszor/rtsp-proxy

VAAPI works nicely with an RGBA target frame and decoding
uses a low 7-10% CPU time on an based Ice Lake machine
for 4K RTSP streams.

The problem occurs when using Vulkan accelerated decoding.
av_hwframe_transfer_data() doesn't work if the target AVFrame
is pre-allocated as RGBA, so sws_scale() has to be used.

Which in turn makes the ~3% CPU usage for pure Vulkan based
decoding (when the frame is not used, just decoding is exercised)
into ~30% CPU usage to get a usable frame.

How can I use libavfilter in a way to use Vulkan accelerated
conversion to RGBA? There is no clear example to do this (i.e.
combining accelerated decoding and filtering) in the FFmpeg
sources and more than a decade old Stackoverflow answers for
FFmpeg 3.x-4.x are not too helpful for recent versions.

Thanks in advance,
Zoltán Böszörményi