[Libav-user] Unable to convert frame format on GPU while decoding

[Nikita Vaňků]

Hello everyone,
I'm trying to achieve a simple use case, I want to use an HW decoder,
for as many codecs as possible, currently, on Linux, I'm trying
vaapi/cuda/vdpau, and on windows cuda/dxva2/direct3d to decode the
video frame in GPU and convert it before transferring it to CPU to
RGB0 or RGBA on GPU.
Ultimately, I want an RGB0/RGBA frame in system memory without using sws_scale.

I'm using FFmpeg 5.1.2 and reading through docs of
av_hwframe_transfer_data
https://ffmpeg.org/doxygen/5.1/hwcontext_8c.html#abf1b1664b8239d953ae2cac8b643815a
It says:
"the format of dst (if set) must use one of the formats returned by
av_hwframe_transfer_get_formats(src,
AV_HWFRAME_TRANSFER_DIRECTION_FROM)."
In my case, I'm using vaapi and this returns various formats like
nv12, yuv420/422, bgra, rgba, rgb0, etc. but when I actually do use
sw_frame->format and set it to any other than nv12 the function fails
and my dreams are crushed.

For a minimal reproduction example, I have used hw_decode.c example
and tweak it a little:
/* retrieve data from GPU to CPU */
enum AVPixelFormat* formats;
if (av_hwframe_transfer_get_formats(frame->hw_frames_ctx,
AV_HWFRAME_TRANSFER_DIRECTION_FROM, &formats, 0) == 0) {

    for (int i = 0; formats[i] != AV_PIX_FMT_NONE; i++) {
        if (formats[i] == AV_PIX_FMT_RGBA) {
            sw_frame->format = AV_PIX_FMT_RGBA;
            break;
        }
    }

    av_freep(&formats);
}

if ((ret = av_hwframe_transfer_data(sw_frame, frame, 0)) < 0) {
    fprintf(stderr, "Error transferring the data to system memory\n");
    goto fail;
}

When run on h264 video it outputs:
[AVHWFramesContext @ 0x113be00] Failed to read image from surface
0x14: 1 (operation failed).
Error transferring the data to system memory
[AVHWFramesContext @ 0x113be00] Failed to read image from surface
0x12: 1 (operation failed).
Error transferring the data to system memory

I have tried other formats, NV12 works fine, but I need RGBA or RGB0.

I have included complete source code and other related information
into this gist https://gist.github.com/Zaraka/e12d5f6ca1c15706613ba5f6539559c3

What am I doing wrong? Did I misunderstand some steps? If this is not
possible, can I use some other HW-accelerated filters that FFmpeg
provides to do the format conversion? Or do I have to implement my own
compute shaders?

Thank you!
Nikita