[Libav-user] Using VAAPI to decode to RGB colors and to DRM PRIME fd

[M. Stoeckl]

Hello,

I have the following problem: there is a program outside of my control
which uses and produces DRM PRIME fds only with a specific format (for
example, DRM_FORMAT_XRGB8888/AV_PIX_FMT_BGR0), and I'd like to decode
an input video format (such as H264, or VP9) in YUV colorspace and have
its frames be written to the DRM PRIME fds, without copying the entire
frame back to the CPU to do a color format conversion.

With the h264_vaapi encoder, I can use av_hwframe_map with
AV_PIX_FMT_DRM_PRIME frames (with format DRM_FORMAT_XRGB8888) to produce
AV_PIX_FMT_VAAPI frames with the software format AV_PIX_FMT_NV12, that
the encoder then uses successfully. [Unless I am mistaken, this calls
vaCreateSurfaces using the DRM frame data, and the VAAPI driver does
the format conversion work.]

However, I'm not sure how best to decode with VAAPI; a similar approach
trying av_hwframe_map before the decoder started produced
AVERROR(ENOSYS). When I configure AVVAAPIFramesContext to have the
frames be produced from an existing DRM PRIME file descriptor, the 
decoder runs, but the original file descriptor contents aren't updated.
Alternatively, converting the frames produced by the decoder, with
av_hwframe_map, to DRM PRIME frames gives me data in NV12 format, which
isn't what I want.

Has anyone managed to get something similar working, or even any
GPU-only decoding pipeline that can decode YUV video into RGB images,
handling any conversions on-GPU?

Any additional advice (or pointers to relevant example code not already
in FFMPEG) would be appreciated.