[FFmpeg-devel] [PATCH] lavu/hwcontext_vaapi: fix a race mapping to allow CSC to YUV420P (PR #20158)

[nyanmisaka]

PR #20158 opened by nyanmisaka
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/20158
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/20158.patch

lavu/hwcontext_vaapi: fix a race mapping to allow CSC to YUV420P

There's a race condition for YUV420P when mapping from pix_fmt
to VA fourcc, both I420 and YV12 could be found by pix_fmt.

Currently, vaapi_get_image_format() iterates over the query results
of pix_fmt and returns the first matching result in the order
declared in the driver. This may result in an incorrect image_format.

Now use fourcc to find the image_format.

Fixes:
ffmpeg -hwaccel vaapi -hwaccel_output_format vaapi -i INPUT \
-vf scale_vaapi=format=yuv420p,hwmap,format=yuv420p \
-vframes 1 -f rawvideo -y yuv420p.yuv

Signed-off-by: nyanmisaka <nst799610810 at gmail.com>

This was discovered because some NVR software's object recognition requires fully-planar YUV420P instead of NV12.


From 4b69c844b03da78cd12205e7f8865cc8f16c1a67 Mon Sep 17 00:00:00 2001
From: nyanmisaka <nst799610810 at gmail.com>
Date: Thu, 7 Aug 2025 22:32:20 +0800
Subject: [PATCH 1/2] lavu/hwcontext_vaapi: fix a race mapping to allow CSC to
 YUV420P

There's a race condition for YUV420P when mapping from pix_fmt
to VA fourcc, both I420 and YV12 could be found by pix_fmt.

Currently, vaapi_get_image_format() iterates over the query results
of pix_fmt and returns the first matching result in the order
declared in the driver. This may result in an incorrect image_format.

Now use fourcc to find the image_format.

Fixes:
ffmpeg -hwaccel vaapi -hwaccel_output_format vaapi -i INPUT \
-vf scale_vaapi=format=yuv420p,hwmap,format=yuv420p \
-vframes 1 -f rawvideo -y yuv420p.yuv

Signed-off-by: nyanmisaka <nst799610810 at gmail.com>
---
 libavutil/hwcontext_vaapi.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/libavutil/hwcontext_vaapi.c b/libavutil/hwcontext_vaapi.c
index 95aa38d9d2..7c243ec1ef 100644
--- a/libavutil/hwcontext_vaapi.c
+++ b/libavutil/hwcontext_vaapi.c
@@ -72,6 +72,7 @@ typedef struct VAAPIDevicePriv {
 typedef struct VAAPISurfaceFormat {
     enum AVPixelFormat pix_fmt;
     VAImageFormat image_format;
+    unsigned int fourcc;
 } VAAPISurfaceFormat;
 
 typedef struct VAAPIDeviceContext {
@@ -218,15 +219,21 @@ static int vaapi_get_image_format(AVHWDeviceContext *hwdev,
                                   VAImageFormat **image_format)
 {
     VAAPIDeviceContext *ctx = hwdev->hwctx;
+    const VAAPIFormatDescriptor *desc;
     int i;
 
+    desc = vaapi_format_from_pix_fmt(pix_fmt);
+    if (!desc || !image_format)
+        goto fail;
+
     for (i = 0; i < ctx->nb_formats; i++) {
-        if (ctx->formats[i].pix_fmt == pix_fmt) {
-            if (image_format)
-                *image_format = &ctx->formats[i].image_format;
+        if (ctx->formats[i].fourcc == desc->fourcc) {
+            *image_format = &ctx->formats[i].image_format;
             return 0;
         }
     }
+
+fail:
     return AVERROR(ENOSYS);
 }
 
@@ -435,6 +442,7 @@ static int vaapi_device_init(AVHWDeviceContext *hwdev)
             av_log(hwdev, AV_LOG_DEBUG, "Format %#x -> %s.\n",
                    fourcc, av_get_pix_fmt_name(pix_fmt));
             ctx->formats[ctx->nb_formats].pix_fmt      = pix_fmt;
+            ctx->formats[ctx->nb_formats].fourcc       = fourcc;
             ctx->formats[ctx->nb_formats].image_format = image_list[i];
             ++ctx->nb_formats;
         }
-- 
2.49.1


From fcceafc72c1c6116011a1a34911a5b9cb1026de5 Mon Sep 17 00:00:00 2001
From: nyanmisaka <nst799610810 at gmail.com>
Date: Thu, 7 Aug 2025 22:48:04 +0800
Subject: [PATCH 2/2] lavu/hwcontext_vaapi: drop a redundant check

vaapi_get_image_format() will be called in vaapi_map_frame().

So it's a double check, drop it to avoid redundancy.

Signed-off-by: nyanmisaka <nst799610810 at gmail.com>
---
 libavutil/hwcontext_vaapi.c | 6 ------
 1 file changed, 6 deletions(-)

diff --git a/libavutil/hwcontext_vaapi.c b/libavutil/hwcontext_vaapi.c
index 7c243ec1ef..51af22a607 100644
--- a/libavutil/hwcontext_vaapi.c
+++ b/libavutil/hwcontext_vaapi.c
@@ -1019,12 +1019,6 @@ static int vaapi_map_to_memory(AVHWFramesContext *hwfc, AVFrame *dst,
 {
     int err;
 
-    if (dst->format != AV_PIX_FMT_NONE) {
-        err = vaapi_get_image_format(hwfc->device_ctx, dst->format, NULL);
-        if (err < 0)
-            return err;
-    }
-
     err = vaapi_map_frame(hwfc, dst, src, flags);
     if (err)
         return err;
-- 
2.49.1