[FFmpeg-devel] [PATCH] avcodec/cuviddec: fix AV1 decoding error

[Zhao Zhili]

From: Zhao Zhili <zhilizhao at tencent.com>

cuvidParseVideoData only supports pure OBUs, it report unknown
error with AV1CodecConfigurationRecord. Check whether extradata
is AV1CodecConfigurationRecord and skip the first 4 bytes to fix
the issue.

The bug is revealed in ffmpeg cmd since 45e3b6a68 and ffd1316e.
---
 libavcodec/cuviddec.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/libavcodec/cuviddec.c b/libavcodec/cuviddec.c
index cb3cda7e24..fc9c0515fe 100644
--- a/libavcodec/cuviddec.c
+++ b/libavcodec/cuviddec.c
@@ -953,6 +953,16 @@ static av_cold int cuvid_decode_init(AVCodecContext *avctx)
         extradata_size = avctx->extradata_size;
     }
 
+    // Check first bit to determine whether it's AV1CodecConfigurationRecord.
+    // Skip first 4 bytes of AV1CodecConfigurationRecord to keep configOBUs
+    // only, otherwise cuvidParseVideoData report unknown error.
+    if (avctx->codec->id == AV_CODEC_ID_AV1 &&
+            extradata_size > 4 &&
+            extradata && extradata[0] & 0x80) {
+        extradata += 4;
+        extradata_size -= 4;
+    }
+
     ctx->cuparse_ext = av_mallocz(sizeof(*ctx->cuparse_ext)
             + FFMAX(extradata_size - (int)sizeof(ctx->cuparse_ext->raw_seqhdr_data), 0));
     if (!ctx->cuparse_ext) {
-- 
2.34.1