[FFmpeg-devel] [PATCH v2] lavc/vp9: reset segmentation fields when segmentation isn't enabled

[llyyr.public at gmail.com]

From: llyyr <llyyr.public at gmail.com>

Fields under the segmentation switch are never reset on a new frame, and
retain the value from the previous frame. This bugs out a bunch of
hwaccel drivers when segmentation is disabled but update_map isn't
reset because they don't ignore values behind switches. This commit also
resets the temporal field, though it may not be required.

We also do this for vp8 [1] so this commit is just mirroring the vp8
logic.

This fixes an issue with certain samples [2] that causes blocky
artifacts with vaapi, d3d11va and cuda (and possibly others).
Mesa worked around [3] this by ignoring these fields if
segmentation.enabled is 0, but d3d11va still displays blocky artifacts.

[1] https://git.ffmpeg.org/gitweb/ffmpeg.git/blob/HEAD:/libavcodec/vp8.c#l811
[2] https://github.com/mpv-player/mpv/issues/13533
[3] https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27816

Signed-off-by: llyyr <llyyr.public at gmail.com>
---
 libavcodec/vp9.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/libavcodec/vp9.c b/libavcodec/vp9.c
index 6e2d18bf9595..8ede2e2eb358 100644
--- a/libavcodec/vp9.c
+++ b/libavcodec/vp9.c
@@ -709,6 +709,12 @@ static int decode_frame_header(AVCodecContext *avctx,
                 s->s.h.segmentation.feat[i].skip_enabled = get_bits1(&s->gb);
             }
         }
+    } else {
+        // Reset fields under segmentation switch if segmentation is disabled.
+        // This is necessary because some hwaccels don't ignore these fields
+        // if segmentation is disabled.
+        s->s.h.segmentation.temporal = 0;
+        s->s.h.segmentation.update_map = 0;
     }
 
     // set qmul[] based on Y/UV, AC/DC and segmentation Q idx deltas
-- 
2.45.1