[FFmpeg-cvslog] avcodec/av1: only set the private context pix_fmt field if get_pixel_format() succeeds

[James Almer]

ffmpeg | branch: release/5.0 | James Almer <jamrial at gmail.com> | Tue Mar 22 15:35:19 2022 -0300| [fd4121a0aa1906f8cc653a0efc2c85c4a35235fe] | committer: James Almer

avcodec/av1: only set the private context pix_fmt field if get_pixel_format() succeeds

Otherwise get_pixel_format() will not be called when parsing a subsequent Sequence
Header in non hwaccel enabled scenarios, allowing frame parsing when it shouldn't.

This prevents the scenario seqhdr -> frame_hdr/redundant_frame_hdr -> seqhdr ->
redundant_frame_hdr from having the latter redundant frame header parsed as if it
was a frame header by the decoder because the former was discarded.
Since CBS did not discard it, the latter redundant frame header is output with a
zeroed AV1RawFrameHeader struct, which can have undesired results, like division
by zero with fields normally guaranteed to be anything else.

Fixes: division by zero
Fixes: 43769/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_AV1_fuzzer-5392562205097984
Fixes: 43950/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_AV1_fuzzer-5769210217758720

Reviewed-by: Michael Niedermayer <michael at niedermayer.cc>
Signed-off-by: James Almer <jamrial at gmail.com>
(cherry picked from commit 5670eddf8cd3907f9c0a9e626b5698d27c81c81b)

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=fd4121a0aa1906f8cc653a0efc2c85c4a35235fe
---

 libavcodec/av1dec.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/libavcodec/av1dec.c b/libavcodec/av1dec.c
index 09df2bf421..81f65ff2fb 100644
--- a/libavcodec/av1dec.c
+++ b/libavcodec/av1dec.c
@@ -499,9 +499,8 @@ static int get_pixel_format(AVCodecContext *avctx)
 
     if (pix_fmt == AV_PIX_FMT_NONE)
         return -1;
-    s->pix_fmt = pix_fmt;
 
-    switch (s->pix_fmt) {
+    switch (pix_fmt) {
     case AV_PIX_FMT_YUV420P:
 #if CONFIG_AV1_DXVA2_HWACCEL
         *fmtp++ = AV_PIX_FMT_DXVA2_VLD;
@@ -544,7 +543,7 @@ static int get_pixel_format(AVCodecContext *avctx)
         break;
     }
 
-    *fmtp++ = s->pix_fmt;
+    *fmtp++ = pix_fmt;
     *fmtp = AV_PIX_FMT_NONE;
 
     ret = ff_thread_get_format(avctx, pix_fmts);
@@ -562,6 +561,7 @@ static int get_pixel_format(AVCodecContext *avctx)
         return AVERROR(ENOSYS);
     }
 
+    s->pix_fmt = pix_fmt;
     avctx->pix_fmt = ret;
 
     return 0;