[FFmpeg-cvslog] avcodec/dnxhdenc: fix possible out of bound writes for big w/h

[Paul B Mahol]

ffmpeg | branch: master | Paul B Mahol <onemda at gmail.com> | Fri Mar  4 23:25:48 2022 +0100| [37480b1b85b0405563962b581dc2899b1b4bec59] | committer: Paul B Mahol

avcodec/dnxhdenc: fix possible out of bound writes for big w/h

It was caused by integer overflows.

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=37480b1b85b0405563962b581dc2899b1b4bec59
---

 libavcodec/dnxhdenc.c | 3 +++
 libavcodec/dnxhdenc.h | 2 +-
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/libavcodec/dnxhdenc.c b/libavcodec/dnxhdenc.c
index ac92474e56..374fec499b 100644
--- a/libavcodec/dnxhdenc.c
+++ b/libavcodec/dnxhdenc.c
@@ -1228,6 +1228,9 @@ static int dnxhd_encode_fast(AVCodecContext *avctx, DNXHDEncContext *ctx)
             ctx->mb_qscale[mb] = ctx->qscale + 1;
             ctx->mb_bits[mb]   = ctx->mb_rc[rc + ctx->m.mb_num].bits;
         }
+
+        if (max_bits > ctx->frame_bits)
+            return AVERROR(EINVAL);
     }
     return 0;
 }
diff --git a/libavcodec/dnxhdenc.h b/libavcodec/dnxhdenc.h
index 9e4c869bc4..30ae8c15e3 100644
--- a/libavcodec/dnxhdenc.h
+++ b/libavcodec/dnxhdenc.h
@@ -34,7 +34,7 @@
 #include "dnxhddata.h"
 
 typedef struct RCCMPEntry {
-    uint16_t mb;
+    uint32_t mb;
     int value;
 } RCCMPEntry;