[FFmpeg-cvslog] avcodec/mpegvideo_enc: Fix potential overflow in RD

[Michael Niedermayer]

ffmpeg | branch: release/7.0 | Michael Niedermayer <michael at niedermayer.cc> | Sun May 12 00:13:58 2024 +0200| [435b74c6a5d4e457d9526aa21e7a6e68926bf52c] | committer: Michael Niedermayer

avcodec/mpegvideo_enc: Fix potential overflow in RD

Fixes: CID1500285 Unintentional integer overflow

Sponsored-by: Sovereign Tech Fund
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
(cherry picked from commit b6b2b01025e016ce29e5add57305384a663edcfc)
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=435b74c6a5d4e457d9526aa21e7a6e68926bf52c
---

 libavcodec/mpegvideo_enc.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/libavcodec/mpegvideo_enc.c b/libavcodec/mpegvideo_enc.c
index 9d048e3dec..c4c174a02e 100644
--- a/libavcodec/mpegvideo_enc.c
+++ b/libavcodec/mpegvideo_enc.c
@@ -1433,7 +1433,7 @@ static int estimate_best_b_count(MpegEncContext *s)
                 goto fail;
             }
 
-            rd += (out_size * lambda2) >> (FF_LAMBDA_SHIFT - 3);
+            rd += (out_size * (uint64_t)lambda2) >> (FF_LAMBDA_SHIFT - 3);
         }
 
         /* get the delayed frames */
@@ -1442,7 +1442,7 @@ static int estimate_best_b_count(MpegEncContext *s)
             ret = out_size;
             goto fail;
         }
-        rd += (out_size * lambda2) >> (FF_LAMBDA_SHIFT - 3);
+        rd += (out_size * (uint64_t)lambda2) >> (FF_LAMBDA_SHIFT - 3);
 
         rd += c->error[0] + c->error[1] + c->error[2];