[FFmpeg-cvslog] dnxhdenc: fix signed overflows

[Mans Rullgard]

ffmpeg | branch: master | Mans Rullgard <mans at mansr.com> | Sat Nov 26 20:11:12 2011 +0000| [3a83b2461e4ce9d48ad6ab037eb14569d0e53506] | committer: Mans Rullgard

dnxhdenc: fix signed overflows

Signed-off-by: Mans Rullgard <mans at mansr.com>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=3a83b2461e4ce9d48ad6ab037eb14569d0e53506
---

 libavcodec/dnxhdenc.c |    5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/libavcodec/dnxhdenc.c b/libavcodec/dnxhdenc.c
index 89e5c08..d1ab597 100644
--- a/libavcodec/dnxhdenc.c
+++ b/libavcodec/dnxhdenc.c
@@ -682,7 +682,8 @@ static int dnxhd_encode_rdo(AVCodecContext *avctx, DNXHDEncContext *ctx)
                 int qscale = 1;
                 int mb = y*ctx->m.mb_width+x;
                 for (q = 1; q < avctx->qmax; q++) {
-                    unsigned score = ctx->mb_rc[q][mb].bits*lambda+(ctx->mb_rc[q][mb].ssd<<LAMBDA_FRAC_BITS);
+                    unsigned score = ctx->mb_rc[q][mb].bits*lambda+
+                        ((unsigned)ctx->mb_rc[q][mb].ssd<<LAMBDA_FRAC_BITS);
                     if (score < min) {
                         min = score;
                         qscale = q;
@@ -709,7 +710,7 @@ static int dnxhd_encode_rdo(AVCodecContext *avctx, DNXHDEncContext *ctx)
                 lambda = (lambda+last_higher)>>1;
             else
                 lambda -= down_step;
-            down_step *= 5; // XXX tune ?
+            down_step = FFMIN((int64_t)down_step*5, INT_MAX);
             up_step = 1<<LAMBDA_FRAC_BITS;
             lambda = FFMAX(1, lambda);
             if (lambda == last_lower)