[FFmpeg-cvslog] avcodec/ra144: Fix integer overflow in ff_eval_refl()

[Michael Niedermayer]

ffmpeg | branch: release/3.3 | Michael Niedermayer <michael at niedermayer.cc> | Thu Jun 21 23:08:32 2018 +0200| [eedde18f1a8f5cb32883d0bf529f400b82d80d1c] | committer: Michael Niedermayer

avcodec/ra144: Fix integer overflow in ff_eval_refl()

Fixes: signed integer overflow: -4096 * -524288 cannot be represented in type 'int'
Fixes: 8650/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_RA_144_fuzzer-5734816036159488

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
(cherry picked from commit b31189881a4cf54b0057ecf3eab917ad56eecfea)
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=eedde18f1a8f5cb32883d0bf529f400b82d80d1c
---

 libavcodec/ra144.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/libavcodec/ra144.c b/libavcodec/ra144.c
index c077b7b327..cf8127c236 100644
--- a/libavcodec/ra144.c
+++ b/libavcodec/ra144.c
@@ -1569,11 +1569,11 @@ int ff_eval_refl(int *refl, const int16_t *coefs, AVCodecContext *avctx)
         b = 0x1000000 / b;
         for (j=0; j <= i; j++) {
 #if CONFIG_FTRAPV
-            int a = bp2[j] - ((refl[i+1] * bp2[i-j]) >> 12);
+            int a = bp2[j] - ((int)(refl[i+1] * (unsigned)bp2[i-j]) >> 12);
             if((int)(a*(unsigned)b) != a*(int64_t)b)
                 return 1;
 #endif
-            bp1[j] = (int)((bp2[j] - ((refl[i+1] * bp2[i-j]) >> 12)) * (unsigned)b) >> 12;
+            bp1[j] = (int)((bp2[j] - ((int)(refl[i+1] * (unsigned)bp2[i-j]) >> 12)) * (unsigned)b) >> 12;
         }
 
         if ((unsigned) bp1[i] + 0x1000 > 0x1fff)