[FFmpeg-cvslog] avcodec/vp9block: fix runtime error: signed integer overflow: 196675 * 20670 cannot be represented in type 'int'

[Michael Niedermayer]

ffmpeg | branch: release/3.3 | Michael Niedermayer <michael at niedermayer.cc> | Sun May 21 02:12:21 2017 +0200| [d11c686204b40921152fa6fd56e6b4b171ea39c5] | committer: Michael Niedermayer

avcodec/vp9block: fix runtime error: signed integer overflow: 196675 * 20670 cannot be represented in type 'int'

Fixes: 1710/clusterfuzz-testcase-minimized-4837032931098624

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Reviewed-by: "Ronald S. Bultje" <rsbultje at gmail.com>
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
(cherry picked from commit d4ee76780869c659a5d3b0815c56024ab260a81d)
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=d11c686204b40921152fa6fd56e6b4b171ea39c5
---

 libavcodec/vp9block.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/libavcodec/vp9block.c b/libavcodec/vp9block.c
index ae2f0e4c6f..a16ccdccdb 100644
--- a/libavcodec/vp9block.c
+++ b/libavcodec/vp9block.c
@@ -915,9 +915,9 @@ skip_eob:
         if (!--band_left)
             band_left = band_counts[++band];
         if (is_tx32x32)
-            STORE_COEF(coef, rc, ((vp8_rac_get(c) ? -val : val) * qmul[!!i]) / 2);
+            STORE_COEF(coef, rc, (int)((vp8_rac_get(c) ? -val : val) * (unsigned)qmul[!!i]) / 2);
         else
-            STORE_COEF(coef, rc, (vp8_rac_get(c) ? -val : val) * qmul[!!i]);
+            STORE_COEF(coef, rc, (vp8_rac_get(c) ? -val : val) * (unsigned)qmul[!!i]);
         nnz = (1 + cache[nb[i][0]] + cache[nb[i][1]]) >> 1;
         tp = p[band][nnz];
     } while (++i < n_coeffs);