[FFmpeg-cvslog] avcodec/hqxdsp: Fix runtime error: signed integer overflow: -196264 * 11585 cannot be represented in type 'int'

[Michael Niedermayer]

ffmpeg | branch: master | Michael Niedermayer <michael at niedermayer.cc> | Sun May 14 14:06:56 2017 +0200| [b923213276777f33d6366b1cb9d1845a8658f365] | committer: Michael Niedermayer

avcodec/hqxdsp: Fix runtime error: signed integer overflow: -196264 * 11585 cannot be represented in type 'int'

Fixes: 1568/clusterfuzz-testcase-minimized-5944868608147456

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpeg
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=b923213276777f33d6366b1cb9d1845a8658f365
---

 libavcodec/hqxdsp.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/libavcodec/hqxdsp.c b/libavcodec/hqxdsp.c
index 04a65e7767..7f8044e463 100644
--- a/libavcodec/hqxdsp.c
+++ b/libavcodec/hqxdsp.c
@@ -47,8 +47,8 @@ static inline void idct_col(int16_t *blk, const uint8_t *quant)
     t5  = t1 * 2 + t3;
     t6  = t2 - t3;
     t7  = t3 * 2 + t6;
-    t8  = (t6 * 11585) >> 14;
-    t9  = (t7 * 11585) >> 14;
+    t8  = (int)(t6 * 11585U) >> 14;
+    t9  = (int)(t7 * 11585U) >> 14;
     tA  = (int)(s2 * 8867U - s6 * 21407U) >> 14;
     tB  = (int)(s6 * 8867U + s2 * 21407U) >> 14;
     tC  = (s0 >> 1) - (s4 >> 1);