[FFmpeg-cvslog] avcodec/ralf: Fix integer overflows with the filter coefficient in decode_channel()

[Michael Niedermayer]

ffmpeg | branch: master | Michael Niedermayer <michael at niedermayer.cc> | Tue Nov  5 22:27:04 2019 +0100| [721624c2f67545989626ba4413f7b8dbd7dff678] | committer: Michael Niedermayer

avcodec/ralf: Fix integer overflows with the filter coefficient in decode_channel()

Fixes: signed integer overflow: 1145975808 - -1146173210 cannot be represented in type 'int'
Fixes: 18616/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_RALF_fuzzer-5121296757424128

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=721624c2f67545989626ba4413f7b8dbd7dff678
---

 libavcodec/ralf.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/libavcodec/ralf.c b/libavcodec/ralf.c
index ca8817aa21..d8f1803086 100644
--- a/libavcodec/ralf.c
+++ b/libavcodec/ralf.c
@@ -264,8 +264,8 @@ static int decode_channel(RALFContext *ctx, GetBitContext *gb, int ch,
             t = get_vlc2(gb, vlc[cmode].table, vlc[cmode].bits, 2);
             t = extend_code(gb, t, 21, add_bits);
             if (!cmode)
-                coeff -= 12 << add_bits;
-            coeff = t - coeff;
+                coeff -= 12U << add_bits;
+            coeff = (unsigned)t - coeff;
             ctx->filter[i] = coeff;
 
             cmode = coeff >> add_bits;