[FFmpeg-cvslog] avcodec/flacdec: reduce limit for golomb so that the max value does not overflow

[Michael Niedermayer]

ffmpeg | branch: master | Michael Niedermayer <michael at niedermayer.cc> | Mon Feb 20 23:59:26 2017 +0100| [0a65dae9d0c01730323695fdc45eb1c1f5a978f7] | committer: Michael Niedermayer

avcodec/flacdec: reduce limit for golomb so that the max value does not overflow

Fixes: runtime error: left shift of 32 by 26 places cannot be represented in type 'int'

Fixes: 628/clusterfuzz-testcase-6187747641393152
Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/targets/ffmpeg
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=0a65dae9d0c01730323695fdc45eb1c1f5a978f7
---

 libavcodec/flacdec.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/libavcodec/flacdec.c b/libavcodec/flacdec.c
index 6ea86d4..f73a32b 100644
--- a/libavcodec/flacdec.c
+++ b/libavcodec/flacdec.c
@@ -258,8 +258,9 @@ static int decode_residuals(FLACContext *s, int32_t *decoded, int pred_order)
             for (; i < samples; i++)
                 *decoded++ = get_sbits_long(&s->gb, tmp);
         } else {
+            int real_limit = tmp ? (INT_MAX >> tmp) + 2 : INT_MAX;
             for (; i < samples; i++) {
-                int v = get_sr_golomb_flac(&s->gb, tmp, INT_MAX, 0);
+                int v = get_sr_golomb_flac(&s->gb, tmp, real_limit, 0);
                 if (v == 0x80000000){
                     av_log(s->avctx, AV_LOG_ERROR, "invalid residual\n");
                     return AVERROR_INVALIDDATA;