[FFmpeg-cvslog] avcodec/mpegvideo_enc: Check for integer overflow in ff_mpv_reallocate_putbitbuffer()

[Michael Niedermayer]

ffmpeg | branch: release/2.6 | Michael Niedermayer <michael at niedermayer.cc> | Thu Jan 21 15:39:43 2016 +0100| [302a3269d6231dc276956b4e44e5e4b4bf7bc454] | committer: Michael Niedermayer

avcodec/mpegvideo_enc: Check for integer overflow in ff_mpv_reallocate_putbitbuffer()

Fixes assertion failure
Fixes: 6568d187979ce17878b6fe5fbbb89142/signal_sigabrt_7ffff6ae7cb7_7176_564bbc6741bdcf907f5c4e685c9a77a2.mpg

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
(cherry picked from commit b65efbc0f4195421c15d2a6c228d331eec5b31c3)

Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=302a3269d6231dc276956b4e44e5e4b4bf7bc454
---

 libavcodec/mpegvideo_enc.c |    5 +++++
 1 file changed, 5 insertions(+)

diff --git a/libavcodec/mpegvideo_enc.c b/libavcodec/mpegvideo_enc.c
index 2a00c84..d27d54e 100644
--- a/libavcodec/mpegvideo_enc.c
+++ b/libavcodec/mpegvideo_enc.c
@@ -2733,6 +2733,11 @@ int ff_mpv_reallocate_putbitbuffer(MpegEncContext *s, size_t threshold, size_t s
         uint8_t *new_buffer = NULL;
         int new_buffer_size = 0;
 
+        if ((s->avctx->internal->byte_buffer_size + size_increase) >= INT_MAX/8) {
+            av_log(s->avctx, AV_LOG_ERROR, "Cannot reallocate putbit buffer\n");
+            return AVERROR(ENOMEM);
+        }
+
         av_fast_padded_malloc(&new_buffer, &new_buffer_size,
                               s->avctx->internal->byte_buffer_size + size_increase);
         if (!new_buffer)