[FFmpeg-cvslog] vmnc: Check for integer overflow

[Michael Niedermayer]

ffmpeg | branch: master | Michael Niedermayer <michaelni at gmx.at> | Sat Dec  1 22:09:14 2012 +0100| [aae478036223ae16c24b9890d087feda2efbe38a] | committer: Michael Niedermayer

vmnc: Check for integer overflow

Fixes null pointer dereference and potential out of array accesses.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni at gmx.at>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=aae478036223ae16c24b9890d087feda2efbe38a
---

 libavcodec/vmnc.c |    4 ++++
 1 file changed, 4 insertions(+)

diff --git a/libavcodec/vmnc.c b/libavcodec/vmnc.c
index d3c86f1..d465f30 100644
--- a/libavcodec/vmnc.c
+++ b/libavcodec/vmnc.c
@@ -345,6 +345,10 @@ static int decode_frame(AVCodecContext *avctx, void *data, int *data_size, AVPac
         size_left = buf_size - (src - buf);
         switch(enc) {
         case MAGIC_WMVd: // cursor
+            if (w*(int64_t)h*c->bpp2 > INT_MAX/2 - 2) {
+                av_log(avctx, AV_LOG_ERROR, "dimensions too large\n");
+                return AVERROR_INVALIDDATA;
+            }
             if(size_left < 2 + w * h * c->bpp2 * 2) {
                 av_log(avctx, AV_LOG_ERROR, "Premature end of data! (need %i got %i)\n", 2 + w * h * c->bpp2 * 2, size_left);
                 return -1;