[FFmpeg-cvslog] avcodec/xsubdec: Check that RLE coded image and colors fit in the buffer

[Michael Niedermayer]

ffmpeg | branch: master | Michael Niedermayer <michael at niedermayer.cc> | Tue May 23 00:20:25 2017 +0200| [2002436b0c914e18eda4f45ef816b14501a1dbae] | committer: Michael Niedermayer

avcodec/xsubdec: Check that RLE coded image and colors fit in the buffer

Fixes: Timeout
Fixes: 1747/clusterfuzz-testcase-minimized-6035451213250560

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=2002436b0c914e18eda4f45ef816b14501a1dbae
---

 libavcodec/xsubdec.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/libavcodec/xsubdec.c b/libavcodec/xsubdec.c
index 2a766b33f7..93fd0f4d50 100644
--- a/libavcodec/xsubdec.c
+++ b/libavcodec/xsubdec.c
@@ -91,6 +91,9 @@ static int decode_frame(AVCodecContext *avctx, void *data, int *data_size,
     // we just ignore it
     bytestream_get_le16(&buf);
 
+    if (buf_end - buf < h + 3*4)
+        return AVERROR_INVALIDDATA;
+
     // allocate sub and set values
     sub->rects =  av_mallocz(sizeof(*sub->rects));
     if (!sub->rects)