[FFmpeg-cvslog] smackerdemuxer: check some values before instead of just after malloc()

[Michael Niedermayer]

ffmpeg | branch: release/0.7 | Michael Niedermayer <michaelni at gmx.at> | Fri Dec 16 02:57:22 2011 +0100| [127a2902533cfb6d55cf1960d5f184eb4f02211f] | committer: Michael Niedermayer

smackerdemuxer: check some values before instead of just after malloc()
Fixes Ticket777
Bug Found by: Diana Elena Muscalu

Signed-off-by: Michael Niedermayer <michaelni at gmx.at>
(cherry picked from commit c402c1c976dc5bd63908d1aaff5b60521cbbee92)

Signed-off-by: Michael Niedermayer <michaelni at gmx.at>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=127a2902533cfb6d55cf1960d5f184eb4f02211f
---

 libavformat/smacker.c |    6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/libavformat/smacker.c b/libavformat/smacker.c
index b2c442e..faf85a5 100644
--- a/libavformat/smacker.c
+++ b/libavformat/smacker.c
@@ -251,6 +251,8 @@ static int smacker_read_packet(AVFormatContext *s, AVPacket *pkt)
             memcpy(oldpal, pal, 768);
             size = avio_r8(s->pb);
             size = size * 4 - 1;
+            if(size + 1 > frame_size)
+                return AVERROR_INVALIDDATA;
             frame_size -= size;
             frame_size--;
             sz = 0;
@@ -292,10 +294,12 @@ static int smacker_read_packet(AVFormatContext *s, AVPacket *pkt)
         /* if audio chunks are present, put them to stack and retrieve later */
         for(i = 0; i < 7; i++) {
             if(flags & 1) {
-                int size;
+                unsigned int size;
                 uint8_t *tmpbuf;
 
                 size = avio_rl32(s->pb) - 4;
+                if(size + 4L > frame_size)
+                    return AVERROR_INVALIDDATA;
                 frame_size -= size;
                 frame_size -= 4;
                 smk->curstream++;