[FFmpeg-cvslog] theora: check that pix fmt is valid, fix null ptr deref

[Michael Niedermayer]

ffmpeg | branch: master | Michael Niedermayer <michaelni at gmx.at> | Mon Nov 12 18:04:12 2012 +0100| [d1493d2ce5f598016adff8cda8484529a560fb0d] | committer: Michael Niedermayer

theora: check that pix fmt is valid, fix null ptr deref

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni at gmx.at>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=d1493d2ce5f598016adff8cda8484529a560fb0d
---

 libavcodec/vp3.c |    7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/libavcodec/vp3.c b/libavcodec/vp3.c
index 2105bb5..8a8c4ce 100644
--- a/libavcodec/vp3.c
+++ b/libavcodec/vp3.c
@@ -2175,6 +2175,10 @@ static int theora_decode_header(AVCodecContext *avctx, GetBitContext *gb)
     {
         skip_bits(gb, 5); /* keyframe frequency force */
         avctx->pix_fmt = theora_pix_fmts[get_bits(gb, 2)];
+        if (avctx->pix_fmt == AV_PIX_FMT_NONE) {
+            av_log(avctx, AV_LOG_ERROR, "Invalid pixel format\n");
+            return AVERROR_INVALIDDATA;
+        }
         skip_bits(gb, 3); /* reserved */
     }
 
@@ -2349,7 +2353,8 @@ static av_cold int theora_decode_init(AVCodecContext *avctx)
     switch(ptype)
     {
         case 0x80:
-            theora_decode_header(avctx, &gb);
+            if (theora_decode_header(avctx, &gb) < 0)
+                return -1;
                 break;
         case 0x81:
 // FIXME: is this needed? it breaks sometimes