[FFmpeg-devel] [PATCH] avcodec/mpeg4videodec: Clear bits_per_raw_sample if it has originated from a previous instance

[Michael Niedermayer]

Fixes: assertion failure
Fixes: ffmpeg_crash_5.avi

Found-by: Thuan Pham <thuanpv at comp.nus.edu.sg>, Marcel Böhme, Andrew Santosa and Alexandru Razvan Caciulescu with AFLSmart
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
---
 libavcodec/mpeg4videodec.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/libavcodec/mpeg4videodec.c b/libavcodec/mpeg4videodec.c
index 51b58fac02..d0ebaac6e8 100644
--- a/libavcodec/mpeg4videodec.c
+++ b/libavcodec/mpeg4videodec.c
@@ -3109,6 +3109,12 @@ int ff_mpeg4_decode_picture_header(Mpeg4DecContext *ctx, GetBitContext *gb)
     /* search next start code */
     align_get_bits(gb);
 
+    // If we have not switched to studio profile than we also did not switch bps
+    // that means something else (like a previous instance) outside set bps which
+    // would be inconsistant with the currect state, thus reset it
+    if (!s->studio_profile && s->avctx->bits_per_raw_sample != 8)
+        s->avctx->bits_per_raw_sample = 0;
+
     if (s->codec_tag == AV_RL32("WV1F") && show_bits(gb, 24) == 0x575630) {
         skip_bits(gb, 24);
         if (get_bits(gb, 8) == 0xF0)
-- 
2.17.1