[FFmpeg-devel] [PATCH] VP3: check that init_vlc with user-supplied data is successful

[Reimar Döffinger]

On Mon, Jul 06, 2009 at 05:12:53AM +0200, Michael Niedermayer wrote:
> On Sun, Jul 05, 2009 at 12:00:29PM +0200, Reimar D?ffinger wrote:
> > sample is ogv/smclock.ogv.1.101.ogv from issue 1240.
> > Attached patch makes decode_init fail if the coded huffman tables are
> > invalid and thus init_vlc fails.
> > There remains one huge WTF: although decode_init returns -1 each and
> > every time, with this command-line
> > ./ffmpeg_g -i crashers/ogv/smclock.ogv.1.101.ogv -f framecrc -
> > FFmpeg will still call the decode function, obviously causing a crash
> > since the decoder was never correctly initialized.
> > stack trace:
> > ==26703== Invalid write of size 1
> > ==26703==    at 0x4A09F98: memset (mc_replace_strmem.c:471)
> > ==26703==    by 0x6B9947: vp3_decode_frame (string3.h:85)
> > ==26703==    by 0x493EEC: avcodec_decode_video2 (utils.c:577)
> > ==26703==    by 0x407226: output_packet (ffmpeg.c:1314)
> > ==26703==    by 0x40A3F3: av_encode (ffmpeg.c:2281)
> > ==26703==    by 0x40ACFB: main (ffmpeg.c:3997)
> > ==26703==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
> 
> maybe caused by combination of try_decode_frame() and codec being non null
> even on failure ...

No, it was due to theora_decode_init stupidly ignoring the return value
of vp3_decode_init.

> >  vp3.c |   29 +++++++++++++++++++----------
> >  1 file changed, 19 insertions(+), 10 deletions(-)
> > 3a53df8f442531ecbd4d4f3ff82118b5a1974698  vp3vlc_check.diff
> 
> looks ok

Applied btw.