[FFmpeg-devel] [PATCH] check for failed extradata malloc in avidec

[Reimar Döffinger]

On Wed, Jul 01, 2009 at 11:27:02PM +0200, Michael Niedermayer wrote:
> On Wed, Jul 01, 2009 at 08:16:56PM +0200, Reimar D?ffinger wrote:
> > Hello,
> > this fixes one of the files in issue1240.
> > Not sure if this is the best solution (the extradata size limit of 2^30 seems
> > rather extreme) but anyway...
> > Index: libavformat/avidec.c
> > ===================================================================
> > --- libavformat/avidec.c	(revision 19317)
> > +++ libavformat/avidec.c	(working copy)
> > @@ -478,6 +478,10 @@
> >                      if(size > 10*4 && size<(1<<30)){
> >                          st->codec->extradata_size= size - 10*4;
> >                          st->codec->extradata= av_malloc(st->codec->extradata_size + FF_INPUT_BUFFER_PADDING_SIZE);
> > +                        if (!st->codec->extradata) {
> > +                            st->codec->extradata_size= 0;
> > +                            return AVERROR(ENOMEM);
> > +                        }
> >                          get_buffer(pb, st->codec->extradata, st->codec->extradata_size);
> >                      }
> 
> hmm, id treat malloc failure like a too large extradata

I was thinking about it, but the reason I don't like it is that when
someone slowly runs out of RAM, things will just start failing weirdly
(due to missing extradata) but never give an AVERROR(ENOMEM).
I don't like it when things like available RAM change behaviour in such
a "creepy" way.