[FFmpeg-devel] [PATCH] Check malloc values in swscale.

[Michael Niedermayer]

On Wed, Aug 26, 2009 at 05:43:45PM -0300, Ramiro Polla wrote:
> On Sun, Aug 23, 2009 at 7:05 PM, Michael Niedermayer<michaelni at gmx.at> wrote:
> > On Sun, Aug 23, 2009 at 06:50:09PM -0300, Ramiro Polla wrote:
> [...]
> >> @@ -1737,7 +1738,7 @@
> >>
> >> ? ? ?// Note the +1 is for the MMX scaler which reads over the end
> >> ? ? ?/* align at 16 for AltiVec (needed by hScale_altivec_real) */
> >> - ? ?*outFilter= av_mallocz(*outFilterSize*(dstW+1)*sizeof(int16_t));
> >> + ? ?CHECKED_ALLOCZ(*outFilter, *outFilterSize*(dstW+1)*sizeof(int16_t));
> >>
> >> ? ? ?/* normalize & store in outFilter */
> >> ? ? ?for (i=0; i<dstW; i++) {
> >> @@ -1764,7 +1765,7 @@
> >> ? ? ?}
> >>
> >> ? ? ?ret=0;
> >> -error:
> >> +fail:
> >> ? ? ?av_free(filter);
> >> ? ? ?av_free(filter2);
> >> ? ? ?return ret;
> >
> > i doubt, this with the lack of checks of the return value wil work
> 
> Do you mean that the return value of initFilter() is not checked?
> Patch attached.

>  swscale.c |   20 ++++++++++++--------
>  1 file changed, 12 insertions(+), 8 deletions(-)
> 03f034dc778772549431ead6bb9bd67aa41ecab0  initFilter_check_return.diff
> --- swscale.c.orig	2009-08-26 17:37:15.000000000 -0300
> +++ swscale.c	2009-08-26 17:38:10.000000000 -0300
> @@ -2812,14 +2812,16 @@
>              (flags & SWS_CPU_CAPS_ALTIVEC) ? 8 :
>              1;
>  
> -        initFilter(&c->hLumFilter, &c->hLumFilterPos, &c->hLumFilterSize, c->lumXInc,
> +        if (initFilter(&c->hLumFilter, &c->hLumFilterPos, &c->hLumFilterSize, c->lumXInc,
>                     srcW      ,       dstW, filterAlign, 1<<14,
>                     (flags&SWS_BICUBLIN) ? (flags|SWS_BICUBIC)  : flags,
> -                   srcFilter->lumH, dstFilter->lumH, c->param);
> -        initFilter(&c->hChrFilter, &c->hChrFilterPos, &c->hChrFilterSize, c->chrXInc,
> +                   srcFilter->lumH, dstFilter->lumH, c->param) == -1)

i prefer <0 checks especially when the return value might be ENOMEM

besides this, its ok if tested (manually forced malloc fail under valgrind)

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

In fact, the RIAA has been known to suggest that students drop out
of college or go to community college in order to be able to afford
settlements. -- The RIAA
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20090827/3201d9ea/attachment.pgp>