[MPlayer-dev-eng] [patch] invalid reads in vf_spp & vf_fspp

[Jindrich Makovicka]

On 11/1/06, Michael Niedermayer <michaelni at gmx.at> wrote:
> Hi
>
> On Wed, Nov 01, 2006 at 01:52:19PM +0100, Jindrich Makovicka wrote:
> > Hi,
> >
> > fspp & fspp currently allocate and copy 16 times larger buffer for
> > quantizers, which can cause invalid read accesses.
> >
> > Trivial fix (divide image height with the height of a macroblock) attached.
> > --
> > Jindrich Makovicka
>
> > Index: libmpcodecs/vf_fspp.c
> > ===================================================================
> > --- libmpcodecs/vf_fspp.c     (revision 20577)
> > +++ libmpcodecs/vf_fspp.c     (working copy)
> > @@ -531,8 +531,8 @@
> >      vf->priv->mpeg2= mpi->qscale_type;
> >      if(mpi->pict_type != 3 && mpi->qscale && !vf->priv->qp){
> >       if(!vf->priv->non_b_qp)
> > -         vf->priv->non_b_qp= malloc(mpi->qstride * mpi->h);
> > -     memcpy(vf->priv->non_b_qp, mpi->qscale, mpi->qstride * mpi->h);
> > +         vf->priv->non_b_qp= malloc(mpi->qstride * (mpi->h >> 4));
> > +     memcpy(vf->priv->non_b_qp, mpi->qscale, mpi->qstride * (mpi->h >> 4));
>
> height doesnt need to be a mutiple of 16

ack... filter() would fail in this case.

Should I use (mpi->h + 15) >> 4 or mpi->height >> 4? The former is
consistent with the rest of the filter so I'd prefer that.

-- 
Jindrich Makovicka