[MPlayer-dev-eng] [PATCH] altivec fixes, part 3: array overrun/underrun

[Alan Curry]

The initialization loops for vYCoeffsBank and vCCoeffsBank used incorrect
boundaries based on the image width instead of height. This patch corrects
the loops.

A related point: the usage pattern of the v{Y,C}CoeffsBank arrays appears to
be very inefficient.

altivec_yuv2packedX() is called once for each row of the output image. Each
time, it builds the large vYCoeffsBank and vCCoeffsBank, containing at least
one altivec vector for each row of the output image. It then uses only a
small part of the *CoeffsBank arrays (corresponding to the single output row
being generated), and destroys them. Why?

It would be better, I think, to just build the vectors that you're actually
going to use. Or build them all just once and preserve them in the
SwsContext, since they are just expanded versions of the v{Lum,Chr}Filter
that is already in there.

I'll work on that later. This patch is just to prevent overruning the end of
the array when dstW>dstH and incomplete initialization when dstW<dstH.

diff -u postproc/yuv2rgb_altivec.c postproc/yuv2rgb_altivec.c
--- postproc/yuv2rgb_altivec.c	6 Feb 2006 02:15:27 -0000
+++ postproc/yuv2rgb_altivec.c	6 Feb 2006 02:57:51 -0000
@@ -795,14 +795,14 @@
   vYCoeffsBank = memalign (16, sizeof (vector signed short)*lumFilterSize*c->dstH);
   vCCoeffsBank = memalign (16, sizeof (vector signed short)*chrFilterSize*c->dstH);
 
-  for (i=0;i<lumFilterSize*dstW;i++) {
+  for (i=0;i<lumFilterSize*c->dstH;i++) {
     tmp = c->vLumFilter[i];
     p = &vYCoeffsBank[i];
     for (j=0;j<8;j++)
       p[j] = tmp;
   }
 
-  for (i=0;i<chrFilterSize*dstW;i++) {
+  for (i=0;i<chrFilterSize*c->dstH;i++) {
     tmp = c->vChrFilter[i];
     p = &vCCoeffsBank[i];
     for (j=0;j<8;j++)