[MPlayer-dev-eng] altivec patch 2/5: wrong array boundaries

[Alan Curry]

vYCoeffsBank and vCCoeffsBank are allocated and initialized using incorrect
sizes based on the image width instead of height. This patch corrects the
sizes.

A related point: the usage pattern of the v{Y,C}CoeffsBank arrays appears to
be very inefficient.

altivec_yuv2packedX() is called once for each row of the output image. Each
time, it builds the large vYCoeffsBank and vCCoeffsBank, containing at least
one altivec vector for each row of the output image. It then uses only a
small part of the *CoeffsBank arrays (corresponding to the single output row
being generated), and destroys them. Why?

It would be better, I think, to just build the vectors that you're actually
going to use. Or build them all just once and preserve them in the
SwsContext, since they are just expanded versions of the v{Lum,Chr}Filter
that is already in there.

I'll work on that later. This patch is just to prevent overruning the end of
the array when dstW>dstH and incomplete initialization when dstW<dstH.

-------------- next part --------------
diff -u postproc/yuv2rgb_altivec.c postproc/yuv2rgb_altivec.c
--- postproc/yuv2rgb_altivec.c	6 Feb 2006 21:44:19 -0000
+++ postproc/yuv2rgb_altivec.c	6 Feb 2006 22:00:23 -0000
@@ -789,17 +789,17 @@
 
   vector signed short *YCoeffs, *CCoeffs;
 
-  vYCoeffsBank = malloc (sizeof (vector signed short)*lumFilterSize*dstW);
-  vCCoeffsBank = malloc (sizeof (vector signed short)*chrFilterSize*dstW);
+  vYCoeffsBank = malloc (sizeof (vector signed short)*lumFilterSize*c->dstH);
+  vCCoeffsBank = malloc (sizeof (vector signed short)*chrFilterSize*c->dstH);
 
-  for (i=0;i<lumFilterSize*dstW;i++) {
+  for (i=0;i<lumFilterSize*c->dstH;i++) {
     tmp = c->vLumFilter[i];
     p = &vYCoeffsBank[i];
     for (j=0;j<8;j++)
       p[j] = tmp;
   }
 
-  for (i=0;i<chrFilterSize*dstW;i++) {
+  for (i=0;i<chrFilterSize*c->dstH;i++) {
     tmp = c->vChrFilter[i];
     p = &vCCoeffsBank[i];
     for (j=0;j<8;j++)