[MPlayer-dev-eng] possible bugs in vf_decimate filter

[Trent Piepho]

The decimate filter calculates 8x8 SADs over the image.  The loop that
calls the SAD function increments x and y by 4 each time, rather than 8.
This means all the pixels, except for the outer four, are included in four
SAD calculations instead of one.

It's not clear if this is intentional or not, the code being 100%
comment-free.

The filter also skips the leftmost 8 columns.  Nothing is skipped on the
top, bottom, or right side, and this behavior is not documented.

The calculation for the number of blocks that need to exceed the 'lo' value
is wrong too.  It's using (w/16)*(h/16)*frac, but, if the code is adjusted
to not do overlapping SADs, it should be (w/8)*(h/8)*frac.  If the code is
supposed to do SADs that overlap by four pixels, then it would be ((w-4)/4)
* ((h-4)/4) * frac.

Patch attached that removes the skipping of the left 8 columns, stops doing
overlapping SADs, and calculates the number of 'lo' blocks correctly.

BTW, the code doesn't cope with mpi->x or mpi->y != 0 either, but neither
do most filters.
-------------- next part --------------
Index: libmpcodecs/vf_decimate.c
===================================================================
--- libmpcodecs/vf_decimate.c	(revision 20132)
+++ libmpcodecs/vf_decimate.c	(working copy)
@@ -80,9 +80,9 @@
 {
 	int x, y;
 	int d, c=0;
-	int t = (w/16)*(h/16)*frac;
-	for (y = 0; y < h-7; y += 4) {
-		for (x = 8; x < w-7; x += 4) {
+	int t = (w/8)*(h/8)*frac;
+	for (y = 0; y < h-7; y += 8) {
+		for (x = 0; x < w-7; x += 8) {
 			d = diff(old+x+y*os, new+x+y*ns, os, ns);
 			if (d > hi) return 0;
 			if (d > lo) {