[FFmpeg-cvslog] swscale/utils: More carefully merge and clear coefficients outside the input

[Michael Niedermayer]

ffmpeg | branch: release/2.4 | Michael Niedermayer <michaelni at gmx.at> | Tue Feb 24 00:32:39 2015 +0100| [2dde6d5d367e853a085abb595cf6fbd26bda5fc8] | committer: Michael Niedermayer

swscale/utils: More carefully merge and clear coefficients outside the input

Fixes out of array read
Fixes: asan_heap-oob_35ca682_1474_cov_3230122439_aletrek_tga_16bit.mov

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni at gmx.at>
(cherry picked from commit 1895d414aaacece3b57d7bf19502305e9a064fae)

Signed-off-by: Michael Niedermayer <michaelni at gmx.at>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=2dde6d5d367e853a085abb595cf6fbd26bda5fc8
---

 libswscale/utils.c |   20 +++++++++++++++-----
 1 file changed, 15 insertions(+), 5 deletions(-)

diff --git a/libswscale/utils.c b/libswscale/utils.c
index 858c5c2..ff82dee 100644
--- a/libswscale/utils.c
+++ b/libswscale/utils.c
@@ -623,14 +623,24 @@ static av_cold int initFilter(int16_t **outFilter, int32_t **filterPos,
 
         if ((*filterPos)[i] + filterSize > srcW) {
             int shift = (*filterPos)[i] + FFMIN(filterSize - srcW, 0);
+            int64_t acc = 0;
 
-            // move filter coefficients right to compensate for filterPos
-            for (j = filterSize - 2; j >= 0; j--) {
-                int right = FFMIN(j + shift, filterSize - 1);
-                filter[i * filterSize + right] += filter[i * filterSize + j];
-                filter[i * filterSize + j]      = 0;
+            for (j = filterSize - 1; j >= 0; j--) {
+                if ((*filterPos)[i] + j >= srcW) {
+                    acc += filter[i * filterSize + j];
+                    filter[i * filterSize + j] = 0;
+                }
             }
+            for (j = filterSize - 1; j >= 0; j--) {
+                if (j < shift) {
+                    filter[i * filterSize + j] = 0;
+                } else {
+                    filter[i * filterSize + j] = filter[i * filterSize + j - shift];
+                }
+            }
+
             (*filterPos)[i]-= shift;
+            filter[i * filterSize + srcW - 1 - (*filterPos)[i]] += acc;
         }
     }