[FFmpeg-cvslog] avutil/fixed_dsp: Fix integer overflows in butterflies_fixed_c()

[Michael Niedermayer]

ffmpeg | branch: master | Michael Niedermayer <michael at niedermayer.cc> | Thu Jul 16 22:58:13 2020 +0200| [4a02ae49c26395fc3ae2d38c733a2a13bd3080e7] | committer: Michael Niedermayer

avutil/fixed_dsp: Fix integer overflows in butterflies_fixed_c()

Fixes: signed integer overflow: 0 - -2147483648 cannot be represented in type 'int'
Fixes: 23646/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_AAC_FIXED_fuzzer-5480991098667008

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=4a02ae49c26395fc3ae2d38c733a2a13bd3080e7
---

 libavutil/fixed_dsp.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/libavutil/fixed_dsp.c b/libavutil/fixed_dsp.c
index 8c018581df..f1b195f184 100644
--- a/libavutil/fixed_dsp.c
+++ b/libavutil/fixed_dsp.c
@@ -134,9 +134,10 @@ static int scalarproduct_fixed_c(const int *v1, const int *v2, int len)
     return (int)(p >> 31);
 }
 
-static void butterflies_fixed_c(int *v1, int *v2, int len)
+static void butterflies_fixed_c(int *v1s, int *v2, int len)
 {
     int i;
+    unsigned int *v1 = v1s;
 
     for (i = 0; i < len; i++){
         int t = v1[i] - v2[i];