[FFmpeg-cvslog] avcodec/aacsbr_fixed: Fix overflows in rounding in sbr_hf_assemble()

[Michael Niedermayer]

ffmpeg | branch: release/3.3 | Michael Niedermayer <michael at niedermayer.cc> | Wed Jan 31 18:13:07 2018 +0100| [a97335b1b3aec9fb449ddf1a8b33cdaf418bf1a2] | committer: Michael Niedermayer

avcodec/aacsbr_fixed: Fix overflows in rounding in sbr_hf_assemble()

Fixes: runtime error: signed integer overflow: 2052929346 + 204817098 cannot be represented in type 'int'
Fixes: 5275/clusterfuzz-testcase-minimized-5367635958038528

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
(cherry picked from commit b1bef755f617af9685b592d866b3eb7f3c4b02b1)
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=a97335b1b3aec9fb449ddf1a8b33cdaf418bf1a2
---

 libavcodec/aacsbr_fixed.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/libavcodec/aacsbr_fixed.c b/libavcodec/aacsbr_fixed.c
index 2a679491b0..0622d2ff7c 100644
--- a/libavcodec/aacsbr_fixed.c
+++ b/libavcodec/aacsbr_fixed.c
@@ -572,7 +572,8 @@ static void sbr_hf_assemble(int Y1[38][64][2],
                 int A = (1-((indexsine+(kx & 1))&2));
                 int B = (A^(-idx)) + idx;
                 int *out = &Y1[i][kx][idx];
-                int shift, round;
+                int shift;
+                unsigned round;
 
                 SoftFloat *in  = sbr->s_m[e];
                 for (m = 0; m+1 < m_max; m+=2) {
@@ -585,12 +586,12 @@ static void sbr_hf_assemble(int Y1[38][64][2],
                     }
                     if (shift < 32) {
                         round = 1 << (shift-1);
-                        out[2*m  ] += (in[m  ].mant * A + round) >> shift;
+                        out[2*m  ] += (int)(in[m  ].mant * A + round) >> shift;
                     }
 
                     if (shift2 < 32) {
                         round = 1 << (shift2-1);
-                        out[2*m+2] += (in[m+1].mant * B + round) >> shift2;
+                        out[2*m+2] += (int)(in[m+1].mant * B + round) >> shift2;
                     }
                 }
                 if(m_max&1)
@@ -601,7 +602,7 @@ static void sbr_hf_assemble(int Y1[38][64][2],
                         return;
                     } else if (shift < 32) {
                         round = 1 << (shift-1);
-                        out[2*m  ] += (in[m  ].mant * A + round) >> shift;
+                        out[2*m  ] += (int)(in[m  ].mant * A + round) >> shift;
                     }
                 }
             }