[FFmpeg-cvslog] avutil/mathematics: Do not treat INT64_MIN as positive in av_rescale_rnd

[Michael Niedermayer]

ffmpeg | branch: release/2.8 | Michael Niedermayer <michael at niedermayer.cc> | Tue Dec  1 12:44:23 2015 +0100| [460710500e172042144784a3152aa8d1c5c29cf6] | committer: Michael Niedermayer

avutil/mathematics: Do not treat INT64_MIN as positive in av_rescale_rnd

The code expects actual positive numbers and gives completely wrong
results if INT64_MIN is treated as positive
Instead clip it into the valid range that is add 1 and treat it as
negative

Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
(cherry picked from commit 25e37f5ea92d4201976a59ae306ce848d257a7e6)

Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=460710500e172042144784a3152aa8d1c5c29cf6
---

 libavutil/mathematics.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/libavutil/mathematics.c b/libavutil/mathematics.c
index 126cffc..b1ffd65 100644
--- a/libavutil/mathematics.c
+++ b/libavutil/mathematics.c
@@ -76,8 +76,8 @@ int64_t av_rescale_rnd(int64_t a, int64_t b, int64_t c, enum AVRounding rnd)
         rnd -= AV_ROUND_PASS_MINMAX;
     }
 
-    if (a < 0 && a != INT64_MIN)
-        return -av_rescale_rnd(-a, b, c, rnd ^ ((rnd >> 1) & 1));
+    if (a < 0)
+        return -av_rescale_rnd(-FFMAX(a, -INT64_MAX), b, c, rnd ^ ((rnd >> 1) & 1));
 
     if (rnd == AV_ROUND_NEAR_INF)
         r = c / 2;