[FFmpeg-devel] [PATCH] avcodec/apedec: use int64_t for FFABS

[Ganesh Ajjanagadde]

res, absres are currently int's, which on most platforms is 32 bits.
Unfortunately, data is untrusted, and on line 1267 res is manipulated
with data. Thus, res can take on INT32_MIN/INT_MIN with crafted data,
making FFABS on line 1282 unsafe.

Once again, using FFNABS will make it less readable: logic is less
clear, diff is bigger, and there is scope for mistakes during the
expression rewrites.

Tested with FATE.

Signed-off-by: Ganesh Ajjanagadde <gajjanagadde at gmail.com>
---
 libavcodec/apedec.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/libavcodec/apedec.c b/libavcodec/apedec.c
index fcccfbe..e46558e 100644
--- a/libavcodec/apedec.c
+++ b/libavcodec/apedec.c
@@ -1254,8 +1254,8 @@ static void init_filter(APEContext *ctx, APEFilter *f, int16_t *buf, int order)
 static void do_apply_filter(APEContext *ctx, int version, APEFilter *f,
                             int32_t *data, int count, int order, int fracbits)
 {
-    int res;
-    int absres;
+    int64_t res;
+    int64_t absres;
 
     while (count--) {
         /* round fixedpoint scalar product */
-- 
2.6.1