[MPlayer-dev-eng] [PATCH 2/7] af_scale*: af_calc_insize_constrained_overflow_quickfix

[Robert Juliano]

af_calc_insize_constrained_overflow_quickfix
In-Reply-To: <loom.20070611T054310-358 at post.gmane.org>
References: <loom.20070611T054310-358 at post.gmane.org>
Follow-Up: 43703
X-Follow-Up: 43703
Message-ID: <20070611004646.nomail at ent.52.off>
Date: Mon, 11 Jun 2007 00:46:46 -0400

The later patches have floating point ratios of input to
output which causes af.c:af_calc_insize_constrained to integer
overflow causing dropped data in libmpcodecs/dec_audio.c:decode_audio.
See my post from 5/30 "What's up with frac_t af->mul?"  This
is a quickfix to make these overflows less annoying and isn't
necessarily part of larger patch series.

Files:
  libaf/af.c

Index: mplayer-HEAD/libaf/af.c
===================================================================
--- mplayer-HEAD.orig/libaf/af.c	2007-06-10 16:33:13.000000000 -0400
+++ mplayer-HEAD/libaf/af.c	2007-06-10 16:34:07.000000000 -0400
@@ -582,13 +582,13 @@
   if(!mul.n || !mul.d) 
     return -1;
 
-  in = t * (((len/t) * mul.d - 1)/mul.n);
+  in = t * (((len/t) * (int64_t)mul.d - 1)/mul.n);
   
   if(in>max_insize) in=t*(max_insize/t);
 
   // Try to meet constraint nr 3. 
-  while((out=t * (((in/t+1)*mul.n - 1)/mul.d)) <= max_outsize && in<=max_insize){
-    if( (t * (((in/t)*mul.n))/mul.d) >= len) return in;
+  while((out=t * (((in/t+1)*(int64_t)mul.n - 1)/mul.d)) <= max_outsize && in<=max_insize){
+    if( (t * (((in/t)*(int64_t)mul.n))/mul.d) >= len) return in;
     in+=t;
   }
   
@@ -596,7 +596,7 @@
   while(out > max_outsize || in > max_insize){
     in-=t;
     if(in<t) return -1; // Input parameters are probably incorrect
-    out = t * (((in/t)*mul.n + 1)/mul.d);
+    out = t * (((in/t)*(int64_t)mul.n + 1)/mul.d);
   }
   return in;
 }