[FFmpeg-devel] [PATCH] ac3enc: fix 16-bit signed overflow in mdct

[Justin Ruggles]

On 12/20/2010 07:47 PM, Michael Niedermayer wrote:

> On Mon, Dec 20, 2010 at 11:40:51PM +0000, Mans Rullgard wrote:
> 
> Welcome back! :)
> 
> 
>> One of the cos table values is -32768 and must not be negated.
>> ---
>>  libavcodec/ac3enc.c |    2 +-
>>  1 files changed, 1 insertions(+), 1 deletions(-)
>>
>> diff --git a/libavcodec/ac3enc.c b/libavcodec/ac3enc.c
>> index d372e6a..1191d16 100644
>> --- a/libavcodec/ac3enc.c
>> +++ b/libavcodec/ac3enc.c
>> @@ -405,7 +405,7 @@ static void mdct512(AC3MDCTContext *mdct, int32_t *out, int16_t *in)
>>      for (i = 0; i < n4; i++) {
>>          re =  ((int)rot[   2*i] - (int)rot[ n-1-2*i]) >> 1;
>>          im = -((int)rot[n2+2*i] - (int)rot[n2-1-2*i]) >> 1;
>> -        CMUL(x[i].re, x[i].im, re, im, -mdct->xcos1[i], mdct->xsin1[i]);
>> +        CMUL(x[i].re, x[i].im, -re, -im, mdct->xcos1[i], -mdct->xsin1[i]);
> 
> this contains more negates than before and i suspect the amount can be reduced
> if it is slower


I slightly prefer to change the table generation back to clipping -32767
to 32767.  Anyone against?

-Justin