[FFmpeg-devel] [PATCH] avcodec/aacdec_fixed: Fix integer overflow

[Nedeljko Babic]

>>                      for(j = 0; j<samples; j++){
>> -                        che->ch[0].ret[j] = (int32_t)av_clipl_int32((int64_t)che->ch[0].ret[j]<<7)+0x8000;
>> -                        che->ch[1].ret[j] = (int32_t)av_clipl_int32((int64_t)che->ch[1].ret[j]<<7)+0x8000;
>> +                        che->ch[0].ret[j] = (int32_t)((int64_t)av_clipl_int32((int64_t)che->ch[0].ret[j]<<7)+0x8000);
>> +                        che->ch[1].ret[j] = (int32_t)((int64_t)av_clipl_int32((int64_t)che->ch[1].ret[j]<<7)+0x8000);
>
>why does it overflow ?
>also all overflows during fate occur in the 2nd channel only
>are the affected case maybe all just using the first channel, or
>something like that ?
>it seems enough to skip the 2nd depending on type

You are correct and I should have look more deeply at this problem.
2nd channel has sense only for CPE type.
No work should be done on it for other types and they should be filtered out...
I am abandoning this patch and I will create new patch that correctly fixes this.

Thanks,
Nedeljko