[FFmpeg-devel] [PATCH] ac3enc: fix 16-bit signed overflow in mdct

[Michael Niedermayer]

On Mon, Dec 20, 2010 at 11:40:51PM +0000, Mans Rullgard wrote:

Welcome back! :)


> One of the cos table values is -32768 and must not be negated.
> ---
>  libavcodec/ac3enc.c |    2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
> 
> diff --git a/libavcodec/ac3enc.c b/libavcodec/ac3enc.c
> index d372e6a..1191d16 100644
> --- a/libavcodec/ac3enc.c
> +++ b/libavcodec/ac3enc.c
> @@ -405,7 +405,7 @@ static void mdct512(AC3MDCTContext *mdct, int32_t *out, int16_t *in)
>      for (i = 0; i < n4; i++) {
>          re =  ((int)rot[   2*i] - (int)rot[ n-1-2*i]) >> 1;
>          im = -((int)rot[n2+2*i] - (int)rot[n2-1-2*i]) >> 1;
> -        CMUL(x[i].re, x[i].im, re, im, -mdct->xcos1[i], mdct->xsin1[i]);
> +        CMUL(x[i].re, x[i].im, -re, -im, mdct->xcos1[i], -mdct->xsin1[i]);

this contains more negates than before and i suspect the amount can be reduced
if it is slower

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Opposition brings concord. Out of discord comes the fairest harmony.
-- Heraclitus
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20101221/2b3dda1e/attachment.pgp>