[FFmpeg-devel] [PATCH] avcodec/jpeg2000dsp: Use unsigned to avoid overflow

[Tomas Härdin]

tis 2022-09-27 klockan 03:47 +0200 skrev Andreas Rheinhardt:
> Affected the jpeg2000dsp checkasm test.
> 
> Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt at outlook.com>
> ---
>  libavcodec/jpeg2000dsp.c | 9 ++++-----
>  1 file changed, 4 insertions(+), 5 deletions(-)
> 
> diff --git a/libavcodec/jpeg2000dsp.c b/libavcodec/jpeg2000dsp.c
> index b61be3b72f..b1bff6d5b1 100644
> --- a/libavcodec/jpeg2000dsp.c
> +++ b/libavcodec/jpeg2000dsp.c
> @@ -76,14 +76,13 @@ static void ict_int(void *_src0, void *_src1,
> void *_src2, int csize)
>  
>  static void rct_int(void *_src0, void *_src1, void *_src2, int
> csize)
>  {
> -    int32_t *src0 = _src0, *src1 = _src1, *src2 = _src2;
> -    int32_t i0, i1, i2;
> +    uint32_t *src0 = _src0, *src1 = _src1, *src2 = _src2;
>      int i;
>  
>      for (i = 0; i < csize; i++) {
> -        i1 = *src0 - (*src2 + *src1 >> 2);
> -        i0 = i1 + *src2;
> -        i2 = i1 + *src1;
> +        uint32_t i1 = *src0 - ((int32_t)(*src2 + *src1) >> 2);

The addition could conceivably overflow. Also could just use / 4
instead of >> 2.

> +        int32_t i0 = i1 + *src2;
> +        int32_t i2 = i1 + *src1;

These could also overflow. And agian, not in typical use obviously
because this is for lossless, but for malicious files possibly.

/Tomas