[FFmpeg-devel] [PATCH 2/3] avcodec/exr: Fix undefined integer multiplication

[James Almer]

On 9/13/2021 7:48 PM, Michael Niedermayer wrote:
> Fixes: signed integer overflow: 7020950083487072256 * 2 cannot be represented in type 'long long'
> Fixes: 37523/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_EXR_fuzzer-5133634955771904
> 
> Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
> Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
> ---
>   libavcodec/exr.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/libavcodec/exr.c b/libavcodec/exr.c
> index 67340c892da..c395b6098df 100644
> --- a/libavcodec/exr.c
> +++ b/libavcodec/exr.c
> @@ -1062,7 +1062,7 @@ static int dwa_uncompress(EXRContext *s, const uint8_t *src, int compressed_size
>       }
>   
>       {
> -        unsigned long dest_len = dc_count * 2LL;
> +        unsigned long dest_len = dc_count * 2ULL;

You could instead move this multiplication after the check below. If 
dc_count is equal to dc_w * dc_h * 3, multiplying it by 2 will never 
overflow an int64_t.

Also, you may want to do the same for ac_count earlier in this function. 
It's also an int64_t set with AV_RL64() and the multiplied by 2LL.

>           GetByteContext agb = gb;
>   
>           if (dc_count != dc_w * dc_h * 3)