[FFmpeg-devel] [PATCH 3/4] avformat/rmdec: Use 64bit for intermediate for DEINT_ID_INT4

[James Almer]

On 4/15/2021 5:44 PM, Michael Niedermayer wrote:
> Fixes: runtime error: signed integer overflow: 65312 * 65535 cannot be represented in type 'int'
> Fixes: 32832/clusterfuzz-testcase-minimized-ffmpeg_dem_RM_fuzzer-4817710040088576
> 
> Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
> Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
> ---
>   libavformat/rmdec.c | 4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/libavformat/rmdec.c b/libavformat/rmdec.c
> index fc3bff4859..af032ed90a 100644
> --- a/libavformat/rmdec.c
> +++ b/libavformat/rmdec.c
> @@ -269,9 +269,9 @@ static int rm_read_audio_stream_info(AVFormatContext *s, AVIOContext *pb,
>           case DEINT_ID_INT4:
>               if (ast->coded_framesize > ast->audio_framesize ||
>                   sub_packet_h <= 1 ||
> -                ast->coded_framesize * sub_packet_h > (2 + (sub_packet_h & 1)) * ast->audio_framesize)
> +                ast->coded_framesize * (uint64_t)sub_packet_h > (2 + (sub_packet_h & 1)) * ast->audio_framesize)

This check seems superfluous with the one below right after it. 
ast->coded_framesize * sub_packet_h must be equal to 2 * 
ast->audio_framesize. It can be removed.

>                   return AVERROR_INVALIDDATA;
> -            if (ast->coded_framesize * sub_packet_h != 2*ast->audio_framesize) {
> +            if (ast->coded_framesize * (uint64_t)sub_packet_h != 2*ast->audio_framesize) {
>                   avpriv_request_sample(s, "mismatching interleaver parameters");
>                   return AVERROR_INVALIDDATA;
>               }

How about something like

> diff --git a/libavformat/rmdec.c b/libavformat/rmdec.c
> index fc3bff4859..09880ee3fe 100644
> --- a/libavformat/rmdec.c
> +++ b/libavformat/rmdec.c
> @@ -269,7 +269,7 @@ static int rm_read_audio_stream_info(AVFormatContext *s, AVIOContext *pb,
>          case DEINT_ID_INT4:
>              if (ast->coded_framesize > ast->audio_framesize ||
>                  sub_packet_h <= 1 ||
> -                ast->coded_framesize * sub_packet_h > (2 + (sub_packet_h & 1)) * ast->audio_framesize)
> +                ast->audio_framesize > INT_MAX / sub_packet_h)
>                  return AVERROR_INVALIDDATA;
>              if (ast->coded_framesize * sub_packet_h != 2*ast->audio_framesize) {
>                  avpriv_request_sample(s, "mismatching interleaver parameters");

Instead?

We already know that ast->coded_framesize is not bigger than 
ast->audio_framesize, and with this change we'll also know that 
ast->audio_framesize * sub_packet_h can't overflow, so neither will 
ast->coded_framesize * sub_packet_h.