[FFmpeg-devel] [PATCH 1/2] avutil/video_enc_params: Combine overflow checks

[Andreas Rheinhardt]

Andreas Rheinhardt:
> This patch also fixes a -Wtautological-constant-out-of-range-compare
> warning from Clang and a -Wtype-limits warning from GCC on systems
> where size_t is 64bits and unsigned 32bits. The reason for this seems
> to be that variable (whose value derives from sizeof() and can therefore
> be known at compile-time) is used instead of using sizeof() directly in
> the comparison.
> 
> Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt at gmail.com>
> ---
> Using (UINT_MAX > SIZE_MAX / sizeof(AVVideoBlockParams) && nb_blocks >
> SIZE_MAX / sizeof(AVVideoBlockParams)) would have fixed the warning with
> Clang, but not GCC.
> 
>  libavutil/video_enc_params.c | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
> 
> diff --git a/libavutil/video_enc_params.c b/libavutil/video_enc_params.c
> index c46c0f1dc6..b9cdafddbb 100644
> --- a/libavutil/video_enc_params.c
> +++ b/libavutil/video_enc_params.c
> @@ -33,8 +33,7 @@ AVVideoEncParams *av_video_enc_params_alloc(enum AVVideoEncParamsType type,
>      size_t size;
>  
>      size = sizeof(*par);
> -    if (nb_blocks > SIZE_MAX / sizeof(AVVideoBlockParams) ||
> -        nb_blocks * sizeof(AVVideoBlockParams) > SIZE_MAX - size)
> +    if (nb_blocks > (SIZE_MAX - size) / sizeof(AVVideoBlockParams))
>          return NULL;
>      size += sizeof(AVVideoBlockParams) * nb_blocks;
>  
> 
Will apply this patchset tomorrow unless there are objections.

- Andreas