[FFmpeg-devel] [PATCH 1/2] avcodec/dvbsubenc: Use 64bits in nb_colors and num_rects checks

[Andreas Rheinhardt]

Michael Niedermayer:
> Fixes: integer overflow
> 
> No testcase
> 
> Found-by: 김승호 <kimsho98 at naver.com>
> Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
> ---
>  libavcodec/dvbsubenc.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/libavcodec/dvbsubenc.c b/libavcodec/dvbsubenc.c
> index 822e3a53099..53f0f6412fa 100644
> --- a/libavcodec/dvbsubenc.c
> +++ b/libavcodec/dvbsubenc.c
> @@ -302,7 +302,7 @@ static int dvbsub_encode(AVCodecContext *avctx, uint8_t *outbuf, int buf_size,
>  
>      /* page composition segment */
>  
> -    if (buf_size < 8 + h->num_rects * 6)
> +    if (buf_size < 8 + h->num_rects * 6LL)
>          return AVERROR_BUFFER_TOO_SMALL;
>      *q++ = 0x0f; /* sync_byte */
>      *q++ = 0x10; /* segment_type */
> @@ -326,7 +326,7 @@ static int dvbsub_encode(AVCodecContext *avctx, uint8_t *outbuf, int buf_size,
>  
>      if (h->num_rects) {
>          for (clut_id = 0; clut_id < h->num_rects; clut_id++) {
> -            if (buf_size < 6 + h->rects[clut_id]->nb_colors * 6)
> +            if (buf_size < 6 + h->rects[clut_id]->nb_colors * 6LL)
>                  return AVERROR_BUFFER_TOO_SMALL;
>  
>              /* CLUT segment */

I just sent different patches for this:
https://ffmpeg.org/pipermail/ffmpeg-devel/2025-April/342314.html
They are intended to make your patches obsolete.

- Andreas