[FFmpeg-devel] [PATCH 1/2] lavc/hevc_mp4toannexb: Fix interger overflow

[Andriy Gelman]

From: Andriy Gelman <andriy.gelman at gmail.com>

Check packet grow size against INT_MAX instead of SIZE_MAX.

Found with libFuzzer:
4294967044 cannot be represented as int.

Signed-off-by: Andriy Gelman <andriy.gelman at gmail.com>
---
 libavcodec/hevc_mp4toannexb_bsf.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/libavcodec/hevc_mp4toannexb_bsf.c b/libavcodec/hevc_mp4toannexb_bsf.c
index 09bce5b34c2..36fd6f0b15c 100644
--- a/libavcodec/hevc_mp4toannexb_bsf.c
+++ b/libavcodec/hevc_mp4toannexb_bsf.c
@@ -152,8 +152,8 @@ static int hevc_mp4toannexb_filter(AVBSFContext *ctx, AVPacket *out)
         extra_size    = add_extradata * ctx->par_out->extradata_size;
         got_irap     |= is_irap;
 
-        if (SIZE_MAX - nalu_size < 4 ||
-            SIZE_MAX - 4 - nalu_size < extra_size) {
+        if (INT_MAX < 4 + nalu_size ||
+            INT_MAX - 4 < extra_size + nalu_size) {
             ret = AVERROR_INVALIDDATA;
             goto fail;
         }
-- 
2.24.0