[FFmpeg-cvslog] avformat/rpl: Use 64bit for total_audio_size and check it

[Michael Niedermayer]

ffmpeg | branch: release/4.4 | Michael Niedermayer <michael at niedermayer.cc> | Tue Mar 26 00:36:40 2024 +0100| [a5e39d0a27b307cfae3c5b8f5b69bb0b5b3bac4c] | committer: Michael Niedermayer

avformat/rpl: Use 64bit for total_audio_size and check it

Fixes: 62276/clusterfuzz-testcase-minimized-ffmpeg_dem_RPL_fuzzer-4677434693517312
Fixes: signed integer overflow: 5555555555555555556 * 8 cannot be represented in type 'long long'

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
(cherry picked from commit 878625812f164fbb733f442965235656d9eaccc8)
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=a5e39d0a27b307cfae3c5b8f5b69bb0b5b3bac4c
---

 libavformat/rpl.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/libavformat/rpl.c b/libavformat/rpl.c
index 36daa1cad0..ac82940b7a 100644
--- a/libavformat/rpl.c
+++ b/libavformat/rpl.c
@@ -117,7 +117,7 @@ static int rpl_read_header(AVFormatContext *s)
     AVIOContext *pb = s->pb;
     RPLContext *rpl = s->priv_data;
     AVStream *vst = NULL, *ast = NULL;
-    int total_audio_size;
+    int64_t total_audio_size;
     int error = 0;
     const char *endptr;
     char audio_type[RPL_LINE_LENGTH];
@@ -299,6 +299,8 @@ static int rpl_read_header(AVFormatContext *s)
         if (ast)
             av_add_index_entry(ast, offset + video_size, total_audio_size,
                                audio_size, audio_size * 8, 0);
+        if (total_audio_size/8 + (uint64_t)audio_size >= INT64_MAX/8)
+            return AVERROR_INVALIDDATA;
         total_audio_size += audio_size * 8;
     }