[FFmpeg-cvslog] avcodec/h264_parser: Fix undefined left shift

[Andreas Rheinhardt]

ffmpeg | branch: master | Andreas Rheinhardt <andreas.rheinhardt at outlook.com> | Tue May 26 20:26:28 2020 +0200| [c83a7dd25fdf9636e7e5da3fad5e0d82035b9c86] | committer: Andreas Rheinhardt

avcodec/h264_parser: Fix undefined left shift

Use an uint32_t for the NAL unit size of an AVC H.264 NAL unit instead
of an int as a left shift of a signed value is undefined behaviour
if the result doesn't fit into the target type.

Also make the log message never output negative lengths.

Fixes: left shift of 16711968 by 8 places cannot be represented in type 'int'
Fixes: 36601/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_H264_fuzzer-6581933285965824

Reviewed-by: Michael Niedermayer <michael at niedermayer.cc>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt at outlook.com>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=c83a7dd25fdf9636e7e5da3fad5e0d82035b9c86
---

 libavcodec/h264_parser.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/libavcodec/h264_parser.c b/libavcodec/h264_parser.c
index d3c56cc188..01ea016409 100644
--- a/libavcodec/h264_parser.c
+++ b/libavcodec/h264_parser.c
@@ -83,12 +83,13 @@ static int h264_find_frame_end(H264ParseContext *p, const uint8_t *buf,
 
     for (i = 0; i < buf_size; i++) {
         if (i >= next_avc) {
-            int nalsize = 0;
+            uint32_t nalsize = 0;
             i = next_avc;
             for (j = 0; j < p->nal_length_size; j++)
                 nalsize = (nalsize << 8) | buf[i++];
-            if (nalsize <= 0 || nalsize > buf_size - i) {
-                av_log(logctx, AV_LOG_ERROR, "AVC-parser: nal size %d remaining %d\n", nalsize, buf_size - i);
+            if (!nalsize || nalsize > buf_size - i) {
+                av_log(logctx, AV_LOG_ERROR, "AVC-parser: nal size %"PRIu32" "
+                       "remaining %d\n", nalsize, buf_size - i);
                 return buf_size;
             }
             next_avc = i + nalsize;