[FFmpeg-cvslog] avformat/hlsenc: Fix extradata length check

[Andreas Rheinhardt]

ffmpeg | branch: master | Andreas Rheinhardt <andreas.rheinhardt at gmail.com> | Tue Oct 20 12:16:27 2020 +0200| [96ad55df5bfa594defa2d57970686df3106a9ffa] | committer: Andreas Rheinhardt

avformat/hlsenc: Fix extradata length check

Commit a2b1dd0ce301450a47c972745a6b33c4c273aa5d added support for
parsing annex B HEVC extradata to extract profile and level information.
Yet it only checks for there to be enough data left for the startcode
and the first byte of the NAL unit header and not for the full NAL unit
header; it simply presumes the second byte of the NAL unit header to be
present and skips it. Then the remaining size of the extradata is calculated
which ends up negative if the second byte of the NAL unit header is not
present. Yet when calling ff_nal_unit_extract_rbsp() it
will be converted to an uint32_t and end up as UINT32_MAX which
will cause mayhem.

This is solved by making sure that there is always enough remaining
extradata that could (pending 0x03 escapes) contain the data that we
are interested in.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt at gmail.com>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=96ad55df5bfa594defa2d57970686df3106a9ffa
---

 libavformat/hlsenc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libavformat/hlsenc.c b/libavformat/hlsenc.c
index 8e4cc36d50..49c4ab5966 100644
--- a/libavformat/hlsenc.c
+++ b/libavformat/hlsenc.c
@@ -349,7 +349,7 @@ static void write_codec_attr(AVStream *st, VariantStream *vs)
             level = st->codecpar->level;
 
         /* check the boundary of data which from current position is small than extradata_size */
-        while (data && (data - st->codecpar->extradata + 5) < st->codecpar->extradata_size) {
+        while (data && (data - st->codecpar->extradata + 19) < st->codecpar->extradata_size) {
             /* get HEVC SPS NAL and seek to profile_tier_level */
             if (!(data[0] | data[1] | data[2]) && data[3] == 1 && ((data[4] & 0x42) == 0x42)) {
                 int remain_size = 0;