[FFmpeg-cvslog] avformat/utils: Prevent undefined shift with wrap_bits > 64.

[Dale Curtis]

ffmpeg | branch: release/3.3 | Dale Curtis <dalecurtis at chromium.org> | Fri Nov 17 13:35:56 2017 -0800| [85ea121684a7b128c39373845506e6016daa60cc] | committer: Michael Niedermayer

avformat/utils: Prevent undefined shift with wrap_bits > 64.

2LL << (wrap_bits=64 - 1) does not fit in int64_t; change the
code to use a uint64_t (2ULL) and add an av_assert2() to
ensure wrap_bits <= 64.

Signed-off-by: Dale Curtis <dalecurtis at chromium.org>
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
(cherry picked from commit 03fbc0daa7e37af024f8b017a28105c32bbe25ca)
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=85ea121684a7b128c39373845506e6016daa60cc
---

 libavformat/utils.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/libavformat/utils.c b/libavformat/utils.c
index 5200c7d2e7..2c622d2c56 100644
--- a/libavformat/utils.c
+++ b/libavformat/utils.c
@@ -1732,13 +1732,14 @@ int av_read_frame(AVFormatContext *s, AVPacket *pkt)
 
             if (next_pkt->dts != AV_NOPTS_VALUE) {
                 int wrap_bits = s->streams[next_pkt->stream_index]->pts_wrap_bits;
+                av_assert2(wrap_bits <= 64);
                 // last dts seen for this stream. if any of packets following
                 // current one had no dts, we will set this to AV_NOPTS_VALUE.
                 int64_t last_dts = next_pkt->dts;
                 while (pktl && next_pkt->pts == AV_NOPTS_VALUE) {
                     if (pktl->pkt.stream_index == next_pkt->stream_index &&
-                        (av_compare_mod(next_pkt->dts, pktl->pkt.dts, 2LL << (wrap_bits - 1)) < 0)) {
-                        if (av_compare_mod(pktl->pkt.pts, pktl->pkt.dts, 2LL << (wrap_bits - 1))) {
+                        av_compare_mod(next_pkt->dts, pktl->pkt.dts, 2ULL << (wrap_bits - 1)) < 0) {
+                        if (av_compare_mod(pktl->pkt.pts, pktl->pkt.dts, 2ULL << (wrap_bits - 1))) {
                             // not B-frame
                             next_pkt->pts = pktl->pkt.dts;
                         }