[FFmpeg-devel] [avformat] Prevent undefined shift with wrap_bits > 63.

[Michael Niedermayer]

On Fri, Nov 17, 2017 at 01:46:39PM -0800, Dale Curtis wrote:
> Derp, actually, 2 << 63 doesn't fit in int64_t either, this check should be
> < 63. Fixed.
> 
> 
> 
> On Fri, Nov 17, 2017 at 1:38 PM, Dale Curtis <dalecurtis at chromium.org>
> wrote:
> 
> > 2 << (wrap_bits=64 - 1) does not fit in int64_t; apply the check
> > used in other places that handle wrap bits to ensure the values
> > are <= 63.
> >
> > Signed-off-by: Dale Curtis <dalecurtis at chromium.org>
> >
> >

>  utils.c |    4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 2f61647f4321bebca803154960f74643d4670ee8  wrap_bits_v2.patch
> From 4ae4992326487ba0e42fa7fcf2a53fe3d4564780 Mon Sep 17 00:00:00 2001
> From: Dale Curtis <dalecurtis at chromium.org>
> Date: Fri, 17 Nov 2017 13:35:56 -0800
> Subject: [PATCH] [avformat] Prevent undefined shift with wrap_bits >= 63.
> 
> 2 << (wrap_bits=63 - 1) does not fit in int64_t; apply the check
> used in other places that handle wrap bits to ensure the values
> are < 63.
> 
> Signed-off-by: Dale Curtis <dalecurtis at chromium.org>
> ---
>  libavformat/utils.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/libavformat/utils.c b/libavformat/utils.c
> index ff5e14df6c..65d111459f 100644
> --- a/libavformat/utils.c
> +++ b/libavformat/utils.c
> @@ -1738,8 +1738,8 @@ int av_read_frame(AVFormatContext *s, AVPacket *pkt)
>                  // current one had no dts, we will set this to AV_NOPTS_VALUE.
>                  int64_t last_dts = next_pkt->dts;
>                  while (pktl && next_pkt->pts == AV_NOPTS_VALUE) {
> -                    if (pktl->pkt.stream_index == next_pkt->stream_index &&
> -                        (av_compare_mod(next_pkt->dts, pktl->pkt.dts, 2LL << (wrap_bits - 1)) < 0)) {
> +                    if (pktl->pkt.stream_index == next_pkt->stream_index && wrap_bits < 63 &&
> +                        av_compare_mod(next_pkt->dts, pktl->pkt.dts, 2LL << (wrap_bits - 1)) < 0) {

this would skip the code for wrap_bits >= 63, this does not look
correct

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Those who are best at talking, realize last or never when they are wrong.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: Digital signature
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20171118/4d340588/attachment.sig>