[FFmpeg-devel] [PATCH 4/5] avcodec/apedec: Fix integer overflow in intermediate

[Michael Niedermayer]

On Wed, Sep 15, 2021 at 10:09:49PM +0200, Andreas Rheinhardt wrote:
> Michael Niedermayer:
> > Fixes: signed integer overflow: 559334865 * 4 cannot be represented in type 'int'
> > Fixes: 37929/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_APE_fuzzer-6751932295806976
> > 
> > Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
> > Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
> > ---
> >  libavcodec/apedec.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/libavcodec/apedec.c b/libavcodec/apedec.c
> > index bf481ba3549..23318be0613 100644
> > --- a/libavcodec/apedec.c
> > +++ b/libavcodec/apedec.c
> > @@ -1337,7 +1337,7 @@ static void do_apply_filter(APEContext *ctx, int version, APEFilter *f,
> >              absres = FFABSU(res);
> >              if (absres)
> >                  *f->adaptcoeffs = APESIGN(res) *
> > -                                  (8 << ((absres > f->avg * 3) + (absres > f->avg * 4 / 3)));
> > +                                  (8 << ((absres > f->avg * 3) + (absres > f->avg * 4LL / 3)));
> >                  /* equivalent to the following code
> >                      if (absres <= f->avg * 4 / 3)
> >                          *f->adaptcoeffs = APESIGN(res) * 8;
> > 
> How about using f->avg + f->avg / 3? Given that you haven't modified the
> f->avg * 3 before, this should be save.

not anymore, the *3 overflows too in 39172/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_APE_fuzzer-638602483033702

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Many that live deserve death. And some that die deserve life. Can you give
it to them? Then do not be too eager to deal out death in judgement. For
even the very wise cannot see all ends. -- Gandalf
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20210929/7555a6fd/attachment.sig>