[FFmpeg-devel] [PATCH] aaccoder: prevent crash of anmr coder

[Claudio Freire]

On Fri, Dec 4, 2015 at 2:23 PM, Andreas Cadhalpun
<andreas.cadhalpun at googlemail.com> wrote:
> If minq is negative, the range of sf_idx can be larger than
> SCALE_MAX_DIFF allows, causing assertion failures later in
> encode_scale_factors.
>
> Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun at googlemail.com>
> ---
>  libavcodec/aaccoder.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/libavcodec/aaccoder.c b/libavcodec/aaccoder.c
> index 2a0cb1f..e8a61ce 100644
> --- a/libavcodec/aaccoder.c
> +++ b/libavcodec/aaccoder.c
> @@ -370,7 +370,7 @@ static void search_for_quantizers_anmr(AVCodecContext *avctx, AACEncContext *s,
>      }
>      while (idx) {
>          sce->sf_idx[bandaddr[idx]] = minq + q0;
> -        minq = paths[idx][minq].prev;
> +        minq = FFMAX(paths[idx][minq].prev, 0);
>          idx--;
>      }
>      //set the same quantizers inside window groups

Actually, a negative .prev signals an impossible path.

So perhaps the attached patch (well, git diff) would go better? (if
you confirm it passes fuzzing I'll push it)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-AAC-encoder-anmr-fixfuzz.patch
Type: text/x-patch
Size: 536 bytes
Desc: not available
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20151204/789ce070/attachment.bin>