[FFmpeg-devel] [PATCH] avcodec/ac3enc_float: fix fdsp memory leak

[James Almer]

On 10/17/2019 6:20 AM, Paul B Mahol wrote:
> Fixes #8294
> 
> Signed-off-by: Paul B Mahol <onemda at gmail.com>
> ---
>  libavcodec/ac3enc_float.c | 10 ++++++++--
>  1 file changed, 8 insertions(+), 2 deletions(-)
> 
> diff --git a/libavcodec/ac3enc_float.c b/libavcodec/ac3enc_float.c
> index d6e658b2b4..18df313c1a 100644
> --- a/libavcodec/ac3enc_float.c
> +++ b/libavcodec/ac3enc_float.c
> @@ -132,10 +132,16 @@ av_cold int ff_ac3_float_mdct_init(AC3EncodeContext *s)
>  av_cold int ff_ac3_float_encode_init(AVCodecContext *avctx)
>  {
>      AC3EncodeContext *s = avctx->priv_data;
> +    int ret = ff_ac3_encode_init(avctx);
> +
> +    if (ret < 0)
> +        return ret;
>      s->fdsp = avpriv_float_dsp_alloc(avctx->flags & AV_CODEC_FLAG_BITEXACT);
> -    if (!s->fdsp)
> +    if (!s->fdsp) {
> +        ff_ac3_encode_close(avctx);
>          return AVERROR(ENOMEM);
> -    return ff_ac3_encode_init(avctx);
> +    }
> +    return 0;

Nit: Do it the other way around instead. Call avpriv_float_dsp_alloc()
first, then ff_ac3_encode_init(). That way you'll only call an
av_free(s->fdsp) on failure rather than ff_ac3_encode_close().

LGTM either way.

>  }
>  
>  AVCodec ff_ac3_encoder = {
>