[FFmpeg-devel] [PATCH 04/21] avformat: Redo cleanup of demuxer upon read_header() failure

[James Almer]

On 3/22/2020 12:47 AM, Andreas Rheinhardt wrote:
> If reading the header fails, the demuxer's read_close() function (if
> existing) is not called automatically; instead several demuxers call it
> via "goto fail" in read_header().
> 
> This commit intends to change this by adding a flag to AVInputFormat
> that can be used to set on a per-AVInputFormat basis whether read_close()
> should be called generically after an error during read_header().
> 
> The flag controlling this behaviour has been added because it might be
> unsafe to call read_close() generally (e.g. this might lead to
> read_close() being called twice and this might e.g. lead to double-frees
> if av_free() instead of av_freep() is used; or a size field has not
> been reset after freeing the elements (see the mov demuxer for an
> example of this)). Yet the intention is to check and fix all demuxers
> and make the flag redundant in the medium run.
> 
> The flag has been added to a new internal field of AVInputFormat,
> flags_internal. When it has become redundant, it can be removed again
> at the next major version bump.

Fields considered not public (And thus added below the relevant notice
in the struct) can be added and removed at any time. So this sentence is
not needed.

> 
> Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt at gmail.com>
> ---
>  libavformat/avformat.h |  7 +++++++
>  libavformat/utils.c    | 11 +++++++++--
>  2 files changed, 16 insertions(+), 2 deletions(-)
> 
> diff --git a/libavformat/avformat.h b/libavformat/avformat.h
> index 9b9b634ec3..50b90788b1 100644
> --- a/libavformat/avformat.h
> +++ b/libavformat/avformat.h
> @@ -780,6 +780,13 @@ typedef struct AVInputFormat {
>       * @see avdevice_capabilities_free() for more details.
>       */
>      int (*free_device_capabilities)(struct AVFormatContext *s, struct AVDeviceCapabilitiesQuery *caps);
> +
> +#define FF_INPUTFORMAT_HEADER_CLEANUP 0x0001 /**< read_close() should be called
> +                                                  on read_header() failure */

If this is internal, then it should be defined in internal.h, same as
the caps_internal flags for AVCodecContext.

> +    /**
> +     * Can use flags: FF_INPUTFORMAT_HEADER_CLEANUP
> +     */
> +    int flags_internal;
>  } AVInputFormat;
>  /**
>   * @}
> diff --git a/libavformat/utils.c b/libavformat/utils.c
> index a58e47fabc..87807a7841 100644
> --- a/libavformat/utils.c
> +++ b/libavformat/utils.c
> @@ -400,8 +400,12 @@ int av_demuxer_open(AVFormatContext *ic) {
>  
>      if (ic->iformat->read_header) {
>          err = ic->iformat->read_header(ic);
> -        if (err < 0)
> +        if (err < 0) {
> +            if (ic->iformat->read_close &&
> +                ic->iformat->flags_internal & FF_INPUTFORMAT_HEADER_CLEANUP)
> +                ic->iformat->read_close(ic);
>              return err;
> +        }
>      }
>  
>      if (ic->pb && !ic->internal->data_offset)
> @@ -628,8 +632,11 @@ FF_ENABLE_DEPRECATION_WARNINGS
>  
>  
>      if (!(s->flags&AVFMT_FLAG_PRIV_OPT) && s->iformat->read_header)
> -        if ((ret = s->iformat->read_header(s)) < 0)
> +        if ((ret = s->iformat->read_header(s)) < 0) {
> +            if (s->iformat->flags_internal & FF_INPUTFORMAT_HEADER_CLEANUP)
> +                goto close;
>              goto fail;
> +        }
>  
>      if (!s->metadata) {
>          s->metadata = s->internal->id3v2_meta;
>