[FFmpeg-devel] [PATCH 2/4] libavutil/hwcontext_qsv: fix a bug when malloc handle_pairs_internal

[Soft Works]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of
> Chen, Wenbin
> Sent: Wednesday, November 10, 2021 4:03 AM
> To: ffmpeg-devel at ffmpeg.org
> Subject: Re: [FFmpeg-devel] [PATCH 2/4] libavutil/hwcontext_qsv: fix
> a bug when malloc handle_pairs_internal
> 
> > This commandline cause core dumped:
> > ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 \
> > -hwaccel_output_format vaapi -i input.264 \
> > -vf "hwmap=derive_device=qsv,format=qsv" \
> > -c:v h264_qsv output.264
> >
> > reason: We use nb_surfaces to assign surface to
> handle_pairs_internal
> > but handle_pairs_internal is alloced with the size of
> init_pool_size.
> > This lead to access to illegal address.
> >
> > Now change it to use nb_surfaces to allocate handle_pairs_internal
> and the

I'm not sure about whether this is right.

When we look at the top of the qsv_frames_derive_to function that you 
are changing, there is this:


    if (src_ctx->initial_pool_size == 0) {
        av_log(dst_ctx, AV_LOG_ERROR, "Only fixed-size pools can be "
            "mapped to QSV frames.\n");
        return AVERROR(EINVAL);
    }

It's because QSV doesn't support dynamic pool sizes.

When we look at the vaapi_pool_alloc function in hwcontext_vaapi.c, we
can see that:

  when  initial_pool_size is > 0, the pool cannot grow beyond this value,
  so nb_surfaces cannot be > initial_pool_size

So I'm wondering what could have caused the segfault? Which values did
you have there for nb_surfaces and initial_pool_size?


> > core dumped error is unseen. Also change D3D11VA to use nb_surfaces
> > to align to VAAPI and DXVA2.

Those changes are unrelated to fixing the issue with VAAPI.
(besides that I don't think these are needed at all)

Kind regards,
softworkz