[FFmpeg-devel] [PATCH 2/4] libavutil/hwcontext_qsv: fix a bug when malloc handle_pairs_internal

[Chen, Wenbin]

> > -----Original Message-----
> > From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of
> > Chen, Wenbin
> > Sent: Wednesday, November 10, 2021 4:03 AM
> > To: ffmpeg-devel at ffmpeg.org
> > Subject: Re: [FFmpeg-devel] [PATCH 2/4] libavutil/hwcontext_qsv: fix
> > a bug when malloc handle_pairs_internal
> >
> > > This commandline cause core dumped:
> > > ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 \
> > > -hwaccel_output_format vaapi -i input.264 \
> > > -vf "hwmap=derive_device=qsv,format=qsv" \
> > > -c:v h264_qsv output.264
> > >
> > > reason: We use nb_surfaces to assign surface to
> > handle_pairs_internal
> > > but handle_pairs_internal is alloced with the size of
> > init_pool_size.
> > > This lead to access to illegal address.
> > >
> > > Now change it to use nb_surfaces to allocate handle_pairs_internal
> > and the
> 
> I'm not sure about whether this is right.
> 
> When we look at the top of the qsv_frames_derive_to function that you
> are changing, there is this:
> 
> 
>     if (src_ctx->initial_pool_size == 0) {
>         av_log(dst_ctx, AV_LOG_ERROR, "Only fixed-size pools can be "
>             "mapped to QSV frames.\n");
>         return AVERROR(EINVAL);
>     }
> 
> It's because QSV doesn't support dynamic pool sizes.
> 
> When we look at the vaapi_pool_alloc function in hwcontext_vaapi.c, we
> can see that:
> 
>   when  initial_pool_size is > 0, the pool cannot grow beyond this value,
>   so nb_surfaces cannot be > initial_pool_size
> 
> So I'm wondering what could have caused the segfault? Which values did
> you have there for nb_surfaces and initial_pool_size?
> 
> 
> > > core dumped error is unseen. Also change D3D11VA to use nb_surfaces
> > > to align to VAAPI and DXVA2.
> 
> Those changes are unrelated to fixing the issue with VAAPI.
> (besides that I don't think these are needed at all)
> 
> Kind regards,
> softworkz

You are right. The real cause is that vaapi_decode_make_config() is called twice.
The init_pool_size is changed on the second call. I will resubmit patch to fix this

Thanks
Wenbin
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".