[FFmpeg-devel] [PATCH] libavutil/hwcontext_qsv: fix a bug for mapping qsv frame to vaapi

[Xiang, Haihao]

On Fri, 2021-09-10 at 02:19 +0000, Chen, Wenbin wrote:
> > -----Original Message-----
> > From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of
> > James Almer
> > Sent: Thursday, September 9, 2021 8:48 PM
> > To: ffmpeg-devel at ffmpeg.org
> > Subject: Re: [FFmpeg-devel] [PATCH] libavutil/hwcontext_qsv: fix a bug for
> > mapping qsv frame to vaapi
> > 
> > On 9/9/2021 6:13 AM, Wenbin Chen wrote:
> > > Command below failed.
> > > ffmpeg -v verbose -init_hw_device vaapi=va:/dev/dri/renderD128
> > > -init_hw_device qsv=qs at va -hwaccel qsv -hwaccel_device qs
> > > -filter_hw_device va -c:v h264_qsv
> > > -i 1080P.264 -vf "hwmap,format=vaapi" -c:v h264_vaapi output.264
> > > 
> > > Cause: Assign pair->first directly to data[3] in vaapi frame.
> > > pair->first is *VASurfaceID while data[3] in vaapi frame is
> > > VASurfaceID. I fix this line of code. Now the command above works.
> > > 
> > > Signed-off-by: Wenbin Chen <wenbin.chen at intel.com>
> > > ---
> > >   libavutil/hwcontext_qsv.c | 2 +-
> > >   1 file changed, 1 insertion(+), 1 deletion(-)
> > > 
> > > diff --git a/libavutil/hwcontext_qsv.c b/libavutil/hwcontext_qsv.c
> > > index d431e71eab..6539cae619 100644
> > > --- a/libavutil/hwcontext_qsv.c
> > > +++ b/libavutil/hwcontext_qsv.c
> > > @@ -781,7 +781,7 @@ static int qsv_map_from(AVHWFramesContext *ctx,
> > >       case AV_HWDEVICE_TYPE_VAAPI:
> > >       {
> > >           mfxHDLPair *pair = (mfxHDLPair*)surf->Data.MemId;
> > > -        child_data = pair->first;
> > > +        child_data = (uint8_t*)(intptr_t)*(VASurfaceID*)pair->first;
> > 
> > You can probably remove the intptr_t casting.
> 
> If intptr_t is removed, it will report compile warning
> "cast to pointer from integer of different size"

How about to add a comment here? If so, others can understand why this casting
is needed.

Thanks
Haihao


>  
> > 
> > Also, shouldn't this same fix be done for all three child device types
> > used in this function? Which for that matter, child_data seems to be set
> > for a d3d11va child device, but then never used.
> 
> Dxva and d3d11 frames store the pointer to surface while vaapi frames store
> surface, 
> so this part of code for dxva and d3d11va should be correct.
> 
> D3d11 and dxva share dxva codec, the child_data is used in this part of code.
> > 
> > >           break;
> > >       }
> > >   #endif
> > > 
> > 
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel at ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> > 
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".