[FFmpeg-cvslog] libavutil/hwcontext_qsv: fix a bug for mapping qsv frame to vaapi

[Wenbin Chen]

ffmpeg | branch: master | Wenbin Chen <wenbin.chen at intel.com> | Mon Sep 13 14:22:22 2021 +0800| [f2891fbdeddd9049ba03b600d6c93a1ab732df66] | committer: James Almer

libavutil/hwcontext_qsv: fix a bug for mapping qsv frame to vaapi

Command below failed.
ffmpeg -v verbose -init_hw_device vaapi=va:/dev/dri/renderD128
-init_hw_device qsv=qs at va -hwaccel qsv -hwaccel_device qs
-filter_hw_device va -c:v h264_qsv
-i 1080P.264 -vf "hwmap,format=vaapi" -c:v h264_vaapi output.264

Cause: Assign pair->first directly to data[3] in vaapi frame.
pair->first is *VASurfaceID while data[3] in vaapi frame is
VASurfaceID. I fix this line of code. Now the command above works.

Signed-off-by: Wenbin Chen <wenbin.chen at intel.com>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=f2891fbdeddd9049ba03b600d6c93a1ab732df66
---

 libavutil/hwcontext_qsv.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/libavutil/hwcontext_qsv.c b/libavutil/hwcontext_qsv.c
index 225ae768a9..c18747f7eb 100644
--- a/libavutil/hwcontext_qsv.c
+++ b/libavutil/hwcontext_qsv.c
@@ -781,7 +781,11 @@ static int qsv_map_from(AVHWFramesContext *ctx,
     case AV_HWDEVICE_TYPE_VAAPI:
     {
         mfxHDLPair *pair = (mfxHDLPair*)surf->Data.MemId;
-        child_data = pair->first;
+        /* pair->first is *VASurfaceID while data[3] in vaapi frame is VASurfaceID, so
+         * we need this casting for vaapi.
+         * Add intptr_t to force cast from VASurfaceID(uint) type to pointer(long) type
+         * to avoid compile warning */
+        child_data = (uint8_t*)(intptr_t)*(VASurfaceID*)pair->first;
         break;
     }
 #endif