[FFmpeg-devel] [PATCH] hwcontext_vulkan: workaround MoltenVK's bug which leads to segmentation fault

[Wu, Jianhua]

Zhao Zhili wrote:
> Sent: Thursday, January 27, 2022 4:11 PM
> To: ffmpeg-devel at ffmpeg.org
> Cc: Zhao Zhili <quinkblack at foxmail.com>
> Subject: [FFmpeg-devel] [PATCH] hwcontext_vulkan: workaround
> MoltenVK's bug which leads to segmentation fault
> 
> MoltenVK doesn't reset instance pointer when CreateInstance() failed, then
> DestroyInstance() leads to segmentation fault. MoltenVK's bug has been
> fixed by [1], which doesn't available on homebrew yet.
> Regardless MoltenVK's bug, we shouldn't call DestroyInstance() in the case
> of CreateInstance() failed, so reset instance making sense.
> 
> [1] https://github.com/KhronosGroup/MoltenVK/commit/86a1fbdb8
> ---
>  libavutil/hwcontext_vulkan.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c
> index 2e219511c9..ac8e00003a 100644
> --- a/libavutil/hwcontext_vulkan.c
> +++ b/libavutil/hwcontext_vulkan.c
> @@ -719,6 +719,8 @@ static int create_instance(AVHWDeviceContext *ctx,
> AVDictionary *opts)
>      if (ret != VK_SUCCESS) {
>          av_log(ctx, AV_LOG_ERROR, "Instance creation failure: %s\n",
>                 vk_ret2str(ret));
> +        /* Workaround MoltenVK's bug which doesn't reset instance pointer.
> */
> +        hwctx->inst = (VkInstance) { 0 };

Hi,

It's no need to use the explicit cast and use hwctx->inst = VK_NULL_HANDLE instead, which is the null context defined by Vulkan spec.

Thanks,
Jianhua