[FFmpeg-devel] [PATCH] hwcontext_vulkan: workaround MoltenVK's bug which leads to segmentation fault

[Zhao Zhili]

MoltenVK doesn't reset instance pointer when CreateInstance() failed,
then DestroyInstance() leads to segmentation fault. MoltenVK's bug
has been fixed by [1], which doesn't available on homebrew yet.
Regardless MoltenVK's bug, we shouldn't call DestroyInstance()
in the case of CreateInstance() failed, so reset instance making
sense.

[1] https://github.com/KhronosGroup/MoltenVK/commit/86a1fbdb8
---
 libavutil/hwcontext_vulkan.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c
index 2e219511c9..ac8e00003a 100644
--- a/libavutil/hwcontext_vulkan.c
+++ b/libavutil/hwcontext_vulkan.c
@@ -719,6 +719,8 @@ static int create_instance(AVHWDeviceContext *ctx, AVDictionary *opts)
     if (ret != VK_SUCCESS) {
         av_log(ctx, AV_LOG_ERROR, "Instance creation failure: %s\n",
                vk_ret2str(ret));
+        /* Workaround MoltenVK's bug which doesn't reset instance pointer. */
+        hwctx->inst = (VkInstance) { 0 };
         err = AVERROR_EXTERNAL;
         goto fail;
     }
-- 
2.31.1