[FFmpeg-devel] [PATCH v3] avformat/avisynth: fix segfault when also compiled with libvmaf

[Daniel Bermond]

When FFmpeg is compiled with support for both avisynth and libvmaf,
a segmentation fault occurs when using avisynth (.avs) input.

This happens because both avisynthplus and vmaf have the exactly
same C++ symbol 'Cache::~Cache()'[1][2], which is a C++ destructor.
When using avisynth input, this destructor is also called in the
vmaf side, although no vmaf object was created, thus causing a
segmentation fault.

Without changing this conflicting symbol in avisynthplus and/or
vmaf upstream code, a solution is to switch the 'RTLD_LOCAL'[3]
flag for 'RTLD_DEEPBIND'[4] when calling 'dlopen()' on the
avisynthplus library. This will make the avisynthplus symbols
loaded by 'dlopen()' to be ahead of the symbols loaded in the
global scope, and consequently avoid the segmentation fault by
preventing the conflicting vmaf C++ destructor to be called.

More details can be seen in the discussion of the Arch Linux bug
report[5] about this issue.

[1] https://github.com/AviSynth/AviSynthPlus/blob/v3.7.5/avs_core/core/cache.h#L62
[2] https://github.com/Netflix/vmaf/blob/v3.0.0/libvmaf/src/svm.cpp#L75
[3] https://man.archlinux.org/man/dlopen.3#RTLD_LOCAL
[4] https://man.archlinux.org/man/dlopen.3#RTLD_DEEPBIND
[5] https://gitlab.archlinux.org/archlinux/packaging/packages/ffmpeg/-/issues/19

Signed-off-by: Daniel Bermond <danielbermond at gmail.com>
---
 libavformat/avisynth.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libavformat/avisynth.c b/libavformat/avisynth.c
index cb2be10925..bef61b4a60 100644
--- a/libavformat/avisynth.c
+++ b/libavformat/avisynth.c
@@ -120,7 +120,7 @@ typedef struct AviSynthContext {
 
 static av_cold int avisynth_load_library(AviSynthContext *avs)
 {
-    avs->avs_library.library = dlopen(AVISYNTH_LIB, RTLD_NOW | RTLD_LOCAL);
+    avs->avs_library.library = dlopen(AVISYNTH_LIB, RTLD_NOW | RTLD_DEEPBIND);
     if (!avs->avs_library.library)
         return AVERROR_UNKNOWN;
 
-- 
2.50.1