[FFmpeg-devel] [PATCH] avfilter/vf_overlay: Fix the calculation of average alpha with alpha composition and threads > 1

[Linjie Fu]

The calculation of average alpha utilizes the value of next line(alpha[x+linesize]):
    alpha[0]            alpha[1]
    alpha[0+linesize]   alpha[1+linesize]

However, alpha[x+linesize] could be either an alpha_composited value or
an original value if it locates at next slice, which may lead to an
incorrect output and run2run changed MD5.

Hence, only utilizes the alpha within one slice on the boundary to fix
the calculation of average alpha.

CMD:
    $ffmpeg -i https://samples.ffmpeg.org/FLV/flash_with_alpha/laraShadow_dl.flv \
            -filter_complex "nullsrc=1920x1080[arena];[arena][0:v]overlay" \
                                                    -vframes 10 -an -f md5 -

Before:
Incorrect output and changed MD5 at each run.

After this patch:
Consistent output and MD5.

Signed-off-by: Linjie Fu <linjie.justin.fu at gmail.com>
---
[RFC]This is a partial fix. A fully-fixed method may be don't overwrite dst_alpha
or preserve the original alpha value in composite_alpha(). 
Then we can use the original alpha value for calculation in blend_plane().

 libavfilter/vf_overlay.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/libavfilter/vf_overlay.c b/libavfilter/vf_overlay.c
index f560d54dae..9488ccd0d7 100644
--- a/libavfilter/vf_overlay.c
+++ b/libavfilter/vf_overlay.c
@@ -515,13 +515,13 @@ static av_always_inline void blend_plane_##depth##_##nbits##bits(AVFilterContext
             if (main_has_alpha && alpha != 0 && alpha != max) {                                            \
                 /* average alpha for color components, improve quality */                                  \
                 uint8_t alpha_d;                                                                           \
-                if (hsub && vsub && j+1 < src_hp && k+1 < src_wp) {                                        \
+                if (hsub && vsub && j+1 < src_hp && k+1 < src_wp && j+1 < slice_end) {                     \
                     alpha_d = (da[0] + da[dst->linesize[3]] +                                              \
                                da[1] + da[dst->linesize[3]+1]) >> 2;                                       \
                 } else if (hsub || vsub) {                                                                 \
                     alpha_h = hsub && k+1 < src_wp ?                                                       \
                         (da[0] + da[1]) >> 1 : da[0];                                                      \
-                    alpha_v = vsub && j+1 < src_hp ?                                                       \
+                    alpha_v = vsub && j+1 < src_hp && j+1 < slice_end ?                                    \
                         (da[0] + da[dst->linesize[3]]) >> 1 : da[0];                                       \
                     alpha_d = (alpha_v + alpha_h) >> 1;                                                    \
                 } else                                                                                     \
-- 
2.31.1