[FFmpeg-user] Blending two inputs with custom expression

[SviMik]

2017-05-09 18:50 GMT+03:00 Gyan <gyandoshi at gmail.com>:
>
> On Tue, May 9, 2017 at 7:47 AM, SviMik <svimik at gmail.com> wrote:
>
> > Let's assume A is a video, and B is a png image with alpha channel. I
need
> > to do the following blending (assuming the format is rgba and B_alpha
is in
> > 0...1 range):
> >
> > red = (A_red - B_red * B_alpha) / (1 - B_alpha);
> > green = (A_green - B_green * B_alpha) / (1 - B_alpha);
> > blue = (A_blue - B_blue * B_alpha) / (1 - B_alpha);
> >
>
> You can achieve this using a sequence of filters, where the above
> expression is realized piecemeal. I haven't tested this, but it should
work.
>
> I assume the first input to ffmpeg is the video, and the (unlooped) image
> second. And both are 8-bit RGBA.
>
> -filter_complex
"[1]geq=r='p(X,Y)*alpha(X,Y)/255':g='p(X,Y)*alpha(X,Y)/255':
> b='p(X,Y)*alpha(X,Y)/255'[imgpremult]; [imgpremult][0]blend=all_expr=
> B-A:c3_expr=A,lutrgb=a=maxval-val,geq=r='255*p(X,Y)/alpha(X,
> Y)':g='255*p(X,Y)/alpha(X,Y)':b='255*p(X,Y)/alpha(X,Y)'"
>
> The video output will have an alpha plane populated during the blend
> operation. If you need to preserve the original alpha, insert
> [0]blend=all_expr=B:c3_expr=A at the end. Only one input is specified, the
> 2nd is the unconnected output from the last geq filter.
>
> There's no attempt to clip or validate the values from any of the
> expressions e.g. A_red - B_red * B_alpha = -0.5 if A_red = 0.5, B_red=1,
> B_alpha=1. Which is invalid as it's out of range [0,1] and will remain so
> even after division by 1 - B_alpha. Not to mention how you wish to handle
> B_alpha = 1.

Great answer, thank you for the idea! I have tried to run it, and it
produced me a black screen (all zeroes), but I think it may be my fault
somewhere. Maybe the alpha needs to be inverted? In my equation the B_alpha
assumes 0.0=transparent and 1.0=opaque, I really forgot to mention this.
[UPD: tried to invert the alpha channel in png image, no change]
I'm still looking at your code, and it will take me some time to understand
it. Sometimes the complex filters *are* complex :)

Just to make sure we're on the same page, I'm attaching the working php
code. Don't be confused if you're not writing in php, it's much like C and
easy to read. Except the way php operating with alpha value, I tried to
make it explicit by commenting.

for($x=0;$x<$w;$x++){
for($y=0;$y<$h;$y++){
    /* A screencap from the video */
    $rgb=imagecolorat($im, $x, $y);
    $r = ($rgb >> 16) & 0xFF;
    $g = ($rgb >> 8) & 0xFF;
    $b = $rgb & 0xFF;

    /* The png image */
    $rgb=imagecolorat($imm, $x, $y);
    $mt = ($rgb >> 24) & 0xFF; /* in php the alpha range is 0...127 */
    $mr = ($rgb >> 16) & 0xFF;
    $mg = ($rgb >> 8) & 0xFF;
    $mb = $rgb & 0xFF;

    /* in php the 0 is opaque and 127 is transparent */
    /* invert and rescale it to 0.0=transparent, and 1.0=opaque */
    $mt = (127-$mt)/127;

    /* Yes, I'm aware that 100% opaque pixel will cause div by zero */
    /* No need to validate it, the png does not have such values */
    $r = ($r - $mr*$mt) / (1-$mt);
    $g = ($g - $mg*$mt) / (1-$mt);
    $b = ($b - $mb*$mt) / (1-$mt);

    /* In real screencaps a pixel or two gets occasionally clipped */
    $r=max(0, min(255, round($r)));
    $g=max(0, min(255, round($g)));
    $b=max(0, min(255, round($b)));

    imagesetpixel($im2, $x, $y, ($r<<16)|($g<<8)|$b);
}
}