[FFmpeg-user] Stitching two overlapping fisheye videos with maskedmerge

[Michael Koch]

Am 03.03.2020 um 10:52 schrieb Paul B Mahol:
> Hi,
>
> On 3/3/20, Michael Koch <astroelectronic at t-online.de> wrote:
>> I'd like to stitch two fisheye videos together to an equirectangular
>> video. The input videos have larger than 180° field of view, so that
>> there is an overlap region and I want to use the maskedmerge filter for
>> merging the videos.
>> My first (successful) test is with png images:
>>
>> First create the mask file for maskedmerge:
>> ffmpeg -f lavfi -i nullsrc=size=640x640 -vf
>> "format=gray8,geq='clip(128-128/10*(180-191.5/(640/2)*hypot(X-640/2,Y-640/2)),0,255)',v360=fisheye:e:ih_fov=191.5:iv_fov=191.5,format=rgb24"
>> -frames 1 -y mergemap.png
>> The pixel values in this image are 0 for one fisheye region, 255 for the
>> other fisheye region and a 0-255 ramp in the overlap region.
>>
>> Now create two fisheye inputs (one is red and the other is green) and
>> transform them to equirectangular:
>> ffmpeg -f lavfi -i color=red:size=640x640 -lavfi
>> "drawbox=w=1:h=1:color=black,v360=input=fisheye:output=e:ih_fov=191.5:iv_fov=191.5"
>> -frames 1 -y eq1.png
>> ffmpeg -f lavfi -i color=green:size=640x640 -lavfi
>> "drawbox=w=1:h=1:color=black,v360=input=fisheye:output=e:ih_fov=191.5:iv_fov=191.5:yaw=180"
>> -frames 1 -y eq2.png
>>
>> Now merge the two images with maskedmerge:
>> ffmpeg -i eq1.png -i eq2.png -i mergemap.png -lavfi "maskedmerge"
>> -frames 1 -y out.png
>>
>> Up to this point, everything did work as expected. The output image
>> looks correct. As the next step, I want to do the same thing with
>> videos. I replace the file extensions by *.mp4, I replace "-frames 1" by
>> "-t 5", and I add "-loop 1" for the third input file (which is the same
>> as before):
>> ffmpeg -f lavfi -i color=red:size=640x640 -lavfi
>> "drawbox=w=1:h=1:color=black,v360=input=fisheye:output=e:ih_fov=191.5:iv_fov=191.5"
>> -t 5 -y eq1.mp4
>> ffmpeg -f lavfi -i color=green:size=640x640 -lavfi
>> "drawbox=w=1:h=1:color=black,v360=input=fisheye:output=e:ih_fov=191.5:iv_fov=191.5:yaw=180"
>> -t 5 -y eq2.mp4
>> ffmpeg -i eq1.mp4 -i eq2.mp4 -loop 1 -i mergemap.png -lavfi
>> "maskedmerge" -t 5 -y out.mp4
>>
>> The intermediate videos eq1.mp4 and eq2.mp4 look correct, but the output
>> video is wrong.
>> I did already try many format conversions, but I haven't yet figured out
>> what's going wrong here.
>> Below is the console output for the last command line.
> ffmpeg -i eq1.mp4 -i eq2.mp4 -loop 1 -i mergemap.png -lavfi
> "[0:v]format=gbrp[a],[1:v]format=gbrp[b],[2:v]format=gbrp[c],[a][b][c]maskedmerge,format=gbrp"
> -t 5 -y out.mp4

yes, this works. The trick is to use the format filter after 
maskedmerge. Then it also works with rgb24.
I did expect that the output of maskedmerge is in the same format as the 
three inputs (I did already test with three rgb24 inputs). Apparently 
this is not the case and another format=rgb24 is required after 
maskedmerge. There should be a warning in the documentation that 
maskedmerge has a different output format, even if all three inputs are 
rgb24.

Thanks,
Michael