[FFmpeg-devel] Mixture of homographies video stabilization

[Matthew Lai]

Hello!

I want to write a more advanced video stabilizer for libavfilter (*),
implementing the algorithm described here -
https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37744.pdf
The focus of the paper is rolling shutter removal, but it builds on top of
another algorithm that does full frame stabilization, and the new algorithm
does that as well.

This is the algorithm used in YouTube's stabilizing filter, and is state of
the art. Adobe calls it Warp Stabilizer (it's the same thing as far as I
can tell from public information anyways).

3 questions:
1. Is there a linear algebra library already in use? I didn't see anything
in configure, but would be surprised if none of the existing filters work
with matrices?
2. Is there anything to watch out for re. a high frame delay (say a few
hundred frames)? Looking at the API, I don't see a callback to flush out
remaining frames when input frames are finished? Is doing it in two passes
the only option?
3. doc/writing_filters.txt says only slice threading is available. That's
not really possible with this filter, but frame threading is. Can I just
buffer frames internally (which I need to do anyways to smooth out motion),
and do my own threading?

* vid.stab is good for what it does, but it only does rotation and
translation, and doesn't handle zoom, perspective distortion, or rolling
shutter. This means it's limited when it comes to things like scuba diving
videos, where the camera is filming the seabed at a small distance and at
an angle.

Thanks!
Matthew