[FFmpeg-devel] [PATCH] MMX/floating-point crash issue

[Ray Simard]

The following requires the patch for the uninitialized variable above to
be applied first ("Odd, random-appearing crashes"), so I'm mentioning
this here, after that.

As Reimar Doeffinger pointed out, there is no reason to use doubles in
these loops. As it happens, the result is worse than needless overhead;
the assignment of the result of the MMX calculations called by the CMP
macro to the double floating-point members of mv without an intervening
call to emms_c() was causing bogus data to be assigned, resulting in the
crashes. In fact, the values in mv are never used for anything except
integer values.

(My earlier, rather naive suggestion, to add emms_c() calls in the
loops, should be disregarded.  It was a good test to verify the cause of
the data-corruption and crash problem, but not a solution.)

The attached patch implements his suggestion to replace the MotionVector
variables in these cases with an integer-only version. I've tried it out
and it seems to be working perfectly.  At the moment I'm running FATE on
it. It's very straightforward and I'd be very surprised if there were
any problem with it.


Note: This does not affect the MotionVector member of the Transform
struct passed to avfilter_transform(). That is still double.
Determining whether or not that should also be changed requires further
scrutiny.


One of the loops in question. Others are similar. The struct mv points
to was a MotionVector struct in which x and y were double.

    if (deshake->search == EXHAUSTIVE) {
        // Compare every possible position - this is sloooow!
        for (y = -deshake->ry; y <= deshake->ry; y++) {
            for (x = -deshake->rx; x <= deshake->rx; x++) {
                diff = CMP(cx - x, cy - y);
                if (diff < smallest) {
                    smallest = diff;
                    mv->x = x;  <=== BOGUS.
                    mv->y = y;  <===  ""
                }
            }
        }

Ray Simard
rhs.ffmpeg at sylvan-glade.com