[FFmpeg-devel] [PATCH] VC-1 MMX DSP functions

[Christophe GISQUET]

infernix a ?crit :
> Zuxy Meng wrote:
>> I wonder if it's caused by "m"(sstr) and things like that. Maybe it
>> can be corrected by using "g"((intptr_t)sstr).
> 
> IANAC, but i applied:
> 
> sed -i 's|\([SDrmRg]" *((\)long)|\1intptr_t)|g' libavcodec/i386/*.[ch]
> 
> and changed libavcodec/i386/vc1dsp_mmx.c line 252-253 to:
> 
>          : "g"((intptr_t)sstr), "g"((intptr_t)dstr), 
> "r"((intptr_t)offset), "r"((intptr_t)3*offset), \
>            "g"((intptr_t)rnd) 

The crash occurs in vc1_put_shift2_mmx, so line
        : "m"(src_stride), "m"(dst_stride),
becomes:
        : "m"((intptr_t)src_stride), "m"((intptr_t)dst_stride),

But anyway, it still sound strange:
rax is 3*offset and rcx is offset, harmless

In the backtrace, only the value of rdx seems to be able to cause a
crash. And its value is that of src, that gdb warns as:
src=0x2ab2aaefdfee <Address 0x2ab2aaefdfee out of bounds>

Same for dst, that we can trace back to vc1_decode_p_blocks. I'm not
sure how to interpret this, but that does sound bad.

I have to admit I'm clumsy with gas contraints (though the
put_no_rnd_h264 was a particularly bad example), so I would rather
suspect that (src register being kept across function calls) causing the
problem.

Best regards,
-- 
Christophe GISQUET