[MPlayer-dev-eng] segv in swscaler YV12->YUY2 with SIMD on x86_64

[Michael Niedermayer]

Hi

On Sat, Apr 15, 2006 at 06:56:34PM -0400, Jason Tackaberry wrote:
> I can consistently reproduce a crash in swscaler on amd64 with
> YV12->YUY2 conversion.  I have a filter that supports only YUY2:
>         
>         static int
>         query_format(struct vf_instance_s* vf, unsigned int fmt)
>         {
>             if (fmt == IMGFMT_YUY2)
>                 return vf_next_query_format(vf, fmt);
>             return 0;
>         }
> 
> Mplayer will helpfully do the colorspace conversion for me, but it
> crashes on am64. :)  It will crash with MMX2, 3DNOW, and MMX, but if I
> disable all three, there's no crash.  Here's a traceback:
> 
> 
>         SwScaler: BICUBIC scaler, from Planar YV12 to Packed YUY2 using MMX2
>         VO: [null] 618x464 => 618x464 Packed YUY2
>         A:   0.0 V:   0.0 A-V:  0.009 ct:  0.000   1/  1 ??% ??% ??,?% 0 0
>         Program received signal SIGSEGV, Segmentation fault.
>         [Switching to Thread 46912507208448 (LWP 23628)]
>         0x0000000000594654 in swScale_MMX2 ()
>         (gdb) bt
>         #0  0x0000000000594654 in swScale_MMX2 ()
>         #1  0x000000000058b7f3 in sws_scale_ordered ()
>         #2  0x0000000000495198 in scale ()
>         #3  0x00000000004951f0 in draw_slice ()
>         #4  0x000000000046de98 in draw_slice ()
>         #5  0x00000000005b4831 in ff_draw_horiz_band ()
>         #6  0x00000000006aa4d5 in decode_slice ()
>         #7  0x00000000006ab439 in ff_h263_decode_frame ()
>         #8  0x00000000005b0e80 in avcodec_decode_video ()
>         #9  0x000000000046e6d6 in decode ()
>         #10 0x000000000046b404 in decode_video ()
>         #11 0x0000000000415f20 in main ()
>         
> Happy to provide more info if needed.

no linenumbers, no disassembly, read:
http://www.mplayerhq.hu/DOCS/HTML/en/bugreports_what.html#bugreports_crash

[...]
-- 
Michael

In the past you could go to a library and read, borrow or copy any book
Today you'd get arrested for mere telling someone where the library is