[MPlayer-users] Segmentation fault in ff_fdct_sse2

[Guillaume POIRIER]

Hi,

On 3/5/07, mslama at email.cz <mslama at email.cz> wrote:
>
>    Hi.
>
> I get reproducibly segmentation fault when running second pass of mencoder. I have Digital Video from camera. If necessary I can try to get sample and upload it somewhere. Unfortunately it does not happen when compiled with debug info (so I suspect it is connected with optimization). I had to use -fomit-frame-pointer to be able to compile in debug mode otherwise compilation fails as I described in my previous email.)
>
> Workaround is to either use use debug version or disable SSE2 during configuration.
>
> Comand line:
>
> nice -+19 mencoder all.avi -oac mp3lame -lameopts cbr:br=256 -ovc lavc -lavcopts vcodec=mpeg4:mbd=2:mv0:trell:v4mv:cbp:last_pred=3:predia=2:dia=2:vmax_b_frames=2:vb_strategy=1:precmp=3:cmp=3:subcmp=3:preme=2:vqcomp=0.6:autoaspect:vbitrate=6400:turbo:vpass=1 -o out.avi
> nice -+19 mencoder all.avi -oac mp3lame -lameopts cbr:br=256 -ovc lavc -lavcopts vcodec=mpeg4:mbd=2:mv0:trell:v4mv:cbp:last_pred=3:predia=2:dia=2:vmax_b_frames=2:precmp=3:cmp=3:subcmp=3:preme=2:vqcomp=0.6:autoaspect:vbitrate=6400:vpass=2:psnr -o out.avi
>
> gcc:
> Using built-in specs.
> Target: i486-linux-gnu
> Configured with: ../src/configure -v --enable-languages=c,c++,fortran,objc,obj-c++,treelang --prefix=/usr --enable-shared --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --enable-nls --program-suffix=-4.1 --enable-__cxa_atexit --enable-clocale=gnu --enable-libstdcxx-debug --enable-mpfr --enable-checking=release i486-linux-gnu
> Thread model: posix
> gcc version 4.1.2 (Ubuntu 4.1.2-0ubuntu3)
>
> machine is AMD64
>
> OS:
> Linux 2.6.20-9-386 #2 Mon Feb 26 02:58:41 UTC 2007 i686 GNU/Linux
>
> Info from gdb:
> #0  0x08537267 in ff_fdct_sse2 ()
> #1  0xfff3fffd in ?? ()
> #2  0xfff5fff3 in ?? ()
> #3  0xfff90003 in ?? ()
> #4  0x00070009 in ?? ()
> #5  0x00150019 in ?? ()
> #6  0x000dfff9 in ?? ()
> #7  0x001d0009 in ?? ()
> #8  0xfff90009 in ?? ()
> #9  0x0020fffd in ?? ()
> #10 0xffd50003 in ?? ()
> #11 0x0017ffe0 in ?? ()
> #12 0xfffc001e in ?? ()
> #13 0xfff0fff8 in ?? ()
> #14 0x00180010 in ?? ()
> #15 0xfff80010 in ?? ()
> #16 0xffe0fff8 in ?? ()
> #17 0x00060003 in ?? ()
> #18 0x00120013 in ?? ()
> #19 0x00130007 in ?? ()
> #20 0x0004000f in ?? ()
> #21 0x00110009 in ?? ()
> #22 0x000f0011 in ?? ()
> #23 0x00150017 in ?? ()
> #24 0x00110003 in ?? ()
> #25 0x00140014 in ?? ()
> #26 0xfffe0009 in ?? ()
> #27 0xfff10004 in ?? ()
> #28 0xffe5ffed in ?? ()
> #29 0x00000000 in ?? ()

That's really strange. Looks like your stack is corrupt. This should not happen.




> (gdb) disass $pc-32 $pc+32
> Dump of assembler code from 0x8537247 to 0x8537287:
> 0x08537247 <ff_fdct_sse2+727>:  cmc
> 0x08537248 <ff_fdct_sse2+728>:  int3
> 0x08537249 <ff_fdct_sse2+729>:  paddd  %xmm7,%xmm3
> 0x0853724d <ff_fdct_sse2+733>:  paddd  %xmm2,%xmm1
> 0x08537251 <ff_fdct_sse2+737>:  paddd  %xmm6,%xmm3
> 0x08537255 <ff_fdct_sse2+741>:  paddd  %xmm6,%xmm1
> 0x08537259 <ff_fdct_sse2+745>:  psrad  $0x11,%xmm3
> 0x0853725e <ff_fdct_sse2+750>:  psrad  $0x11,%xmm1
> 0x08537263 <ff_fdct_sse2+755>:  packssdw %xmm3,%xmm1
> 0x08537267 <ff_fdct_sse2+759>:  movdqa %xmm1,(%ecx)
> 0x0853726b <ff_fdct_sse2+763>:  movq   0x40(%esp),%xmm2
> 0x08537271 <ff_fdct_sse2+769>:  movq   0x48(%esp),%xmm0
> 0x08537277 <ff_fdct_sse2+775>:  movdqa 0x20(%edx),%xmm3
> 0x0853727c <ff_fdct_sse2+780>:  movdqa 0x30(%edx),%xmm7
> 0x08537281 <ff_fdct_sse2+785>:  movq   %xmm2,%xmm1
> 0x08537285 <ff_fdct_sse2+789>:  pshuflw $0x1b,%xmm0,%xmm0
> End of assembler dump.
> (gdb) info all-registers
> eax            0x8719720        141661984
> ecx            0xbfdfe388       -1075846264

Looks like an unaligned mem access to me.

Please try attached patch. It's not a proper fix I think, but It's the
best I could do within 2min of free time ;-)

Guillaume
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ff_fdct_sse2_segfault_fix.diff
Type: text/x-patch
Size: 540 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/mplayer-users/attachments/20070305/b4e36e34/attachment.bin>