[FFmpeg-user] Segfault in mpdecimate filter when using ssse3/mmx extensions

Ed Torbett ed.torbett at simulation-systems.co.uk
Thu Dec 19 11:43:00 CET 2013


> I cannot reproduce your crash, valgrind does not report
> anything suspicious.

Here's relevant output of valgrind for me: 
vex x86->IR: unhandled instruction bytes: 0xF3 0xF 0xBC 0xF6
==9526== valgrind: Unrecognised instruction at address 0x8a4728e.
==9526== Your program just tried to execute an instruction that Valgrind
==9526== did not recognise.  There are two possible reasons for this.
==9526== 1. Your program has a bug and erroneously jumped to a non-code
==9526==    location.  If you are running Memcheck and you just saw a
==9526==    warning about a bad jump, it's probably your program's fault.
==9526== 2. The instruction is legitimate but Valgrind doesn't handle it,
==9526==    i.e. it's Valgrind's fault.  If you think this is the case or
==9526==    you are not sure, please let us know and we'll try to fix it.
==9526== Either way, Valgrind will now raise a SIGILL signal which will
==9526== probably kill your program.
==9526== 
==9526== Process terminating with default action of signal 4 (SIGILL)
==9526==  Illegal opcode at address 0x8A4728E
==9526==    at 0x8A4728E: ??? (in /ldisk/ffmpeg/ffmpeg_sources/ffmpeg/ffmpeg_g)

Full output available at http://edward.torbett.co.uk/valgrind.log

> Could you confirm the md5sum of
> 56f371bc6e11583275d76288dc0ad9c7 for your file?

This is the correct checksum.

> This is at least unusual, is the crash reproducible with:
> $ ffmpeg -i combined.mp4 -vf mpdecimate -f null -

Yes, still crashes

> $ ffmpeg -i combined.mp4 -vf mpdecimate -vcodec mpeg4 out.avi

Still crashes.

> or does a shorter configure line help?
> $ ./configure --enable-gpl --enable-libx264 && make

Still crashes:

$ /ldisk/ffmpeg/ffmpeg_sources/ffmpeg/ffmpeg -v debug -i combined.mp4 -vf mpdecimate decimated.mp4
ffmpeg version N-41485-gf5d0398 Copyright (c) 2000-2013 the FFmpeg developers
  built on Dec 19 2013 10:39:35 with gcc 4.1.2 (GCC) 20080704 (Red Hat 4.1.2-54)
  configuration: --enable-gpl --enable-libx264
  libavutil      52. 58.101 / 52. 58.101
  libavcodec     55. 45.102 / 55. 45.102
  libavformat    55. 22.100 / 55. 22.100
  libavdevice    55.  5.102 / 55.  5.102
  libavfilter     4.  0.100 /  4.  0.100
  libswscale      2.  5.101 /  2.  5.101
  libswresample   0. 17.104 /  0. 17.104
  libpostproc    52.  3.100 / 52.  3.100
Splitting the commandline.
Reading option '-v' ... matched as option 'v' (set logging level) with argument 'debug'.
Reading option '-i' ... matched as input file with argument 'combined.mp4'.
Reading option '-vf' ... matched as option 'vf' (set video filters) with argument 'mpdecimate'.
Reading option 'decimated.mp4' ... matched as output file.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option v (set logging level) with argument debug.
Successfully parsed a group of options.
Parsing a group of options: input file combined.mp4.
Successfully parsed a group of options.
Opening an input file: combined.mp4.
[mov,mp4,m4a,3gp,3g2,mj2 @ 0xa0d1d80] Format mov,mp4,m4a,3gp,3g2,mj2 probed with size=2048 and score=100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0xa0d1d80] ISO: File Type Major Brand: isom
[mov,mp4,m4a,3gp,3g2,mj2 @ 0xa0d1d80] Before avformat_find_stream_info() pos: 2826966 bytes read:41094 seeks:1
[h264 @ 0xa0d2760] no picture
[mov,mp4,m4a,3gp,3g2,mj2 @ 0xa0d1d80] All info found
[mov,mp4,m4a,3gp,3g2,mj2 @ 0xa0d1d80] After avformat_find_stream_info() pos: 64663 bytes read:105709 seeks:2 frames:1
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'combined.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf55.9.100
  Duration: 00:00:50.08, start: 0.000000, bitrate: 451 kb/s
    Stream #0:0(und), 1, 1/12800: Video: h264 (High) (avc1 / 0x31637661), yuv420p, 720x576 [SAR 16:15 DAR 4:3], 1/25, 450 kb/s, 12.50 fps, 12.50 tbr, 12800 tbn, 25 tbc (default)
    Metadata:
      handler_name    : VideoHandler
Successfully opened the file.
Parsing a group of options: output file decimated.mp4.
Applying option vf (set video filters) with argument mpdecimate.
Successfully parsed a group of options.
Opening an output file: decimated.mp4.
Successfully opened the file.
detected 2 logical cores
[Parsed_mpdecimate_0 @ 0xa0f2280] max_drop_count:0 hi:768 lo:320 frac:0.330000
[graph 0 input from stream 0:0 @ 0xa0da0a0] Setting 'video_size' to value '720x576'
[graph 0 input from stream 0:0 @ 0xa0da0a0] Setting 'pix_fmt' to value '0'
[graph 0 input from stream 0:0 @ 0xa0da0a0] Setting 'time_base' to value '1/12800'
[graph 0 input from stream 0:0 @ 0xa0da0a0] Setting 'pixel_aspect' to value '16/15'
[graph 0 input from stream 0:0 @ 0xa0da0a0] Setting 'sws_param' to value 'flags=2'
[graph 0 input from stream 0:0 @ 0xa0da0a0] Setting 'frame_rate' to value '25/2'
[graph 0 input from stream 0:0 @ 0xa0da0a0] w:720 h:576 pixfmt:yuv420p tb:1/12800 fr:25/2 sar:16/15 sws_param:flags=2
[format @ 0xa0d9d00] compat: called with args=[yuv420p|yuvj420p|yuv422p|yuvj422p|yuv444p|yuvj444p|nv12|nv16]
[format @ 0xa0d9d00] Setting 'pix_fmts' to value 'yuv420p|yuvj420p|yuv422p|yuvj422p|yuv444p|yuvj444p|nv12|nv16'
[AVFilterGraph @ 0xa0f2980] query_formats: 4 queried, 3 merged, 0 already done, 0 delayed
[libx264 @ 0xa0d8ca0] using mv_range_thread = 88
[libx264 @ 0xa0d8ca0] using SAR=16/15
[libx264 @ 0xa0d8ca0] using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64 SlowShuffle
[libx264 @ 0xa0d8ca0] profile High, level 2.2
[libx264 @ 0xa0d8ca0] 264 - core 133 r2334 a3ac64b - H.264/MPEG-4 AVC codec - Copyleft 2003-2013 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=3 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=12 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'decimated.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf55.22.100
    Stream #0:0(und), 0, 1/12800: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv420p, 720x576 [SAR 16:15 DAR 4:3], 2/25, q=-1--1, 12800 tbn, 12.50 tbc (default)
    Metadata:
      handler_name    : VideoHandler
Stream mapping:
  Stream #0:0 -> #0:0 (h264 -> libx264)
Press [q] to stop, [?] for help
[h264 @ 0xa841260] no picture
[h264 @ 0xa886100] no picture
[Parsed_mpdecimate_0 @ 0xa0f2280] keep pts:0 pts_time:0 drop_count:-1
Segmentation fault


Here's some more info from the previous compilation (I didn't include debug symbols this time)


(gdb) disass $pc-32,$pc+32
Dump of assembler code for function sum_abs_dctelem_sse2:
0x08714d60 <sum_abs_dctelem_sse2+0>:    mov    0x4(%esp),%eax
0x08714d64 <sum_abs_dctelem_sse2+4>:    pxor   %xmm0,%xmm0
0x08714d68 <sum_abs_dctelem_sse2+8>:    pxor   %xmm1,%xmm1
0x08714d6c <sum_abs_dctelem_sse2+12>:   movdqa (%eax),%xmm2
0x08714d70 <sum_abs_dctelem_sse2+16>:   movdqa 0x10(%eax),%xmm3
0x08714d75 <sum_abs_dctelem_sse2+21>:   movdqa 0x20(%eax),%xmm4
0x08714d7a <sum_abs_dctelem_sse2+26>:   movdqa 0x30(%eax),%xmm5
0x08714d7f <sum_abs_dctelem_sse2+31>:   pxor   %xmm6,%xmm6
0x08714d83 <sum_abs_dctelem_sse2+35>:   psubw  %xmm2,%xmm6
0x08714d87 <sum_abs_dctelem_sse2+39>:   pmaxsw %xmm6,%xmm2
0x08714d8b <sum_abs_dctelem_sse2+43>:   paddusw %xmm2,%xmm0
0x08714d8f <sum_abs_dctelem_sse2+47>:   pxor   %xmm7,%xmm7
0x08714d93 <sum_abs_dctelem_sse2+51>:   psubw  %xmm3,%xmm7
0x08714d97 <sum_abs_dctelem_sse2+55>:   pmaxsw %xmm7,%xmm3
0x08714d9b <sum_abs_dctelem_sse2+59>:   paddusw %xmm3,%xmm1
0x08714d9f <sum_abs_dctelem_sse2+63>:   pxor   %xmm6,%xmm6
0x08714da3 <sum_abs_dctelem_sse2+67>:   psubw  %xmm4,%xmm6
0x08714da7 <sum_abs_dctelem_sse2+71>:   pmaxsw %xmm6,%xmm4
0x08714dab <sum_abs_dctelem_sse2+75>:   paddusw %xmm4,%xmm0
0x08714daf <sum_abs_dctelem_sse2+79>:   pxor   %xmm7,%xmm7
0x08714db3 <sum_abs_dctelem_sse2+83>:   psubw  %xmm5,%xmm7
0x08714db7 <sum_abs_dctelem_sse2+87>:   pmaxsw %xmm7,%xmm5
0x08714dbb <sum_abs_dctelem_sse2+91>:   paddusw %xmm5,%xmm1
0x08714dbf <sum_abs_dctelem_sse2+95>:   movdqa 0x40(%eax),%xmm2
0x08714dc4 <sum_abs_dctelem_sse2+100>:  movdqa 0x50(%eax),%xmm3
0x08714dc9 <sum_abs_dctelem_sse2+105>:  movdqa 0x60(%eax),%xmm4
0x08714dce <sum_abs_dctelem_sse2+110>:  movdqa 0x70(%eax),%xmm5
0x08714dd3 <sum_abs_dctelem_sse2+115>:  pxor   %xmm6,%xmm6
0x08714dd7 <sum_abs_dctelem_sse2+119>:  psubw  %xmm2,%xmm6
0x08714ddb <sum_abs_dctelem_sse2+123>:  pmaxsw %xmm6,%xmm2
0x08714ddf <sum_abs_dctelem_sse2+127>:  paddusw %xmm2,%xmm0
0x08714de3 <sum_abs_dctelem_sse2+131>:  pxor   %xmm7,%xmm7
0x08714de7 <sum_abs_dctelem_sse2+135>:  psubw  %xmm3,%xmm7
0x08714deb <sum_abs_dctelem_sse2+139>:  pmaxsw %xmm7,%xmm3
0x08714def <sum_abs_dctelem_sse2+143>:  paddusw %xmm3,%xmm1
0x08714df3 <sum_abs_dctelem_sse2+147>:  pxor   %xmm6,%xmm6
0x08714df7 <sum_abs_dctelem_sse2+151>:  psubw  %xmm4,%xmm6
0x08714dfb <sum_abs_dctelem_sse2+155>:  pmaxsw %xmm6,%xmm4
0x08714dff <sum_abs_dctelem_sse2+159>:  paddusw %xmm4,%xmm0
0x08714e03 <sum_abs_dctelem_sse2+163>:  pxor   %xmm7,%xmm7
0x08714e07 <sum_abs_dctelem_sse2+167>:  psubw  %xmm5,%xmm7
0x08714e0b <sum_abs_dctelem_sse2+171>:  pmaxsw %xmm7,%xmm5
0x08714e0f <sum_abs_dctelem_sse2+175>:  paddusw %xmm5,%xmm1
0x08714e13 <sum_abs_dctelem_sse2+179>:  paddusw %xmm1,%xmm0
0x08714e17 <sum_abs_dctelem_sse2+183>:  movhlps %xmm0,%xmm1
0x08714e1a <sum_abs_dctelem_sse2+186>:  paddusw %xmm1,%xmm0
0x08714e1e <sum_abs_dctelem_sse2+190>:  pshuflw $0xe,%xmm0,%xmm1
0x08714e23 <sum_abs_dctelem_sse2+195>:  paddusw %xmm1,%xmm0
0x08714e27 <sum_abs_dctelem_sse2+199>:  pshuflw $0x1,%xmm0,%xmm1
0x08714e2c <sum_abs_dctelem_sse2+204>:  paddusw %xmm1,%xmm0
0x08714e30 <sum_abs_dctelem_sse2+208>:  movd   %xmm0,%eax
0x08714e34 <sum_abs_dctelem_sse2+212>:  and    $0xffff,%eax
0x08714e39 <sum_abs_dctelem_sse2+217>:  ret
End of assembler dump.

(gdb) info all-registers
eax            0xbfff8fac       -1073770580
ecx            0x9e65038        166088760
edx            0x94ca404        156017668
ebx            0x9b669d8        162949592
esp            0xbfff8f2c       0xbfff8f2c
ebp            0xbfff8fac       0xbfff8fac
esi            0x9e63838        166082616
edi            0x8      8
eip            0x8714d6c        0x8714d6c <sum_abs_dctelem_sse2+12>
eflags         0x10247  [ CF PF ZF IF RF ]
cs             0x73     115
ss             0x7b     123
ds             0x7b     123
es             0x7b     123
fs             0x0      0
gs             0x33     51
st0            -nan(0x200040003)        (raw 0xffff0000000200040003)
st1            -nan(0x100030003)        (raw 0xffff0000000100030003)
st2            -nan(0x9300c000d800bf)   (raw 0xffff009300c000d800bf)
st3            -nan(0x48003f0052006d)   (raw 0xffff0048003f0052006d)
st4            -nan(0x1111111111111111) (raw 0xffff1111111111111111)
st5            1000000  (raw 0x4012f424000000000000)
st6            240000   (raw 0x4010ea60000000000000)
st7            -inf     (raw 0xffff0000000000000000)
fctrl          0x37f    895
fstat          0x120    288
ftag           0x82aa   33450
fiseg          0x73     115
fioff          0x80ad18f        134926735
foseg          0x7b     123
fooff          0xbfff8f7c       -1073770628
fop            0x35c    860
xmm0           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
    0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, uint128 = 0x00000000000000000000000000000000}
xmm1           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x0 <repeats 16 times>}, v8_int16 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
    0x0, 0x0}, v4_int32 = {0x0, 0x0, 0x0, 0x0}, v2_int64 = {0x0, 0x0}, uint128 = 0x00000000000000000000000000000000}
xmm2           {v4_float = {0x0, 0xffa8c4d3, 0x0, 0x0}, v2_double = {0x8000000000000000, 0x0}, v16_int8 = {0x18, 0x18, 0x18, 0x18, 0x5a, 0x76, 0xae, 0xca,
    0xcb, 0xcc, 0xb7, 0xa2, 0x75, 0x5c, 0x40, 0x3d}, v8_int16 = {0x1818, 0x1818, 0x765a, 0xcaae, 0xcccb, 0xa2b7, 0x5c75, 0x3d40}, v4_int32 = {0x18181818,
    0xcaae765a, 0xa2b7cccb, 0x3d405c75}, v2_int64 = {0xcaae765a18181818, 0x3d405c75a2b7cccb}, uint128 = 0x3d405c75a2b7cccbcaae765a18181818}
xmm3           {v4_float = {0x0, 0x0, 0x32121, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x18, 0x18, 0x68, 0xbc, 0xcc, 0xad, 0x69, 0x3f, 0x40, 0x48, 0x48,
    0x48, 0x43, 0x43, 0x3b, 0x38}, v8_int16 = {0x1818, 0xbc68, 0xadcc, 0x3f69, 0x4840, 0x4848, 0x4343, 0x383b}, v4_int32 = {0xbc681818, 0x3f69adcc,
    0x48484840, 0x383b4343}, v2_int64 = {0x3f69adccbc681818, 0x383b434348484840}, uint128 = 0x383b4343484848403f69adccbc681818}
xmm4           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x45, 0x0, 0x48, 0x0, 0x48, 0x0, 0x46, 0x0, 0x43, 0x0, 0x3c, 0x0,
    0x40, 0x0, 0x30, 0x0}, v8_int16 = {0x45, 0x48, 0x48, 0x46, 0x43, 0x3c, 0x40, 0x30}, v4_int32 = {0x480045, 0x460048, 0x3c0043, 0x300040}, v2_int64 = {
    0x46004800480045, 0x300040003c0043}, uint128 = 0x00300040003c00430046004800480045}
xmm5           {v4_float = {0xfedb8dd0, 0x0, 0x3212, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0x18, 0x39, 0x92, 0xcb, 0xc2, 0x8c, 0x4e, 0x3e, 0x45, 0x48,
    0x48, 0x46, 0x43, 0x3c, 0x40, 0x30}, v8_int16 = {0x3918, 0xcb92, 0x8cc2, 0x3e4e, 0x4845, 0x4648, 0x3c43, 0x3040}, v4_int32 = {0xcb923918, 0x3e4e8cc2,
    0x46484845, 0x30403c43}, v2_int64 = {0x3e4e8cc2cb923918, 0x30403c4346484845}, uint128 = 0x30403c43464848453e4e8cc2cb923918}
xmm6           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x8000000000000000, 0x0}, v16_int8 = {0xd2, 0x7, 0x15, 0x0, 0xd2, 0x7, 0x3, 0xdf, 0x5c,
    0xc1, 0xb, 0x0, 0x36, 0xff, 0xa, 0x0}, v8_int16 = {0x7d2, 0x15, 0x7d2, 0xdf03, 0xc15c, 0xb, 0xff36, 0xa}, v4_int32 = {0x1507d2, 0xdf0307d2, 0xbc15c,
    0xaff36}, v2_int64 = {0xdf0307d2001507d2, 0xaff36000bc15c}, uint128 = 0x000aff36000bc15cdf0307d2001507d2}
xmm7           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x0, 0x0}, v16_int8 = {0xff, 0x0, 0xff, 0x0, 0xff, 0x0, 0xff, 0x0, 0xff, 0x0, 0xff, 0x0,
    0xff, 0x0, 0xff, 0x0}, v8_int16 = {0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff}, v4_int32 = {0xff00ff, 0xff00ff, 0xff00ff, 0xff00ff}, v2_int64 = {
    0xff00ff00ff00ff, 0xff00ff00ff00ff}, uint128 = 0x00ff00ff00ff00ff00ff00ff00ff00ff}
mxcsr          0x1fa0   [ PE IM DM ZM OM UM PM ]
mm0            {uint64 = 0x200040003, v2_int32 = {0x40003, 0x2}, v4_int16 = {0x3, 0x4, 0x2, 0x0}, v8_int8 = {0x3, 0x0, 0x4, 0x0, 0x2, 0x0, 0x0, 0x0}}
mm1            {uint64 = 0x100030003, v2_int32 = {0x30003, 0x1}, v4_int16 = {0x3, 0x3, 0x1, 0x0}, v8_int8 = {0x3, 0x0, 0x3, 0x0, 0x1, 0x0, 0x0, 0x0}}
---Type <return> to continue, or q <return> to quit---
mm2            {uint64 = 0x9300c000d800bf, v2_int32 = {0xd800bf, 0x9300c0}, v4_int16 = {0xbf, 0xd8, 0xc0, 0x93}, v8_int8 = {0xbf, 0x0, 0xd8, 0x0, 0xc0,
    0x0, 0x93, 0x0}}
mm3            {uint64 = 0x48003f0052006d, v2_int32 = {0x52006d, 0x48003f}, v4_int16 = {0x6d, 0x52, 0x3f, 0x48}, v8_int8 = {0x6d, 0x0, 0x52, 0x0, 0x3f,
    0x0, 0x48, 0x0}}
mm4            {uint64 = 0x1111111111111111, v2_int32 = {0x11111111, 0x11111111}, v4_int16 = {0x1111, 0x1111, 0x1111, 0x1111}, v8_int8 = {0x11, 0x11,
    0x11, 0x11, 0x11, 0x11, 0x11, 0x11}}
mm5            {uint64 = 0xf424000000000000, v2_int32 = {0x0, 0xf4240000}, v4_int16 = {0x0, 0x0, 0x0, 0xf424}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
    0x24, 0xf4}}
mm6            {uint64 = 0xea60000000000000, v2_int32 = {0x0, 0xea600000}, v4_int16 = {0x0, 0x0, 0x0, 0xea60}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
    0x60, 0xea}}
mm7            {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0, 0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}




More information about the ffmpeg-user mailing list