[MPlayer-dev-eng] [PATCH] vf_screenshot.c: segfault (buffer overflow)
Stanislav Maslovski
stanislav.maslovski at gmail.com
Wed Nov 15 22:38:50 CET 2006
Hello,
Currently, mplayer started with -vf screenshot segfaults on some video
files of unusual resolution when "s" is pressed. I have seen this on some of
my AVIs. An investigation has shown that this happens in yuv420_rgb24_MMX2
conversion code:
######################### GDB log ########################################
(gdb) run
Starting program: /home/stas/src/svn/mplayer/gmplayer
[Thread debugging using libthread_db enabled]
[New Thread -1230068000 (LWP 10628)]
MPlayer dev-SVN-r20944-4.1.2 (C) 2000-2006 MPlayer Team
CPU: AMD Athlon(tm) XP 2500+ (Family: 6, Model: 10, Stepping: 0)
CPUflags: MMX: 1 MMX2: 1 3DNow: 1 3DNow2: 1 SSE: 1 SSE2: 0
Compiled for x86 CPU with extensions: MMX MMX2 3DNow 3DNowEx SSE
xscreensaver_disable: Could not find XScreenSaver window.
/usr/share/fonts/truetype/msttcorefonts/Verdana.ttf doesn't look like a
bitmap font description, ignoring.
Cannot load bitmap font: /usr/share/fonts/truetype/msttcorefonts/Verdana.ttf
Playing /mnt/f/ANIME/Complete Series/Gankutsuou/Gankutsuou 01
[DeathSquad].avi.
AVI file format detected.
VIDEO: [DX50] 708x392 24bpp 23,976 fps 947,9 kbps (115,7 kbyte/s)
xscreensaver_disable: Could not find XScreenSaver window.
Opening video filter: [screenshot]
Opening video filter: [fspp=4:0:0]
==========================================================================
Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family
Selected video codec: [ffodivx] vfm: ffmpeg (FFmpeg MPEG-4)
==========================================================================
==========================================================================
Opening audio decoder: [mp3lib] MPEG layer-2, layer-3
AUDIO: 48000 Hz, 2 ch, s16le, 96,0 kbit/6,25% (ratio: 12000->192000)
Selected audio codec: [mp3] afm: mp3lib (mp3lib MPEG layer-2, layer-3)
==========================================================================
AO: [alsa] 48000Hz 2ch s16le (2 bytes per sample)
Starting playback...
VDec: vo config request - 708 x 392 (preferred colorspace: Planar YV12)
[PP] Using external postprocessing filter, max q = 5.
VDec: using Planar YV12 as output csp (no 0)
Movie-Aspect is 1,81:1 - prescaling to correct movie aspect.
SwScaler: using unscaled yuv420p -> bgr24 special converter
VO: [xv] 708x392 => 708x392 Planar YV12
sending VFCTRL_SCREENSHOT!007 ct: 0,002 84/ 84 5% 46% 0,6% 0 0
*** screenshot 'shot0001.png' ***
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1230068000 (LWP 10628)]
0x0820d295 in yuv420_rgb24_MMX2 (c=0x8a286e0, src=0x89bead0,
srcStride=0xbfa7f08c, srcSliceY=0, srcSliceH=400, dst=0xbfa7f10c,
dstStride=0xbfa7f098) at yuv2rgb_template.c:333
333 __asm__ __volatile__ (
(gdb) bt
#0 0x0820d295 in yuv420_rgb24_MMX2 (c=0x8a286e0, src=0x89bead0,
srcStride=0xbfa7f08c, srcSliceY=0, srcSliceH=400, dst=0xbfa7f10c,
dstStride=0xbfa7f098) at yuv2rgb_template.c:333
#1 0x081f92db in sws_scale_ordered (c=0x8a286e0, src=0x89bead0,
srcStride=0x89beae0, srcSliceY=0, srcSliceH=400, dst=0xbfa7f10c,
dstStride=0xbfa7f100) at swscale.c:2438
#2 0x0814ec97 in put_image (vf=0x88fcef0, mpi=0x89caaf0,
pts=3.5034720897674561) at vf_screenshot.c:142
#3 0x0810ec6e in filter_video (sh_video=0x88f8860, frame=0x890a858,
pts=3.5034720897674561) at dec_video.c:388
#4 0x080871ef in main (argc=-1079503820, argv=0x3fa55ac0) at mplayer.c:4376
(gdb) disas 0x0820d295 0x0820d295+100
Dump of assembler code from 0x820d295 to 0x820d2f9:
0x0820d295 <yuv420_rgb24_MMX2+453>: movntq %mm6,0x8(%edi)
0x0820d299 <yuv420_rgb24_MMX2+457>: pshufw $0xff,%mm0,%mm5
0x0820d29d <yuv420_rgb24_MMX2+461>: pshufw $0xfa,%mm2,%mm3
0x0820d2a1 <yuv420_rgb24_MMX2+465>: pshufw $0xfa,%mm1,%mm6
0x0820d2a5 <yuv420_rgb24_MMX2+469>: movd 0x4(%edx,%ecx,1),%mm0
0x0820d2aa <yuv420_rgb24_MMX2+474>: pand %mm7,%mm5
0x0820d2ad <yuv420_rgb24_MMX2+477>: pand %mm4,%mm3
0x0820d2b0 <yuv420_rgb24_MMX2+480>: pand 0x868fe60,%mm6
0x0820d2b7 <yuv420_rgb24_MMX2+487>: movd 0x4(%eax,%ecx,1),%mm1
0x0820d2bc <yuv420_rgb24_MMX2+492>: por %mm5,%mm3
0x0820d2bf <yuv420_rgb24_MMX2+495>: por %mm3,%mm6
0x0820d2c2 <yuv420_rgb24_MMX2+498>: movntq %mm6,0x10(%edi)
0x0820d2c6 <yuv420_rgb24_MMX2+502>: movq 0x8(%ebx,%ecx,2),%mm6
0x0820d2cb <yuv420_rgb24_MMX2+507>: pxor %mm4,%mm4
0x0820d2ce <yuv420_rgb24_MMX2+510>: add $0x18,%edi
0x0820d2d1 <yuv420_rgb24_MMX2+513>: add $0x4,%ecx
0x0820d2d4 <yuv420_rgb24_MMX2+516>: js 0x820d1b7
<yuv420_rgb24_MMX2+231>
0x0820d2da <yuv420_rgb24_MMX2+522>: incl 0x8(%esp)
0x0820d2de <yuv420_rgb24_MMX2+526>: add 0x20(%esp),%ebx
0x0820d2e2 <yuv420_rgb24_MMX2+530>: mov 0x14(%esp),%eax
0x0820d2e6 <yuv420_rgb24_MMX2+534>: mov 0x8(%esp),%edx
0x0820d2ea <yuv420_rgb24_MMX2+538>: add %eax,(%esp)
0x0820d2ed <yuv420_rgb24_MMX2+541>: cmp %edx,0x48(%esp)
0x0820d2f1 <yuv420_rgb24_MMX2+545>: jne 0x820d184
<yuv420_rgb24_MMX2+180>
0x0820d2f7 <yuv420_rgb24_MMX2+551>: emms
End of assembler dump.
(gdb) print /x $edi
$1 = 0xb5d7dff8
(gdb) print /x dst[0]
$2 = 0xb5cb2020
(gdb) print ((unsigned)$edi - (unsigned)dst[0])/dstStride[0]
$3 = 392
##########################################################################
From here we see that the conversion code tries to continue writing to the
dst buffer after 392nd stride [notice the height of the original image;
the magic 400 comes from mpi->height, see patch] and then segfaults.
The attached patch solves this problem.
--
Станислав
-------------- next part --------------
Index: libmpcodecs/vf_screenshot.c
===================================================================
--- libmpcodecs/vf_screenshot.c (revision 20944)
+++ libmpcodecs/vf_screenshot.c (working copy)
@@ -139,7 +139,7 @@
dst[0] = priv->buffer;
dst[1] = dst[2] = 0;
- sws_scale_ordered(priv->ctx, mpi->planes, mpi->stride, 0, mpi->height, dst, dst_stride);
+ sws_scale_ordered(priv->ctx, mpi->planes, mpi->stride, 0, priv->dh, dst, dst_stride);
}
static void start_slice(struct vf_instance_s* vf, mp_image_t *mpi){
More information about the MPlayer-dev-eng
mailing list