[MPlayer-dev-eng] [PATCH] vf_screenshot.c: segfault (buffer overflow)

[Guillaume POIRIER]

Hi,

On 11/15/06, Stanislav Maslovski <stanislav.maslovski at gmail.com> wrote:
> Hello,
>
> Currently, mplayer started with -vf screenshot segfaults on some video
> files of unusual resolution when "s" is pressed. I have seen this on some of
> my AVIs. An investigation has shown that this happens in yuv420_rgb24_MMX2
> conversion code:
>
> ######################### GDB log ########################################
> (gdb) run
> Starting program: /home/stas/src/svn/mplayer/gmplayer
> [Thread debugging using libthread_db enabled]
> [New Thread -1230068000 (LWP 10628)]
> MPlayer dev-SVN-r20944-4.1.2 (C) 2000-2006 MPlayer Team
> CPU: AMD Athlon(tm) XP 2500+ (Family: 6, Model: 10, Stepping: 0)
> CPUflags:  MMX: 1 MMX2: 1 3DNow: 1 3DNow2: 1 SSE: 1 SSE2: 0
> Compiled for x86 CPU with extensions: MMX MMX2 3DNow 3DNowEx SSE
> xscreensaver_disable: Could not find XScreenSaver window.
> /usr/share/fonts/truetype/msttcorefonts/Verdana.ttf doesn't look like a
> bitmap font description, ignoring.
> Cannot load bitmap font: /usr/share/fonts/truetype/msttcorefonts/Verdana.ttf
>
> Playing /mnt/f/ANIME/Complete Series/Gankutsuou/Gankutsuou 01
> [DeathSquad].avi.
> AVI file format detected.
> VIDEO:  [DX50]  708x392  24bpp  23,976 fps  947,9 kbps (115,7 kbyte/s)
> xscreensaver_disable: Could not find XScreenSaver window.
> Opening video filter: [screenshot]
> Opening video filter: [fspp=4:0:0]
> ==========================================================================
> Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family
> Selected video codec: [ffodivx] vfm: ffmpeg (FFmpeg MPEG-4)
> ==========================================================================
> ==========================================================================
> Opening audio decoder: [mp3lib] MPEG layer-2, layer-3
> AUDIO: 48000 Hz, 2 ch, s16le, 96,0 kbit/6,25% (ratio: 12000->192000)
> Selected audio codec: [mp3] afm: mp3lib (mp3lib MPEG layer-2, layer-3)
> ==========================================================================
> AO: [alsa] 48000Hz 2ch s16le (2 bytes per sample)
> Starting playback...
> VDec: vo config request - 708 x 392 (preferred colorspace: Planar YV12)
> [PP] Using external postprocessing filter, max q = 5.
> VDec: using Planar YV12 as output csp (no 0)
> Movie-Aspect is 1,81:1 - prescaling to correct movie aspect.
> SwScaler: using unscaled yuv420p -> bgr24 special converter
> VO: [xv] 708x392 => 708x392 Planar YV12
> sending VFCTRL_SCREENSHOT!007 ct:  0,002  84/ 84  5% 46%  0,6% 0 0
> *** screenshot 'shot0001.png' ***
>
> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread -1230068000 (LWP 10628)]
> 0x0820d295 in yuv420_rgb24_MMX2 (c=0x8a286e0, src=0x89bead0,
>     srcStride=0xbfa7f08c, srcSliceY=0, srcSliceH=400, dst=0xbfa7f10c,
>     dstStride=0xbfa7f098) at yuv2rgb_template.c:333
> 333                 __asm__ __volatile__ (
> (gdb) bt
> #0  0x0820d295 in yuv420_rgb24_MMX2 (c=0x8a286e0, src=0x89bead0,
>     srcStride=0xbfa7f08c, srcSliceY=0, srcSliceH=400, dst=0xbfa7f10c,
>     dstStride=0xbfa7f098) at yuv2rgb_template.c:333
> #1  0x081f92db in sws_scale_ordered (c=0x8a286e0, src=0x89bead0,
>     srcStride=0x89beae0, srcSliceY=0, srcSliceH=400, dst=0xbfa7f10c,
>     dstStride=0xbfa7f100) at swscale.c:2438
> #2  0x0814ec97 in put_image (vf=0x88fcef0, mpi=0x89caaf0,
>     pts=3.5034720897674561) at vf_screenshot.c:142
> #3  0x0810ec6e in filter_video (sh_video=0x88f8860, frame=0x890a858,
>     pts=3.5034720897674561) at dec_video.c:388
> #4  0x080871ef in main (argc=-1079503820, argv=0x3fa55ac0) at mplayer.c:4376
> (gdb) disas 0x0820d295 0x0820d295+100
> Dump of assembler code from 0x820d295 to 0x820d2f9:
> 0x0820d295 <yuv420_rgb24_MMX2+453>:     movntq %mm6,0x8(%edi)
> 0x0820d299 <yuv420_rgb24_MMX2+457>:     pshufw $0xff,%mm0,%mm5
> 0x0820d29d <yuv420_rgb24_MMX2+461>:     pshufw $0xfa,%mm2,%mm3
> 0x0820d2a1 <yuv420_rgb24_MMX2+465>:     pshufw $0xfa,%mm1,%mm6
> 0x0820d2a5 <yuv420_rgb24_MMX2+469>:     movd   0x4(%edx,%ecx,1),%mm0
> 0x0820d2aa <yuv420_rgb24_MMX2+474>:     pand   %mm7,%mm5
> 0x0820d2ad <yuv420_rgb24_MMX2+477>:     pand   %mm4,%mm3
> 0x0820d2b0 <yuv420_rgb24_MMX2+480>:     pand   0x868fe60,%mm6
> 0x0820d2b7 <yuv420_rgb24_MMX2+487>:     movd   0x4(%eax,%ecx,1),%mm1
> 0x0820d2bc <yuv420_rgb24_MMX2+492>:     por    %mm5,%mm3
> 0x0820d2bf <yuv420_rgb24_MMX2+495>:     por    %mm3,%mm6
> 0x0820d2c2 <yuv420_rgb24_MMX2+498>:     movntq %mm6,0x10(%edi)
> 0x0820d2c6 <yuv420_rgb24_MMX2+502>:     movq   0x8(%ebx,%ecx,2),%mm6
> 0x0820d2cb <yuv420_rgb24_MMX2+507>:     pxor   %mm4,%mm4
> 0x0820d2ce <yuv420_rgb24_MMX2+510>:     add    $0x18,%edi
> 0x0820d2d1 <yuv420_rgb24_MMX2+513>:     add    $0x4,%ecx
> 0x0820d2d4 <yuv420_rgb24_MMX2+516>:     js     0x820d1b7
> <yuv420_rgb24_MMX2+231>
> 0x0820d2da <yuv420_rgb24_MMX2+522>:     incl   0x8(%esp)
> 0x0820d2de <yuv420_rgb24_MMX2+526>:     add    0x20(%esp),%ebx
> 0x0820d2e2 <yuv420_rgb24_MMX2+530>:     mov    0x14(%esp),%eax
> 0x0820d2e6 <yuv420_rgb24_MMX2+534>:     mov    0x8(%esp),%edx
> 0x0820d2ea <yuv420_rgb24_MMX2+538>:     add    %eax,(%esp)
> 0x0820d2ed <yuv420_rgb24_MMX2+541>:     cmp    %edx,0x48(%esp)
> 0x0820d2f1 <yuv420_rgb24_MMX2+545>:     jne    0x820d184
> <yuv420_rgb24_MMX2+180>
> 0x0820d2f7 <yuv420_rgb24_MMX2+551>:     emms
> End of assembler dump.
> (gdb) print /x $edi
> $1 = 0xb5d7dff8
> (gdb) print  /x dst[0]
> $2 = 0xb5cb2020
> (gdb) print  ((unsigned)$edi - (unsigned)dst[0])/dstStride[0]
> $3 = 392
> ##########################################################################
>
> From here we see that the conversion code tries to continue writing to the
> dst buffer after 392nd stride [notice the height of the original image;
> the magic 400 comes from mpi->height, see patch] and then segfaults.
>
> The attached patch solves this problem.

Good patch, applied, thanks.

Guillaume
-- 
With DADVSI (http://en.wikipedia.org/wiki/DADVSI), France finally has
a lead on USA on selling out individuals right to corporations!
Vive la France!