[MPlayer-users] SIGSEGV: mencoder + mpeg2video + vpass=2

[byavl]

Hi All.

It's me again with SIGSEGV :)

Troble is a function in mjpeg.c:
static int mjpeg_decode_frame(AVCodecContext *avctx,
                               void *data, int *data_size,
                               uint8_t *buf, int buf_size)

There is code where don't check buffer pointer position
to leave bounds into "while{}" construction.
Of course SIGSEGV welcome.

[avl at avl argasi]$ gdb mencoder --core=core.14215
GNU gdb Red Hat Linux (5.3post-0.20021129.18rh)
This GDB was configured as "i386-redhat-linux-gnu"...
Core was generated by `mencoder argasi96_raw1_25.mjpg -o 
argasi96_raw1_25.mjpg.mpeg -vf hqdn3d=2:1:2,p'.
Program terminated with signal 11, Segmentation fault.

#0  0x081b575f in mjpeg_decode_frame (avctx=0x865fe20, data=0x865fd30, 
data_size=0x30, buf=0x881aec0 "ЪьЪЮ",
     buf_size=48) at mjpeg.c:1837
1837    mjpeg.c: No such file or directory.
         in mjpeg.c

mjpeg.c
line 1822:

    /* unescape buffer of SOS */
    if (start_code == SOS)
    {
        uint8_t *src = buf_ptr;
        uint8_t *dst = s->buffer;

        while (src<buf_end)
        {
            uint8_t x = *(src++);

            *(dst++) = x;
            if (x == 0xff)
            {
                while(*src == 0xff) src++;

line 1837:    x = *(src++); <-- SIGSEGV here,but causer is prev. while.
                if (x >= 0xd0 && x <= 0xd7)
                    *(dst++) = x;
                else if (x)
                    break;
            }
        }
        init_get_bits(&s->gb, s->buffer, (dst - s->buffer)*8);

        dprintf("escaping removed %d bytes\n",
            (buf_end - buf_ptr) - (dst - s->buffer));
    }
    else
        init_get_bits(&s->gb, buf_ptr, (buf_end - buf_ptr)*8);



===
Aleksey Lekhtin
byavl at papillon.ru   ICQ: 15146863, AIM: byavl