[MPlayer-dev-eng] demux_audio segfault

[Alan Curry]

        The story of the bug:

Once upon a time, when I was looking for endianness bugs in swscale, I had a
script that ran through various -vf format options with this loop header:

  for fmt in none rgb24 bgr24 rgb32 bgr32 yv12 yvu9

("none" was a special case where -vf format was omitted.)

This script was created in the directory where the test movies were located.

After the endianness bugs were mostly taken care of, I forgot about that old
test script, until one day I ran "mplayer *" in the video directory, and it
segfaulted. Do you see the bug yet?

It turns out the string "fmt " being anywhere near the beginning of the file
is good enough for demux_audio to think it's a WAV. The next 4 bytes are
expected to be an integer, which is immediately passed to malloc.

In this case the next 4 bytes are "in n", corresponding to a malloc of
0x6e206e69 bytes - well over 1.5 gigabytes of memory. malloc failed, NULL was
dereferenced, SEGV was received; all because I didn't call that loop variable
"format" or "f".

        The bug:

demux_audio.c:409:

    l = stream_read_dword_le(s);
    if(l < 16) {
      mp_msg(MSGT_DEMUX,MSGL_ERR,"[demux_audio] Bad wav header length: too short
 (%d)!!!\n",l);
      free_sh_audio(sh_audio);
      return 0;
    }
    sh_audio->wf = w = (WAVEFORMATEX*)malloc(l > sizeof(WAVEFORMATEX) ? l : size
of(WAVEFORMATEX));
    w->wFormatTag = sh_audio->format = stream_read_word_le(s);

There should probably be an upper limit sanity check on the size of l. There
should definitely be a don't-dereference-NULL check on the result of the
malloc.