[MPlayer-users] patch to fix bad code (UMR) in mp3lib/layer2.c causing signal 11 in mplayer

[Nilmoni Deb]

This is in reference to the bug reported in
http://mplayerhq.hu/pipermail/mplayer-users/2003-January/027281.html .
The bug appearred for a movie whose audio is mp2 (so thats all is needed 
to reproduce the problem since the bug is in mp3lib/layer2.c).

The problem is very easy to see as follows:

In mp3lib/sr1.c, look at the line 226:

  if(fr->sampling_frequency>8) return FALSE;  // valid: 0..8

which allows fr->sampling_frequency to go up to 8.

Now, in mp3lib/layer2.c, in the function definition of II_select_table,
this code exists:

  if(fr->lsf)
    table = 4;
  else
    table = translate[fr->sampling_frequency][2-fr->stereo][fr->bitrate_index];

The problem is that in II_select_table function definition, translate is
declared as:

  static int translate[3][2][16] = .....

which means fr->sampling_frequency must be < 3 to prevent illegal memory
reads.

Obviously, the code does not bother about what would happen if
fr->sampling_frequency lies in the range [3,8]. In my test case,
fr->sampling_frequency = 3 and naturally there is a problem.

Here is a patch to fix this:

----- PATCH STARTS NEXT LINE ---------------
--- layer2.c    2003-01-08 02:20:23.000000000 +0000
+++ layer2.c.new        2003-01-23 01:49:55.000000000 +0000
@@ -241,13 +241,28 @@

 static void II_select_table(struct frame *fr)
 {
-  static int translate[3][2][16] =
-   { { { 0,2,2,2,2,2,2,0,0,0,1,1,1,1,1,0 } ,
-       { 0,2,2,0,0,0,1,1,1,1,1,1,1,1,1,0 } } ,
-     { { 0,2,2,2,2,2,2,0,0,0,0,0,0,0,0,0 } ,
-       { 0,2,2,0,0,0,0,0,0,0,0,0,0,0,0,0 } } ,
-     { { 0,3,3,3,3,3,3,0,0,0,1,1,1,1,1,0 } ,
-       { 0,3,3,0,0,0,1,1,1,1,1,1,1,1,1,0 } } };
+  static int translate[8][2][16] =
+   { { { 0,2,2,2,2,2,2,0,0,0,1,1,1,1,1,0 } ,   /*44.1 stereo*/
+       { 0,2,2,0,0,0,1,1,1,1,1,1,1,1,1,0 } } , /*44.1 mono*/
+     { { 0,2,2,2,2,2,2,0,0,0,0,0,0,0,0,0 } ,   /*48 stereo*/
+       { 0,2,2,0,0,0,0,0,0,0,0,0,0,0,0,0 } } , /*48 mono*/
+     { { 0,3,3,3,3,3,3,0,0,0,1,1,1,1,1,0 } ,   /*32 stereo*/
+       { 0,3,3,0,0,0,1,1,1,1,1,1,1,1,1,0 } } , /*32 mono*/
+     { { 2,2,2,2,2,2,2,3,3,3,3,3,3,3,3,0 } ,   /*22.05 stereo*/
+       { 2,2,2,3,3,3,3,3,3,3,3,3,3,3,3,0 } } , /*22.05 mono*/
+     { { 2,2,2,2,2,2,2,3,3,3,3,3,3,3,3,0 } ,   /*24 stereo*/
+       { 2,2,2,3,3,3,3,3,3,3,3,3,3,3,3,0 } } , /*24 mono*/
+     { { 2,2,2,2,2,2,2,3,3,3,3,3,3,3,3,0 } ,   /*16 stereo*/
+       { 2,2,2,3,3,3,3,3,3,3,3,3,3,3,3,0 } } , /*16 mono*/
+     { { 2,2,2,2,2,2,2,3,3,3,3,3,3,3,3,0 } ,   /*11.025 stereo*/
+       { 2,2,2,3,3,3,3,3,3,3,3,3,3,3,3,0 } } , /*11.025 mono*/
+     { { 2,2,2,2,2,2,2,3,3,3,3,3,3,3,3,0 } ,   /*12 stereo*/
+       { 2,2,2,3,3,3,3,3,3,3,3,3,3,3,3,0 } } , /*12 mono*/
+     { { 2,2,2,2,2,2,2,3,3,3,3,3,3,3,3,0 } ,   /*8 stereo*/
+       { 2,2,2,3,3,3,3,3,3,3,3,3,3,3,3,0 } }   /*8 mono*/
+/*       0  48  64  96 128 192 256 384 */
+/*        32  56  80 112 160 224 320  XX*/
+       };

   int table,sblim;
   static struct al_table *tables[5] =
----- PATCH ENDED PREVIOUS LINE ---------------

This is from Nick.

thanks
- Nil