[MPlayer-dev-eng] [patch] backports/fixes from uoti demux_mkv.c for comp_algo==3

[Yuriy Kaminskiy]

Reimar Döffinger wrote:
> On Sat, Aug 14, 2010 at 03:38:24AM +0400, Yuriy Kaminskiy wrote:
>> PS By the way, I quickly looked at lavf matroska demuxer, EXACTLY SAME integer
>> overflow problems present there...
> 
> I don't see that. I think there is one, but at least
> 1) It is not so extremely obvious as the one in the first
>    patch
I'd say less noticeable bugs are worse :-)

> 2) It can only happen if a malloc of > 1.3 GB succeeds before
Same with patch from OP :-) - to be exploitable it requires two malloc() for
more than 2G (combined) to succeed :-)

> This I think should fix it:
> Index: libavformat/matroskadec.c
> ===================================================================
> --- libavformat/matroskadec.c   (revision 24788)
> +++ libavformat/matroskadec.c   (working copy)
> @@ -901,6 +901,9 @@
>      int result = 0;
>      int olen;
>  
> +    if (pkt_size >= 10000000)
> +        return -1;
> +
>      switch (encodings[0].compression.algo) {
>      case MATROSKA_TRACK_ENCODING_COMP_HEADERSTRIP:
>          return encodings[0].compression.settings.size;

Umm... from first glance, likely yes, should be enough; there are few suspicious
places with unchecked casts uint64->int(32)->uint(32), but (UNDOCUMENTED!)
'length < INT_MAX-16' check in av_malloc should protect them from potential
exploitability (it was not obvious at first :-\).