[FFmpeg-devel] Output buffer size in ffmpeg.c

[Michael Niedermayer]

On Sat, Jun 13, 2009 at 01:12:29PM +0100, M?ns Rullg?rd wrote:
> In ffmpeg.c, line 1926, the size of the output buffer for compressed
> data is calculated as 6*width*height+200.  The subsequent malloc is
> failing on my 64MB systems for some inputs, and is the cause of the
> redcode-demux FATE failure on AVR32.  This particular test passes if I
> enable memory overcommit.  This will not help on blackfin, however,
> since it has no MMU and hence overcommit is impossible.
> 
> Does the output buffer really need to be this big?  The only case I
> can think of that would need it is 48-bit raw.  Perhaps we should use
> a more intelligent estimate taking the codec into account.

at a low quantizer mpeg1/2 can generate quite a bit of data given
mallicious input, just consider that each dct coeff is coded as 
escape value.

Before tha allocated amount can de decreased all encoders must be
checked and possibly fixed to not overflow the buffer with any input.
(some per row or per MB check like:
  if(remaining_size < max_size_per_mb)
    fail
)

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Old school: Use the lowest level language in which you can solve the problem
            conveniently.
New school: Use the highest level language in which the latest supercomputer
            can solve the problem without the user falling asleep waiting.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20090613/dd6d956c/attachment.pgp>