[FFmpeg-devel] [PATCH] Fix quadratic memory use in ff_h2645_extract_rbsp() when multiple NALUs exist in packet.

[Derek Buitenhuis]

On 11/1/2017 1:36 AM, Michael Niedermayer wrote:
> The idea would be that there would only be one uint8_t buffer and the
> 2000 entries from te pool would point into that.
> So as a larger NAL shifts through the 2000 the pointers would get
> distributed differently but the size would not grow
> Any variable size buffer the H2645NAL needs would be such a "shared"
> buffer

This fixes the type of case I mentioned, but it doesn't fix something, for
example, on a never-ending stream (e.g. a TV channel), if you hit a bunch
of huge NALs in a packet, the single buffer will be resizes to fit them all,
and so instead of a momentary memory usage spike, it lasts forever (or until
you restart the stream / reinit the decoder).

- Derek