[FFmpeg-devel] [PATCH] Fix quadratic memory use in ff_h2645_extract_rbsp() when multiple NALUs exist in packet.

[Kieran Kunhya]

On Tue, 31 Oct 2017 at 22:36 Niki Bowe <nbowe-at-google.com at ffmpeg.org>
wrote:

> I was going to prepare a patch, but it sounds like Kieran is going to.
> Thank you Kieran. Let me know if there's anything you want me to do, or if
> you would prefer me to do it for the experience.
>
> As for the NAL memory pool, I think I see what you mean. You are saying
> h264dec.c is reusing the H2645Packet packet in decode_nal_units, so any
> growth in nals_allocated from previous packets carries over to later
> packets, and also the nals themselves also carry over even if the current
> packet has small pkt->nb_nals.
> Moving rbsp_buffer (and maybe skipped_bytes_pos) out of the nals and into
> H2645Packet should fix the worst of this.
>

I am not sure this is possible. I will see tomorrow.
Just FYI we are regularly seeing OOMs on some 24/7 live streams as a result
of h2645_parser.

Kieran