[FFmpeg-devel] [PATCH] Unbreak av_malloc_max(0) API/ABI

[Andreas Rheinhardt]

Joakim Tjernlund:
> On Tue, 2020-10-13 at 15:24 +0100, Derek Buitenhuis wrote:
>>
>> On 13/10/2020 15:19, Joakim Tjernlund wrote:
>>> For now just fixing av_malloc_max(0) will do, av_max_malloc2() etc. is beyond this patch.
>>
>> As Ronald mentioned, if you're going to fix the ABI/API, you should actually
>> fix every use of that ABI/API, not just the one you care about (0).
>>
>> So max values of 1-31 should also be handled.
> 
> OK, how far do you want to take this, will this do?
>     if (max < 32)
>         max = INT_MAX;
> 

The old behaviour was to effectively limit allocations to max - 32 in
this case (near to SIZE_MAX), not to INT_MAX. And it does not restore
default behaviour (the effective default limit was INT_MAX - 32).

> Complete API fix would need to revert the patch that caused the break to begin with and I don't think you want that ?
> 
Actually the old behaviour was against the documented API (or where do
you see the 32 mentioned in the docs?).

- Andreas

PS: My rationale for this patch was to match actual behaviour and
documented behaviour and to make the full range of INT available for
allocations if the maximum has not been overridden by av_max_alloc().
Otherwise allocations of e.g. packets with sizes in the range (INT_MAX -
31 - AV_INPUT_BUFFER_PADDING_SIZE)..(INT_MAX -
AV_INPUT_BUFFER_PADDING_SIZE) will fail.