[FFmpeg-devel] [PATCH v3 1/4] libavutil/imgutils: add utility to get plane sizes

[James Almer]

On 7/13/2020 9:19 PM, Brian Kim wrote:
> On Mon, Jul 13, 2020 at 11:22 AM James Almer <jamrial at gmail.com> wrote:
> [...]
>> You would need to cast height to size_t for this, i think, but seeing
>> av_image_check_size() currently rejects line sizes and plane sizes
>> bigger than INT_MAX, maybe we should just keep INT_MAX in the above
>> check instead (No need to send a new revision for this, i can amend it
>> before pushing with either solution. I just want your opinion).
> 
> If we move towards using size_t/ptrdiff_t, it seems to make sense to
> use SIZE_MAX so that we do not have to come back and update this
> afterwards. It seems like even if we limited the values to INT_MAX
> users would think that they should check the range again anyways based
> on the type.
> 
> However, I am fine with keeping it limited to INT_MAX to keep things
> consistent until we can actually use the full range in other places.

Left it with SIZE_MAX checks and height cast to size_t, and pushed the set.

Thanks