[FFmpeg-devel] [PATCH 12/19] avcodec/mpegvideo_enc: Don't segfault on unorthodox mpeg_quant

[Andreas Rheinhardt]

Michael Niedermayer:
> On Wed, Apr 07, 2021 at 11:07:54PM +0200, Andreas Rheinhardt wrote:
>> Michael Niedermayer:
>>> On Tue, Apr 06, 2021 at 11:55:09PM +0200, Andreas Rheinhardt wrote:
>>>> The (deprecated) field AVCodecContext.mpeg_quant has no range
>>>> restriction; MpegEncContext.mpeg_quant is restricted to 0..1.
>>>> If the former is set, the latter is overwritten with it without
>>>> checking the range. This can trigger an av_assert2() with the MPEG-4
>>>> encoder when writing said field.
>>>>
>>>> Fix this by just setting MpegEncContext.mpeg_quant to 1 if
>>>> AVCodecContext.mpeg_quant is set.
>>>>
>>>> Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt at outlook.com>
>>>> ---
>>>> This can not be triggered if one only sets options via the dictionary in
>>>> avcodec_open2(); one needs to set it directly in the AVCodecContext or
>>>> use the private class of AVCodecContext.
>>>
>>> I tend toward checking mpeg_quant to be valid in init somewhere but this
>>> here is fine too or in addition
>>>
>> Do you want to error out if AVCodecContext.mpeg_quant is outside of
>> 0..1? 
> 
> that was the idea, i did not test this though 
> 

I think it's not worth it for an option that will be removed on the next
bump (AVCodecContext.mpeg_quant is deprecated).

> 
>> Or do you want another check for MpegEncContext.mpeg_quant? The
>> option for the latter already uses the proper range, so I don't think we
>> need a check for that.
>>
>> - Andreas