[FFmpeg-devel] [PATCH 12/19] avcodec/mpegvideo_enc: Don't segfault on unorthodox mpeg_quant

[Michael Niedermayer]

On Wed, Apr 07, 2021 at 11:07:54PM +0200, Andreas Rheinhardt wrote:
> Michael Niedermayer:
> > On Tue, Apr 06, 2021 at 11:55:09PM +0200, Andreas Rheinhardt wrote:
> >> The (deprecated) field AVCodecContext.mpeg_quant has no range
> >> restriction; MpegEncContext.mpeg_quant is restricted to 0..1.
> >> If the former is set, the latter is overwritten with it without
> >> checking the range. This can trigger an av_assert2() with the MPEG-4
> >> encoder when writing said field.
> >>
> >> Fix this by just setting MpegEncContext.mpeg_quant to 1 if
> >> AVCodecContext.mpeg_quant is set.
> >>
> >> Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt at outlook.com>
> >> ---
> >> This can not be triggered if one only sets options via the dictionary in
> >> avcodec_open2(); one needs to set it directly in the AVCodecContext or
> >> use the private class of AVCodecContext.
> > 
> > I tend toward checking mpeg_quant to be valid in init somewhere but this
> > here is fine too or in addition
> > 
> Do you want to error out if AVCodecContext.mpeg_quant is outside of
> 0..1? 

that was the idea, i did not test this though 


> Or do you want another check for MpegEncContext.mpeg_quant? The
> option for the latter already uses the proper range, so I don't think we
> need a check for that.
> 
> - Andreas
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Rewriting code that is poorly written but fully understood is good.
Rewriting code that one doesnt understand is a sign that one is less smart
than the original author, trying to rewrite it will not make it better.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20210408/ccaf192c/attachment.sig>