[FFmpeg-devel] [PATCH 1/2] avcodec/videotoolboxenc: fix undefined behavior with rc_max_rate=0

[Kári Helgason]

Sorry for jumping in with a slightly OT question,

Am I understanding correctly that you have noticed that when using kVTCompre
ssionPropertyKey_DataRateLimits you will sometimes get random garbage from
the encoder in the output callback?

How frequently does this happen, and can you reproduce it reliably, or do
you only see it in production?

Has a radar been filed with Apple about this?

On Wed, Jul 4, 2018 at 1:44 PM Thomas Guillem <thomas at gllm.fr> wrote:

> On Wed, Jul 4, 2018, at 09:05, Thomas Guillem wrote:
> > On macOS, a zero rc_max_rate cause an error from
> > VTSessionSetProperty(kVTCompressionPropertyKey_DataRateLimits).
> >
> > on iOS (depending on device/version), a zero rc_max_rate cause invalid
> > arguments from the vtenc_output_callback after few frames and then a
> crash
> > within the VideoToolbox library.
>
> In fact, when setting a correct max_rate on iOS, you could still get
> random crashes the same way. It's happening on ios 11.4 but seems to be OK
> on iOS 12 Beta.
>
> > ---
> >  libavcodec/videotoolboxenc.c | 2 ++
> >  1 file changed, 2 insertions(+)
> >
> > diff --git a/libavcodec/videotoolboxenc.c b/libavcodec/videotoolboxenc.c
> > index ac847358ab..aa9aae7e05 100644
> > --- a/libavcodec/videotoolboxenc.c
> > +++ b/libavcodec/videotoolboxenc.c
> > @@ -1019,6 +1019,7 @@ static int vtenc_create_encoder(AVCodecContext
>  *avctx,
> >
> >      if (vtctx->codec_id == AV_CODEC_ID_H264) {
> >          // kVTCompressionPropertyKey_DataRateLimits is not available
> > for HEVC
> > +        if (max_rate > 0) {
> >          bytes_per_second_value = max_rate >> 3;
> >          bytes_per_second = CFNumberCreate(kCFAllocatorDefault,
> >                                            kCFNumberSInt64Type,
> > @@ -1058,6 +1059,7 @@ static int vtenc_create_encoder(AVCodecContext
> > *avctx,
> >              av_log(avctx, AV_LOG_ERROR, "Error setting max bitrate
> > property: %d\n", status);
> >              return AVERROR_EXTERNAL;
> >          }
> > +        }
> >
> >          if (profile_level) {
> >              status = VTSessionSetProperty(vtctx->session,
> > --
> > 2.18.0
> >
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel at ffmpeg.org
> > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>