[FFmpeg-devel] [PATCH v2] avcodec/h264, videotoolbox: fix crash after VT decoder fails

[Aman Gupta]

On Tue, Feb 21, 2017 at 1:04 PM, Ronald S. Bultje <rsbultje at gmail.com>
wrote:

> Hi,
>
> On Tue, Feb 21, 2017 at 1:48 PM, Aman Gupta <ffmpeg at tmm1.net> wrote:
>
>> diff --git a/libavcodec/h264dec.c b/libavcodec/h264dec.c
>> index 41c0964392..a0ae632fed 100644
>> --- a/libavcodec/h264dec.c
>> +++ b/libavcodec/h264dec.c
>> @@ -850,7 +850,12 @@ static int output_frame(H264Context *h, AVFrame
>> *dst, H264Picture *srcp)
>>      AVFrame *src = srcp->f;
>>      const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(src->format);
>>      int i;
>> -    int ret = av_frame_ref(dst, src);
>> +    int ret;
>> +
>> +    if (src->format == AV_PIX_FMT_VIDEOTOOLBOX && src->buf[0]->size == 1)
>> +        return AVERROR_INVALIDDATA;
>> +
>> +    ret = av_frame_ref(dst, src);
>>      if (ret < 0)
>>          return ret;
>
>
> This is a total hack :) Is there a way to hide this into VT-specific code
> outside h264*.[ch]?
>

The way the VT hwaccel works is a total hack, as noted in my commit message.

AFAICT, given how the hwaccel APIs work, there's no way to do this outside
the h264 decoder. But I'm happy to try fixing this a different way if
anyone has a suggestion.


>
> Ronald
>