[FFmpeg-user] avcodec_open2() fails with error "Operation not permitted"

[Juha Heinanen]

Juha Heinanen via ffmpeg-user writes:

> And in h264_set_extradata(), the failing call is the first one:
> 
>     ret = ff_h264_decode_extradata(avctx->extradata, avctx->extradata_size,
>         &ps, &is_avc, &nal_length_size, 0, avctx);

Since decode with standard h264 decoder works, I checked why and found
in h264_decode_init() this:

    if (avctx->extradata_size > 0 && avctx->extradata) {
        ret = ff_h264_decode_extradata(avctx->extradata, avctx->extradata_size,
                                       &h->ps, &h->is_avc, &h->nal_length_size,
                                       avctx->err_recognition, avctx);
        if (ret < 0) {
            h264_decode_end(avctx);
            return ret;
        }
    }

So it works, because it decodes extradata only if extradata exists.
mediacodec_decode_init(), on the other hand, always tries to decode
extradata no matter if it exists or not.

-- Juha