[FFmpeg-user] avcodec_open2() fails with error "Operation not permitted"

[Juha Heinanen]

Juha Heinanen via ffmpeg-user writes:

> When it is executed, I get to log:
> 
> 05-20 15:27:56.638 13664 13664 D Baresip Lib: found h264_mediacodec
> 05-20 15:27:56.638 13664 13664 D Baresip Lib: allocated context
> 05-20 15:27:56.638 13664 13664 D Baresip Lib: avcodec_open2 error: -1 "Operation not permitted"
> 
> Any ideas why avcodec_open2() fails?

I traced this further and found that avcodec_open2() calls
mediacodec_decode_init() defined in libavcodec/mediacodecdec.c and there
the failing call is h264_set_extradata(), which returns -1:

    switch (avctx->codec_id) {
#if CONFIG_H264_MEDIACODEC_DECODER
    case AV_CODEC_ID_H264:
        codec_mime = "video/avc";
        ret = h264_set_extradata(avctx, format);
        /* ret is -1 */

And in h264_set_extradata(), the failing call is the first one:

    ret = ff_h264_decode_extradata(avctx->extradata, avctx->extradata_size,
        &ps, &is_avc, &nal_length_size, 0, avctx);

Finally, ff_h264_decode_extradata() fails in the first test:

    if (!data || size <= 0)
        return -1;

because data, i.e, avctx->extradata is NULL and its size is 0.

I haven't yet figured out why there is no extradata in avctx. Perhaps I
have missed some API call in my code?

-- Juha