[FFmpeg-user] avcodec_decode_video2 crash HEVC with get_buffer2

[Никита Скиба]

Hello to all!

Recently I have faced a problem with decoding hevc annexB stream. Crash
happens inside avcodec_decode_video2. I also use deprecated
get_buffer/release_buffer method to decrease memory copy operations count.
While decoding h264 or jpeg, or mpeg-4 there is no problems. Crash happens
only with hevc.

I assumed that reason of those crashes is in deprecated
get_buffer/release_buffer usage and tried to upgrade sources to get_buffer2
usage. But after upgrade decoder works well with h264 and still crashes on
hevc. Without get_buffer2 it decodes well.

Could you please point me to what I am doing wrong?

Sources of "upgraded" version of get_buffer2.

static void release_buffer2(void* opaque, uint8_t* data)
{
    // pSample into opaque.
    data[0] = 0;
    data[1] = 0;
    data[2] = 0;

    if (0 != opaque)
    {
        Sample* pSample = (Sample*)(opaque);
        pSample->Release();
    }
}

static int get_buffer2(struct AVCodecContext *c, AVFrame *pic, int flags)
{
    pic->width = c->width;
    pic->height = c->height;
    pic->format = c->pix_fmt;

    CFFmpegAllocator* pObj = (CFFmpegAllocator*)(c->opaque);
    return pObj ? pObj->get_buffer2_impl(c, pic, flags) : -1;
}

int get_buffer2_impl(AVCodecContext* avctx, AVFrame *pic, int flags)
{
    int h_chroma_shift, v_chroma_shift;
    int i, unaligned;
    int size[4] = { 0 };
    int offset[4] = { 0 };

    if (!m_allocator)
    {
        return -1;
    }

    int w = pic->width,
        h = pic->height;

    const AVPixelFormat pix_fmt = static_cast<AVPixelFormat>(pic->format);

    if (av_image_check_size(w, h, 0, NULL) < 0)
    {
        return -1;
    }

    avcodec_get_chroma_sub_sample(pix_fmt, &h_chroma_shift,
&v_chroma_shift);

    // Maximum alignment required is for AVX; use it to be on the safe side
    const int stride_align[4] = { 32, 32, 32, 32 };
    {
        // inlined stripped-down version of avcodec_align_dimensions2
        const int PIXEL_PER_MACROBLOCK = 16;            // assume 16 pixel
per macroblock
        const int w_align = PIXEL_PER_MACROBLOCK;
        const int h_align = PIXEL_PER_MACROBLOCK * 2;   // interlaced needs
2 macroblocks height
        w = FFALIGN(w, w_align);
        h = FFALIGN(h, h_align);
    }

    do{
        av_image_fill_linesizes(pic->linesize, pix_fmt, w);
        w += w & ~(w - 1);

        unaligned = 0;
        for (i = 0; i < 4; i++){
            unaligned |= pic->linesize[i] % stride_align[i];
        }
    } while (unaligned);


    int image_size = av_image_fill_pointers(pic->data, pix_fmt,
        h, NULL, pic->linesize);

    if (image_size < 0)
        return -1;

    for (i = 0; i < 3 && pic->data[i + 1]; i++)
        size[i] = pic->data[i + 1] - pic->data[i];
    size[i] = image_size - (pic->data[i] - pic->data[0]);

    image_size = 0;
    for (i = 0; i < 4 && size[i]; ++i)
    {
        const int h_shift = (0 == i) ? 0 : h_chroma_shift;
        const int v_shift = (0 == i) ? 0 : v_chroma_shift;

        offset[i] = 0;

        size[i] += 16; //16 bytes at the end of each plane
        image_size += size[i] + offset[i];
    }

    Sample* pSample = m_allocator->Alloc(image_size);

    if (!pSample)
    {
        return -1;
    }

    SampleHeader& header = pSample->Header();
    header.BodySize = image_size;

    uint8_t *body = pSample->GetBody();

    pic->data[0] = body + offset[0];
    pic->buf[0] = av_buffer_create(pic->data[0] + offset[0], size[0],
release_buffer2, NULL, 0);
    for (i = 1; i < 4 && size[i]; ++i)
    {
        pic->data[i] = pic->data[i - 1] + size[i - 1] + offset[i];
        pic->buf[i] = av_buffer_create(pic->data[i] + offset[i], size[i],
release_buffer2, NULL, 0);
    }

#if (LIBAVCODEC_VERSION_MAJOR < 54)
    pic->age = INT_MAX;
#endif

    pic->type = FF_BUFFER_TYPE_USER;
    pic->opaque = pSample;

    return 0;
}

Looking forward to your answer,
Nikita