[Libav-user] avcodec_decode_video2 crash with HEVC and get_buffer2

[Никита Скиба]

Hello to all!

Recently I have faced a problem with decoding hevc annexB stream. Crash
happens inside avcodec_decode_video2. I also use deprecated
get_buffer/release_buffer method to decrease memory copy operations count.
While decoding h264 or jpeg, or mpeg-4 there is no problems. Crash happens
only with hevc.

I assumed that reason of those crashes is in deprecated
get_buffer/release_buffer usage and tried to upgrade sources to get_buffer2
usage. But after upgrade decoder works well with h264 and still crashes on
hevc. Without get_buffer2 it decodes well.

Could you please point me to what I am doing wrong with get_buffer2
implementation?

Sources of "upgraded" version of get_buffer2.

static void release_buffer2(void* opaque, uint8_t* data)
{
    // pSample into opaque.
    data[0] = 0;

    if (0 != opaque)
    {
        Sample* pSample = (Sample*)(opaque);
        pSample->Release();
    }
}

static int get_buffer2(struct AVCodecContext *c, AVFrame *pic, int flags)
{
    switch (c->codec_type)
            {
            case AVMEDIA_TYPE_VIDEO:
            {
                pic->width = c->width;
                pic->height = c->height;
                pic->format = c->pix_fmt;

                CFFmpegAllocator* pObj = (CFFmpegAllocator*)(c->opaque);
                return pObj ? pObj->get_buffer2_impl(c, pic, flags) : -1;
            }
            case AVMEDIA_TYPE_AUDIO:
                return avcodec_default_get_buffer2(c, pic, flags);
            default:
                return -1;
            }
}

int get_buffer2_impl(AVCodecContext* avctx, AVFrame *pic, int flags)
{
    int h_chroma_shift, v_chroma_shift;
    int i, unaligned;
    int size[4] = { 0 };
    int offset[4] = { 0 };

    if (!m_allocator)
    {
        return -1;
    }

    int w = pic->width,
        h = pic->height;

    const AVPixelFormat pix_fmt = static_cast<AVPixelFormat>(pic->format);

    if (av_image_check_size(w, h, 0, NULL) < 0)
    {
        return -1;
    }

    avcodec_get_chroma_sub_sample(pix_fmt, &h_chroma_shift,
&v_chroma_shift);

    // Maximum alignment required is for AVX; use it to be on the safe side
    const int stride_align[4] = { 32, 32, 32, 32 };
    {
        // inlined stripped-down version of avcodec_align_dimensions2
        const int PIXEL_PER_MACROBLOCK = 16;            // assume 16 pixel
per macroblock
        const int w_align = PIXEL_PER_MACROBLOCK;
        const int h_align = PIXEL_PER_MACROBLOCK * 2;   // interlaced needs
2 macroblocks height
        w = FFALIGN(w, w_align);
        h = FFALIGN(h, h_align);
    }

    do{
        av_image_fill_linesizes(pic->linesize, pix_fmt, w);
        w += w & ~(w - 1);

        unaligned = 0;
        for (i = 0; i < 4; i++){
            unaligned |= pic->linesize[i] % stride_align[i];
        }
    } while (unaligned);


    int image_size = av_image_fill_pointers(pic->data, pix_fmt,
        h, NULL, pic->linesize);

    if (image_size < 0)
        return -1;

    for (i = 0; i < 3 && pic->data[i + 1]; i++)
        size[i] = pic->data[i + 1] - pic->data[i];
    size[i] = image_size - (pic->data[i] - pic->data[0]);

    image_size = 0;
    for (i = 0; i < 4 && size[i]; ++i)
    {
        const int h_shift = (0 == i) ? 0 : h_chroma_shift;
        const int v_shift = (0 == i) ? 0 : v_chroma_shift;

        offset[i] = 0;

        size[i] += 16; //16 bytes at the end of each plane
        image_size += size[i] + offset[i];
    }

    Sample* pSample = m_allocator->Alloc(image_size);

    if (!pSample)
    {
        return -1;
    }

    SampleHeader& header = pSample->Header();
    header.BodySize = image_size;

    uint8_t *body = pSample->GetBody();

    pic->data[0] = body + offset[0];
    pic->buf[0] = av_buffer_create(pic->data[0] + offset[0], image_size
/*size[0]*/, release_buffer2, pSample, 0);
    for (i = 1; i < 4 && size[i]; ++i)
    {
        pic->data[i] = pic->data[i - 1] + size[i - 1] + offset[i];
        // pic->buf[i] = av_buffer_create(pic->data[i] + offset[i],
size[i], release_buffer2, NULL, 0);
    }

#if (LIBAVCODEC_VERSION_MAJOR < 54)
    pic->age = INT_MAX;
#endif

    pic->type = FF_BUFFER_TYPE_USER;
    pic->opaque = pSample;

    return 0;
}

I also found while debugging, that get_buffer2 is called two times both for
codec_type AVMEDIA_TYPE_VIDEO. Why is that second call could be happen?

Stack trace don't show anything useful cause of missing symbols for ffmpeg
libs. Crash happens inside avcodec_decode_video2 call. Will be also very
appreciated if you also point me how to make windbg accept libav symbols
and whow to build ffmpeg with symbols for Win (I use cygwin+mingw cross
compiler).

AVPacket data contain only sequentually VPS, SPS, PPS and a single
IDR_W_RADL.

Version:
ffmpeg version 0.11.5 Copyright (c) 2000-2014 the FFmpeg developers
built on Sep 26 2014 01:00:11 with gcc 4.9-win32 (GCC)
configuration: --arch=x86 --target-os=mingw32
--cross-prefix=i686-w64-mingw32- -
-disable-pthreads --enable-w32threads --disable-ffprobe --disable-ffplay
--build
-suffix=-ovs-3.1 --enable-shared --disable-static
--extra-libs='-Wl,-Bstatic -lw
inpthread -Wl,-Bdynamic' --enable-libass
--prefix=/home/gzh/sdk/mingw-i686/ffmpe
g-ovs-3.1
libavutil      54.  7.100 / 54.  7.100
libavcodec     56.  1.100 / 56.  1.100
libavformat    56.  4.101 / 56.  4.101
libavdevice    56.  0.100 / 56.  0.100
libavfilter     5.  1.100 /  5.  1.100
libswscale      3.  0.100 /  3.  0.100
libswresample   1.  1.100 /  1.  1.100

Looking forward to your answer,
Nikita
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://ffmpeg.org/pipermail/libav-user/attachments/20160212/90ba8491/attachment.html>