[Libav-user] Buffer overruns on new API

Daykin, Evan daykin at frib.msu.edu
Fri Apr 22 23:20:40 EEST 2022 

Hi,

I am having an odd issue migrating a module  to the new libav/libavcodec API. In the old code, a frame of raw data was encoded to JPEG as follows, after initializing the codec:

/* lock the output plugin mutex */
    this->mutex.lock();

    /* Release the last jpeg created */
    if (this->jpeg) {
        this->jpeg->release();
    }

    /* Convert it to a jpeg */
    this->jpeg = this->pNDArrayPool->alloc(1, &size, NDInt8, 0, NULL);

    AVPacket pkt;
    int got_output;
    av_init_packet(&pkt);
    pkt.data = (uint8_t*)this->jpeg->pData;    // packet data will be allocated by the encoder
    pkt.size = c->width * c->height;

    //c: AVCodecContext*
    //scPicture: AVPacket*
    //jpeg: Pointer to wrapper for an N-dimensional array
    //pkt: An AVPacket object initialized above
    //scPicture: AVFrame*
    if (avcodec_encode_video2(c, &pkt, scPicture, &got_output)) {
        asynPrint(this->pasynUserSelf, ASYN_TRACE_ERROR,
            "%s:%s: Encoding jpeg failed\n",
            driverName, functionName);
    }

    this->jpeg->dims[0].size = pkt.size;

    /* signal fresh_frame to output plugin and unlock mutex */
    epicsEvent *evt;
    while(this->waiting.tryReceive(&evt, sizeof(evt)) != -1)
        evt->signal();

    this->mutex.unlock();

This runs fine all day. I updated this snippet to use avcodec_send_frame/avcodec_receive_packet below:
    // segfault here after a few successful frames
    if (avcodec_send_frame(c,scPicture) < 0) {
        asynPrint(this->pasynUserSelf, ASYN_TRACE_ERROR,
            "%s:%s: Encoding jpeg failed\n",
            driverName, functionName);
    }

    if (avcodec_receive_packet(c, &pkt) < 0){
        asynPrint(this->pasynUserSelf, ASYN_TRACE_ERROR,
            "%s:%s: Encoding jpeg failed\n",
            driverName, functionName);
    }

    this->jpeg->dims[0].size = pkt.size;
    this->jpeg->pData = pkt.data;
In this case, a varying number of frames are correctly encoded, usually about 10. After that, something in libav is overrunning a buffer that it previously malloc'd. Here is the output from gdb and Valgrind - there are no other issues or overruns until the faulty frame occurs:

Thread 151 "MJPG_Plugin_1" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff6891fb00 (LWP 936195)]
tcache_get (tc_idx=<optimized out>) at malloc.c:2937
2937    malloc.c: No such file or directory.
(gdb) bt
#0  tcache_get (tc_idx=<optimized out>) at malloc.c:2937
#1  __GI___libc_malloc (bytes=40) at malloc.c:3051
#2  0x00007ffff78e8b42 in __posix_memalign (size=<optimized out>, alignment=<optimized out>, memptr=0x7fff6891e768) at malloc.c:5366
#3  __posix_memalign (memptr=0x7fff6891e768, alignment=<optimized out>, size=<optimized out>) at malloc.c:5353
#4  0x00007ffff5193911 in av_malloc (size=40) at libavutil/mem.c:86
#5  0x00007ffff5193ae3 in av_mallocz (size=40) at libavutil/mem.c:239
#6  0x00007ffff5183d9a in av_buffer_create
    (data=0x2f2f303033313031 <error: Cannot access memory at address 0x2f2f303033313031>, size=609047, free=0x7ffff5183d1b <pool_release_buffer>, opaque=0x7fffdc455b50, flags=825307440) at libavutil/buffer.c:36
#7  0x00007ffff5184313 in av_buffer_pool_get (pool=0x7fffdc0bdcd0) at libavutil/buffer.c:345
#8  0x00007ffff55d7f42 in video_get_buffer (s=0x7fffdc013ee0, pic=0x7fffdc01f690) at libavcodec/decode.c:1626
#9  0x00007ffff55d9dde in avcodec_default_get_buffer2 (avctx=0x7fffdc013ee0, frame=0x7fffdc01f690, flags=2) at libavcodec/decode.c:1665
#10 0x00007ffff55da3bd in ff_get_buffer (avctx=0x7fffdc013ee0, frame=0x7fffdc01f690, flags=2) at libavcodec/decode.c:1898
#11 0x00007ffff5868121 in thread_get_buffer_internal (avctx=0x7fffdc013ee0, f=0x7fffdc01c790, flags=2) at libavcodec/pthread_frame.c:924
#12 0x00007ffff5868bce in ff_thread_get_buffer (avctx=0x7fffdc013ee0, f=0x28, flags=2) at libavcodec/pthread_frame.c:1000
#13 0x00007ffff57faf03 in alloc_frame_buffer (avctx=0x7fffdc013ee0, pic=0x7fffdc01c788, me=0x2, sc=0x11, chroma_x_shift=1, chroma_y_shift=1, linesize=1952, uvlinesize=976)
    at libavcodec/mpegpicture.c:117
#14 0x00007ffff57fb2ff in ff_alloc_picture
    (avctx=0x7fffdc013ee0, pic=0x7fffdc01c788, me=0x2, sc=0x7fffdc015500, shared=0, encoding=1, chroma_x_shift=1, chroma_y_shift=1, out_format=3, mb_stride=121, mb_width=120, mb_height=75, b8_stride=241, linesize=0x7fffdc014db0, uvlinesize=0x7fffdc014db8) at libavcodec/mpegpicture.c:254
#15 0x00007ffff58086db in alloc_picture (s=0x28, pic=0x28, shared=2) at libavcodec/mpegvideo_enc.c:1149
#16 0x00007ffff5808843 in load_input_picture (s=0x7fffdc014ac0, pic_arg=0x5555557509d0) at libavcodec/mpegvideo_enc.c:1222
#17 0x00007ffff580f982 in ff_mpv_encode_picture (avctx=0x7fffdc013ee0, pkt=0x7fffdc014a10, pic_arg=0x28, got_packet=0x11) at libavcodec/mpegvideo_enc.c:1835
#18 0x00007ffff56115f1 in avcodec_encode_video2 (avctx=0x7fffdc013ee0, avpkt=0x7fffdc014a10, frame=0x5555557509d0, got_packet_ptr=0x7fff6891ec7c) at libavcodec/encode.c:300
#19 0x00007ffff56118e6 in do_encode (avctx=0x7fffdc013ee0, frame=0x5555557509d0, got_packet=0x7fff6891ec7c) at libavcodec/encode.c:370
#20 0x00007ffff5611a49 in avcodec_send_frame (avctx=0x7fffdc013ee0, frame=0x5555557509d0) at libavcodec/encode.c:419
#21 0x00007ffff7ccd1f6 in ffmpegStream::processCallbacks(NDArray*) (this=0x555555754270, pArray=0x7fff6891ece0) at ../ffmpegServer.cpp:589
#22 0x00007ffff7dc9f6f in NDPluginDriver::processTask() (this=0x555555754270) at ../NDPluginDriver.cpp:518
#23 0x00007ffff7dcb7be in NDPluginDriver::run() (this=0x555555754270) at ../NDPluginDriver.cpp:935
#24 0x00007ffff7a87baa in epicsThreadCallEntryPoint(void*) (pPvt=0x55555571b720) at ../../../src/libCom/osi/epicsThread.cpp:83
#25 0x00007ffff7a9067a in start_routine (arg=0x555557847320) at ../../../src/libCom/osi/os/posix/osdThread.c:403
#26 0x00007ffff7446ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#27 0x00007ffff795adef in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Valgrind output:
==1146775==    at 0x483F803: memmove (vg_replace_strmem.c:1270)
==1146775==    by 0x531953F: ??? (in /usr/lib/x86_64-linux-gnu/libaravis-0.6.so.0.0.0)
==1146775==    by 0x7BF30BC: ??? (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6600.8)
==1146775==    by 0x53B9EA6: start_thread (pthread_create.c:477)
==1146775==    by 0x4EECDEE: clone (clone.S:95)
==1146775==  Address 0x22d6bac8 is 5 bytes after a block of size 51,635 alloc'd
==1146775==    at 0x48386AF: malloc (vg_replace_malloc.c:306)
==1146775==    by 0x483ADE7: realloc (vg_replace_malloc.c:834)
==1146775==    by 0x7443958: av_realloc (in /usr/local/lib/libavutil.so.56.51.100)
==1146775==    by 0x743413B: av_buffer_realloc (in /usr/local/lib/libavutil.so.56.51.100)
==1146775==    by 0x637F0C7: packet_alloc (in /usr/local/lib/libavcodec.so.58.91.100)
==1146775==    by 0x638039E: av_packet_make_refcounted (in /usr/local/lib/libavcodec.so.58.91.100)
==1146775==    by 0x643F847: avcodec_encode_video2 (in /usr/local/lib/libavcodec.so.58.91.100)
==1146775==    by 0x643F8E5: do_encode (in /usr/local/lib/libavcodec.so.58.91.100)
==1146775==    by 0x643FA48: avcodec_send_frame (in /usr/local/lib/libavcodec.so.58.91.100)
==1146775==    by 0x4B491F5: ffmpegStream::processCallbacks(NDArray*) (ffmpegServer.cpp:589)
==1146775==    by 0x49E4F6E: NDPluginDriver::processTask() (NDPluginDriver.cpp:518)
==1146775==    by 0x49E67BD: NDPluginDriver::run() (NDPluginDriver.cpp:935)

Is there some memory management that happened in the old libraries, that I now need to perform manually?

Thanks,
Evan

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://ffmpeg.org/pipermail/libav-user/attachments/20220422/88981a33/attachment.htm>

More information about the Libav-user mailing list