[FFmpeg-user] av_parser_parse2() and memory allocations

[Paul B Mahol]

On Mon, Jun 12, 2023 at 4:41 PM <kamildobk at poczta.onet.pl> wrote:

> Hello,
>
> I have an application which parses ~250 realtime video streams using
> av_parser_parse2(), mostly h264 but also MJPEG.  After a few days of
> continuous operation I see a significant performace degradation due to heap
> allocations, most threads are waiting for the heap's critical section with
> a stacktrace like this:
>
>   ntdll.dll!NtWaitForSingleObject()
>  ntdll.dll!RtlpWaitForCriticalSection()
>   ntdll.dll!RtlEnterCriticalSection()
>   ntdll.dll!RtlFlushHeaps()
>   ntdll.dll!RtlAllocateHeap()
>   avutil-56.dll!_heap_alloc(unsigned __int64)
>   avutil-56.dll!malloc(unsigned __int64 size=33173)
>   avutil-56.dll!_aligned_offset_malloc(unsigned __int64 align=31, unsigned
> __int64)
>   avutil-56.dll!_aligned_malloc(unsigned __int64 size, unsigned __int64
> alignment)
>   avutil-56.dll!av_mallocz(unsigned __int64 size=33134)
>   avcodec-58.dll!ff_fast_malloc(void * ptr=0x000000ac8906c410, unsigned
> int * size=0x000000ac8906c409, unsigned __int64 min_size=741033296920, int
> zero_realloc=1712319235)
>   avcodec-58.dll!av_fast_padded_malloc(void * ptr=0x000000ad65b1e040,
> unsigned int * size=0x0000000000000000, unsigned __int64 min_size=0)
>   avcodec-58.dll!parse_nal_units(AVCodecParserContext *
> s=0x000000ac00000000, AVCodecContext * avctx=0x000000ac8906c650, const
> unsigned char * const buf=0x000000ad65b1e040, int buf_size=-1836013760)
>   avcodec-58.dll!h264_parse(AVCodecParserContext * s=0x000000acfb8cf010,
> AVCodecContext * avctx=0x0000000066100807, const unsigned char * *
> poutbuf=0x000000ac8906c658, int * poutbuf_size=0x000000ac00000001, const
> unsigned char * buf=0x000000ad65b1e040, int buf_size=31091)
> avcodec-58.dll!av_parser_parse2(AVCodecParserContext *
> s=0x000000acf10d2750, AVCodecContext * avctx=0x000007f787adf99a, unsigned
> char * * poutbuf=0x000000ac97d073d0, int * poutbuf_size=0x000007f7871205ff,
> const unsigned char * buf=0x000000ac97d0ed5b, int buf_size=3910, __int64
> pts=-9223372036854775808, __int64 dts=-9223372036854775808, __int64 pos=0)
>
>
> Restarting the application solves the problem for the next few days. It
> looks like av_parser_parse2() does a quite a lot of memory allocations
> during parsing a single frame. Smaller allocations are handled by Windows
> Low Fragmentation Heap, but parse_nal_units() always allocates a memory
> equal to the size of the packet ( 20kB - 200kB ) and then immediately frees
> it so the heap gets fragmented ( I think ).  I think this is the reason (??)
> Is there a way to avoid these allocations ? To make av_parser_parse2() use
> the previously allocated buffer or to use a custom allocator ?  Or maybe
> even a small modification of ffmpeg source code ?
>

This is using older version, can you try with 6.0 or newer?


>
>
> Thanks,
> Kamil
>
> _______________________________________________
> ffmpeg-user mailing list
> ffmpeg-user at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
> To unsubscribe, visit link above, or email
> ffmpeg-user-request at ffmpeg.org with subject "unsubscribe".
>