[FFmpeg-user] av_parser_parse2() and memory allocations

[kamildobk at poczta.onet.pl]

Hello,
 
I have an application which parses ~250 realtime video streams using av_parser_parse2(), mostly h264 but also MJPEG.  After a few days of continuous operation I see a significant performace degradation due to heap allocations, most threads are waiting for the heap's critical section with a stacktrace like this: 
 
  ntdll.dll!NtWaitForSingleObject() 
 ntdll.dll!RtlpWaitForCriticalSection()
  ntdll.dll!RtlEnterCriticalSection()
  ntdll.dll!RtlFlushHeaps() 
  ntdll.dll!RtlAllocateHeap() 
  avutil-56.dll!_heap_alloc(unsigned __int64)
  avutil-56.dll!malloc(unsigned __int64 size=33173) 
  avutil-56.dll!_aligned_offset_malloc(unsigned __int64 align=31, unsigned __int64)
  avutil-56.dll!_aligned_malloc(unsigned __int64 size, unsigned __int64 alignment) 
  avutil-56.dll!av_mallocz(unsigned __int64 size=33134) 
  avcodec-58.dll!ff_fast_malloc(void * ptr=0x000000ac8906c410, unsigned int * size=0x000000ac8906c409, unsigned __int64 min_size=741033296920, int zero_realloc=1712319235) 
  avcodec-58.dll!av_fast_padded_malloc(void * ptr=0x000000ad65b1e040, unsigned int * size=0x0000000000000000, unsigned __int64 min_size=0) 
  avcodec-58.dll!parse_nal_units(AVCodecParserContext * s=0x000000ac00000000, AVCodecContext * avctx=0x000000ac8906c650, const unsigned char * const buf=0x000000ad65b1e040, int buf_size=-1836013760)
  avcodec-58.dll!h264_parse(AVCodecParserContext * s=0x000000acfb8cf010, AVCodecContext * avctx=0x0000000066100807, const unsigned char * * poutbuf=0x000000ac8906c658, int * poutbuf_size=0x000000ac00000001, const unsigned char * buf=0x000000ad65b1e040, int buf_size=31091) 
avcodec-58.dll!av_parser_parse2(AVCodecParserContext * s=0x000000acf10d2750, AVCodecContext * avctx=0x000007f787adf99a, unsigned char * * poutbuf=0x000000ac97d073d0, int * poutbuf_size=0x000007f7871205ff, const unsigned char * buf=0x000000ac97d0ed5b, int buf_size=3910, __int64 pts=-9223372036854775808, __int64 dts=-9223372036854775808, __int64 pos=0) 
 
 
Restarting the application solves the problem for the next few days. It looks like av_parser_parse2() does a quite a lot of memory allocations during parsing a single frame. Smaller allocations are handled by Windows Low Fragmentation Heap, but parse_nal_units() always allocates a memory equal to the size of the packet ( 20kB - 200kB ) and then immediately frees it so the heap gets fragmented ( I think ).  I think this is the reason (??)
Is there a way to avoid these allocations ? To make av_parser_parse2() use the previously allocated buffer or to use a custom allocator ?  Or maybe even a small modification of ffmpeg source code ?
 
 
Thanks,
Kamil