[Libav-user] Possible infinite loop in libavcodec/h264.c

[Carl Eugen Hoyos]

 <DSchere at ...> writes:

> I am tracking down a memory leak in vlc where within 
> a matter of seconds vlc consumes over 14G of memory,

Is there a sample that allows to reproduce this?
If yes, please provide it.

> the problem is intermittent.

(That is not a good sign.)

> I think that the problem may be an infinite loop in 
> libavcodec/h264.c at around line 1894 inside a while 
> loop that allocates frames:

(If you found a work-around that fixes the issue, please 
post it, even if not correct it may lead to the source 
of the problem.)

[...]

> What I did was created some diagnostic code which 
> tracked the pointers from p_picture structures 
> allocated from within VLC per thread. If the number of 
> pictures allocated over one second exceeded a threshold 
> (100 pictures) a flag was set that would raise an 
> exception on the next allocation, this would in turn 
> force a stack trace in gdb:

Is it not possible to see the backtrace without additional 
changes?
Did you already try to reproduce the issue with FFmpeg 
alone / without vlc?

[...]

> CONFIDENTIALITY NOTICE: The information contained in this electronic message
is confidential information

Please understand that this makes no sense in an email 
sent to a public mailing list that is mirrored many 
times on the internet.

Carl Eugen