[Libav-user] H.264 decoder shooting "no picture " and "Internal error, picture buffer overflow"

[Roman Ryltsov]

Hello,

(sorry if it comes twice - the previous one was sent from wrong address)

I am seeing a problem with libavcodec's H.264 decoder, which I need some
help with.

I was using the codec to decode H.264 video in a straightforward way,
and at some point I realized that it fails to decode specific content
coming from x264 encoder. After some investigation I found that with all
the rest unchanged, if encoder applies a constraint and limits output to
be baseline compatible, things work perfectly. Should this be removed
and content is e.g. High at 3.1, or if the constraint it set to output Main
profile stream, the problem immediately comes up. I might suppose that
presence of B frames is the cause, however I have not yet checked that deep.

This "bad" input however, muxed into MP4 file, is played back well with
Windows Media Player, and Media Player Classic HC. Not VLC though. I had
a chance to pass it through another decoder in Windows and the content
was perfectly playable, which makes me think it is not that bad.

With libavcodec however I see multiple "no picture " log outputs, and
eventually fatal "Internal error, picture buffer overflow". Moreover,
the problem starts comes up immediately with the first video frame of
the stream, which contains SPS, PPS, IDR NALs and in my understanding is
supposed to be decodable immediately. Some frames are still decoded (and
without artifacts, though might be in wrong order), however the rate of
erroneous frames is pretty high (perhaps one half).

Finding this out, I could isolate it all into a few tens of lines of
code (+ embedded AVPacket data inline):

http://www.alax.info/trac/public/browser/trunk/Utilities/FFmpeg/NoPictureIssue/NoPictureIssue.cpp

I wonder if anyone is aware of this problem and there is any additional
information out there to work it around? This occurs with both Zeranoe
pre-built libraries, and those I built myself with MinGW.

This stream is Annex B formatted, without global headers (extradata),
could this be a problem and/or it is safer or otherwise recommended to
convert to length prefixes? I looked into code and I see that there is
support for both.

Any specific initialization I am missing? Since all the data is
basically one packet with a few NALs, the sequence of calls is as simple as:

av_lockmgr_register(...)
avcodec_register_all()
av_log_set_level(AV_LOG_DEBUG)
avcodec_alloc_context()
avcodec_find_decoder(CODEC_ID_H264)
avcodec_open(...)
av_init_packet(...)
avcodec_decode_video2(...)

with consumed data in avcodec_decode_video2, error logged, and no frame
availability on the output.

Thanks,
Roman