[Libav-user] avcodec_decode_video2 strange behaviour on lost frame

[serdel]

I implemented a simple algorithm which kept track of previous frame and the
current one and checked if the sequence is correct. If it wasn't I didn't
put the frame to the avcodec_decode_video2 function and stayed that way
until next sequence has started. It brought the desired results until one
unlucky moment where there was no missing packets in live555, but the main
frame still got decoded incorrectly (positive return value and 0 as
got_picture_ptr) and the same situation all over again. Why doesn't
flushing the buffer help here? I am now thinking about reinitializing the
whole ffmpeg on that occurrence, but it seems brutal...


On Wed, 28 Nov 2012 18:48:45 +0200, Alex Cohn <alexcohn at netvision.net.il>
wrote:
> On Wed, Nov 28, 2012 at 2:27 PM, serdel <serdel at toya.net.pl> wrote:
> 
>> Hi,
>>
>> I am using ffmpeg to decode an rtps (live555) streaming. I decode each
>> frame separately in a standard sequence. The normal sequence is SPS and
>> PPS, then NAL_IDR_SLICE and finaly NAL_SLICE until next SPS&PPS. On
>> normal
>> work, avcodec_decode_video2 returns positive value of bytes used and
sets
>> the got_picture_ptr to the positive value. On SPS&PPS the return value
is
>> -1 and got_picture_ptr is 0. This is a normal sequence and works
>> perfectly
>> without any errors or lags. The problem appears when some packet is lost
>> in
>> frame transmission (WiFi transmission with some noise around). Live555
>> rejects the rest of the packets for this frame so the entire frame is
>> lost.
>> In almost all of the cases it is the  NAL_IDR_SLICE frame since it is
>> usually about 20 times bigger than NAL_SLICE frames. So when the next
>> NAL_SLICE frame arrives complete the avcodec_decode_video2 returns
>> positive
>> value of used bytes, but the got_picture_ptr is 0 - not a normal
>> behaviour.
>> The worse thing is that this continues to happen to every next NAL_SLICE
>> frame and to the next NAL_IDR_SLICE with it's following NAL_SLICE
frames.
>> This keeps on going a couple of 'full seqences' (SPS&PPS,NAL_IDR_SLIC
and
>> following NAL_SLICE) and finally on some of the frames jumps bakc to
>> normal
>> work (avcodec_decode_video2 return is positive and got_picture_ptr is
>> also
>> non-zero positive). On this occur the screen picture freezes for 2-3
>> seconds and continuous running with no jump_over! You don't see any jump
>> in
>> the screen which would indicate some lost frames - no, the picture keeps
>> on
>> from where it froze and is just delayed to the sender stream by those
2-3
>> seconds.
>>
>> I tried detecting this situation and when it occurs I run
>> avcodec_flush_buffers() followed by  avcodec_decode_video2 with a dummy
>> empty AVPacket to somehow reset the coded state but it didn't help. Does
>> any one have any other suggestions?
>>
> 
> It looks as if the decoder expects an I-frame after SPS&PPS, so it
refuses
> to accept P-frames. This does not readily explain your finding that it
> takes a couple of "full sequences", but this is at least some start.
> 
> The remedy then would be to delay feeding SPS&PPS to decoder until you
have
> a complete I-frame from RTPS. If LIVE555 throwed this frame away, then
> throw away SPS&PPS and proceed with the following frames.
> 
> Note that H264 stream is not error-resilient, and you can see all kinds
of
> artifacts when some frames are missing. Things can become especially
nasty
> if the stream uses B-frames (not in 'Ultrafast' profile). But at least
this
> way you will hopefully not have the movie freeze for few seconds.
> 
> Sincerely,
> Alex