[Ffmpeg-devel] lavc decoding of network transmitted frames - packet loss

[Michael Niedermayer]

Hi

On Sat, Oct 07, 2006 at 09:59:53AM +1000, leon zadorin wrote:
> Hi all
> 
> In short, I am using lavc to
> encode video frames (e.g.  avcodec_encode_video)
> and then transmitting the encoded data over
> network (evenly dividing the data stream into UDP
> packets). Transmitted data is then read by the
> receiving code and frames are reconstructed prior
> to being passed to avcodec_decode_video... Of
> course, the problem is when some UDP packets are
> lost on their way to the receiver.
> 
> Ideally I would like to be able to know which
> parts of the partially decoded frame (i.e. which
> MACROBLOCKS) were decoded OK and then use those in
> combination with OTHER macroblocks from OTHER
> FRAMES to reconstruct an image without any bad
> areas (i.e. this final image might have
> macroblocks from different times/frames, but all
> of such macroblocks represent a correctly decoded
> data at some point in time).
> 
> My next few questions are as follows:
> 
> 1) is there a way to register an error_callback
> with lavc during a decoding process so that lavc
> will call it when it detects an error on a given
> macroblock. For example, when decoding various
> H263 or mpeg2 streams there are numerous printouts
> of text similar to: "Error on MB 123"... it would
> be nice to be able to place a hook into this
> functionality (this way client app can build a
> vector of invalid macroblocks and then do not use
> corresponding image areas from the decoded image).
> In other words, a function  like:
> "void error_callback (int mvNumber)"...  this
> could be matched by an accessor to also get the
> macroblock dimensions (width and height)...

lavc does not know which macroblocks are damaged it just detects errors
and an error means that everything after it until the next slice is 
certainly lost but the detected error is generally a few dozen macroblocks
after the point were the real error is
at the end of the frame if there where any errors the decoder will use
libavcodec/error_resilience.c to fill in the detected damaged areas from 
the spatial surrounding and previous frame

so if you think you can do better, then look at error_resilience.c
and improve it
and dont forget to send a patch if you are successfull in improving it


> 
> 2) I know that we can use rtp callbacks during
> encoding to determine the explicit number of
> macroblocks of a given payload packet prior to
> sending it over the network. But during the
> decoding - is there a way to tell the decoder
> which macroblocks it should be decoding (as
> opposed to decoding the whole, possibly corrupt,
> frame with avcodec_decode_video)... For example,
> if I had received only the macroblocks 1, 2, 3 and
> 5 (thus macroblock 4 has been lost) - is there a
> way for me to present the decoder with only the
> data for macroblocks 1, 2, 3, and 5 and tell it to
> decode those? In other words, if I am able to tell
> which macroblocks were lost during the
> transmission of the network, how could I
> communicate this information to lavc decoding
> processes?

has the frame been encoded with multiple slices?
if not then everything after the first damaged mb is lost and is useless
if it has been encoded with multplie slices then the decoder should find them
and decode them if you just concatenate what you have


[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

In the past you could go to a library and read, borrow or copy any book
Today you'd get arrested for mere telling someone where the library is