[FFmpeg-devel] [RFC] rtpdec: Reordering RTP packets

[Martin Storsjö]

On Mon, 24 May 2010, Michael Niedermayer wrote:

> On Mon, May 24, 2010 at 05:35:50AM +0200, Luca Barbato wrote:
> > On 05/24/2010 01:46 AM, Michael Niedermayer wrote:
> > > this might work with playing a video but for lowdelay communication this
> > > design fails
> > 
> > Let me make a summary:
> > 
> > In rtp you have timestamp and sequence number. rtp mandates that every
> > packets is within the mtu since lower protocols might not be adequate to
> > handle fragmentation. That makes you have a frame spanning multiple rtp
> > packets.
> > 
> > Sometimes for some reasons you might get misordered packets, the simple
> > solution in those case is just discard them, the decoder gets garbage at
> > best and does the concealment. Another solution applicable if you can
> > tolerate some delay is keep fetching some more packets and once you get
> > all you need you feed them to the demuxer so the codec gets sane data
> > with a bit of delay (say 5-10 packets at most)
> > 
> > > There normally is some buffer, to compensate variable network delay
> > > and there is one (possible the same) buffer to compensate bitrate
> > > variation between frames (and all video since mpeg1 has variable sized
> > > frames, keyframes are much larger) thats the vbv buffer, its mandatory.
> > > currently these buffers are in the application and thus cannot be used
> > > by rtpdec.
> > > if rtpdec would return a packet if it has one when the application
> > > requests one then these buffers could be inside rtpdec and rtpdec
> > > could use them for free for reordering packets until the packets are
> > > needed by the application.
> > 
> > The behavior for the application that is accounting for network glitches
> > won't change at all. If you don't have misordering you get the frame as
> > soon you get the last packet containing it; if you have misordering the
> > frame might be ready a bit later or never.
> > 
> > That said _probably_ the current network protocol implementation might
> > be improved, right now we have a separate select loop in each _read
> > function and the application has no way to know which is the network
> > status (e.g. accessing rtcp statistics).
> > 
> > Right now ffplay works between fairly well and really well, even in
> > situations in which mplayer or vlc (with either live555 or libnemesi)
> > have problems.
> 
> please tell me if you
> A. Do understand that your design requires additional delay compared
>    to my suggestion but consider this irrelevant / too difficult to fix
> B. Do not understand that your design requires additional delay compared
>    to reordering and vbv using the same buffer

I do understand that our design requires additional delay, in the cases 
where we actually receive packets out of order - as long as they arrive in 
order, there's no extra delay at all. I also do think it's possible to do 
some kind of reordering even without delay, e.g. by doing something like 
this:

In udp_read_packet, do:
- with a nonblocking read, read all outstanding packets, enqueue and 
  reorder them
- after the nonblocking read doesn't return any more packets, return the 
  first packet in the queue
- if there's no more enqueued packets, do a blocking read
- if the blocking read returns a packet, return it immediately without 
  reordering

In that setup, one would get reordering if the packets are already in the 
kernel buffer, but if we're forced to wait for the packet, we'd just 
return it immediately. Then there's no extra delay.

That said, I'm not in favour of this approach - if we really want 
reordering, we should also be ok with some extra delay, since the out of 
order packets very well can arrive with some delay. And if we want a very 
low delay, set the max_delay parameter to a sufficiently low value to 
avoid all reordering/queuing.

// Martin