[FFmpeg-devel] [RFC] rtpdec: Reordering RTP packets

[Martin Storsjö]

On Wed, 19 May 2010, Michael Niedermayer wrote:

> The obvious way to handle streaming (overall not just libav) is that
> 1. packets are received and put in a buffer
> 2. each packet is decoded and displayed once its time has come
> 3. the initial delay is set so that display likely wont get stuck and
>    that the delay doesnt annoy the user too much
> 4. if the next packet is unavailable by the time it is needed then
>    either one has to decode without it or one has to wait which implicates
>    that the delay between receive and display from now on is larger
> 5. Clocks used to decode/display have to be synchronized between server
>    and client (using SCR/PCR in mpeg and NTP in rtp)
> 
> I might be missing some things and there may be better and different
> approuches and i dont mind at all if its done differently.
> But i do mind if its done in a way that is worse in terms of delay vs.
> reordering resistance because i dont think the world needs another rtp
> implementation that doesnt work properly.
> 
> now if we assume the packet que is inside lavf, with any que between
> lavf and lavc in the user app being as small as possible then the
> application will call av_read_frame() when it _needs_ the next packet
> and the rtp code simply has to return the next packet in this case.
> i see no complexity here.
> This might need minor adjustments in ffplay related to que sizes and
> some 2 line initial delay buildup loop between av_open* and av_read_frame()
> and we may need a way for the user app to indicate which stream it needs
> a packet from first for av_read_frame().
> But it really does not look all that complex.
> As said dont mind at all to do this differently if the end result is
> equally good from the users point of view in performance and features.
> I also dont mind if just a subset of this is implemented to keep work
> and complexity small.
> But i do mind if work is going of toward some dead end that doesnt get
> us closer to a proper implementation

While I agree with you in general, there are a few complications in doing 
it this way in the particular case with RTP/UDP.

Since UDP packets are dropped silently if not received in due time, the 
receiving thread should spend most its time blocking in select()/recv() or 
something similar, so that packets are received by the application and not 
dropped by some kernel queue. But that's no problem with the current 
design, by adjusting the queue sizes, I think.

But for reordering, one frame can be fragmented over numerous packets, 
that all have the same timestamp. When these packets are received out of 
order, they are fed into a parser within lavf, that merges them together 
to reconstruct the full original frame. The only clue on how to reorder 
them properly is the RTP sequence number, which is only available within 
rtpdec and not exposed outside of that - unless you want to expose such 
RTP-specific details in the general code of course.

So given this, I still think the reordering should be done at the rtpdec 
layer.

// Martin