[FFmpeg-devel] [PATCH 1/1] avcodec/libopusdec: Enable FEC/PLC

[Philip-Dylan Gleonec]

Andreas Rheinhardt:

 >>> Could you elaborate?
 >>> I would have expected that the normal use case is not have a
 >>> lossy input and that the new feature is always useful if data
 >>> was lost.
 >>
 >> The use-case for FEC is typically RTP stream where audio is compressed
 >> with opus. In that case, depending on the network conditions, packets
 >> can be lost. From the decoder side, this can be observed as a packet
 >> with a pts that is incremented more than we would expect.
 >>
 >> With this patch, if no discontinuity is detected, the packet is decoded
 >> as usual. If a discontinuity is detected, its duration is checked to
 >> deduct how many samples should be reconstructed. One limitation of opus
 >> FEC is that packets can be reconstructed with a granularity of 2.5ms
 >> (120 samples), so libopus can not jsut reconstruct an arbitrary number
 >> of samples. This patch manages this by "rounding" the number of lost
 >> samples to the closest number of samples that libopsu can reconstruct.
 >>
 >>> Or is there no way to distinguish between a stream with
 >>> unusual timestamps and a stream with packet loss?
 >>
 >> From my understanding, the only information at the decoder side are the
 >> metadata associated with the packet to decode, notably the pts, dts and
 >> packet duration. It does not have information about how the packet has
 >> been received (RTP, SDP, other...). Packet loss can typically be
 >> detected at higher layers (RTCP packets for RTP) but not at the decoder
 >> layer.
 >>
 >> From observation it seemed that a pts different from the last pts +
 >> packet duration would indicate that some samples were lost. The decoder
 >> than tries to restore as much sample as it can.
 >>
 >> Note that packet duration seem to change units depending on how the file
 >> is encapsulated: the same file had a packet->duration of 20 (ms) in a
 >> .mkv container but 960 (samples) in a .opus container. Since the decoder
 >> is not aware of which container is used, it has to "guess" which unit is
 >> used. As FEC is ony used with SILK frames, which have a 10ms (480
 >> samples) minimal duration, and an opus packet has a max duration of
 >> 120ms, a threshold can find how the pts is expressed.
 >
 >There is AVCodecContext.pkt_timebase.

Thanks for the feedback, I had missed that property.
I've attached a reworked version of the patch which makes use of it. It
is indeed more clean and precise. I've left a fallback when computing
the expected_next_pts in case it has not been set after creating the
AVCodecContext and left at the {0, 1} value.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-avcodec-libopusdec-Enable-FEC-PLC.patch
Type: text/x-patch
Size: 7809 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20210218/ae80d2b7/attachment.bin>