[FFmpeg-devel] [PATCH 2/2] avcodec/libopusdec: Enable FEC/PLC

[Michael Niedermayer]

On Wed, Mar 16, 2022 at 03:00:45PM +0100, Philip-Dylan Gleonec wrote:
> Adds FEC/PLC support to libopus. The lost packets are detected as a
> discontinuity in the audio stream. When a discontinuity is used, this
> patch tries to decode the FEC data. If FEC data is present in the
> packet, it is decoded, otherwise audio is re-created through PLC.
> 
> This patch is based on Steinar H. Gunderson contribution, and corrects
> the pts computation: all pts are expressed in samples instead of time.
> This patch also adds an option "decode_fec" which enables or disables
> FEC decoding. This option is disabled by default to keep consistent
> behaviour with former versions.
> 
> A number of checks are made to ensure compatibility with different
> containers. Indeed, video containers seem to have a pts expressed in ms
> while it is expressed in samples for audio containers. It also manages
> the cases where pkt->duration is 0, in some RTP streams. This patch
> ignores data it can not reconstruct, i.e. packets received twice and
> packets with a length that is not a multiple of 2.5ms.
> 
> Signed-off-by: Philip-Dylan Gleonec <philip-dylan.gleonec at savoirfairelinux.com>
> Co-developed-by: Steinar H. Gunderson <steinar+ffmpeg at gunderson.no>
> ---
>  libavcodec/libopusdec.c | 105 +++++++++++++++++++++++++++++++++++-----
>  1 file changed, 94 insertions(+), 11 deletions(-)
> 
> diff --git a/libavcodec/libopusdec.c b/libavcodec/libopusdec.c
> index 86ef715205..66134300d2 100644
> --- a/libavcodec/libopusdec.c
> +++ b/libavcodec/libopusdec.c
> @@ -43,10 +43,15 @@ struct libopus_context {
>  #ifdef OPUS_SET_PHASE_INVERSION_DISABLED_REQUEST
>      int apply_phase_inv;
>  #endif
> +    int decode_fec;
> +    int64_t expected_next_pts;
>  };
>  
>  #define OPUS_HEAD_SIZE 19
>  

> +// Sample rate is constant as libopus always output at 48kHz
> +const AVRational opus_timebase = { 1, 48000 };

static const 


> +
>  static av_cold int libopus_decode_init(AVCodecContext *avc)
>  {
>      struct libopus_context *opus = avc->priv_data;
> @@ -134,6 +139,8 @@ static av_cold int libopus_decode_init(AVCodecContext *avc)
>      /* Decoder delay (in samples) at 48kHz */
>      avc->delay = avc->internal->skip_samples = opus->pre_skip;
>  
> +    opus->expected_next_pts = AV_NOPTS_VALUE;
> +
>      return 0;
>  }
>  
> @@ -155,25 +162,100 @@ static int libopus_decode(AVCodecContext *avc, void *data,
>  {
>      struct libopus_context *opus = avc->priv_data;
>      AVFrame *frame               = data;
> -    int ret, nb_samples;
> +    uint8_t *outptr;
> +    int ret, nb_samples = 0, nb_lost_samples = 0, nb_samples_left;
> +
> +    // If FEC is enabled, calculate number of lost samples
> +    if (opus->decode_fec &&
> +        opus->expected_next_pts != AV_NOPTS_VALUE &&
> +        pkt->pts != AV_NOPTS_VALUE &&
> +        pkt->pts != opus->expected_next_pts) {
> +        // Cap at recovering 120 ms of lost audio.
> +        nb_lost_samples = pkt->pts - opus->expected_next_pts;
> +        nb_lost_samples = FFMIN(nb_lost_samples, MAX_FRAME_SIZE);
> +        // pts is expressed in ms for some containers (e.g. mkv)
> +        // FEC only works for SILK frames (> 10ms)
> +        // Detect if nb_lost_samples is in ms, and convert in samples if it is
> +        if (nb_lost_samples > 0) {
> +            if (avc->pkt_timebase.den != 48000) {
> +                nb_lost_samples = av_rescale_q(nb_lost_samples, avc->pkt_timebase, opus_timebase);
> +            }
> +            // For FEC/PLC, frame_size has to be to have a multiple of 2.5 ms

> +            if (nb_lost_samples % (int)(2.5 / 1000 * opus_timebase.den)) {
> +                nb_lost_samples -= nb_lost_samples % (int)(2.5 / 1000 * opus_timebase.den);

something like this
nb_lost_samples % (5LL * opus_timebase.den / 2000)
would avoid the float

also if noone reacts to your patch keep pinging it

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Into a blind darkness they enter who follow after the Ignorance,
they as if into a greater darkness enter who devote themselves
to the Knowledge alone. -- Isha Upanishad
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20220617/6d7fbfb0/attachment.sig>