[FFmpeg-user] Correct way to use FFmpeg filters in a real time context

[Ronak]

Hi all,

I'm trying to embed some ffmpeg filters, such as loudnorm and silenceremove in a real time context: meaning I have an audio speaker connected at the end of it.

I saw that loudnorm supports single pass mode; where it doesn't require me to pass all the audio in the file through it, before it can start normalizing to a given LUFS value.
I also saw silenceremove is also similar.

However, what I can't seem to figure out is that if I just plug in those filters into my filter chain during playback, I hear a bunch of silence, audio artifacts and random noise, rather than my audio.

I'm thinking this might be an issue due to how I'm inputting data into those filters perhaps?

Here's the code I use to push data into my filter chain. This is code in the Swift language on iOS. Would someone be able to help look over this code and give me any pointers on what I'm doing wrong?

Thanks,

Ronak

private func filterBuffer(_ inputBufferList: UnsafeMutableAudioBufferListPointer, frameCount: UInt32, outputBufferList: UnsafeMutableAudioBufferListPointer) throws -> UInt32 {
    guard let inputAudioFrame = inputAudioFrame, let outputAudioFrame = outputAudioFrame else { throw PlayerError.filterFailure([:]) }

    // copy the pointers to the audio buffer into the frame for manipulation

    // each buffer represents the audio per channel. copy data from the inputBuffer into the input frame.
    for index in 0..<inputBufferList.count {
      let bufferSize = Int(inputBufferList[index].mDataByteSize)
      let buffer = inputBufferList[index].mData?.bindMemory(to: UInt8.self, capacity: bufferSize)

      inputAudioFrame.pointee.nb_samples = Int32(frameCount)
      inputAudioFrame.pointee.extended_data[index] = buffer
      inputAudioFrame.pointee.linesize.0 = Int32(bufferSize)
    }

    // write the audio frame into the audioInputContext so it can be filtered
    let writeResult = av_buffersrc_write_frame(audioInputContext, inputAudioFrame)
    if writeResult == 0 {

      // pull the filtered audio out of the audioOutputContext
      let pullResult = av_buffersink_get_frame(audioOutputContext, outputAudioFrame)

      // if we were able to read the filtered audio without any issue, lets use it
      if pullResult >= 0 {

        // copy the pointers to the filtered audio into the output buffers
        for index in 0..<outputBufferList.count {
          outputBufferList[index].mData = UnsafeMutableRawPointer(outputAudioFrame.pointee.extended_data[index])
          outputBufferList[index].mDataByteSize = UInt32(outputAudioFrame.pointee.linesize.0)
          outputBufferList[index].mNumberChannels = 1
        }

        return UInt32(outputAudioFrame.pointee.nb_samples)
      } else if pullResult == -EAGAIN {
        // if the result was -EAGAIN, it means we need to write additional data into the filter graph to pull data out of it..

        return UInt32(outputAudioFrame.pointee.nb_samples)
      } else {
        let rawErrorString = UnsafeMutablePointer<Int8>.allocate(capacity: 64)
        av_make_error_string(rawErrorString, 64, pullResult)

        ARCLogError("Failed to read audio out of the filter due to error code: \(pullResult): \(String(cString: rawErrorString))")

        // the audio couldn't be filtered properly, throw an error
        throw PlayerError.filterFailure([:])
      }
    } else {
      let rawErrorString = UnsafeMutablePointer<Int8>.allocate(capacity: 64)
      av_make_error_string(rawErrorString, 64, writeResult)

      ARCLogError("Failed to write audio into the filter due to error code: \(writeResult): \(String(cString: rawErrorString))")

      // the audio couldn't be filtered properly, throw an error
      throw PlayerError.filterFailure([:])
    }