[FFmpeg-devel] Audio conversion and floating-point codecs

[Måns Rullgård]

There is a long-standing desire from some to make the floating-point
decoders output float samples instead of converting to int16
internally, and I agree with the reasons for this.  However, making
this change hastily will make decoding orders of magnitude slower on
many CPUs.  The reason is that when a decoder outputs float samples,
the fast asm code for float-to-int conversion is not used.

In order to change the output format of these decoders without
impacting performance, we must first make a few improvements to the
avcodec API and to the generic audio format conversion code.

What we have
------------

- Very fast float-to-int16 conversion code in dsputil.  These
  functions require input scaled to -32k..32k.

- The codecs in question all scale the output to the correct range as
  part of transforms or filters.  The scaling is thus effectively free.

- Generic sample format conversion code (audioconvert.c).  This code
  requires float input in the range -1..1.  It does not use any asm
  and is thus excruciatingly slow.  Decoding wmapro on Cortex-A8
  spends more than 50% of the total time here.

What we need
------------

- The libavcodec API needs to be amended such that a specific scaling
  can be requested of the decoders.  This should probably be done
  similarly to how channel down-mixing is already handled.

- The decoders should output planar audio instead of interleaved for
  multichannel streams.  This probably means introducing
  avcodec_decode_audio4() with an AVFrame output.

- A better audio conversion system needs to be implemented.  Ideally,
  this should be able to reuse existing asm code.  To this end, the
  desired input range should be exported after configuration, allowing
  it to be passed back to decoders.

- The conversion system should be designed to allow up/down-mixing of
  channels within the same API.  This is a feature currently missing
  from FFmpeg, making playback of multi-channel AAC or Vorbis on a
  stereo output difficult.  Implementing this is not a prerequisite
  for switching the output format of the decoders.

- All the pieces above need to be tied together in ffmpeg.c.

None of the above is especially difficult to do, but it is important
that it is done properly, or performance will suffer.

-- 
M?ns Rullg?rd
mans at mansr.com