[FFmpeg-devel] SAMPLE_FMT_FLT bounds

[Michael Niedermayer]

On Wed, Aug 05, 2009 at 05:17:41PM +0100, Colin McQuillan wrote:
> I'm looking for the correct way to clip the output of a floating-point decoder.
> 
> audioconvert.c converts floats thus:
> lrintf(*(const float*)pi * (1<<15))) to convert to int16_t
> lrintf(*(const float*)pi * (1<<7)) + 0x80) to convert to uint8_t
> 
> qcelpdec.c clips to [-1.0, 32767.0/32768.0]. This is alright when
> converting to 16-bit but will overflow when converting to 8-bit.
> Should I clip to [-1, 127.0/128.0]? Or [-128.499/127, 127.499/128]?

id say convert should be
lrintf(*(const float*)pi * 32767.5 - 0.5)
lrintf(*(const float*)pi * 127.5 + 127.5)

that maps max to max and min to min, no cliping

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Frequently ignored awnser#1 FFmpeg bugs should be sent to our bugtracker. User
questions about the command line tools should be sent to the ffmpeg-user ML.
And questions about how to use libav* should be sent to the libav-user ML.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20090806/81af0c0a/attachment.pgp>