[FFmpeg-devel] SAMPLE_FMT_FLT bounds

[Colin McQuillan]

I'm looking for the correct way to clip the output of a floating-point decoder.

audioconvert.c converts floats thus:
lrintf(*(const float*)pi * (1<<15))) to convert to int16_t
lrintf(*(const float*)pi * (1<<7)) + 0x80) to convert to uint8_t

qcelpdec.c clips to [-1.0, 32767.0/32768.0]. This is alright when
converting to 16-bit but will overflow when converting to 8-bit.
Should I clip to [-1, 127.0/128.0]? Or [-128.499/127, 127.499/128]?

The attached patch is just an idea.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: clip-bounds.patch
Type: text/x-diff
Size: 1807 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20090805/aa37c3f9/attachment.patch>