[FFmpeg-devel] [PATCH] libavutil/softfloat: Fix normalizations.

[Nedeljko Babic]

>> diff --git a/libavutil/softfloat.h b/libavutil/softfloat.h
>> index 8097d28..182e517 100644
>> --- a/libavutil/softfloat.h
>> +++ b/libavutil/softfloat.h
>> @@ -47,7 +47,7 @@ static const SoftFloat FLOAT_0999999    = { 0x3FFFFBCE,   0};
>>  static av_const SoftFloat av_normalize_sf(SoftFloat a){
>>      if(a.mant){
>>  #if 1
>> -        while((a.mant + 0x20000000U)<0x40000000U){
>> +        while((FFABS(a.mant) + 0x20000000U)<0x40000000U){
>
>when exactly is this needed and how does a.mant reach that value
>for which this is needed?
>FFABS would significantly slow this down i suspect and it just looks
>wrong
>

The problem occurs for some negative numbers during normalization.

For example, please consider if 0xFFFFFFD80 is sent to av_normalize_sf.

The while loop has a condition: (a.mant + 0x20000000U)<0x40000000U.
0xFFFFD80 is cast to unsigned and we have:
0xFFFFD80 + 0x20000000 = 11D800000
and since it is working on 32 bit, it will be 1D800000 and that is smaller than
0x40000000.
So the number will enter the while loop and it will iterate until the condition
is valid.
That will occur when mantis is 0xD8000000 and this is out of valid range.

As for FFABS, I used it to minimize changes to the code and to make it as much
readable as possible.

I can use "if" before "while" (to check if numbers are positive or
negative) for increased speed, but the code will be bigger.

>
>>              a.mant += a.mant;
>>              a.exp  -= 1;
>>          }
>> @@ -68,7 +68,7 @@ static av_const SoftFloat av_normalize_sf(SoftFloat a){
>>  
>>  static inline av_const SoftFloat av_normalize1_sf(SoftFloat a){
>>  #if 1
>> -    if((int32_t)(a.mant + 0x40000000U) < 0){
>> +    if((int32_t)(a.mant + 0x40000000U) <= 0){
>
>    0x40000000 + 0x40000000U == 0x80000000 not 0 so this shouldnt
>make a difference

There is a typo in comment.
It should be : "when mantis is -0x4000000".

-Nedeljko