[FFmpeg-devel] [PATCH 06/11] avutil/half2float: adjust conversion of NaN

[James Almer]

On 8/10/2022 7:14 PM, Mark Reid wrote:
> On Wed, Aug 10, 2022 at 2:53 PM Timo Rothenpieler <timo at rothenpieler.org>
> wrote:
> 
>> On 10.08.2022 23:43, Andreas Rheinhardt wrote:
>>> Timo Rothenpieler:
>>>> On 10.08.2022 23:24, Andreas Rheinhardt wrote:
>>>>> Timo Rothenpieler:
>>>>>> IEEE-754 differentiates two different kind of NaNs.
>>>>>> Quiet and Signaling ones. They are differentiated by the MSB of the
>>>>>> mantissa.
>>>>>>
>>>>>> For whatever reason, actual hardware conversion of half to single
>> always
>>>>>> sets the signaling bit to 1 if the mantissa is != 0, and to 0 if it's
>> 0.
>>>>>> So our code has to follow suite or fate-testing hardware float16 will
>> be
>>>>>> impossible.
>>>>>
>>>>> What does the exr spec say about quiet and signaling nans?
>>>>
>>>> Not sure how exr would be involved here.
>>>
>>> Your patch changes the output of an exr-test. The output of the exr
>>> decoder is presumably determined by the exr spec. There is after all the
>>> possibility that what hardware does in hardware and what this patch does
>>> in software is incompatible with what exr specifies.
>>
>> The exr spec just says something along the lines of analogous to
>> ieee-754 floats:
>>
>> https://openexr.readthedocs.io/en/latest/TechnicalIntroduction.html?highlight=ieee#the-half-data-type
>> It barely ever mentions NaNs, other than that they exist. Which makes
>> sense, given they don't typically appear in images.
>>
>> The only output changed is that for how NaNs are converted.
>> And given the cross-validation with multiple hardware implementations,
>> I'm confident that it's correct.
>>
> 
> here is openexr implementation
> https://github.com/AcademySoftwareFoundation/Imath/blob/main/src/Imath/toFloat.cpp#L78
> It has been a while since I check but I believe the current implementation
> matches this.
> 
> The fate sample: rgb_scanline_zip_half_float_0x0_to_0xFFFF.exr was created
> to test this.
> it contains every possible float16 value

Then maybe the current implementation should be moved back to exr (it 
used to be internal to exr until Paul made it standalone), so this lavu 
module can match the existing hardware implementations of IEEE-734 half 
floats for the purpose of relevant pixel format support.