[FFmpeg-devel] [PATCH 06/11] avutil/half2float: adjust conversion of NaN

[Mark Reid]

On Wed, Aug 10, 2022 at 2:53 PM Timo Rothenpieler <timo at rothenpieler.org>
wrote:

> On 10.08.2022 23:43, Andreas Rheinhardt wrote:
> > Timo Rothenpieler:
> >> On 10.08.2022 23:24, Andreas Rheinhardt wrote:
> >>> Timo Rothenpieler:
> >>>> IEEE-754 differentiates two different kind of NaNs.
> >>>> Quiet and Signaling ones. They are differentiated by the MSB of the
> >>>> mantissa.
> >>>>
> >>>> For whatever reason, actual hardware conversion of half to single
> always
> >>>> sets the signaling bit to 1 if the mantissa is != 0, and to 0 if it's
> 0.
> >>>> So our code has to follow suite or fate-testing hardware float16 will
> be
> >>>> impossible.
> >>>
> >>> What does the exr spec say about quiet and signaling nans?
> >>
> >> Not sure how exr would be involved here.
> >
> > Your patch changes the output of an exr-test. The output of the exr
> > decoder is presumably determined by the exr spec. There is after all the
> > possibility that what hardware does in hardware and what this patch does
> > in software is incompatible with what exr specifies.
>
> The exr spec just says something along the lines of analogous to
> ieee-754 floats:
>
> https://openexr.readthedocs.io/en/latest/TechnicalIntroduction.html?highlight=ieee#the-half-data-type
> It barely ever mentions NaNs, other than that they exist. Which makes
> sense, given they don't typically appear in images.
>
> The only output changed is that for how NaNs are converted.
> And given the cross-validation with multiple hardware implementations,
> I'm confident that it's correct.
>

here is openexr implementation
https://github.com/AcademySoftwareFoundation/Imath/blob/main/src/Imath/toFloat.cpp#L78
It has been a while since I check but I believe the current implementation
matches this.

The fate sample: rgb_scanline_zip_half_float_0x0_to_0xFFFF.exr was created
to test this.
it contains every possible float16 value


> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
>