[FFmpeg-devel] swscale : add bitexact conv for grayf32 and gray16 to f32 conv

[Michael Niedermayer]

On Mon, Sep 17, 2018 at 09:04:06AM +0200, Martin Vignali wrote:
> > also, have you tried adding a small constant to tmp ?
> > i would expect that this or a similar operation would allow moving
> > away from all "unstable" points without really changing the output in a
> > relevant way.
> >
> >
> Can't find an op, in mult mode, in order to not raise the assert
> But if i use add calc instead of mult,
> I can have a bigger tolerance without raising the assert
> 
> In 8 bits mode
> Tolerance
> 0.0000000000001 : pass
> 0.000000000001 : fail for i = 1
> 0.00000000001 : fail for i in [1, 2, 4, 8]
> 0.0000000001 : fail for i in [1, 2, 4, 8, 16, 32, 64]
> 
> In 16 bits mode
> 0.0000000000001 : pass
> 0.000000000001 : fail for i in [1, 257, 259, 261, 263, 265, 267, 269, 271,
> 273, 275, 277, 279, 281, 65407]
> 
> using this code :
> #define assert_stable_float(x) av_assert0((float)(x+0.0000000000001) ==
> (float)(x-0.0000000000001))
> static void inline fill_uint_to_float_lut(SwsContext *c, int bitdepth) {
>     static const double float_mult8 = 1.0 / 255.0;
>     static const double float_mult16 = 1.0 / 65535.0;
>     int i;
>     double tmp = 0.;
> 
>     if (bitdepth == 8) { /*! fill uint8 to float lut */
>         for (i = 0; i < 256; ++i){
>             c->uint2float_lut[i] = (float)tmp;
>             tmp += float_mult8;
>             assert_stable_float(tmp);
>         }
>     } else if (bitdepth == 16) { /*! fill uint16 to float lut */
>         for (i = 0; i < 65536; ++i){
>             c->uint2float_lut[i] = (float)tmp;
>             tmp += float_mult16;
>             assert_stable_float(tmp);

This is very unstable, alot worse than before, also the C compiler
optimizer can replace it by a float_mult16*(i+1)

The reason why this works is probably because it skips the 0 element
and not for any reason of this being "better" (the table is just shifted by
1 element)
0 will always fail the assert, i missed this before. 0 does not need to
be tested, 0 will stay 0 normally in practice probably

also there are 2 divisions in this that you can trivially eliminate
/255 and /65535 (extra precission beyond IEEE float/double could change
these)

also the whole could be done with fewer floats and no extra complexity
for example:
int64_t tmp2 = 16843009LL * i;
(float)((double)tmp2 / (1LL<<32))
and
int64_t tmp2 = 4295032833LL * i or uint64_t 281479271743489


[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

If you think the mosad wants you dead since a long time then you are either
wrong or dead since a long time.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: not available
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20180918/8bbc61f9/attachment.sig>