[FFmpeg-soc] [soc]: r369 - jpeg2000/j2kenc.c

[Kamil Nowosad]

Hi

On Wed, Jul 11, 2007 at 01:16:04AM +0200, Michael Niedermayer wrote:
> > @@ -817,14 +818,14 @@ static int getnmsedec_ref(int x, int bpn
> >  
> >  static double getwmsedec(int nmsedec, int bandpos, int lev, int bpno)
> >  {
> > -   static const double dwt_norms[4][10] = {
> > -    {1.000, 1.500, 2.750, 5.375, 10.68, 21.34, 42.67, 85.33, 170.7, 341.3},
> > -    {1.038, 1.592, 2.919, 5.703, 11.33, 22.64, 45.25, 90.48, 180.9},
> > -    {1.038, 1.592, 2.919, 5.703, 11.33, 22.64, 45.25, 90.48, 180.9},
> > -    {.7186, .9218, 1.586, 3.043, 6.019, 12.01, 24.00, 47.97, 95.93}};
> > +   static const int dwt_norms[4][10] = { // multiplied by 10000
> > +    {10000, 15000, 27500, 53750, 106800, 213400, 426700, 853300, 1707000, 3413000},
> > +    {10380, 15920, 29190, 57030, 113300, 226400, 452500, 904800, 1809000},
> > +    {10380, 15920, 29190, 57030, 113300, 226400, 452500, 904800, 1809000},
> > +    { 7186,  9218, 15860, 30430,  60190, 120100, 240000, 479700,  959300}};
> >  
> > -    double t = dwt_norms[bandpos][lev] * (1 << bpno);
> > -    return t * t * nmsedec / 8192.0;
> > +    int t = (dwt_norms[bandpos][lev]) * (1 << bpno);
> > +    return (double) t * (double) t * nmsedec;
> >  }
> 
> why is the returned values not int64_t or int?
> same question for the code which uses getwmsedec() ?

the values returned by the function can reach 2^81 (dwt_norms - 2^22,
1<<bpno - 2^7, nmsedec - 2^23); they're then summed up for the whole
code block [about 20 times], and for the whole tile [with current
settings it can contain 2^8 code blocks]. So I should divide all of them by a
big number so as to avoid overflows. But the function returns much
smaller values in many cases. So the precision loss would be (in my
opinion) too big.
If there exists a smart way to replace doubles with ints here, I'll
implement it. However, I don't see any.

-- 
Best regards,
Kamil Nowosad