[FFmpeg-devel] basic DCT question

[Roman V. Shaposhnik]

Guys,

the following is probably going to show just how clueless
I am (still! :-() in the magic of DCT, but I have to ask it 
anyway.

I have always thought that the additional 3 bits that we get
out of ffmpeg's DCT is a "good thing"(tm). They give additional
precision and can be safely included in the calculations, provided
that one scales the result back at the end.

Now I realize that these 3 bits differ between the implementations
and platforms and they seem to be the culprit of DV's regression
test miscompare on PPC. Basically when the AC coefficients are
multiplied by DV's weighting factors the difference in the extra 
3 bits get carried to the visible result.

So, what's the best way to deal with it? Here's the formula that I use:

  level = (FFABS(level) * weight[i] + (1 << (dv_weight_bits+3))) >> (dv_weight_bits+4);

the most obvious/stupid way to make it work so far seems to be:
  
  level = (((FFABS(level)) >> 3) * weight[i] + (1 << (dv_weight_bits))) >> (dv_weight_bits+1);

But is this the right thing to do?

Thanks,
Roman.