[FFmpeg-devel] 10bit DNxHD vs dct_quantize_c

[Stefan Gehrer]

On 03/18/2011 06:24 PM, Joseph Artsimovich wrote:
> Hi,
>
> Having heard someone is working on a 10-bit safe (I)DCT assembly implementation, I put 10bit DNxHD decoding on hold for now.  Instead I've been working on encoding and today I finally got that working.  To get there I had to write my own DCT and quantization code, as I couldn't get the existing code working for 10bit, even having forced DCT to faandct implementation.  The existing code I am refering to is dct_quantize_c().  I've got a couple of questions regarding it.
> First of all, what's the advantage of using that complex looking quantization code in favour of using a dead-simple quantization formula from the VC-3 spec:
> quantized[u, v] = sign(X[u, v]) * floor((abs(X[u, v]) * p) / (qsf * weight_table[u, v]))
> where p is 32 for 8-bit samples and 8 for 10-bit ones.

A division is expensive to compute , the existing code avoids it by
multiplying with the inverse and adjusting things accordingly (I am
just guessing from a glance at the code).