[FFmpeg-devel] 10bit DNxHD vs dct_quantize_c

[Joseph Artsimovich]

Hi,

Having heard someone is working on a 10-bit safe (I)DCT assembly implementation, I put 10bit DNxHD decoding on hold for now.  Instead I've been working on encoding and today I finally got that working.  To get there I had to write my own DCT and quantization code, as I couldn't get the existing code working for 10bit, even having forced DCT to faandct implementation.  The existing code I am refering to is dct_quantize_c().  I've got a couple of questions regarding it.
First of all, what's the advantage of using that complex looking quantization code in favour of using a dead-simple quantization formula from the VC-3 spec:
quantized[u, v] = sign(X[u, v]) * floor((abs(X[u, v]) * p) / (qsf * weight_table[u, v]))
where p is 32 for 8-bit samples and 8 for 10-bit ones.

Second, what's up with faandct()?  It's definitely not equivalent to a textbook DCT transform, as it produces values in the wrong range.  I saw  the FAAN_POSTSCALE definition, but its effect is not documented anywhere and I can't figure out what it does on my own.


Joseph Artsimovich
Software Developer
MirriAd Ltd