[FFmpeg-devel] [libav-devel] [RFC] M-bit to N-bit YUV conversions

[madshi]

> There have been complaints about how ffmpeg/libav
> converts M-bit to N-bit YUV, e.g. see
> http://forum.doom9.org/showthread.php?t=161915 which
> I've been asked to look into.

Glad to see the doom9 thread ending up here!

FWIW, the problem applies to RGB, as well, not just to YUV.

> The current code in swscale is based on how 8-bit to
> 16-bit YUV conversions have been done earlier - basically
> duplicating each input byte in the output (e.g. 0xf0 -> 0xf0f0,
> 0x65 -> 0x6565). I believe this is the right thing for full
> range content.

Not necessarily. If you upscale like that, smooth gradiants
will not be perfectly smooth, anymore. IMHO, there are three
ways to upscale full range content:

(1) 10bit = 8bit << 2
(2) 10bit = round(8bit * 1023 / 255)  (similar to byte duplicating)
(3) 10bit = dither(8bit * 1023 / 255)

Solution (1) keeps gradiants smooth, but loses a bit of
dynamic range. Solution (2) maintains full dynamic range,
but gradiants are not perfectly smooth, anymore. Solution
(3) keep gradiants smooth *and* maintains full dynamic
range. However, if you try to compress (3), compression
efficiency will go down due to the dither noise. Also (3) will
perform slowest, of course.

My personal favorites are either (1) or (3), while swscale
currently does (2).

Furthermore, the chroma channel needs to be scaled
with signed math, to make sure that 8bit 128 gets properly
scaled to 10bit 512. If you use (1), though, signed math
is not necessary.

> As I can see the issues with the current code is mainly:
> 3) Dithering back to the original bit depth may not
> preserve original value.

I don't see 3) as a problem.

> a) What method should be used for conversion?

For limited range content: Simple shifting.
For full range content: Either (1) or (3). See above.

> b) Should we distinguish between limited and full range YUV?

The only way not to would be to use simple shifting everywhere, IMHO.

Best regards, Mathias.