[FFmpeg-devel] [libav-devel] [RFC] M-bit to N-bit YUV conversions

[Ronald S. Bultje]

Hi,

On Wed, Aug 24, 2011 at 5:18 PM, Oskar Arvidsson <oskar at irock.se> wrote:
> There have been complaints about how ffmpeg/libav converts M-bit to N-bit YUV,
> e.g. see http://forum.doom9.org/showthread.php?t=161915 which I've been asked
> to look into.

Cool!

> a) What method should be used for conversion? Current method? Simple shifting?

Neither. I believe the shifting is OK, but we can do it in the same
step as hscale(). E.g. if we have 8bit input and 14bit hscale coeffs
(sum = 0x4000, or 1,0000,0000,0000,00b), then instead of shifting
input, we can scale to sum = 1,0000,0001,0000,00b - i.e. a 1 every 8th
bit, and for 10bit input it'd look like sum = 1,0000,0000,0100,00b
instead) instead to achieve the same effect. (This doesn't solve the
luma problem, which is completely orthogonal, and you already know how
to fix it anyway.) I've sort-of started working towards this by
removing the shifting in the input operations for the scaler, although
they still exist in the unscaled case. They will be reintroduced at a
later point by changing hscale coeffs sum as per above.

Dither is all just because of ordered dither and I don't feel it's
worth looking into. I've been planning to replace it with
Atkinson-style dither (that's like f-s, which x264 uses) so let's just
work towards that and be done with it.

> b) Should we distinguish between limited and full range YUV?

Doesn't convertJpegRange already do that? Are you afraid that we
become "out of range" when we scale between bitdepths with different
ranges? (I admit I haven't looked much into that yet, i.e. don't know
how to handle it yet.)

> c) Other comments, ideas?

Awesome, I appreciate any effort to make sws suck less.

Ronald