[FFmpeg-user] The internal ffmpeg rgb=>yuv pipeline and conversion matrices

[Andy Furniss]

Peter Rabbitson wrote:
> Hello!
>
> I am having trouble finding info on how to properly control color
> reproduction, both in terms of "reference manual" and in terms of
> "best practice guides". If I missed a resource neatly explaining
> everything I am confused about below, please do not hesitate to
> "RTFM" me ;)
>
> == Background

I see others have answered already. I am just a user/lurker who doesn't
do anything "for real" adding some random thoughts.

> I am looking to produce high quality 720p H264 streams derived from
> RGB source material. Both the source and 99.999% of the playback
> environments are 24bit RGB color platforms (pc/mobile/etc displaying
> a youtube/vimeo/etc window in a browser)

Hmm so what is RGB :-) Probably a very corner case but PC screens use
sRGB which AIUI is the same as rec. 709 but with different expected
screen gamma. In theory someone using colour management may see
different things playing 709 as a result.

>
> == Setup
>
> For simplicity let's construct a non-sensical RGB source:
>
> ffmpeg -y -r 30 \ -filter_complex 'color=c=00aa00:s=320x200;
> color=c=00aa00:s=1280x720' \ -c:v libx264 -pix_fmt yuv444p -profile:v
> high444 -qp 0 -preset:v ultrafast \ -t 5 greenz.mkv

As Paul says using color src is special a bit (in what tests I did)
though it's OK for your test, but maybe testing with some real source
would be better as you may see different results.

>
> This will generate two identical 5 second video streams of the
> #00AA00 green, with the only difference being the screen size
> (ultra-sub-SD vs HD). If then we render these streams on-screen, and
> add a pure RGB "control output" and take a screenshot we end up with
> the attached image.
>
> The 320x200 is displayed properly, although the color is slightly
> off: #00AB01. I assume that this loss is due to the
> inherently-imperfect RGB24 => YUV444P transition, and there isn't
> much one can do about it.

I got 00AA01 with mpv for 601 and 00AB02 forcing 709 (720p).

You do loose many colours doing rgb to yuv, but in addition, by default
ffmpeg will scale to tv levels - which is good, as by default players
will assume yuv is tv levels and stretch them back to full range.

>
> However the 720p playback is *way* off, at #009000. I can "correct"
> it by playing via `mpv --vid=2 --vf=format=colormatrix=bt.601
> greenz.mkv` or by explicitly tagging the H264 bitstream at encoding
> time via `... -colorspace bt470bg...`.
>
> == Questions
>
> - What is the "default" conversion matrix used within ffmpeg when
> nothing else is specified. Is it (apparently) bt.601, or does it
> default to different things depending on input?

It seems so.

>
> - If it is bt.601 is it bt470bg or is it smpte170m or in the matrix
> case it doesn't matter?

Pass.

> - Is the visual change during playback (via mpv in this case) of the
>  "untagged" streams based solely on the output size? Or is there
> another reason why stream1 looks different than stream2 with default
> settings?

mpv is guessing based on size as other players may or may not.

IIRC mplayer doesn't, but even then as players offload csc/scale to h/w
if possible there may be variation in behavior as the h/w drivers may
guess based on size.

> - Is it "ok" to produce 720p video explicitly tagged at bt.601? I ran
>  across multiple writeups on the net suggesting 720p not encoded (is
> this the right term?) in bt.709 is a big no-no.

709 would probably be best - but players nay still vary.


> - Regardless of what the answer to the previous question is: If I
> wanted to produce the same example video in bt.709 and have it play
> correctly in both cases - what would the ffmpeg incantation be?

With your example either adding (x2)

,colormatrix=bt601:bt709

or

,scale=in_color_matrix=bt709:out_color_matrix=bt709

worked for me - for real use I guess the latter would be better.

> - Am I correct that the inconsistency #00AA00 => #00AB01 is simply
> "cost of doing YUV business", or is the reason more
> nuanced/involved?

The scale as well, to test you can use the scale filter to ask for full
(though this didn't seem to work in combination with color src)

> - Anything else I missed/need to know? :)

yuv444p is not very compatible it may just fail or the player may need
to convert to 420 anyway.

In the words of Poynton (IIRC) the only reason to convert rgb to yuv is
to enable sub-sampling (though I guess it may be better supported than
if you actually encoded as rgb  with libx264rgb)

> Thank you in advance for your help!