[FFmpeg-user] vp9_vaapi bad color range when using rgb input

[Stéphane Chauveau]

I am trying to encode a rgb input stream in real time using vaapi (so a 
screen recorder). I am trying to avoid the rgb to yuv conversion on the 
cpu. That works fine with 'hwupload,scale_vaapi=format=nv12' except that 
the color range is incorrect. The highlights and shadows are obviously 
clipped.

Here is a simple example.

# generate an input pattern as follow:

convert -negate -size 512x512 pattern:checkerboard A.png
convert -size 512x512 gradient: B.png
convert \
     -negate -size 512x512 pattern:checkerboard \
     -size 512x512 gradient: \
     -compose Mathematics -define compose:args="0,1,0.2,-0.1" -composite \
     -depth 8 -define png:color-type=2 range-pattern.png

# encode it with

ffmpeg -y -loop 1 -t 5 -i range-pattern.png -vaapi_device 
/dev/dri/renderD128  -vf 'showinfo,hwupload,scale_vaapi=format=nv12' 
-c:v h264_vaapi out.mkv

The pattern is completely missing in the top and bottom of the image. 
That is very obvious and that looks like a range issue.

I was able to solve the problem for h264_vaapi and hevc_vaapi with 
-color_range pc but that does not work for vp9_vaapi.

I know that this is not a limitation of the codec because the problem 
does not exist when the yuv conversion is done before the hwupload.

I tried setting the various color options in several filters but so far 
nothing is working (except doing the yuv conversion on cpu). Any idea?