[FFmpeg-devel] [PATCH 0/1] libswscale: add area upscale

[Pavel Klimov]

I often encode pixelated videos from old game consoles emulators.
Usual workaround is upscale with neighbor, or exceeding upscale
followed by lanczos downscale. I knew that area upscale should
give similar effect, but didn't take effort to find out why ffmpeg
with area method doesn't give what I expected to see.
Up until recently.

Recently I did check sources to find out what is going on.
And I found in sources comment that area method falls into bilinear
when it's used for upscale. First thing I thought was: not implemented?
So, just for fun I tried to implement it. Why not? - I thought.

It was a bit challenging, because main function that makes coefficients
is using fixed point arithmetics and has no info about its units.
Also srcPos and dstPos initial values are not described.
Eventually, I just used approach where I don't really need to know much.

Idea is following. In area representation, any point is covered
by color of nearest pixel of source image. So, I just need to
integrate nearest neighbor formula over pixel width. Taking into
assumption that nearest neighborhood is correct, I integrated it.
And it worked. At least as I thought.

Then, I made python script using fractions with built-in big integers
to have perfect area upscale for reference. And made test images.
Also I made script for image comparison. It turned out that image
comming from ffmpeg is off by 4 for some pixels. So I made log of
resulting coefficients, and also made script that does same in python.
It turned out that only 4 most significant decimal digits of integer
coefficients were right. (by the way they are 53 bits of int64_t)

During some of this, I figured out that xInc is rounded fixed
point 16.16 width of pixel. So, assuming that xInc is always
calculated same, I made similar dInc but using type double,
and changed formulas taking it into account. After change
into doubles, only two least significant decimal digits were wrong.

Next, when it was working fine I thought about sharing patch
and read about fate. I've made fate, and it failed.
I didn't mention before, but I was changing existing area upscale
to not fall into bilinear and make it upscale using area method.
Here is list of failed testcases using -k switch:

hevc-paramchange-yuv420p-yuv420p10
vsynth_lena-dv-411
vsynth_lena-dnxhd-edge3-hr
vsynth_lena-dnxhd-edge2-hr
vsynth_lena-dnxhd-edge1-hr
vsynth2-dv-411
vsynth2-dnxhd-edge3-hr
vsynth2-dnxhd-edge2-hr
vsynth2-dnxhd-edge1-hr
vsynth1-dv-411
vsynth1-dnxhd-edge3-hr
vsynth1-dnxhd-edge2-hr
vsynth1-dnxhd-edge1-hr

First thoughts: why do you incorporate rescale into other stuff?
Why don't you use sources as is. But after a bit of thinking,
it didn't matter anymore. Yes you could change references and
behavior of scaling, but it doesn't fit usual cases.

Area scaling falling into bilinear upscale makes sense.
Usually you don't know for sure will you do upscale or downscale.
Or, at least you don't want to care about it. And current behavior
with area scale is win-win: if downscale - area is better,
if upscale - bilinear is better for normal video that you don't
want to be pixelated. Also, image often has planes of different size.
So, I switched to separate option falling into area downscale.

Oh, by the way. Area upscale is identical to theoretical upscale
to infinity by nearest neighbor followed by downscale by area.
But neither holds for neighbor upscale to infinity followed
by lanczos/neighbor downscale to target.

In short. I have:
1) fixed arithmetic version with bad precision
2) python perfect version, coefficients generator
3) floating point version with good precision

I don't insist on addition of area upscale method.
But if you like the idea of having it, then I could probably assist.

I provide patch that modify libswscale, because it was easiest
approach for me. Also, it naturally fits in there.

Any suggestions appreciated.

Things to consider:
Appropriate place: libswscale / .gitignore ?
Name: area-only, area-all, area-both, area-forced, area-up...
Fixed arithmetic / floating point version
Should I include change in docs at once, or in separate commit?
Should I add test in fate? I don't see lanczos or gauss tests.

Pavel Klimov (1):
  libswscale: add area upscale

 libswscale/options.c |  1 +
 libswscale/swscale.h |  1 +
 libswscale/utils.c   | 30 +++++++++++++++++++++++++++++-
 3 files changed, 31 insertions(+), 1 deletion(-)

-- 
2.17.0.windows.1