[FFmpeg-user] The future of video

[Oliver Fromme]

I'm not sure this is on-topic for the ffmpeg users list ...
Anyway, I'll try to comment on some of your points.

Mark Filipak wrote:
 >    I interviewed at NVIDIA when it was a start-up. My interview wound up being an argument over 
 > sprite graphics, which they held in contempt. I did not get the job.
 > 
 >    Well, guess what? The so-called "slice prediction" in AVC is sprite graphics. It's real image but 
 > it's sprites. At Atari, we did sprites in hardware only because the 6502 was 8 bits running at 
 > 1.193[18..] MHz (i.e., 3.579[54..]/3), far too slow.

I'm not sure if there's an official definition, but "sprites" is the
common term for 2D objects in video games that are overlaid over a
background by hardware, usually combined with collision-detection.
(BTW, I programmed sprites on the C64 and Amiga ... long time ago.)

Calling MPEG slices "sprites" is a little far-fetched in my opinion.

By the way, in the new H.266 standard (a.k.a. VVC), slices can be
transformed in various ways, like shearing, scaling and rotation.
An encoder that makes clever use of that feature is able to improve
compression efficiency considerably, although it also increases the
complexity (read: the encoder is slower).

 >    To fill human field of vision, including peripheral vision, screen resolution needs to be 
 > approximately 12000x6000. If viewers are closer to the screen so that FOV+ is filled, then 
 > resolution needs to be proportionately greater.

Typical video content, like movies, documentaries and so on, is not
meant to fill the FOV completely.  This is not the creators' intent.
Actually, the recommended maximum viewing angle of movies is usually
30 degrees, in order to reduce head movement and eye movement, so
watching the movie is comfortable to everyone.  And 4k resolution is
more than enough for that.

Yes, there are video domes, 180° cinemas and similar things, but
they're not intended for normal movies that run 100 minutes or more.
(My favorite theme park -- Europa-Park -- has several of these.)

However, there's one application where the FOV should be as large as
possible:  virtual reality (including augmented and mixed reality).
I own a Meta Quest 3 that has 100° FOV, which is pretty good.
But viewing 2D movies (or normal TV) with the Quest 3 is done in
a virtual cinema (there are actually quite a lot of different apps
for this) that has a screen with a smaller viewing angle that is
comfortable for watching a 100-minutes action movie.

 >    To match human persistence of vision, the refresh rate of individual pels actually needs to be no 
 > greater than 10 or 12 refreshes per second.

I think you're confusing things.

At about 10 fps, the human brain switches from "sequence of separate
frames" to "continuous movement" (although the exact number varies
individually; 10 fps is just an approximate average).
But that doesn't mean it looks good.  It only means that the brain
accepts it as continuous, even though it's far from looking smooth.

The typical cinema frame rate (23.976 fps) is the minimum that's
commonly accepted, even though there are people with sensitive vision
who do not regard this frame rate as smooth.  There have been some
experiments with higher frame rates (like 48 fps), and I think this
will become more common in the future.  Some people criticize it
because it doesn't look "cinema-like", but that's just a matter of
getting used to it.  Like people got used to the fact that CDs don't
have the crackling noise of vinyl records.

 > Only the dots that change need be refreshed and only to 
 > the extent that they change. Of course, if sub-dots are electrically dynamic instead of electrically 
 > static, then TVs need to have some sort of dynamic refresh that's opaque to the input. Bulk screen 
 > refresh at the input is undesirable because it would flash at 10 or 12 Hz.

That's already the case with modern TV screens.  The physical refresh
rate of the panel is usually independent from the frame rate of the
video source (although it *can* be synchronized, e.g. for games).
It's sometimes an integer multiple, but it doesn't have to.  Modern
OLED panels support physical refresh rates of 120 Hz and more -- at
those rates, the frame rate of the video source is pretty much
irrelevant.

I'm not sure if there are static displays that are suitable for movie
playback.  E-paper (a.k.a. e-ink) displays are static and don't have
to refresh regularly at all, but these are too slow for video.

 >    To take best advantage of compression, pels should be HSV, not RGB.

No.  HSV has no meaning for video because it has no advantages.

For video, chroma-luma-separated color models are used, like YUV or
YCbCr.  These have several advantages.  First, they're close to the
perception of the human eyes that also have separate sensory cells
for chrominance and luminance.  Second, they allow for better
compression efficiency because the luma components can be subsampled.
Also, the compression software can take advantage of the separate
luminance component in various ways, for example when optimizing the
search for predictors or when handling film grain.

RGB only has a meaning for output and input.  Display hardware has
red/green/blue subpixels (some -- like WOLED -- also have white
subpixels, but these are handled in hardware), so the color model
that is finally brought to the screen has to be RGB.  Similar on the
input side:  Camera sensors have red/green/blue subpixels, so the
color model of the raw input is RGB.

Other than that, RGB has no meaning for video.  For example, when you
transcode a video with ffmpeg, e.g. re-encode it from H.264 to H.265,
RGB is not involved at all.

 >    TVs need to have built-in hardware to do HSV-to-R -G -B conversions that are weighted to match 
 > the optical properties of their RGB sub-dots.

Yes, they have that, except it's not HSV but YUV/YCbCr.

Best regards
 -- Oliver

PS: Note that I chose not to comment on some of your points, but that
doesn't mean I agree with them.  :-)