[FFmpeg-user] SSIM filter showing (small) differences in identical(?) streams

[Ian Pilcher]

I am trying to understand how the SSIM and VMAF filters work, with an
eye to finding the "best" compression settings for a video which will
be composed from a series of TIFF images.  Unfortunately, I'm stuck at
the beginning, as I can't get the SSIM filter to behave as expected.

./source contains the original sequence of images.

  $ tiffinfo source/000000.tif
  TIFF Directory at offset 0x473108 (4665608)
    Image Width: 1440 Image Length: 1080
    Bits/Sample: 8
    Sample Format: unsigned integer
    Compression Scheme: None
    Photometric Interpretation: RGB color
    Samples/Pixel: 3
    Rows/Strip: 1
    Planar Configuration: single image plane

I attempt to create a lossless video of the first minute.

  $ ffmpeg -start_number 0 -framerate 60000/1001 -i source/%06d.tif -t 
00:01:00 -c:v huffyuv lossless.mkv

The result appears reasonable.

  $ mediainfo lossless.mkv
  General
  Unique ID                                : 
235140899628261703308032414639716345340 (0xB0E67D6EF6B78362D9BCF9EA3080A5FC)
  Complete name                            : lossless.mkv
  Format                                   : Matroska
  Format version                           : Version 4
  File size                                : 8.54 GiB
  Duration                                 : 59 s 994 ms
  Overall bit rate                         : 1 223 Mb/s
  Writing application                      : Lavf58.45.100
  Writing library                          : Lavf58.45.100
  ErrorDetectionType                       : Per level 1

  Video
  ID                                       : 1
  Format                                   : HuffYUV
  Format version                           : Version 2
  Codec ID                                 : V_MS/VFW/FOURCC / HFYU
  Duration                                 : 59 s 994 ms
  Bit rate                                 : 1 199 Mb/s
  Width                                    : 1 440 pixels
  Height                                   : 1 080 pixels
  Display aspect ratio                     : 4:3
  Frame rate mode                          : Constant
  Frame rate                               : 59.940 FPS
  Color space                              : RGB
  Bit depth                                : 8 bits
  Scan type                                : Progressive
  Bits/(Pixel*Frame)                       : 12.860
  Stream size                              : 8.37 GiB (98%)
  Writing library                          : Lavc58.91.100 huffyuv
  Default                                  : Yes
  Forced                                   : No

Now let's see what the SSIM filter says.

  $ ffmpeg -i lossless.mkv -start_number 0 -framerate 60000/1001 -i 
source/%06d.tif -t 00:01:00 -filter_complex ssim -f null -
  ...
  Input #0, matroska,webm, from 'lossless.mkv':
    Metadata:
      ENCODER         : Lavf58.45.100
    Duration: 00:00:59.99, start: 0.000000, bitrate: 1223104 kb/s
      Stream #0:0: Video: huffyuv (HFYU / 0x55594648), bgr0, 1440x1080, 
59.94 fps, 59.94 tbr, 1k tbn, 1k tbc (default)
      Metadata:
        ENCODER         : Lavc58.91.100 huffyuv
        DURATION        : 00:00:59.994000000
  Input #1, image2, from 'source/%06d.tif':
    Duration: 01:47:16.00, start: 0.000000, bitrate: N/A
      Stream #1:0: Video: tiff, rgb24, 1440x1080, 59.94 tbr, 59.94 tbn, 
59.94 tbc
  ...
  [Parsed_ssim_0 @ 0x55fb09c738c0] not matching timebases found between 
first input: 1/1000 and second input 1001/60000, results may be incorrect!
  ...
  [Parsed_ssim_0 @ 0x55fb09c738c0] SSIM R:0.833774 (7.793009) G:0.835401 
(7.835723) B:0.831058 (7.722615) All:0.833411 (7.783532)

That's not what I expected.  My understanding is that the R, G, B, and
All values should all be "1.000000 (inf)".

The "not matching timebases" warning is the obvious thing to look at.
After much searching, I came upon the -video_track_timescale option, but
it seems to only take an integer, and 60 is not the same as 59.94, so it
seems that I simply can't directly compare a video stream with a non-
integer framerate to an image sequence.

As a workaround, I tried extracting the lossless video frames as a
separate image sequence.

  $ ffmpeg -i lossless.mkv -start_number 0 lossless/%06d.png

This created the expected sequence of image files (000000.png -
003595.png).  Since I have both the "source" and the "lossless" streams
as images sequences, I can use ImageMagick to compare them.

  $ for I in `seq -w 0 3595` ; do compare -metric AE source/00${I}.tif 
lossless/00${I}.png /tmp/diff.png 2>/dev/null || echo $I ; done

This produces no output, indicating that ImageMagick thinks that the
TIFF files in ./source and the PNG files in ./lossless contain
completely identical image data.  What does the SSIM filter say?

  $ ffmpeg -framerate 60000/1001 -start_number 0 -i lossless/%06d.png 
-framerate 60000/1001 -start_number 0 -i source/%06d.tif -t 00:01:00 
-filter_complex ssim -f null -
  ...
  Input #0, image2, from 'lossless/%06d.png':
    Duration: 00:00:59.99, start: 0.000000, bitrate: N/A
      Stream #0:0: Video: png, rgb24(pc), 1440x1080, 59.94 fps, 59.94 
tbr, 59.94 tbn, 59.94 tbc
  Input #1, image2, from 'source/%06d.tif':
    Duration: 01:47:16.00, start: 0.000000, bitrate: N/A
      Stream #1:0: Video: tiff, rgb24, 1440x1080, 59.94 tbr, 59.94 tbn, 
59.94 tbc
  ...
  [Parsed_ssim_0 @ 0x556c7e948980] SSIM R:0.999775 (36.484195) 
G:0.999777 (36.508857) B:0.999774 (36.451744) All:0.999775 (36.481536)

Close, but not the "1.000000 (inf)" that I would expect for identical
streams.  I suppose that it may be related to the "rgb24(pc)" vs.
rgb24 pixel formats; it's hard to be sure, as Google isn't turning up
anything that describes what "rgb24(pc)" actually means.

Does anyone know what the heck is going on here?  Am I doing something
wrong?

-- 
========================================================================
                  In Soviet Russia, Google searches you!
========================================================================