[FFmpeg-devel] [PATCH] fate: add tests for psnr and ssim filter

[Michael Niedermayer]

On Fri, Jul 21, 2017 at 04:39:37PM +0200, Tobias Rapp wrote:
> On 20.07.2017 22:21, Michael Niedermayer wrote:
> >On Wed, Jul 19, 2017 at 05:23:07PM +0200, Tobias Rapp wrote:
> >[...]
> >> fate/filter-video.mak           |   16 ++++++++++++++
> >> ref/fate/filter-refcmp-psnr-rgb |   45 ++++++++++++++++++++++++++++++++++++++++
> >> ref/fate/filter-refcmp-psnr-yuv |   45 ++++++++++++++++++++++++++++++++++++++++
> >> ref/fate/filter-refcmp-ssim-rgb |   30 ++++++++++++++++++++++++++
> >> ref/fate/filter-refcmp-ssim-yuv |   30 ++++++++++++++++++++++++++
> >> 5 files changed, 166 insertions(+)
> >>ba25d086250183b279c65419343103872074dab8  0001-fate-add-tests-for-psnr-and-ssim-filter.patch
> >>From 430a2c8cfb455cbd11af119987fffa0919f71cc5 Mon Sep 17 00:00:00 2001
> >>From: Tobias Rapp <t.rapp at noa-archive.com>
> >>Date: Tue, 11 Jul 2017 12:38:06 +0200
> >>Subject: [PATCH] fate: add tests for psnr and ssim filter
> >>
> >>Signed-off-by: Tobias Rapp <t.rapp at noa-archive.com>
> >
> >on x86-32:
> >
> >--- tests/ref/fate/filter-refcmp-psnr-yuv       2017-07-20 22:05:14.887394601 +0200
> >+++ tests/data/fate/filter-refcmp-psnr-yuv      2017-07-20 22:14:51.363406746 +0200
> >@@ -3,43 +3,43 @@
> > lavfi.psnr.psnr.y=24.67
> > lavfi.psnr.mse.u=339.38
> > lavfi.psnr.psnr.u=22.82
> >-lavfi.psnr.mse.v=705.41
> >+lavfi.psnr.mse.v=705.31
> > lavfi.psnr.psnr.v=19.65
> >-lavfi.psnr.mse_avg=372.23
> >+lavfi.psnr.mse_avg=372.20
> > lavfi.psnr.psnr_avg=22.42
> > frame:1    pts:1       pts_time:1
> >-lavfi.psnr.mse.y=236.74
> >+lavfi.psnr.mse.y=236.76
> > lavfi.psnr.psnr.y=24.39
> > lavfi.psnr.mse.u=416.17
> > lavfi.psnr.psnr.u=21.94
> >-lavfi.psnr.mse.v=704.98
> >+lavfi.psnr.mse.v=704.97
> > lavfi.psnr.psnr.v=19.65
> > lavfi.psnr.mse_avg=398.66
> > lavfi.psnr.psnr_avg=22.12
> > frame:2    pts:2       pts_time:2
> >-lavfi.psnr.mse.y=234.79
> >+lavfi.psnr.mse.y=234.83
> > lavfi.psnr.psnr.y=24.42
> > lavfi.psnr.mse.u=435.72
> > lavfi.psnr.psnr.u=21.74
> > lavfi.psnr.mse.v=699.60
> > lavfi.psnr.psnr.v=19.68
> >-lavfi.psnr.mse_avg=401.23
> >+lavfi.psnr.mse_avg=401.24
> > lavfi.psnr.psnr_avg=22.10
> > frame:3    pts:3       pts_time:3
> >-lavfi.psnr.mse.y=250.88
> >+lavfi.psnr.mse.y=250.91
> > lavfi.psnr.psnr.y=24.14
> > lavfi.psnr.mse.u=479.73
> > lavfi.psnr.psnr.u=21.32
> >-lavfi.psnr.mse.v=707.55
> >+lavfi.psnr.mse.v=707.54
> > lavfi.psnr.psnr.v=19.63
> >-lavfi.psnr.mse_avg=422.26
> >-lavfi.psnr.psnr_avg=21.88
> >+lavfi.psnr.mse_avg=422.27
> >+lavfi.psnr.psnr_avg=21.87
> > frame:4    pts:4       pts_time:4
> >-lavfi.psnr.mse.y=241.05
> >+lavfi.psnr.mse.y=241.04
> > lavfi.psnr.psnr.y=24.31
> > lavfi.psnr.mse.u=505.04
> > lavfi.psnr.psnr.u=21.10
> > lavfi.psnr.mse.v=716.00
> > lavfi.psnr.psnr.v=19.58
> >-lavfi.psnr.mse_avg=425.79
> >+lavfi.psnr.mse_avg=425.78
> > lavfi.psnr.psnr_avg=21.84
> >Test filter-refcmp-psnr-yuv failed. Look at tests/data/fate/filter-refcmp-psnr-yuv.err for details.
> >make: *** [fate-filter-refcmp-psnr-yuv] Error 1
> >make: *** Waiting for unfinished jobs....
> >--- tests/ref/fate/filter-refcmp-psnr-rgb       2017-07-20 22:05:14.887394601 +0200
> >+++ tests/data/fate/filter-refcmp-psnr-rgb      2017-07-20 22:14:51.367406746 +0200
> >@@ -1,11 +1,11 @@
> > frame:0    pts:0       pts_time:0
> >-lavfi.psnr.mse.r=1381.80
> >+lavfi.psnr.mse.r=1381.38
> > lavfi.psnr.psnr.r=16.73
> > lavfi.psnr.mse.g=896.00
> > lavfi.psnr.psnr.g=18.61
> > lavfi.psnr.mse.b=277.38
> > lavfi.psnr.psnr.b=23.70
> >-lavfi.psnr.mse_avg=851.73
> >+lavfi.psnr.mse_avg=851.59
> > lavfi.psnr.psnr_avg=18.83
> > frame:1    pts:1       pts_time:1
> > lavfi.psnr.mse.r=1380.37
> >@@ -17,9 +17,9 @@
> > lavfi.psnr.mse_avg=930.67
> > lavfi.psnr.psnr_avg=18.44
> > frame:2    pts:2       pts_time:2
> >-lavfi.psnr.mse.r=1403.20
> >+lavfi.psnr.mse.r=1403.21
> > lavfi.psnr.psnr.r=16.66
> >-lavfi.psnr.mse.g=954.05
> >+lavfi.psnr.mse.g=954.04
> > lavfi.psnr.psnr.g=18.34
> > lavfi.psnr.mse.b=494.22
> > lavfi.psnr.psnr.b=21.19
> >Test filter-refcmp-psnr-rgb failed. Look at tests/data/fate/filter-refcmp-psnr-rgb.err for details.
> >make: *** [fate-filter-refcmp-psnr-rgb] Error 1
> >
> 
> So I assume the lesson to be learned for me should be that it is not
> safe to compare string-ified floats and a sane test should do
> something like "abs(ref_val - curr_val) < eps" for each metadata
> entry?

we might be able to get away with strings and lower precission.
but yes otherwise with inexact code like floats checking the difference
would likely be needed

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

"Nothing to hide" only works if the folks in power share the values of
you and everyone you know entirely and always will -- Tom Scott

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: Digital signature
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20170722/02e0751c/attachment.sig>