[FFmpeg-devel] [PATCH] fate: add tests for psnr and ssim filter

[Tobias Rapp]

On 20.07.2017 22:21, Michael Niedermayer wrote:
> On Wed, Jul 19, 2017 at 05:23:07PM +0200, Tobias Rapp wrote:
> [...]
>>  fate/filter-video.mak           |   16 ++++++++++++++
>>  ref/fate/filter-refcmp-psnr-rgb |   45 ++++++++++++++++++++++++++++++++++++++++
>>  ref/fate/filter-refcmp-psnr-yuv |   45 ++++++++++++++++++++++++++++++++++++++++
>>  ref/fate/filter-refcmp-ssim-rgb |   30 ++++++++++++++++++++++++++
>>  ref/fate/filter-refcmp-ssim-yuv |   30 ++++++++++++++++++++++++++
>>  5 files changed, 166 insertions(+)
>> ba25d086250183b279c65419343103872074dab8  0001-fate-add-tests-for-psnr-and-ssim-filter.patch
>> From 430a2c8cfb455cbd11af119987fffa0919f71cc5 Mon Sep 17 00:00:00 2001
>> From: Tobias Rapp <t.rapp at noa-archive.com>
>> Date: Tue, 11 Jul 2017 12:38:06 +0200
>> Subject: [PATCH] fate: add tests for psnr and ssim filter
>>
>> Signed-off-by: Tobias Rapp <t.rapp at noa-archive.com>
>
> on x86-32:
>
> --- tests/ref/fate/filter-refcmp-psnr-yuv       2017-07-20 22:05:14.887394601 +0200
> +++ tests/data/fate/filter-refcmp-psnr-yuv      2017-07-20 22:14:51.363406746 +0200
> @@ -3,43 +3,43 @@
>  lavfi.psnr.psnr.y=24.67
>  lavfi.psnr.mse.u=339.38
>  lavfi.psnr.psnr.u=22.82
> -lavfi.psnr.mse.v=705.41
> +lavfi.psnr.mse.v=705.31
>  lavfi.psnr.psnr.v=19.65
> -lavfi.psnr.mse_avg=372.23
> +lavfi.psnr.mse_avg=372.20
>  lavfi.psnr.psnr_avg=22.42
>  frame:1    pts:1       pts_time:1
> -lavfi.psnr.mse.y=236.74
> +lavfi.psnr.mse.y=236.76
>  lavfi.psnr.psnr.y=24.39
>  lavfi.psnr.mse.u=416.17
>  lavfi.psnr.psnr.u=21.94
> -lavfi.psnr.mse.v=704.98
> +lavfi.psnr.mse.v=704.97
>  lavfi.psnr.psnr.v=19.65
>  lavfi.psnr.mse_avg=398.66
>  lavfi.psnr.psnr_avg=22.12
>  frame:2    pts:2       pts_time:2
> -lavfi.psnr.mse.y=234.79
> +lavfi.psnr.mse.y=234.83
>  lavfi.psnr.psnr.y=24.42
>  lavfi.psnr.mse.u=435.72
>  lavfi.psnr.psnr.u=21.74
>  lavfi.psnr.mse.v=699.60
>  lavfi.psnr.psnr.v=19.68
> -lavfi.psnr.mse_avg=401.23
> +lavfi.psnr.mse_avg=401.24
>  lavfi.psnr.psnr_avg=22.10
>  frame:3    pts:3       pts_time:3
> -lavfi.psnr.mse.y=250.88
> +lavfi.psnr.mse.y=250.91
>  lavfi.psnr.psnr.y=24.14
>  lavfi.psnr.mse.u=479.73
>  lavfi.psnr.psnr.u=21.32
> -lavfi.psnr.mse.v=707.55
> +lavfi.psnr.mse.v=707.54
>  lavfi.psnr.psnr.v=19.63
> -lavfi.psnr.mse_avg=422.26
> -lavfi.psnr.psnr_avg=21.88
> +lavfi.psnr.mse_avg=422.27
> +lavfi.psnr.psnr_avg=21.87
>  frame:4    pts:4       pts_time:4
> -lavfi.psnr.mse.y=241.05
> +lavfi.psnr.mse.y=241.04
>  lavfi.psnr.psnr.y=24.31
>  lavfi.psnr.mse.u=505.04
>  lavfi.psnr.psnr.u=21.10
>  lavfi.psnr.mse.v=716.00
>  lavfi.psnr.psnr.v=19.58
> -lavfi.psnr.mse_avg=425.79
> +lavfi.psnr.mse_avg=425.78
>  lavfi.psnr.psnr_avg=21.84
> Test filter-refcmp-psnr-yuv failed. Look at tests/data/fate/filter-refcmp-psnr-yuv.err for details.
> make: *** [fate-filter-refcmp-psnr-yuv] Error 1
> make: *** Waiting for unfinished jobs....
> --- tests/ref/fate/filter-refcmp-psnr-rgb       2017-07-20 22:05:14.887394601 +0200
> +++ tests/data/fate/filter-refcmp-psnr-rgb      2017-07-20 22:14:51.367406746 +0200
> @@ -1,11 +1,11 @@
>  frame:0    pts:0       pts_time:0
> -lavfi.psnr.mse.r=1381.80
> +lavfi.psnr.mse.r=1381.38
>  lavfi.psnr.psnr.r=16.73
>  lavfi.psnr.mse.g=896.00
>  lavfi.psnr.psnr.g=18.61
>  lavfi.psnr.mse.b=277.38
>  lavfi.psnr.psnr.b=23.70
> -lavfi.psnr.mse_avg=851.73
> +lavfi.psnr.mse_avg=851.59
>  lavfi.psnr.psnr_avg=18.83
>  frame:1    pts:1       pts_time:1
>  lavfi.psnr.mse.r=1380.37
> @@ -17,9 +17,9 @@
>  lavfi.psnr.mse_avg=930.67
>  lavfi.psnr.psnr_avg=18.44
>  frame:2    pts:2       pts_time:2
> -lavfi.psnr.mse.r=1403.20
> +lavfi.psnr.mse.r=1403.21
>  lavfi.psnr.psnr.r=16.66
> -lavfi.psnr.mse.g=954.05
> +lavfi.psnr.mse.g=954.04
>  lavfi.psnr.psnr.g=18.34
>  lavfi.psnr.mse.b=494.22
>  lavfi.psnr.psnr.b=21.19
> Test filter-refcmp-psnr-rgb failed. Look at tests/data/fate/filter-refcmp-psnr-rgb.err for details.
> make: *** [fate-filter-refcmp-psnr-rgb] Error 1
>

So I assume the lesson to be learned for me should be that it is not 
safe to compare string-ified floats and a sane test should do something 
like "abs(ref_val - curr_val) < eps" for each metadata entry?

Regards,
Tobias