[FFmpeg-devel] MPEG-2 FLAGS2_FAST benchmarks

[Ronald S. Bultje]

Hi,

On Sat, Jun 13, 2020 at 12:11 PM Kieran Kunhya <kierank at obe.tv> wrote:

> Hello,
>
> I have run the following commands on the following CPU which is 7 years
> old:
> Intel(R) Xeon(R) CPU E3-1265L v3
>
> ffmpeg_g -i matrixbench_mpeg2.mpg -f null -
> and
> ffmpeg_g -flags2 fast -i matrixbench_mpeg2.mpg -f null -
>
> Here are the results:
>
> Normal Fast
> 2202 2168
> 2155 2057
> 2210 2128
> 2104 2051
> 2132 2053
> 2035 2014
> 2021 2037
> 2247 1999
> 2095 2015
> 2232 2119
> Mean 2143.3 2064.1
> Stdev 79.82209385 55.9075626
>
> It can therefore be shown that the difference is statistically
> insignificant.


I'm all for removing -flags2 fast, but I do think we should do science
correctly.

The one-tailed (we are not considering whether "fast" is slower than
"normal") homoscedastic ("equal variance sampled, not paired") student t
test has a p value of 0.009638597, which for a typical p value threshold of
0.05 would reject the null hypothesis ("they are the same speed") and
accept that "Fast" is faster than "Normal". Why do you think the difference
is statistically insignificant?

Ronald