[FFmpeg-devel] [PATCH] fate: Add a target for listing failed tests

[Alexander Strasser]

Hi Martin!

On 2024-11-07 11:36 +0200, Martin Storsjö wrote:
> If running tests with "make -j<N> fate", the execution will stop
> after the first failing test. To get an overview of the whole
> test suite, one rather would run "make -k -j<N> fate", which then
> again buries the results about what tests actually failed further
> up in the console log.
>
> Add a target so one can run "make fate-list-failing", to see a list
> of all tests that failed the last time they were executed.
>
> Also add a companion target "fate-clear-results" which removes all
> the old test results. (When executing a subset of tests, the result
> files of all tests that aren't executed stay untouched. This also
> allows getting rid of results for tests that no longer are present
> in the testsuite.)

Looks good and useful to me.

Just a few ideas follow:

> ---
> So far, I've always just manually run
> "cat tests/data/fate/*.rep | cut -f 1-2 -d : | grep -v :0", but
> perhaps we should at least wrap it up in something more convenient
> for the other developers.
> ---
>  doc/build_system.txt | 6 ++++++
>  doc/fate.texi        | 7 +++++++
>  tests/Makefile       | 6 ++++++
>  3 files changed, 19 insertions(+)
>
> diff --git a/doc/build_system.txt b/doc/build_system.txt
> index 0b1b0c2054..91c7a5e9af 100644
> --- a/doc/build_system.txt
> +++ b/doc/build_system.txt
> @@ -30,6 +30,12 @@ fate
>  fate-list
>      List all fate/regression test targets.
>
> +fate-list-failing
> +    List the fate tests that failed the last time they were executed.
> +
> +fate-clear-results
> +    Remove the test results from previous test executions.
> +

Would it be better to use the same description as int fate.texi ?


>  install
>      Install headers, libraries and programs.
>
> diff --git a/doc/fate.texi b/doc/fate.texi
> index 17644ce65a..bf01816af8 100644
> --- a/doc/fate.texi
> +++ b/doc/fate.texi
> @@ -208,6 +208,13 @@ Download/synchronize sample files to the configured samples directory.
>  @item fate-list
>  Will list all fate/regression test targets.
>
> + at item fate-list-failing
> +List the fate tests that failed the last time they were executed.
> +
> + at item fate-clear-results
> +Remove the test results from previous test executions (getting rid of
> +potentially stale results from fate-list-failing).
> +
>  @item fate
>  Run the FATE test suite (requires the fate-suite dataset).
>  @end table
> diff --git a/tests/Makefile b/tests/Makefile
> index 9b70145015..adb0799328 100644
> --- a/tests/Makefile
> +++ b/tests/Makefile
> @@ -313,6 +313,12 @@ $(FATE): $(FATE_UTILS:%=tests/%$(HOSTEXESUF)) | $(FATE_OUTDIRS)
>  fate-list:
>  	@printf '%s\n' $(sort $(FATE))
>
> +fate-list-failing:
> +	@cat tests/data/fate/*.rep | cut -f 1-2 -d : | grep -v :0 | sed 's/:.*//;s/^/fate-/'
> +
> +fate-clear-results:
> +	@rm -f tests/data/fate/*.rep
> +
>  coverage.info: TAG = LCOV
>  coverage.info:
>  	$(M)lcov -q -d $(CURDIR) -b $(patsubst src%,./,$(SRC_LINK)) --capture | \
> --

Maybe the attached patch would make sense on top of your changes?


Best regards,
  Alexander
-------------- next part --------------
From 130ca8d85dcb1fecace596e1f23e0414d56348ef Mon Sep 17 00:00:00 2001
From: Alexander Strasser <eclipse7 at gmx.net>
Date: Sun, 1 Dec 2024 15:06:53 +0100
Subject: [PATCH] fate: Simplify implementation for target fate-list-failing

Make it a little easier to read and use one awk invocation as opposed
to a combination of cat, cut, grep, and sed.

Signed-off-by: Alexander Strasser <eclipse7 at gmx.net>
---
 tests/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/Makefile b/tests/Makefile
index adb0799328..2c4ad7c18f 100644
--- a/tests/Makefile
+++ b/tests/Makefile
@@ -314,7 +314,7 @@ fate-list:
 	@printf '%s\n' $(sort $(FATE))

 fate-list-failing:
-	@cat tests/data/fate/*.rep | cut -f 1-2 -d : | grep -v :0 | sed 's/:.*//;s/^/fate-/'
+	@awk -F: '$$2 != 0 { print "fate-" $$1 }' tests/data/fate/*.rep

 fate-clear-results:
 	@rm -f tests/data/fate/*.rep
--