[FFmpeg-devel] [PATCH] fate: Add a target for listing failed tests

[Martin Storsjö]

If running tests with "make -j<N> fate", the execution will stop
after the first failing test. To get an overview of the whole
test suite, one rather would run "make -k -j<N> fate", which then
again buries the results about what tests actually failed further
up in the console log.

Add a target so one can run "make fate-list-failing", to see a list
of all tests that failed the last time they were executed.

Also add a companion target "fate-clear-results" which removes all
the old test results. (When executing a subset of tests, the result
files of all tests that aren't executed stay untouched. This also
allows getting rid of results for tests that no longer are present
in the testsuite.)
---
So far, I've always just manually run
"cat tests/data/fate/*.rep | cut -f 1-2 -d : | grep -v :0", but
perhaps we should at least wrap it up in something more convenient
for the other developers.
---
 doc/build_system.txt | 6 ++++++
 doc/fate.texi        | 7 +++++++
 tests/Makefile       | 6 ++++++
 3 files changed, 19 insertions(+)

diff --git a/doc/build_system.txt b/doc/build_system.txt
index 0b1b0c2054..91c7a5e9af 100644
--- a/doc/build_system.txt
+++ b/doc/build_system.txt
@@ -30,6 +30,12 @@ fate
 fate-list
     List all fate/regression test targets.
 
+fate-list-failing
+    List the fate tests that failed the last time they were executed.
+
+fate-clear-results
+    Remove the test results from previous test executions.
+
 install
     Install headers, libraries and programs.
 
diff --git a/doc/fate.texi b/doc/fate.texi
index 17644ce65a..bf01816af8 100644
--- a/doc/fate.texi
+++ b/doc/fate.texi
@@ -208,6 +208,13 @@ Download/synchronize sample files to the configured samples directory.
 @item fate-list
 Will list all fate/regression test targets.
 
+ at item fate-list-failing
+List the fate tests that failed the last time they were executed.
+
+ at item fate-clear-results
+Remove the test results from previous test executions (getting rid of
+potentially stale results from fate-list-failing).
+
 @item fate
 Run the FATE test suite (requires the fate-suite dataset).
 @end table
diff --git a/tests/Makefile b/tests/Makefile
index 9b70145015..adb0799328 100644
--- a/tests/Makefile
+++ b/tests/Makefile
@@ -313,6 +313,12 @@ $(FATE): $(FATE_UTILS:%=tests/%$(HOSTEXESUF)) | $(FATE_OUTDIRS)
 fate-list:
 	@printf '%s\n' $(sort $(FATE))
 
+fate-list-failing:
+	@cat tests/data/fate/*.rep | cut -f 1-2 -d : | grep -v :0 | sed 's/:.*//;s/^/fate-/'
+
+fate-clear-results:
+	@rm -f tests/data/fate/*.rep
+
 coverage.info: TAG = LCOV
 coverage.info:
 	$(M)lcov -q -d $(CURDIR) -b $(patsubst src%,./,$(SRC_LINK)) --capture | \
-- 
2.39.5 (Apple Git-154)