[RFC PATCH] tests: print failed test numbers at the end of the test run

On modern multi-core processors "make test" is often run in multiple jobs.
If one of them fails the test run does stop, but the concurrently running
tests finish their run. Finding out what test is broken involves a lot of
scrolling. That gets even worse when the -i option is used.

If one or more tests failed, print a list of them before the test summary:

failed test(s): t1000 t6500

fixed   0
success 7638
failed  3
broken  49
total   7723

This makes it possible to just run the test suite with -i and collect all
failed test scripts at the end for further examination.

Signed-off-by: Jens Lehmann <redacted>
---

Maybe I'm missing something completely obvious, but I always have a hard
time finding out which test scripts did fail in a test run with -j30.

 t/aggregate-results.sh |   12 +++++++++++-
 1 files changed, 11 insertions(+), 1 deletions(-)

diff --git a/t/aggregate-results.sh b/t/aggregate-results.sh
index d206b7c..b8e929a 100755
--- a/t/aggregate-results.sh
+++ b/t/aggregate-results.sh

@@ -1,5 +1,6 @@
 #!/bin/sh

+failed_tests=
 fixed=0
 success=0
 failed=0

@@ -18,7 +19,12 @@ do
 		success)
 			success=$(($success + $value)) ;;
 		failed)
-			failed=$(($failed + $value)) ;;
+			failed=$(($failed + $value))
+			if test $value != 0; then
+				testnum=$(echo $file | cut -b 14-18)
+				failed_tests="$failed_tests $testnum"
+			fi
+			;;
 		broken)
 			broken=$(($broken + $value)) ;;
 		total)

@@ -27,6 +33,10 @@ do
 	done <"$file"
 done

+if [ -n "$failed_tests" ]; then
+	printf "\nfailed test(s):$failed_tests\n\n"
+fi
+
 printf "%-8s%d\n" fixed $fixed
 printf "%-8s%d\n" success $success
 printf "%-8s%d\n" failed $failed

-- 
1.7.6.346.g750efc

On Sun, Jul 24, 2011 at 2:16 AM, Jens Lehmann [off-list ref] wrote:

On modern multi-core processors "make test" is often run in multiple jobs.
If one of them fails the test run does stop, but the concurrently running
tests finish their run.

Somewhat related (or not). I change something. I know it breaks things
and want to know _all_ tests it breaks, but "make test" would stop
early. Is there anyway to make it keep going through all tests even if
some fails? "make -j<big number>" improves the situation but does not
really solve it.
-- 
Duy

On Tue, Jul 26, 2011 at 11:25:21AM +0700, Nguyen Thai Ngoc Duy wrote:

On Sun, Jul 24, 2011 at 2:16 AM, Jens Lehmann [off-list ref] wrote:

quoted

On modern multi-core processors "make test" is often run in multiple jobs.
If one of them fails the test run does stop, but the concurrently running
tests finish their run.

Somewhat related (or not). I change something. I know it breaks things
and want to know _all_ tests it breaks, but "make test" would stop
early. Is there anyway to make it keep going through all tests even if
some fails? "make -j<big number>" improves the situation but does not
really solve it.

Try "make -k", which will keep running rules that don't have failed
dependencies (in this case, all of the tests are independent, so it will
run all of them).

Or use "prove", which can do fancy things like putting the last-failed
(so you can see the likely failures immediately and start to probe them
while the rest of the suite runs).

-Peff

On Mon, Jul 25, 2011 at 11:42:36PM -0600, Jeff King wrote:

Or use "prove", which can do fancy things like putting the last-failed
(so you can see the likely failures immediately and start to probe them
while the rest of the suite runs).

Oops, that should read "...putting the last-failed tests at the
beginning".

-Peff