[PATCH] t/t0000-basic.sh: Don't run a passing TODO unless TEST_PASSING_TODO=1

Change the sanity tests in t/t0000-basic.sh to not to run a passing
TODO test unless the TEST_PASSING_TODO environment variable is set.

The motivation is to have nothing out of the ordinary on a normal test
run for test smoking purposes.

If every normal test run has a passing TODO you're more likely to turn
a blind eye to it and not to investigate cases where things really are
passing unexpectedly.

It also makes the prove(1) output less noisy. Before:

    All tests successful.

    Test Summary Report
    -------------------
    ./t0000-basic.sh                                   (Wstat: 0 Tests: 46 Failed: 0)
      TODO passed:   5
    Files=484, Tests=6229, 143 wallclock secs ( 4.00 usr  4.15 sys + 104.77 cusr 351.57 csys = 464.49 CPU)
    Result: PASS

And after:

    All tests successful.
    Files=484, Tests=6228, 139 wallclock secs ( 4.07 usr  4.25 sys + 104.54 cusr 350.85 csys = 463.71 CPU)
    Result: PASS

Signed-off-by: Ævar Arnfjörð Bjarmason <redacted>
---
 t/t0000-basic.sh |   11 ++++++++---
 1 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/t/t0000-basic.sh b/t/t0000-basic.sh
index f2c7336..2f6a17b 100755
--- a/t/t0000-basic.sh
+++ b/t/t0000-basic.sh

@@ -54,9 +54,14 @@ test_expect_success 'success is reported like this' '
 test_expect_failure 'pretend we have a known breakage' '
     false
 '
-test_expect_failure 'pretend we have fixed a known breakage' '
-    :
-'
+if test -z "$TEST_PASSING_TODO"
+then
+	say "Not testing a known breakage, set TEST_PASSING_TODO=1 to enable"
+else
+	test_expect_failure 'pretend we have fixed a known breakage' '
+	    :
+	'
+fi
 test_set_prereq HAVEIT
 haveit=no
 test_expect_success HAVEIT 'test runs if prerequisite is satisfied' '

-- 
1.7.2.1.389.gc3d0b

Heya,

On Tue, Aug 17, 2010 at 04:43, Ævar Arnfjörð Bjarmason [off-list ref] wrote:

Change the sanity tests in t/t0000-basic.sh to not to run a passing
TODO test unless the TEST_PASSING_TODO environment variable is set.

The motivation is to have nothing out of the ordinary on a normal test
run for test smoking purposes.

If every normal test run has a passing TODO you're more likely to turn
a blind eye to it and not to investigate cases where things really are
passing unexpectedly.

Thanks, this has bothered me ever since I first ran the test suite.
While I understand that the test suite needs testing to, it should be
done in such a way that it doesn't produce false positives. That is,
my preference would be to still keep the test in the suite as-is, but
somehow get it to show up as a passing test instead of a fixed one.
E.g., something like:

test_in_new_harness 0 1 0 0 '
    test_expect_failure 'pretend we have fixed a known breakage' '
           :
     '
'

Where there '0 1 0 0' stands for '0 failures, 1 fixed, 0 broken, 0
success'. Not sure there's any other use for a test_in_new_harness
function though...

PS: To clarify, test_in_new_harness would not change any of the
counters (test_failure, test_count, test_fixed, test_broken,
test_success), but would instead compare those against the values
specified as argument to test_in_new_harness.

PSPS: I have no idea how easy it would be to implement something like this.

-- 
Cheers,

Sverre Rabbelier