Re: [PATCH 2/2] test-lib: GIT_TEST_ONLY to run only specific tests

Eric Sunshine [off-list ref] writes:

On Mon, Mar 3, 2014 at 5:24 AM, Ilya Bobyr [off-list ref] wrote:

quoted

This is a counterpart to GIT_SKIP_TESTS.  Mostly useful when debugging.

To be grammatically similar to GIT_SKIP_TESTS, perhaps name it GIT_RUN_TESTS?

I actually do not like the interface to use two variables very much.
Can't we just allow negative entries on "to be skipped" list?

That is

	GIT_SKIP_TESTS='t9??? !t91??'

would skip nine-thousand series, but would run 91xx series, and all
the others are not excluded.

Simple rules to consider:

 - If the list consists of _only_ negated patterns, pretend that
   there is "unless otherwise specified with negatives, skip all
   tests", i.e. treat GIT_SKIP_TESTS='!t91??' just the same way you
   would treat GIT_SKIP_TESTS='* !t91??'.

 - The orders should not matter for simplicity of the semantics;
   before running each test, check if it matches any negative (and
   run it if it matches, without looking at any positives), and
   otherwise check if it matches any positive (and skip it if it
   does not).

Hmm?

quoted

---
 t/README      |   15 +++++++++++++++
 t/test-lib.sh |    8 ++++++++
 2 files changed, 23 insertions(+), 0 deletions(-)

diff --git a/t/README b/t/README
index caeeb9d..f939987 100644
--- a/t/README
+++ b/t/README

@@ -187,6 +187,21 @@ and either can match the "t[0-9]{4}" part to skip the whole
 test, or t[0-9]{4} followed by ".$number" to say which
 particular test to skip.

+Sometimes the opposite is desired - ability to execute only one or
+several tests.  Mostly while debugging tests.  For that you can say
+
+    $ GIT_TEST_ONLY=t9200.8 sh ./t9200-git-cvsexport-commit.sh
+
+or, similrary to GIT_SKIP_TESTS
+
+    $ GIT_TEST_ONLY='t[0-4]??? t91?? t9200.8' make
+
+In additiona to matching against "<test suite number>.<test number>"

s/additiona/addition/

Plus the other typos already mentioned by Philip...

quoted

+GIT_TEST_ONLY is matched against just the test numbes.  This comes
+handy when you are running only one test:
+
+    $ GIT_TEST_ONLY='[0-8]' sh ./t9200-git-cvsexport-commit.sh
+
 Note that some tests in the existing test suite rely on previous
 test item, so you cannot arbitrarily disable one and expect the
 remainder of test to check what the test originally was intended

diff --git a/t/test-lib.sh b/t/test-lib.sh
index 89a405b..12bf436 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh

@@ -464,6 +464,14 @@ test_skip () {
                fi
                skipped_reason="missing $missing_prereq${of_prereq}"
        fi
+       if test -z "$to_skip" && test -n "$GIT_TEST_ONLY" &&
+               ! match_pattern_list $this_test.$test_count $GIT_TEST_ONLY &&
+               ! match_pattern_list $test_count $GIT_TEST_ONLY
+       then
+               to_skip=t
+               skipped_reason="not in GIT_TEST_ONLY"
+       fi
+
        case "$to_skip" in
        t)
                say_color skip >&3 "skipping test: $@"
--

1.7.9

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On 3/3/2014 3:26 PM, Junio C Hamano wrote:

Eric Sunshine [off-list ref] writes:

quoted

On Mon, Mar 3, 2014 at 5:24 AM, Ilya Bobyr [off-list ref] wrote:

quoted

This is a counterpart to GIT_SKIP_TESTS.  Mostly useful when debugging.

To be grammatically similar to GIT_SKIP_TESTS, perhaps name it GIT_RUN_TESTS?

I actually do not like the interface to use two variables very much.
Can't we just allow negative entries on "to be skipped" list?

That is

	GIT_SKIP_TESTS='t9??? !t91??'

would skip nine-thousand series, but would run 91xx series, and all
the others are not excluded.

Simple rules to consider:

  - If the list consists of _only_ negated patterns, pretend that
    there is "unless otherwise specified with negatives, skip all
    tests", i.e. treat GIT_SKIP_TESTS='!t91??' just the same way you
    would treat GIT_SKIP_TESTS='* !t91??'.

  - The orders should not matter for simplicity of the semantics;
    before running each test, check if it matches any negative (and
    run it if it matches, without looking at any positives), and
    otherwise check if it matches any positive (and skip it if it
    does not).

Hmm?

I can do that.  But I am not sure that matches the use cases I had in 
mind the best.

First use case is that while developing I want to run tests frequently 
and I have a specific test that I am working on at the moment.
That test is broken and I am trying to fix it (TDD).
I want to run just the initialization test(s) and then that specific test.
Running everything is quite slow.

GIT_RUN_ONLY addresses the TDD case.

Second case is when I broke one or more tests and want to figure out 
what is wrong.
In this case running tests after the broken one will clutter the output 
directory and will make debugging somewhat harder, especially if I am 
not familiar with all the tests.

For the second case I was actually thinking that something like 
"<t9100.32" would be useful, where 32 is the broken test.

Maybe we can come up with an interface that covers all 3 cases?

While exclusion can be used it adds an extra step to both cases, as you 
need to mentally negate what you want first.

It might be that we are looking at different use cases, as you are 
talking about whole test suits.