Re: [PATCH 1/2] t0303: set reason for skipping tests

On Tue, Mar 13, 2012 at 05:53:32PM -0400, Jeff King wrote:

The reason is that the individual tests do not verify all of the
preconditions themselves, but rather build on each other.

Right. I added a note based on this sentence in the test description.

In an ideal world, each test snippet would be totally independent and
check its preconditions. That would give us an accurate count of how
many tests actually passed or failed. But fundamentally we only care
about "did they all succeed or not?", which the current script does tell
us (either test 2 fails, or if it succeeds, then we have checked the
precondition for test 4). And the tests end up way shorter, because we
don't repeat the preconditions over and over.

If you want to try to make the tests more robust, you can (for example,
you can tighten the precondition on 4 to check "does it give the right
answer with the right protocol" instead of just "does it ever give us
the right answer"). But personally, I'm not sure it's worth that much
effort.

Yeah.

quoted

quoted

Should they actually say "# SKIP ..." to tell prove what's going on? I
don't know very much about TAP.

# SKIP is used when skipping individual tests (IIUC), but when we
skip a group of tests, we simply jump over them and this message is
purely informative output that is not interpreted by the harness.

Just looking at test-lib.sh, it seems like we output "# SKIP" when we do
skip_all. But I think you would have to give a count of which tests you
skipped (e.g., try "./t5541-http-push.sh" to see its TAP output). Which
means when skipping a subset, you'd have to deal with test numbering,
which is a pain. So it's probably not worth worrying about.

Skipped test numbering could done automatically by using test prereqs,
but (after actually doing that and discarding) I agree that it isn't
worth the trouble.


Jonathan Nieder wrote:

Nice idea, but shouldn't this description be in test_description so I
can view it by running "sh t0303-credential-external.sh --help"?

Done.

Updated patches follow.

(This time I tested with GIT_TEST_CREDENTIAL_HELPER=cache
GIT_TEST_CREDENTIAL_HELPER_TIMEOUT="cache --timeout=1,3" and things
seem to work as expected.)

Zbyszek