Re: [PATCH] tests: add an optional test to test git-annex

On Wed, May 17, 2017 at 4:45 AM, Junio C Hamano [off-list ref] wrote:

Ævar Arnfjörð Bjarmason  [off-list ref] writes:

quoted

Add an optional test to test git-annex. It's guarded by a new
EXTERNAL_TESTS environment variable. Running this test takes me 10
minutes.

[Re-arranged your mail because it worked better with my reply]

I do not mind at all to place the simple reproduction recipe Joey
posted as a new test in our test suite, though.  That kind of test
that catches changes to externally visible behaviour surely belongs
to our test suite.

This is not a replacement for having an isolated test for the issue
Joey noted. We should have a separate patch for that, but I did not
have time/interest in writing that up. This change is orthagonal to
that.

Well, it is one thing to place git-annex under CI to make sure its
latest and greatest works together well with our latest and greatest
(and it may be something we want to see happen), but driving its
tests from our testsuite sounds like a tail wagging the dog, at
least to me.

To me this is just a question of:

* Is it the case that git-annex tests for a lot of edge cases we don't
test for: Yes, probably. As evidenced by them spotting this
regression, and not us.

* We can (and should) add a test for the specific breakage we caused
in 2.13.0, but that's no replacement for other things annex may be
covering & we may be missing which'll catch future breakages.

* It's a pretty established practice to test a library (git) along
with its consumers (e.g. annex) before a major release.

* This allows us to do that at minimal cost. I think it makes sense to
add this and integration tests for other similar utilities if they're
similarly easy to integrate.