Re: [PATCH] tests: add an optional test to test git-annex

Ævar Arnfjörð Bjarmason [off-list ref] writes:

quoted

Well, it is one thing to place git-annex under CI to make sure its
latest and greatest works together well with our latest and greatest
(and it may be something we want to see happen), but driving its
tests from our testsuite sounds like a tail wagging the dog, at
least to me.

To me this is just a question of:

* Is it the case that git-annex tests for a lot of edge cases we don't
test for: Yes, probably. As evidenced by them spotting this
regression, and not us.

And I'd encourage them to keep doing so.

* We can (and should) add a test for the specific breakage we caused
in 2.13.0, but that's no replacement for other things annex may be
covering & we may be missing which'll catch future breakages.

* It's a pretty established practice to test a library (git) along
with its consumers (e.g. annex) before a major release.

I am not so sure about the division of labor.  What you are
advocating would work _ONLY_ if we test with a perfect & bug-free
version of the consumers.  If they are also a moving target, then
I do not think it is worth it.  After all, we are *not* in the
business of testing these consumers.

Unless I misunderstood you and you were saying that we freeze a
version, or a set of versions, of customer that is/are known to pass
their own tests, and test the combination of that frozen version of
the customer with our daily development.  If that is the case, then
I would agree that we are using their test to test us, not them.
But I somehow didn't get that impression, hence my reaction.