From: Junio C Hamano <hidden> Date: 2017-05-17 10:19:10
Ævar Arnfjörð Bjarmason [off-list ref] writes:
quoted
Well, it is one thing to place git-annex under CI to make sure its
latest and greatest works together well with our latest and greatest
(and it may be something we want to see happen), but driving its
tests from our testsuite sounds like a tail wagging the dog, at
least to me.
To me this is just a question of:
* Is it the case that git-annex tests for a lot of edge cases we don't
test for: Yes, probably. As evidenced by them spotting this
regression, and not us.
And I'd encourage them to keep doing so.
* We can (and should) add a test for the specific breakage we caused
in 2.13.0, but that's no replacement for other things annex may be
covering & we may be missing which'll catch future breakages.
* It's a pretty established practice to test a library (git) along
with its consumers (e.g. annex) before a major release.
I am not so sure about the division of labor. What you are
advocating would work _ONLY_ if we test with a perfect & bug-free
version of the consumers. If they are also a moving target, then
I do not think it is worth it. After all, we are *not* in the
business of testing these consumers.
Unless I misunderstood you and you were saying that we freeze a
version, or a set of versions, of customer that is/are known to pass
their own tests, and test the combination of that frozen version of
the customer with our daily development. If that is the case, then
I would agree that we are using their test to test us, not them.
But I somehow didn't get that impression, hence my reaction.
On Wed, May 17, 2017 at 12:19 PM, Junio C Hamano [off-list ref] wrote:
Ævar Arnfjörð Bjarmason [off-list ref] writes:
quoted
quoted
Well, it is one thing to place git-annex under CI to make sure its
latest and greatest works together well with our latest and greatest
(and it may be something we want to see happen), but driving its
tests from our testsuite sounds like a tail wagging the dog, at
least to me.
To me this is just a question of:
* Is it the case that git-annex tests for a lot of edge cases we don't
test for: Yes, probably. As evidenced by them spotting this
regression, and not us.
And I'd encourage them to keep doing so.
The point of this patch is that we can do this more systematically and
reliably, not have people discover this sort of thing after a major
release.
I.e. we can be pro-active about this instead of waiting for bug
reports to roll in.
quoted
* We can (and should) add a test for the specific breakage we caused
in 2.13.0, but that's no replacement for other things annex may be
covering & we may be missing which'll catch future breakages.
* It's a pretty established practice to test a library (git) along
with its consumers (e.g. annex) before a major release.
I am not so sure about the division of labor. What you are
advocating would work _ONLY_ if we test with a perfect & bug-free
version of the consumers. If they are also a moving target, then
I do not think it is worth it. After all, we are *not* in the
business of testing these consumers.
Unless I misunderstood you and you were saying that we freeze a
version, or a set of versions, of customer that is/are known to pass
their own tests, and test the combination of that frozen version of
the customer with our daily development. If that is the case, then
I would agree that we are using their test to test us, not them.
But I somehow didn't get that impression, hence my reaction.
The test I'm adding tests the release version of git-annex, so I think
in practice we don't have to worry about random changes of theirs
producing false positives for us.
The utility of this test is that sometime close to release someone
(e.g. me) can run it, if it fails let's see if it fails on the last
release version of ours, if so it's probably upstream breakage, or
like with the 2.13.0 release if it's OK on 2.12.0 it's our bug.
It'll never trip some random tester up since you need to explicitly
opt-in via EXTERNAL_TESTS=1, so honestly I'm a bit puzzled by these
objections. This incurs no burden on either devs, packagers or users,
and would have demonstrably detected an issue we'd rather have wanted
to know about pre-release than post-release as is the case now.
From: Stefan Beller <hidden> Date: 2017-05-17 23:03:56
On Wed, May 17, 2017 at 12:33 PM, Ævar Arnfjörð Bjarmason
[off-list ref] wrote:
On Wed, May 17, 2017 at 12:19 PM, Junio C Hamano [off-list ref] wrote:
quoted
Ævar Arnfjörð Bjarmason [off-list ref] writes:
quoted
quoted
Well, it is one thing to place git-annex under CI to make sure its
latest and greatest works together well with our latest and greatest
(and it may be something we want to see happen), but driving its
tests from our testsuite sounds like a tail wagging the dog, at
least to me.
To me this is just a question of:
* Is it the case that git-annex tests for a lot of edge cases we don't
test for: Yes, probably. As evidenced by them spotting this
regression, and not us.
And I'd encourage them to keep doing so.
The point of this patch is that we can do this more systematically and
reliably, not have people discover this sort of thing after a major
release.
I.e. we can be pro-active about this instead of waiting for bug
reports to roll in.
We can be pro-active without this patch, too. ;)
I guess this makes it just easier for someone out of the Git community to
be proactive in searching for defects.
The utility of this test is that sometime close to release someone
(e.g. me) can run it, if it fails let's see if it fails on the last
release version of ours, if so it's probably upstream breakage, or
like with the 2.13.0 release if it's OK on 2.12.0 it's our bug.
It'll never trip some random tester up since you need to explicitly
opt-in via EXTERNAL_TESTS=1, so honestly I'm a bit puzzled by these
objections. This incurs no burden on either devs, packagers or users,
and would have demonstrably detected an issue we'd rather have wanted
to know about pre-release than post-release as is the case now.
I think this patch could spark a discussion what we expect from our test suite.
Is it a contract that we promise, as in "Here are some blessed workflows,
our users should see and imitate"?
This patch sort of feels like (a) we let others dictate the contract to us
and (b) we choose the easy way out as we do not write enough tests on
our own, so we force downstream to help us out.
I guess it is appropriate to compare this level of testing to other external
tests, such as cppcheck, travis, coverity. For some of them (travis) we
carry some code with us (.travis), not so for others (coverity would benefit
from [1] that I carry outside of Junios tree). Admittedly these external tools
are all focused on testing, not on building on top of Git, which brings
in other expectations from these tools.
[1] https://github.com/stefanbeller/git/commit/0781fbafb9dd2a995ba62a9af5f7581e3cf05359
---
As mentioned in the other thread[2], the idea was floated to have this test or
the downstream project as a submodule, and if you are interested in the test,
then you would obtain the submodule and run the tests (in there?)
That would have the benefit of less 'uninteresting' glue logic for those who
are not interested in these tests as gitlinks may be easier to ignore
for a developer
than code. (Less data to fetch, git-grep also doesn't yield results
for the uninteresting
parts as these submodules would be uninitialized).
(The more I talk about this "downstream tests in submodules"
idea, the less convinced I am)
[2] https://public-inbox.org/git/20170517113824.31700-1-avarab@gmail.com/
From: Brandon Williams <hidden> Date: 2017-05-17 23:56:34
On 05/17, Junio C Hamano wrote:
Ævar Arnfjörð Bjarmason [off-list ref] writes:
quoted
quoted
Well, it is one thing to place git-annex under CI to make sure its
latest and greatest works together well with our latest and greatest
(and it may be something we want to see happen), but driving its
tests from our testsuite sounds like a tail wagging the dog, at
least to me.
To me this is just a question of:
* Is it the case that git-annex tests for a lot of edge cases we don't
test for: Yes, probably. As evidenced by them spotting this
regression, and not us.
And I'd encourage them to keep doing so.
quoted
* We can (and should) add a test for the specific breakage we caused
in 2.13.0, but that's no replacement for other things annex may be
covering & we may be missing which'll catch future breakages.
* It's a pretty established practice to test a library (git) along
with its consumers (e.g. annex) before a major release.
I am not so sure about the division of labor. What you are
advocating would work _ONLY_ if we test with a perfect & bug-free
version of the consumers. If they are also a moving target, then
I do not think it is worth it. After all, we are *not* in the
business of testing these consumers.
I agree with this. It makes no sense to test consumers of git, its
downstream's job to do that. Though I do think that its perfectly
reasonable to test that our API works as advertised such that consumer's
can rely on git.
Unless I misunderstood you and you were saying that we freeze a
version, or a set of versions, of customer that is/are known to pass
their own tests, and test the combination of that frozen version of
the customer with our daily development. If that is the case, then
I would agree that we are using their test to test us, not them.
But I somehow didn't get that impression, hence my reaction.
From: Brandon Williams <hidden> Date: 2017-05-17 23:59:55
On 05/17, Stefan Beller wrote:
On Wed, May 17, 2017 at 12:33 PM, Ævar Arnfjörð Bjarmason
[off-list ref] wrote:
quoted
On Wed, May 17, 2017 at 12:19 PM, Junio C Hamano [off-list ref] wrote:
quoted
Ævar Arnfjörð Bjarmason [off-list ref] writes:
quoted
quoted
Well, it is one thing to place git-annex under CI to make sure its
latest and greatest works together well with our latest and greatest
(and it may be something we want to see happen), but driving its
tests from our testsuite sounds like a tail wagging the dog, at
least to me.
To me this is just a question of:
* Is it the case that git-annex tests for a lot of edge cases we don't
test for: Yes, probably. As evidenced by them spotting this
regression, and not us.
And I'd encourage them to keep doing so.
The point of this patch is that we can do this more systematically and
reliably, not have people discover this sort of thing after a major
release.
I.e. we can be pro-active about this instead of waiting for bug
reports to roll in.
We can be pro-active without this patch, too. ;)
I guess this makes it just easier for someone out of the Git community to
be proactive in searching for defects.
quoted
The utility of this test is that sometime close to release someone
(e.g. me) can run it, if it fails let's see if it fails on the last
release version of ours, if so it's probably upstream breakage, or
like with the 2.13.0 release if it's OK on 2.12.0 it's our bug.
It'll never trip some random tester up since you need to explicitly
opt-in via EXTERNAL_TESTS=1, so honestly I'm a bit puzzled by these
objections. This incurs no burden on either devs, packagers or users,
and would have demonstrably detected an issue we'd rather have wanted
to know about pre-release than post-release as is the case now.
I think this patch could spark a discussion what we expect from our test suite.
Is it a contract that we promise, as in "Here are some blessed workflows,
our users should see and imitate"?
This patch sort of feels like (a) we let others dictate the contract to us
and (b) we choose the easy way out as we do not write enough tests on
our own, so we force downstream to help us out.
I guess it is appropriate to compare this level of testing to other external
tests, such as cppcheck, travis, coverity. For some of them (travis) we
carry some code with us (.travis), not so for others (coverity would benefit
from [1] that I carry outside of Junios tree). Admittedly these external tools
are all focused on testing, not on building on top of Git, which brings
in other expectations from these tools.
[1] https://github.com/stefanbeller/git/commit/0781fbafb9dd2a995ba62a9af5f7581e3cf05359
---
As mentioned in the other thread[2], the idea was floated to have this test or
the downstream project as a submodule, and if you are interested in the test,
then you would obtain the submodule and run the tests (in there?)
Despite working on improving submodules, I'm still very skeptical of
adding submodules to git.git, at least at this point in time. I just
feel like the submodule experience still has a lot of work before we
begin using it...Though My feeling may be completely unfounded.
That would have the benefit of less 'uninteresting' glue logic for those who
are not interested in these tests as gitlinks may be easier to ignore
for a developer
than code. (Less data to fetch, git-grep also doesn't yield results
for the uninteresting
parts as these submodules would be uninitialized).
(The more I talk about this "downstream tests in submodules"
idea, the less convinced I am)
[2] https://public-inbox.org/git/20170517113824.31700-1-avarab@gmail.com/