reversion in GIT_COMMON_DIR refs path

Bisecting this test suite failure
https://git-annex.branchable.com/git-annex_in_nixpkgs_fails_with_git-2.13.0/
I landed on commit f57f37e2e1bf11ab4cdfd221ad47e961ba9353a0 to git.

It seems that changed resolving refs paths when GIT_DIR and GIT_COMMON_DIR
are both set. While before refs were looked for in GIT_COMMON_DIR,
now they're not.

Test case:

#!/bin/sh
set -e
set -x
rm -rf testdir
git init testdir
cd testdir
echo 1 > foo
git add foo
git commit -m add
mkdir dummy
mkdir dummy/overlay
cp .git/index .git/HEAD dummy/overlay
#cp .git/refs .git/packed-refs dummy/overlay -a
cd dummy
export GIT_COMMON_DIR=`pwd`/../.git
export GIT_DIR=`pwd`/overlay
git rev-parse --git-path refs/heads/master
git show refs/heads/master

This script succeeds with git 2.11.0, but with 2.13.0, it fails:

fatal: ambiguous argument 'refs/heads/master': unknown revision or path not in the working tree.

It seems to be failing to look up refs in GIT_COMMON_DIR.
Note that uncommenting the commented out line in the script, to copy the refs
into GIT_DIR, makes it succeed.

git rev-parse --git-path refs/heads/master shows the GIT_COMMON_DIR/refs path
still (as gitrepository-layout documents). So this reversion made
different parts of git disagreeing about the refs path.

On Tue, May 16, 2017 at 7:10 PM, Joey Hess [off-list ref] wrote:

Bisecting this test suite failure
https://git-annex.branchable.com/git-annex_in_nixpkgs_fails_with_git-2.13.0/
I landed on commit f57f37e2e1bf11ab4cdfd221ad47e961ba9353a0 to git.

That links's broken for me. Looking at your wiki it looks like you
mean: https://git-annex.branchable.com/bugs/git-annex_in_nixpkgs_fails_with_git-2.13.0/

I have no idea what this bug is about, but side-question: It looks
like this is git-annex's own test suite that's failing with 2.13.0, is
that right?

It would be very nice to have a test in git itself to test with
git-annex. I.e. some optional test that just pulls down the latest
git-annex release & runs its tests against the git we're building.

Thanks for annex b.t.w., I use it a lot.

Ævar Arnfjörð Bjarmason wrote:

On Tue, May 16, 2017 at 7:10 PM, Joey Hess [off-list ref] wrote:

quoted

Bisecting this test suite failure
https://git-annex.branchable.com/git-annex_in_nixpkgs_fails_with_git-2.13.0/
I landed on commit f57f37e2e1bf11ab4cdfd221ad47e961ba9353a0 to git.

That links's broken for me. Looking at your wiki it looks like you
mean: https://git-annex.branchable.com/bugs/git-annex_in_nixpkgs_fails_with_git-2.13.0/

Thanks for correcting that

I have no idea what this bug is about, but side-question: It looks
like this is git-annex's own test suite that's failing with 2.13.0, is
that right?

Yes indeed.

It would be very nice to have a test in git itself to test with
git-annex. I.e. some optional test that just pulls down the latest
git-annex release & runs its tests against the git we're building.

Thanks for annex b.t.w., I use it a lot.

If the git devs are ok with this, I certianly would be happy if such
tests were run, at least occasionally, on the git side!

-- 
see shy jo

Add an optional test to test git-annex. It's guarded by a new
EXTERNAL_TESTS environment variable. Running this test takes me 10
minutes.

As reported by Joey Hess in "reversion in GIT_COMMON_DIR refs path"[1]
commit f57f37e2e1 ("files-backend: remove the use of git_path()",
2017-03-26) first released as part of the 2.13.0 broke git-annex's
test suite.

This could have been spotted by us before the release by optionally
running the git-annex test suite as part of git itself. This optional
test does that. It currently fails due to the reported regression,
but, passes on the 2.12.0 release.

The git-annex revision to test can be specified with the
GIT_TEST_GIT_ANNEX_REVISION environment variable. Joey has expressed
interest in testing development versions of git against git-annex[2],
and can now test the latest revision with:

    EXTERNAL_TESTS=1 GIT_TEST_GIT_ANNEX_REVISION='@{u}' ./t9950-git-annex.sh

By default the test finds the latest git-annex release tag and tests
that, since the primary purpose is to test regressions in git which
cause git-annex to fail, not regressions in git-annex itself.

The t9* test namespace is currently full as documented in t/README. In
lie of an empty t9X for "external tools" this change claims t995* for
that purpose.

1. [off-list ref]
   (https://public-inbox.org/git/20170516175906.hdwn4x5md7dj7fo3@kitenet.net/T/)
2. http://git-annex.branchable.com/devblog/day_459__git_bug/

Signed-off-by: Ævar Arnfjörð Bjarmason <redacted>
---

On Tue, May 16, 2017 at 7:59 PM, Joey Hess [off-list ref] wrote:

Ævar Arnfjörð Bjarmason wrote:

quoted

On Tue, May 16, 2017 at 7:10 PM, Joey Hess [off-list ref] wrote:
I have no idea what this bug is about, but side-question: It looks
like this is git-annex's own test suite that's failing with 2.13.0, is
that right?

Yes indeed.

quoted

It would be very nice to have a test in git itself to test with
git-annex. I.e. some optional test that just pulls down the latest
git-annex release & runs its tests against the git we're building.

Thanks for annex b.t.w., I use it a lot.

If the git devs are ok with this, I certianly would be happy if such
tests were run, at least occasionally, on the git side!

I for one would run this test occasionally, and perhaps we could even
run it as part of Travis eventually (although there would be a *lot*
of Haskell deps, on my box "apt build-dep git-annex" brought in 1/2 GB
of packages).

As noted in the commit message, once this is part of git.git you can
easily set an environment variable to test the bleeding edge of git
against any arbitrary git-annex version.

 t/t9950-git-annex.sh | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 52 insertions(+)
 create mode 100755 t/t9950-git-annex.sh

diff --git a/t/t9950-git-annex.sh b/t/t9950-git-annex.sh
new file mode 100755
index 0000000000..2cbc1f4be3
--- /dev/null
+++ b/t/t9950-git-annex.sh

@@ -0,0 +1,52 @@
+#!/bin/sh
+
+test_description='the git-annex test suite'
+. ./test-lib.sh
+
+if test -z "$EXTERNAL_TESTS"
+then
+	skip_all='skipping tests of external tools. EXTERNAL_TESTS not defined'
+	test_done
+fi
+
+if test -n "$NO_CURL"
+then
+	skip_all='skipping test, git built without http support'
+	test_done
+fi
+
+test_expect_success 'clone git-annex' '
+	git clone https://git.joeyh.name/git/git-annex.git
+'
+
+if test -n "$GIT_TEST_GIT_ANNEX_REVISION"
+then
+	test_expect_success "plan to test git-annex $GIT_TEST_GIT_ANNEX_REVISION" "
+		echo '$GIT_TEST_GIT_ANNEX_REVISION' >revision-to-test
+	"
+else
+	test_expect_success "plan to test git-annex's latest release tag" '
+		git -C git-annex tag --sort=version:refname -l "[0-9]*.[0-9]*" |
+			tail -n 1 >revision-to-test
+	'
+fi
+
+test_expect_success 'checkout $(cat revision-to-test) for testing' '
+	git -C git-annex checkout $(cat revision-to-test)
+'
+
+test_expect_success 'build git-annex (if this fails, you are likely missing its Haskell dependencies' '
+	(
+		cd git-annex &&
+		make
+	)
+'
+
+test_expect_success 'test git-annex' '
+	(
+		cd git-annex &&
+		make test
+	)
+'
+
+test_done

-- 
2.13.0.303.g4ebf302169

Nice work.

Note that you can export BUILDER=stack and git-annex will build with a
known good dependency stack, which can be more reliable/cross platform
than using apt to install its build dependencies. That needs
https://docs.haskellstack.org/ installed. Also it currently needs
GIT_TEST_GIT_ANNEX_REVISION=master since I improved git-annex's
Makefile slightly.

-- 
see shy jo

Joey Hess wrote:

Bisecting this test suite failure
https://git-annex.branchable.com/git-annex_in_nixpkgs_fails_with_git-2.13.0/
I landed on commit f57f37e2e1bf11ab4cdfd221ad47e961ba9353a0 to git.

It seems that changed resolving refs paths when GIT_DIR and GIT_COMMON_DIR
are both set. While before refs were looked for in GIT_COMMON_DIR,
now they're not.

In case there's any doubt about whether this is a reversion or an
intentional change, see gitrepository-layout(5):

       refs
           References are stored in subdirectories of this directory. The git
           prune command knows to preserve objects reachable from refs found
           in this directory and its subdirectories. This directory is ignored
           if $GIT_COMMON_DIR is set and "$GIT_COMMON_DIR/refs" will be used
           instead.

So the documented behavior is broken.

-- 
see shy jo

On Fri, May 19, 2017 at 9:37 PM, Joey Hess [off-list ref] wrote:

Joey Hess wrote:

quoted

Bisecting this test suite failure
https://git-annex.branchable.com/git-annex_in_nixpkgs_fails_with_git-2.13.0/
I landed on commit f57f37e2e1bf11ab4cdfd221ad47e961ba9353a0 to git.

It seems that changed resolving refs paths when GIT_DIR and GIT_COMMON_DIR
are both set. While before refs were looked for in GIT_COMMON_DIR,
now they're not.

In case there's any doubt about whether this is a reversion or an
intentional change, see gitrepository-layout(5):

       refs
           References are stored in subdirectories of this directory. The git
           prune command knows to preserve objects reachable from refs found
           in this directory and its subdirectories. This directory is ignored
           if $GIT_COMMON_DIR is set and "$GIT_COMMON_DIR/refs" will be used
           instead.

So the documented behavior is broken.

It's a gray area. When I wrote that I think I forgot about
per-worktree refs (refs/bisect/*) so "This directory is ignored" is
not completely true. The final line (probably won't help you much) is
"per-repo refs must be read from $GIT_COMMON_DIR/refs, per-worktree
from $GIT_DIR". The fact that we looked per-repo (like master) in
$GIT_DIR is probably an unwanted side effect.
-- 
Duy