Re: on Amazon EFS (NFS): "Reference directory conflict: refs/heads/" with status code 128

On 08/24/2016 11:39 PM, Jeff King wrote:

On Wed, Aug 24, 2016 at 04:52:33PM -0400, Alex Nauda wrote:

quoted

Elastic File System (EFS) is Amazon's scalable filesystem product that
is exposed to the OS as an NFS mount. We're using EFS to host the
filesystem used by a Jenkins CI server. Sometimes when Jenkins tries
to git fetch, we get this error:
$ git -c core.askpass=true fetch --tags --progress
git@github.com:mediasilo/dodo.git
+refs/pull/*:refs/remotes/origin/pr/*
fatal: Reference directory conflict: refs/heads/
$ echo $? 128

Has anyone seen anything like this before? Any tips on how to troubleshoot it?

No, I haven't seen it before. That's an internal assertion in the refs
code that shouldn't ever happen. It looks like it happens when the loose
refs end up with duplicate directory entries. While a bug in git is an
obvious culprit, I wonder if it's possible that your filesystem might
expose the same name twice in one set of readdir() results.

+cc Michael, who added this assertion long ago (and since this is the
first report in all these years, it does make me suspect that the
filesystem is a critical part of reproducing).

Thanks for the CC.

I've never heard of this problem before.

What Git version are you using?

I tried to provoke the problem by hand-corrupting the packed-refs file,
but wasn't successful.

So Peff's suggestion that the problem originates in your filesystem
seems to be to be the most likely cause. A quick Google search found,
for example,

    https://bugzilla.redhat.com/show_bug.cgi?id=739222

http://superuser.com/questions/640419/how-can-i-have-two-files-with-the-same-name-in-a-directory-when-mounted-with-nfs

though these reports seem connected with having lots of files in the
directory, which seems unlikely for `$GIT_DIR/refs/`. But I didn't do a
more careful search, and it is easily possible that there are other bugs
in NFS (or EFS) that could be affecting you.

If this were repeatable, you could run Git under strace to test Peff's
hypothesis. But I suppose it only happens rarely, right?

Is it possible that multiple clients have the same NFS filesystem
mounted while Git is running? That would seem like an especially bad
idea and I could imagine it leading to problems like this.

It's surprising that you are seeing this problem in directory `refs`,
because (1) that directory is unlikely to have very many entries, and
(2) as far as I remember, Git will never delete the directories
`refs/heads` and `refs/tags`.

Michael