Re: concurrent access to multiple local git repos is error prone

Alexander Mills wrote:

Yeah in this case what appeared to be happening is that if `git status` was
called concurrently, frequently I wouldnt get any stdout..

Thanks for reporting.  A few questions:

What platform are you on?  What version of Git are you running?  What
is the output of "git version --build-options"?

What is the exit status from these "git status" invocations that
didn't write output?  What options are you passing to "git status"?
Are there any other interesting symptoms?

Is this part of a larger tool or script?  Are there other operations
going on (e.g. something sending signals to these git processes)?

Does the repository pass "git fsck"?  Any other symptoms or hints that
could help in tracking this down?

Can you paste a transcript of the commands or script you ran and what
output it produced?  Is this something I should be able to reproduce?

Thanks and hope that helps,
Jonathan

I am on Ubuntu, I will run the checks that you mentioned. I assume
it's operator error for now though. Like I said, it was just `git
status` that was sometimes failing to send any stdout, presumably when
access to a certain git repo was concurrent, but possibly due to
concurrent access across multiple repos.

Also, as an aside, this seems to be a bug, but probably a known bug:

$ git diff-index  HEAD; echo $?

:100755 100755 60e5d683c1eb3e61381b1a8ec2db822b94b9faec
0000000000000000000000000000000000000000 M      cli/npp_check_merge.sh
:100644 100644 35a453544de41e2227ab0afab31a396d299139e9
0000000000000000000000000000000000000000 M      src/find-projects.ts
:100644 100644 c1ee7bc18e6604cbf0d16653e9366109d6ac2ec9
0000000000000000000000000000000000000000 M      src/tables.ts
:100644 100644 29d9674fbb48f223f3434179d666b2aa991ad05a
0000000000000000000000000000000000000000 M
src/vcs-helpers/git-helpers.ts
0

$ git diff-index --quiet HEAD; echo $?
1

different exit codes depending on whether --quiet was used. In this
case, the exit code should be consistent.
The bug is with the `git diff-index` command, as you can see.

-alex

On Sun, Aug 5, 2018 at 1:11 AM, Jonathan Nieder [off-list ref] wrote:

Alexander Mills wrote:

quoted

Yeah in this case what appeared to be happening is that if `git status` was
called concurrently, frequently I wouldnt get any stdout..

Thanks for reporting.  A few questions:

What platform are you on?  What version of Git are you running?  What
is the output of "git version --build-options"?

What is the exit status from these "git status" invocations that
didn't write output?  What options are you passing to "git status"?
Are there any other interesting symptoms?

Is this part of a larger tool or script?  Are there other operations
going on (e.g. something sending signals to these git processes)?

Does the repository pass "git fsck"?  Any other symptoms or hints that
could help in tracking this down?

Can you paste a transcript of the commands or script you ran and what
output it produced?  Is this something I should be able to reproduce?

Thanks and hope that helps,
Jonathan

-- 
Alexander D. Mills
¡¡¡ New cell phone number: (415)730-1805 !!!
alexander.d.mills@gmail.com

www.linkedin.com/pub/alexander-mills/b/7a5/418/

On 08/05, Alexander Mills wrote:

Also, as an aside, this seems to be a bug, but probably a known bug:

$ git diff-index  HEAD; echo $?

:100755 100755 60e5d683c1eb3e61381b1a8ec2db822b94b9faec
0000000000000000000000000000000000000000 M      cli/npp_check_merge.sh
:100644 100644 35a453544de41e2227ab0afab31a396d299139e9
0000000000000000000000000000000000000000 M      src/find-projects.ts
:100644 100644 c1ee7bc18e6604cbf0d16653e9366109d6ac2ec9
0000000000000000000000000000000000000000 M      src/tables.ts
:100644 100644 29d9674fbb48f223f3434179d666b2aa991ad05a
0000000000000000000000000000000000000000 M
src/vcs-helpers/git-helpers.ts
0

$ git diff-index --quiet HEAD; echo $?
1

different exit codes depending on whether --quiet was used. In this
case, the exit code should be consistent.
The bug is with the `git diff-index` command, as you can see.

This is not a bug. 'git diff-index' (and 'git diff') only give an exit
code other than 0 in the default case if something actually goes wrong
with generating the diff, which in the usual case it shouldn't.

To get an exit code from 'git diff-index' if there are differences,
you'd have to pass the '--exit-code' flag.  The '--quite' flag implies
'--exit-code', as there's not much use in 'git diff --quiet' if
there's not even an exit code showing whether there are differences or
not.

The original patch (and more importantly the reasoning why
'--exit-code' is not the default behaviour for 'git diff') can be
found at [1].

[1]: https://public-inbox.org/git/81b0412b0703131717k7106ee1cg964628f0bda2c83e@mail.gmail.com/

-alex

Hi Johnathan,

Yeah this concurrency problem is real. Not only does it happen with
`git status` the same thing happens with `git rev-parse
--show-toplevel`.
What happens is that I get no stdout when repos are accessed
concurrently (and no stderr). If I limit concurrency to 1, the problem
goes away. When I up the concurrency, the problem is sporadic, which
is the exact signal for a concurrency/race-condition related issue.
The signs are damn clear. I have seen this problem on MacOS I think a
year back on a different project, but I never reported it b/c I hadn't
really verified it.

Like I said I am on Ubuntu. I have 3 git repos that are incorporated
into the tool that's generating the problem. For one repo I got this:

$ git fsck

Checking object directories: 100% (256/256), done.
dangling tree 4b825dc642cb6eb9a060e54bf8d69288fbee4904

For `$ git version --build-options` I have:

git version 2.17.1
cpu: x86_64
no commit associated with this build
sizeof-long: 8


-alex



On Sun, Aug 5, 2018 at 1:11 AM, Jonathan Nieder [off-list ref] wrote:

Alexander Mills wrote:

quoted

Yeah in this case what appeared to be happening is that if `git status` was
called concurrently, frequently I wouldnt get any stdout..

Thanks for reporting.  A few questions:

What platform are you on?  What version of Git are you running?  What
is the output of "git version --build-options"?

What is the exit status from these "git status" invocations that
didn't write output?  What options are you passing to "git status"?
Are there any other interesting symptoms?

Is this part of a larger tool or script?  Are there other operations
going on (e.g. something sending signals to these git processes)?

Does the repository pass "git fsck"?  Any other symptoms or hints that
could help in tracking this down?

Can you paste a transcript of the commands or script you ran and what
output it produced?  Is this something I should be able to reproduce?

Thanks and hope that helps,
Jonathan

-- 
Alexander D. Mills
¡¡¡ New cell phone number: (415)730-1805 !!!
alexander.d.mills@gmail.com

www.linkedin.com/pub/alexander-mills/b/7a5/418/

To add something to the previous message, I have strong evidence that
the problem occurs when *different* repos are accessed concurrency,
not the same repo, as bizarre as that may be.

-alex

On Mon, Aug 6, 2018 at 12:36 AM, Alexander Mills
[off-list ref] wrote:

Hi Johnathan,

Yeah this concurrency problem is real. Not only does it happen with
`git status` the same thing happens with `git rev-parse
--show-toplevel`.
What happens is that I get no stdout when repos are accessed
concurrently (and no stderr). If I limit concurrency to 1, the problem
goes away. When I up the concurrency, the problem is sporadic, which
is the exact signal for a concurrency/race-condition related issue.
The signs are damn clear. I have seen this problem on MacOS I think a
year back on a different project, but I never reported it b/c I hadn't
really verified it.

Like I said I am on Ubuntu. I have 3 git repos that are incorporated
into the tool that's generating the problem. For one repo I got this:

$ git fsck

Checking object directories: 100% (256/256), done.
dangling tree 4b825dc642cb6eb9a060e54bf8d69288fbee4904

For `$ git version --build-options` I have:

git version 2.17.1
cpu: x86_64
no commit associated with this build
sizeof-long: 8


-alex



On Sun, Aug 5, 2018 at 1:11 AM, Jonathan Nieder [off-list ref] wrote:

quoted

Alexander Mills wrote:

quoted

Yeah in this case what appeared to be happening is that if `git status` was
called concurrently, frequently I wouldnt get any stdout..

Thanks for reporting.  A few questions:

What platform are you on?  What version of Git are you running?  What
is the output of "git version --build-options"?

What is the exit status from these "git status" invocations that
didn't write output?  What options are you passing to "git status"?
Are there any other interesting symptoms?

Is this part of a larger tool or script?  Are there other operations
going on (e.g. something sending signals to these git processes)?

Does the repository pass "git fsck"?  Any other symptoms or hints that
could help in tracking this down?

Can you paste a transcript of the commands or script you ran and what
output it produced?  Is this something I should be able to reproduce?

Thanks and hope that helps,
Jonathan

--
Alexander D. Mills
¡¡¡ New cell phone number: (415)730-1805 !!!
alexander.d.mills@gmail.com

www.linkedin.com/pub/alexander-mills/b/7a5/418/

-- 
Alexander D. Mills
¡¡¡ New cell phone number: (415)730-1805 !!!
alexander.d.mills@gmail.com

www.linkedin.com/pub/alexander-mills/b/7a5/418/

(administrivia: please don't top-post)
Hi Alex,

Alexander Mills wrote:

Yeah this concurrency problem is real. Not only does it happen with
`git status` the same thing happens with `git rev-parse
--show-toplevel`.

Sorry for the confusion --- I didn't mean to claim your experience was
not real!

What I wanted to make clear is that

 1. Git is designed to allow concurrent reads of a repository (and
    pushes to a repository).  If it doesn't work, that is a simple bug,
    not a design goal.

 2. Plenty of people rely on concurrently accessing repositories, so
    if it doesn't work, then (i) we definitely want to know and (ii)
    we're going to need a lot of detail to figure out what's happening,
    so we can fix it.

Does that make it clearer?

What happens is that I get no stdout when repos are accessed
concurrently (and no stderr). If I limit concurrency to 1, the problem
goes away. When I up the concurrency, the problem is sporadic, which
is the exact signal for a concurrency/race-condition related issue.
The signs are damn clear. I have seen this problem on MacOS I think a
year back on a different project, but I never reported it b/c I hadn't
really verified it.

Like I said I am on Ubuntu. I have 3 git repos that are incorporated
into the tool that's generating the problem. For one repo I got this:

$ git fsck

Checking object directories: 100% (256/256), done.
dangling tree 4b825dc642cb6eb9a060e54bf8d69288fbee4904

For `$ git version --build-options` I have:

git version 2.17.1
cpu: x86_64
no commit associated with this build
sizeof-long: 8

Thanks.  My best idea for a next step is that if you can come up with a
reproduction recipe, that would be very helpful.

It doesn't have to reproduce 100% of the time, but e.g. if you have a
script that reproduces it 50% of the time, I can run that script in a
loop.

Thanks,
Jonathan

On Mon, Aug 6, 2018 at 9:38 AM Alexander Mills
[off-list ref] wrote:

Yeah this concurrency problem is real. Not only does it happen with
`git status` the same thing happens with `git rev-parse
--show-toplevel`.

"git rev-parse --show-toplevel" having this same problem helps. This
command should never make any update in the repository, not even
taking any lock and very basic access to the repository (I think it
just needs to resolve HEAD, it does not even access object database).
It's so "simple" [1] that makes me think this problem is something not
really related to git. Perhaps the process is terminated abnormally
because it hits some system limits?

[1] well the repo discovery steps are a bit messy  but I think it's
unlikely we have any racing issues there.

What happens is that I get no stdout when repos are accessed
concurrently (and no stderr).

-- 
Duy