Hey,
We have a bunch of Windows users, unfortunately, and they're using the
latest msysGit release (Git-1.7.3.1-preview20101002).
An interesting issue we've noticed is that the Time To Complete of
their common operations start deteriorating inexplicably, and
severely, some time after the clone.
For instance, immediately after a clone, `git status` takes about
5-6s. Which is slow compared to Linux (consistent 1-2s), but still
usable (it's a BIG repo).
However, after a reboot (of all things), `git status` latency
skyrockets to 14-15s, making the repo unusable.
Any idea what's going on? We just recently switched from SVN, and
those users are getting really frustrated. BTW, the only real
alternative I'm aware of, Cygwin's git, is even slower.
Thanks, D
From: Wilbert van Dolleweerd <hidden> Date: 2016-06-15 22:50:05
We have a bunch of Windows users, unfortunately, and they're using the
latest msysGit release (Git-1.7.3.1-preview20101002).
An interesting issue we've noticed is that the Time To Complete of
their common operations start deteriorating inexplicably, and
severely, some time after the clone.
For instance, immediately after a clone, `git status` takes about
5-6s. Which is slow compared to Linux (consistent 1-2s), but still
usable (it's a BIG repo).
However, after a reboot (of all things), `git status` latency
skyrockets to 14-15s, making the repo unusable.
Any idea what's going on? We just recently switched from SVN, and
those users are getting really frustrated. BTW, the only real
alternative I'm aware of, Cygwin's git, is even slower.
From: Stephen Bash <hidden> Date: 2016-06-15 22:50:05
----- Original Message -----
From: "Dun Peal" <redacted>
To: "Wilbert van Dolleweerd" <redacted>
Cc: "Git ML" <redacted>
Sent: Tuesday, November 23, 2010 2:59:17 PM
Subject: Re: Inexplicably deteriorating performance of Git repositories on Windows
On Tue, Nov 23, 2010 at 7:12 PM, Wilbert van Dolleweerd
[off-list ref] wrote:
quoted
How big is your repository? We're using some fairly big repositories
over here but I haven't seen this behavior with the latest version
of msysgit.
The working copy totals about 4GB. The .git directory, tightly packed,
is 1GB.
We're working with about a 1.5GB repository, and while I haven't seen an specific msysgit slow downs, I did run into build issues due to Windows anti-virus programs (on-access scans, new files scans, etc). I had to add my development directory to the anti-virus exception list to speed things back up.
That being said, I do most of my development on Mac and Linux, and msysgit is noticeably slower across the board for me...
Thanks,
Stephen
On Tue, Nov 23, 2010 at 8:25 PM, Stephen Bash [off-list ref] wrote:
----- Original Message -----
We're working with about a 1.5GB repository, and while I haven't seen an specific msysgit slow downs, I did run into build issues due to Windows anti-virus programs (on-access scans, new files scans, etc). I had to add my development directory to the anti-virus exception list to speed things back up.
Yeah, the performance numbers I mentioned are *after* excluding our AV
software from that directory.
That being said, I do most of my development on Mac and Linux, and msysgit is noticeably slower across the board for me...
Yup, as mentioned, even under the best case Windows scenario (freshly
cloned repo) I'm still seeing `git status` latencies that are x2-3
times those of Linux machines.
I don't hope to get Windows' git to be as fast as on Linux; at this
point, just making it fast enough to be usable would be an
achievement: I can't tell developers who just switched from a fast SVN
setup to wait for 15s for `git status`, an operation they perform
dozens of times per day (other operations, like stash, also take
involve long, unworkable waits).
.D
From: Martin Langhoff <hidden> Date: 2016-06-15 22:50:05
On Tue, Nov 23, 2010 at 2:08 PM, Dun Peal [off-list ref] wrote:
However, after a reboot (of all things), `git status` latency
skyrockets to 14-15s, making the repo unusable.
A reboot clears your cache. Reboot and time the first and second run
of git status.
The first one should be slow (due to cold cache), the second one
should be roughly as fast as right after a clone.
If it's not, more debugging may be neeed. But always take your timings
on a well seeded ('hot') cache.
cheers,
m
--
martin.langhoff@gmail.com
martin@laptop.org -- School Server Architect
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
On Tue, Nov 23, 2010 at 9:13 PM, Martin Langhoff
[off-list ref] wrote:
A reboot clears your cache. Reboot and time the first and second run
of git status.
Thanks, after all this benchmarking I'm well aware of cold and warm
caches. The 14-15s timing is unfortunately stable across multiple
runs, after the cache was warm (without it, we got times much longer
than that).
.D
Are your users using the 'show my status in the prompt' feature?
If so, then disable all but showing the current branch, it makes a whole
lot of difference :-)
On 11/23/2010 08:08 PM, Dun Peal wrote:
Hey,
We have a bunch of Windows users, unfortunately, and they're using the
latest msysGit release (Git-1.7.3.1-preview20101002).
An interesting issue we've noticed is that the Time To Complete of
their common operations start deteriorating inexplicably, and
severely, some time after the clone.
For instance, immediately after a clone, `git status` takes about
5-6s. Which is slow compared to Linux (consistent 1-2s), but still
usable (it's a BIG repo).
However, after a reboot (of all things), `git status` latency
skyrockets to 14-15s, making the repo unusable.
Any idea what's going on? We just recently switched from SVN, and
those users are getting really frustrated. BTW, the only real
alternative I'm aware of, Cygwin's git, is even slower.
Thanks, D
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, Nov 23, 2010 at 9:49 PM, Ferry Huberts [off-list ref] wrote:
Are your users using the 'show my status in the prompt' feature?
If so, then disable all but showing the current branch, it makes a whole
lot of difference :-)
From: Andreas Ericsson <hidden> Date: 2016-06-15 22:50:06
On 11/23/2010 08:08 PM, Dun Peal wrote:
Hey,
We have a bunch of Windows users, unfortunately, and they're using the
latest msysGit release (Git-1.7.3.1-preview20101002).
An interesting issue we've noticed is that the Time To Complete of
their common operations start deteriorating inexplicably, and
severely, some time after the clone.
For instance, immediately after a clone, `git status` takes about
5-6s. Which is slow compared to Linux (consistent 1-2s), but still
usable (it's a BIG repo).
How many refs (tags and branches) do you have?
Are the refs packed or loose?
If they are loose, does packing them resolve the issue?
Are you using network-mounted or local storage?
What does the .git/config file look like for a user where git status
is excruciatingly slow?
Does copying the config file from a windows user to a linux user make
timings somewhat consistent between various systems?
Do older version of git perform as poorly?
How is the repository laid out (ie, are there any directories with
a ton of files in, or are they spread across multiple directories)?
How many .gitignore files are you using, and what do they look like?
However, after a reboot (of all things), `git status` latency
skyrockets to 14-15s, making the repo unusable.
That's just plain weird, and is almost certainly a system issue.
--
Andreas Ericsson andreas.ericsson@op5.se
OP5 AB www.op5.se
Tel: +46 8-230225 Fax: +46 8-230231
Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.
On Wed, Nov 24, 2010 at 2:08 AM, Dun Peal [off-list ref] wrote:
Hey,
We have a bunch of Windows users, unfortunately, and they're using the
latest msysGit release (Git-1.7.3.1-preview20101002).
An interesting issue we've noticed is that the Time To Complete of
their common operations start deteriorating inexplicably, and
severely, some time after the clone.
For instance, immediately after a clone, `git status` takes about
5-6s. Which is slow compared to Linux (consistent 1-2s), but still
usable (it's a BIG repo).
However, after a reboot (of all things), `git status` latency
skyrockets to 14-15s, making the repo unusable.
Does assume-unchanged bit (see "git update-index") help? I'm not
suggesting to use it but it would help determine if the slowdown is
worktree-related.
--
Duy
----- Original Message -----
From: Tay Ray Chuan
Date: 11/24/2010 7:16 AM
On Wed, Nov 24, 2010 at 3:59 AM, Dun Peal[off-list ref] wrote:
quoted
On Tue, Nov 23, 2010 at 7:12 PM, Wilbert van Dolleweerd
[off-list ref] wrote:
The working copy totals about 4GB. The .git directory, tightly
packed, is 1GB.
What does the structure of your working tree look like? I think the
depth might be affecting performance
Whenever I want to know exactly what is going on with disk access, I
download Process Monitor from http://sysinternals.com/.
In order to just show disk access, I filter entries that begin with TCP,
UDP, and Reg out.
Josh
On Wed, Nov 24, 2010 at 11:34 AM, Andreas Ericsson [off-list ref] wrote:
How many refs (tags and branches) do you have?
We have about 15 branches, but thousands of tags (we tag each release,
and we make those often). However, I'm not sure how that could make
`git status` slow, since it shouldn't care about tags at all?
Are the refs packed or loose?
Packed.
If they are loose, does packing them resolve the issue?
Are you using network-mounted or local storage?
Local storage on all workstations. One machine with those performance
issues actually utilizes an incredibly fast, cutting edge SSD. It's
twice as fast as the other slow machine, but 8s for `git status` is
still slow.
What does the .git/config file look like for a user where git status
is excruciatingly slow?
It's the normal .git/config you get by default on Git 1.7.x when
cloning from a remote. I can paste it if you want, but doubt it will
interest anyone: we didn't modify it.
Does copying the config file from a windows user to a linux user make
timings somewhat consistent between various systems?
No.
Do older version of git perform as poorly?
Yes.
How is the repository laid out (ie, are there any directories with
a ton of files in, or are they spread across multiple directories)?
Generally spread across multiple directories.
How many .gitignore files are you using, and what do they look like?
We're using 25 of those,
quoted
However, after a reboot (of all things), `git status` latency
skyrockets to 14-15s, making the repo unusable.
That's just plain weird, and is almost certainly a system issue.
Yes, it's weird. I'm not sure it's a "system" issue since we're seeing
it across a diverse set of Windows systems: Vista, Windows 7, pretty
sure I saw XP as well, different hardware and software setups
(different programs installed and running).
.D
On Wed, Nov 24, 2010 at 2:16 PM, Tay Ray Chuan [off-list ref] wrote:
What does the structure of your working tree look like? I think the
depth might be affecting performance.
I don't think there's anything special about it; it's a big tree, so
naturally we have some deep (15-level) directory branches. Any
particular reason you suspect the depth?
.D
On Wed, Nov 24, 2010 at 5:16 PM, Joshua Jensen
[off-list ref] wrote:
Whenever I want to know exactly what is going on with disk access, I
download Process Monitor from http://sysinternals.com/.
In order to just show disk access, I filter entries that begin with TCP,
UDP, and Reg out.
Josh
Thanks, we tried that and we don't see a whole lot of disk activity on
the "fast" machines.
One emerging theory is that the "slow" Windows machines differ from
the "fast" ones by how their disk cache works.
So `git status` on a large tree heavily depends on caching. Without
it, it would be slow; with it, it's much faster.
We verified that part since when we reboot a fast Windows machine, the
first run of `git status` is slow (~30s) but the next one is much
faster (~5s).
We see a similar phenomenon on Linux: the first run is always
significantly slower than the others.
On slow Windows machines, this difference is much less pronounced.
On a typical "slow" machine, if you clone the repo, the first run of
`git status` on it would already be fast (5s). But then your reboot,
and the first run is slow, but then it only gets up to 14s. And you
can't get back the 5s latency unless you re-clone the repo and status
the fresh clone.
So my theory is that there's a cache that on the "fast" machines
aggressively caches the entire tree on a regular `git status` run. On
such a machine, it's enough to run `git status` once, and after that
initial cold run, the rest will be warm... until you reboot the
machine, rinse, repeat.
On a slow machine, however, cache isn't so aggressive. It might be
write-oriented. So when you write out a whole new working tree, that
tree gets cached as it is written. And for the remainder of the
lifetime of that cache, you get the fully-cached performance you see
on the "fast" machines. But then you reboot the machine, and lose the
cache. And since the caching process isn't aggressive, any number of
`git status` runs won't get you back to the fully cached state. You
will only get that on a newly written working copy.
What do you think?
.D
From: A Large Angry SCM <hidden> Date: 2016-06-15 22:50:06
On 11/24/2010 04:00 PM, Dun Peal wrote:
On Wed, Nov 24, 2010 at 5:16 PM, Joshua Jensen
[off-list ref] wrote:
quoted
Whenever I want to know exactly what is going on with disk access, I
download Process Monitor from http://sysinternals.com/.
In order to just show disk access, I filter entries that begin with TCP,
UDP, and Reg out.
Josh
Thanks, we tried that and we don't see a whole lot of disk activity on
the "fast" machines.
One emerging theory is that the "slow" Windows machines differ from
the "fast" ones by how their disk cache works.
So `git status` on a large tree heavily depends on caching. Without
it, it would be slow; with it, it's much faster.
We verified that part since when we reboot a fast Windows machine, the
first run of `git status` is slow (~30s) but the next one is much
faster (~5s).
We see a similar phenomenon on Linux: the first run is always
significantly slower than the others.
On slow Windows machines, this difference is much less pronounced.
On a typical "slow" machine, if you clone the repo, the first run of
`git status` on it would already be fast (5s). But then your reboot,
and the first run is slow, but then it only gets up to 14s. And you
can't get back the 5s latency unless you re-clone the repo and status
the fresh clone.
So my theory is that there's a cache that on the "fast" machines
aggressively caches the entire tree on a regular `git status` run. On
such a machine, it's enough to run `git status` once, and after that
initial cold run, the rest will be warm... until you reboot the
machine, rinse, repeat.
On a slow machine, however, cache isn't so aggressive. It might be
write-oriented. So when you write out a whole new working tree, that
tree gets cached as it is written. And for the remainder of the
lifetime of that cache, you get the fully-cached performance you see
on the "fast" machines. But then you reboot the machine, and lose the
cache. And since the caching process isn't aggressive, any number of
`git status` runs won't get you back to the fully cached state. You
will only get that on a newly written working copy.
What do you think?
How much memory do the fast and slow machines have? How much memory will
windows use for disk caching? Is it possible that your normal work flow
between status' are forcing the caches to pruned due to memory pressure?
From: Johannes Sixt <hidden> Date: 2016-06-15 22:50:06
On Mittwoch, 24. November 2010, Dun Peal wrote:
So my theory is that there's a cache that on the "fast" machines
aggressively caches the entire tree on a regular `git status` run. On
such a machine, it's enough to run `git status` once, and after that
initial cold run, the rest will be warm... until you reboot the
machine, rinse, repeat.
On a slow machine, however, cache isn't so aggressive. It might be
write-oriented. So when you write out a whole new working tree, that
tree gets cached as it is written. And for the remainder of the
lifetime of that cache, you get the fully-cached performance you see
on the "fast" machines. But then you reboot the machine, and lose the
cache. And since the caching process isn't aggressive, any number of
`git status` runs won't get you back to the fully cached state. You
will only get that on a newly written working copy.
You can test this theory on a slow machine: You have cloned a repository,
rebooted, and you are now at 14s per 'git status'. Now:
$ git rm -r .
$ git reset --hard
erases the worktree and writes it out again (do this only on a clean
checkout!). Are you now at 5s per 'git status'?
-- Hannes
From: Robin Rosenberg <hidden> Date: 2016-06-15 22:50:07
23 nov 2010 kl. 20:08 skrev Dun Peal:
Hey,
We have a bunch of Windows users, unfortunately, and they're using the
latest msysGit release (Git-1.7.3.1-preview20101002).
An interesting issue we've noticed is that the Time To Complete of
their common operations start deteriorating inexplicably, and
severely, some time after the clone.
For instance, immediately after a clone, `git status` takes about
5-6s. Which is slow compared to Linux (consistent 1-2s), but still
usable (it's a BIG repo).
However, after a reboot (of all things), `git status` latency
skyrockets to 14-15s, making the repo unusable.
Any idea what's going on? We just recently switched from SVN, and
those users are getting really frustrated. BTW, the only real
alternative I'm aware of, Cygwin's git, is even slower.
Is the file system badly fragmented? The worst kind is when the MFT is fragmented,
which may give you very bad performance and the defrag that comes with Windows does
not fix MFT fragmentation.
Is NTFS compression enabled? I doubt its helpfulness with Git.
-- robin