From: Thomas Rast <hidden> Date: 2016-06-15 22:53:04
Piotr Krukowiecki [off-list ref] writes:
On Fri, Feb 10, 2012 at 10:42 AM, Piotr Krukowiecki
[off-list ref] wrote:
quoted
I compared stating whole tree vs one small subdirectory, and I
expected that for the subdirectory status will be very very fast.
After all, it has only few files to stat. But it's not fast. Why?
With cold cache (echo 3 | sudo tee /proc/sys/vm/drop_caches):
$ time git status > /dev/null
real 0m41.670s
user 0m0.980s
sys 0m2.908s
$ time git status -- src/.../somedir > /dev/null
real 0m17.380s
user 0m0.748s
sys 0m0.328s
[...]
I can't reproduce this behavior at the moment. 'status' on the
directory takes about 1.5s instead of 17s. status on whole repository
takes 27s.
This is my work repository, so it was changed today.
To me these timings smell like a combination of either a network
filesystem or a slow/busy disk, and non-packed repositories. Next time
this happens look at 'git count-objects', run 'git gc' and redo the
timings.
If you are indeed on a network filesystem, also look at the
core.preloadIndex setting.
--
Thomas Rast
trast@{inf,student}.ethz.ch
From: Piotr Krukowiecki <hidden> Date: 2016-06-15 22:53:05
On Tue, Feb 14, 2012 at 12:34 PM, Thomas Rast [off-list ref] wrote:
Piotr Krukowiecki [off-list ref] writes:
quoted
On Fri, Feb 10, 2012 at 10:42 AM, Piotr Krukowiecki
[off-list ref] wrote:
quoted
I compared stating whole tree vs one small subdirectory, and I
expected that for the subdirectory status will be very very fast.
After all, it has only few files to stat. But it's not fast. Why?
With cold cache (echo 3 | sudo tee /proc/sys/vm/drop_caches):
$ time git status > /dev/null
real 0m41.670s
user 0m0.980s
sys 0m2.908s
$ time git status -- src/.../somedir > /dev/null
real 0m17.380s
user 0m0.748s
sys 0m0.328s
[...]
quoted
I can't reproduce this behavior at the moment. 'status' on the
directory takes about 1.5s instead of 17s. status on whole repository
takes 27s.
This is my work repository, so it was changed today.
To me these timings smell like a combination of either a network
filesystem or a slow/busy disk, and non-packed repositories. Next time
this happens look at 'git count-objects', run 'git gc' and redo the
timings.
If you are indeed on a network filesystem, also look at the
core.preloadIndex setting.
All is on local disk and system is idle.
Indeed, after gc the times went down:
10s -> 2.3s (subdirectory)
17s -> 9.5s (whole repo)
2 seconds is much better and I'd say acceptable for me. But my questions are:
- why is it so slow with not packed repo?
- can it be faster without repacking?
- even with packed repo, the time on small subdirectory is much higher
than I'd expect given time on whole repo and subdirectory size - why?
$ git count-objects -v
count: 5095
size: 37084
in-pack: 755364
packs: 21
size-pack: 2398468
prune-packable: 0
garbage: 0
$ git gc
Counting objects: 760212, done.
Compressing objects: 100% (158651/158651), done.
Writing objects: 100% (760212/760212), done.
Total 760212 (delta 535848), reused 736108 (delta 513257)
Checking connectivity: 760212, done.
$ time git status -- .
real 0m2.503s
user 0m0.160s
sys 0m0.096s
$ time git status
real 0m9.663s
user 0m0.232s
sys 0m0.556s
--
Piotr Krukowiecki
On Wed, Feb 15, 2012 at 3:57 PM, Piotr Krukowiecki
[off-list ref] wrote:
Indeed, after gc the times went down:
10s -> 2.3s (subdirectory)
17s -> 9.5s (whole repo)
2 seconds is much better and I'd say acceptable for me. But my questions are:
- why is it so slow with not packed repo?
- can it be faster without repacking?
gc does more than just repacking. If you still have the un-gc'd repo,
Try these commands one by one, and time "git status" after each:
- git pack-refs --all --prune
- git reflog expire --all
- git repack -d -l
- git prune --expire
- git rerere gc
I'd be more interested in why auto-gc does not kick in (or whther it should).
- even with packed repo, the time on small subdirectory is much higher
than I'd expect given time on whole repo and subdirectory size - why?
Hard to say without measuring. I just notice that I missed your mail
with profiling results. I will have a look, but just in case, is the
repository publicly available?
--
Duy
From: Piotr Krukowiecki <hidden> Date: 2016-06-15 22:53:05
On Wed, Feb 15, 2012 at 12:01 PM, Nguyen Thai Ngoc Duy
[off-list ref] wrote:
On Wed, Feb 15, 2012 at 3:57 PM, Piotr Krukowiecki
[off-list ref] wrote:
quoted
Indeed, after gc the times went down:
10s -> 2.3s (subdirectory)
17s -> 9.5s (whole repo)
2 seconds is much better and I'd say acceptable for me. But my questions are:
- why is it so slow with not packed repo?
- can it be faster without repacking?
gc does more than just repacking. If you still have the un-gc'd repo,
Try these commands one by one, and time "git status" after each:
- git pack-refs --all --prune
- git reflog expire --all
- git repack -d -l
- git prune --expire
- git rerere gc
It will take some time but hopefully I'll have the stats for tomorrow.
I'd be more interested in why auto-gc does not kick in (or whther it should).
I don't have any specific options set, so default values should be used.
I'm using git-svn though, so my workflow looks like this:
git svn fetch + git svn rebase
... git operations like commit, cherry-pick, rebase ...
git svn dcommit
Not sure if that matters. I remember that I've seen auto-gc being run
several times in the past - I think after svn fetch/rebase.
I'm also using git-new-workdir and have 2 extra workdirs.
quoted
- even with packed repo, the time on small subdirectory is much higher
than I'd expect given time on whole repo and subdirectory size - why?
Hard to say without measuring. I just notice that I missed your mail
with profiling results. I will have a look, but just in case, is the
repository publicly available?
Unfortunately it's not public. I can do some measuring if someone
tells me what to do.
--
Piotr Krukowiecki
From: Jeff King <hidden> Date: 2016-06-15 22:53:05
On Wed, Feb 15, 2012 at 09:57:29AM +0100, Piotr Krukowiecki wrote:
All is on local disk and system is idle.
Indeed, after gc the times went down:
10s -> 2.3s (subdirectory)
17s -> 9.5s (whole repo)
2 seconds is much better and I'd say acceptable for me. But my questions are:
Obviously these answers didn't come from any deep analysis, but are
educated guesses from me based on previous performance patterns we've
seen on the list:
- why is it so slow with not packed repo?
Your numbers show that you're I/O-bound:
quoted
quoted
$ time git status > /dev/null
real 0m41.670s
user 0m0.980s
sys 0m2.908s
$ time git status -- src/.../somedir > /dev/null
real 0m17.380s
user 0m0.748s
sys 0m0.328s
which is not surprising, since you said you dropped caches before-hand.
Repacking probably reduced your disk footprint by a lot, which meant
less I/O.
I notice that you're still I/O bound even after the repack:
$ time git status -- .
real 0m2.503s
user 0m0.160s
sys 0m0.096s
$ time git status
real 0m9.663s
user 0m0.232s
sys 0m0.556s
Did you drop caches here, too? Usually that would not be the case on a
warm cache. If it is, then it sounds like you are short on memory to
actually hold the directory tree and object db in cache. If not, what do
the warm cache numbers look like?
- can it be faster without repacking?
Not really. You're showing an I/O problem, and repacking is git's way of
reducing I/O.
- even with packed repo, the time on small subdirectory is much higher
than I'd expect given time on whole repo and subdirectory size - why?
Hard to say without profiling. It may be that we reduced the object db
lookups, saving some time, but still end up stat()ing the whole tree.
The optimization to stat only the directories of interest was in 688cd6d
(status: only touch path we may need to check, 2010-01-14), which went
into v1.7.0. What version of git are you using?
-Peff
From: Piotr Krukowiecki <hidden> Date: 2016-06-15 22:53:05
On Wed, Feb 15, 2012 at 4:14 PM, Piotr Krukowiecki
[off-list ref] wrote:
On Wed, Feb 15, 2012 at 12:01 PM, Nguyen Thai Ngoc Duy
[off-list ref] wrote:
quoted
On Wed, Feb 15, 2012 at 3:57 PM, Piotr Krukowiecki
[off-list ref] wrote:
quoted
Indeed, after gc the times went down:
10s -> 2.3s (subdirectory)
17s -> 9.5s (whole repo)
2 seconds is much better and I'd say acceptable for me. But my questions are:
- why is it so slow with not packed repo?
- can it be faster without repacking?
gc does more than just repacking. If you still have the un-gc'd repo,
Try these commands one by one, and time "git status" after each:
- git pack-refs --all --prune
- git reflog expire --all
- git repack -d -l
- git prune --expire
- git rerere gc
It will take some time but hopefully I'll have the stats for tomorrow.
Here they are. I did 'status' three times to get reliable results and
before each run have dropped caches. Backed up repository was copied
before each 'status'. Full log is at http://pastebin.com/VmB7J9CJ
git version 1.7.9.rc0.10.gbeecc
Results after each command:
status on whole repo:
18.5s - after git count-objects -v
16.0s - after git pack-refs --all --prune
20.2s - after git reflog expire --all
13.0s - after git repack -d -l
16.8s - after git prune --expire now
19.7s - after git rerere gc
status on subdir:
9.7s - after git count-objects -v
9.2s - after git pack-refs --all --prune
9.3s - after git reflog expire --all
4.4s - after git repack -d -l
9.2s - after git prune --expire now
9.0s - after git rerere gc
--
Piotr Krukowiecki
From: Piotr Krukowiecki <hidden> Date: 2016-06-15 22:53:05
On Wed, Feb 15, 2012 at 8:03 PM, Jeff King [off-list ref] wrote:
On Wed, Feb 15, 2012 at 09:57:29AM +0100, Piotr Krukowiecki wrote:
quoted
I notice that you're still I/O bound even after the repack:
quoted
$ time git status -- .
real 0m2.503s
user 0m0.160s
sys 0m0.096s
$ time git status
real 0m9.663s
user 0m0.232s
sys 0m0.556s
Did you drop caches here, too?
Yes I did - with cache the status takes something like 0.1-0.3s on whole repo.
Usually that would not be the case on a
warm cache. If it is, then it sounds like you are short on memory to
actually hold the directory tree and object db in cache. If not, what do
the warm cache numbers look like?
Not really. You're showing an I/O problem, and repacking is git's way of
reducing I/O.
So if I understand correctly, the reason is because git must compare
workspace files with packed objects - and the problem is
reading/seeking/searching in the packs?
Is there a way to make packs better? I think most operations are on
workdir files - so maybe it'd be possible to tell gc/repack/whatever
to optimize access to files which I currently have in workdir?
quoted
- even with packed repo, the time on small subdirectory is much higher
than I'd expect given time on whole repo and subdirectory size - why?
Hard to say without profiling. It may be that we reduced the object db
lookups, saving some time, but still end up stat()ing the whole tree.
The optimization to stat only the directories of interest was in 688cd6d
(status: only touch path we may need to check, 2010-01-14), which went
into v1.7.0. What version of git are you using?
For latest tests I've used 1.7.9.rc0.10.gbeecc, for profiling - 1.7.9.188.g12766
Is there anything else I could do?
--
Piotr Krukowiecki
From: Jeff King <hidden> Date: 2016-06-15 22:53:05
On Thu, Feb 16, 2012 at 02:37:47PM +0100, Piotr Krukowiecki wrote:
quoted
quoted
$ time git status -- .
real 0m2.503s
user 0m0.160s
sys 0m0.096s
$ time git status
real 0m9.663s
user 0m0.232s
sys 0m0.556s
Did you drop caches here, too?
Yes I did - with cache the status takes something like 0.1-0.3s on whole repo.
OK, then that makes sense. It's pretty much just I/O on the filesystem
and on the object db.
You can break status down a little more to see which is which. Try "git
update-index --refresh" to see just how expensive the lstat and index
handling is.
And then try "git diff-index HEAD" for an idea of how expensive it is to
just read the objects and compare to the index.
quoted
Not really. You're showing an I/O problem, and repacking is git's way of
reducing I/O.
So if I understand correctly, the reason is because git must compare
workspace files with packed objects - and the problem is
reading/seeking/searching in the packs?
Mostly reading (we keep a sorted index and access the packfiles via
mmap, so we only touch the pages we need). But you're also paying to
lstat() the directory tree, too. And you're paying to load (probably)
the whole index into memory, although it's relatively compact compared
to the actual file data.
Is there a way to make packs better? I think most operations are on
workdir files - so maybe it'd be possible to tell gc/repack/whatever
to optimize access to files which I currently have in workdir?
It already does optimize for that case. If you can make it even better,
I'm sure people would be happy to see the numbers.
Mostly I think it is just the case that disk I/O is slow, and the
operation you're asking for has to do a certain amount of it. What kind
of disk/filesystem are you pulling off of?
It's not a fuse filesystem by any chance, is it? I have a repo on an
encfs-mounted filesystem, and the lstat times are absolutely horrific.
-Peff
From: Piotr Krukowiecki <hidden> Date: 2016-06-15 22:53:05
On Thu, Feb 16, 2012 at 8:20 PM, Jeff King [off-list ref] wrote:
On Thu, Feb 16, 2012 at 02:37:47PM +0100, Piotr Krukowiecki wrote:
quoted
quoted
quoted
$ time git status -- .
real 0m2.503s
user 0m0.160s
sys 0m0.096s
$ time git status
real 0m9.663s
user 0m0.232s
sys 0m0.556s
Did you drop caches here, too?
Yes I did - with cache the status takes something like 0.1-0.3s on whole repo.
OK, then that makes sense. It's pretty much just I/O on the filesystem
and on the object db.
You can break status down a little more to see which is which. Try "git
update-index --refresh" to see just how expensive the lstat and index
handling is.
"git update-index --refresh" with dropped cache took
real 0m3.726s
user 0m0.024s
sys 0m0.404s
while "git status" with dropped cache takes
real 0m13.578s
user 0m0.240s
sys 0m0.600s
I'm not sure why it takes more than the 9s reported before - IIRC I
did the previous test in single mode under bare shell and this time
I'm testing under gnome. This or it's the effect of running
update-index :/
Now status on subdir takes 9.5s. But still the
not-much-faster-status-on-subdir rule is true.
And then try "git diff-index HEAD" for an idea of how expensive it is to
just read the objects and compare to the index.
The diff-index after dropping cache takes
real 0m14.095s
user 0m0.268s
sys 0m0.564s
quoted
quoted
Not really. You're showing an I/O problem, and repacking is git's way of
reducing I/O.
So if I understand correctly, the reason is because git must compare
workspace files with packed objects - and the problem is
reading/seeking/searching in the packs?
Mostly reading (we keep a sorted index and access the packfiles via
mmap, so we only touch the pages we need). But you're also paying to
lstat() the directory tree, too. And you're paying to load (probably)
the whole index into memory, although it's relatively compact compared
to the actual file data.
If the index is the objects/pack/*.idx files than it's 21MB
quoted
Is there a way to make packs better? I think most operations are on
workdir files - so maybe it'd be possible to tell gc/repack/whatever
to optimize access to files which I currently have in workdir?
It already does optimize for that case. If you can make it even better,
I'm sure people would be happy to see the numbers.
If I understand correctly, you only need to compute sha1 on the
workdir files and compare it with sha1 files recorded in index/gitdir.
It seems that to get the sha1 from index/gitdir I need to read the
packfiles? Maybe it'd be possible to cache/index it somehow, for
example in separate and smaller file?
Mostly I think it is just the case that disk I/O is slow, and the
operation you're asking for has to do a certain amount of it. What kind
of disk/filesystem are you pulling off of?
It's not a fuse filesystem by any chance, is it? I have a repo on an
encfs-mounted filesystem, and the lstat times are absolutely horrific.
No, it's ext4 and the disk Seagate Barracuda 7200.12 500GB, as it
reads on the cover :)
But IMO faster disk won't help with this - times will be smaller, but
you'll still have to read the same data, so the subdir times will be
just 2x faster than whole repo, won't it? So maybe in my case it will
go down to e.g. 2s on subdir, but for someone with larger repository
it will still be 10s...
--
Piotr Krukowiecki
From: Jeff King <hidden> Date: 2016-06-15 22:53:06
On Fri, Feb 17, 2012 at 06:19:06PM +0100, Piotr Krukowiecki wrote:
"git update-index --refresh" with dropped cache took
real 0m3.726s
user 0m0.024s
sys 0m0.404s
[...]
The diff-index after dropping cache takes
real 0m14.095s
user 0m0.268s
sys 0m0.564s
OK, that suggests to me that the real culprit is the I/O we spend in
accessing the object db, since that is the main I/O that happens in the
second command but not the first.
quoted
Mostly reading (we keep a sorted index and access the packfiles via
mmap, so we only touch the pages we need). But you're also paying to
lstat() the directory tree, too. And you're paying to load (probably)
the whole index into memory, although it's relatively compact compared
to the actual file data.
If the index is the objects/pack/*.idx files than it's 21MB
Yes, that's it. Though we don't necessarily read the whole thing. The
sorted list of sha1s is only a part of that. And we mmap and
binary-search that, so we only have to fault in pages that are actually
used in our binary search.
However, we're faulting in random pages of the index in series, so it
may actually have a lot of latency. You can see how expensive the I/O on
the index is with something like this:
[whole operation, for reference]
$ sudo sh -c 'echo 3 >/proc/sys/vm/drop_caches'
$ time git diff-index HEAD
real 0m2.636s
user 0m0.248s
sys 0m0.392s
[prime the cache with just the index]
$ sudo sh -c 'echo 3 >/proc/sys/vm/drop_caches'
$ time cat .git/objects/pack/*.idx >/dev/null
real 0m0.288s
user 0m0.000s
sys 0m0.028s
$ time git diff-index HEAD
real 0m2.175s
user 0m0.272s
sys 0m0.320s
So roughly 20% of the I/O time in my case went to faulting in the index.
You could pre-fault in the index, which would give the OS a chance to do
read-ahead caching. You can see that the combined cat and diff-index
times are still lower than the raw diff-index time. You could also do
them in parallel, but that will create some additional seeks as the
threads fight for the disk, but may be a win in the long run because we
can read bigger chunks. You can roughly simulate it by running the "cat"
and the "diff-index" above in parallel. I get:
real 0m2.464s
user 0m0.284s
sys 0m0.372s
which is almost exactly the same as doing them separately (though note
that this is on an SSD, so seeking is very cheap).
But the bulk of the time still goes to actually retrieving the object
data, so that's probably a more interesting area to focus, anyway (and
if we can reduce object accesses, we reduce their lookup, too :) ).
If I understand correctly, you only need to compute sha1 on the
workdir files and compare it with sha1 files recorded in index/gitdir.
It seems that to get the sha1 from index/gitdir I need to read the
packfiles? Maybe it'd be possible to cache/index it somehow, for
example in separate and smaller file?
There are two diffs going on in "git status". One is a comparison
between index and worktree. In that one, you need to lstat each file to
make sure the cached sha1 we have in the index is up to date. Assuming
it is, you don't need to touch the file data at all. Then you compare
that sha1 to the stage 0 sha1 (i.e., what we typically think of as
"staged for commit"). If they match, you don't need to do more work.
But the expensive diff-index we've been doing above is comparing the
index to the HEAD tree. And doing that is a little trickier. The index
is a flat list of files with their sha1s. But the HEAD tree is stored
hierarchically. So to get the sha1 of foo/bar/baz, we have to access the
root tree object, find the "foo" entry, access its tree object, find the
"bar" entry, access its tree object, and then find the "baz" entry. Then
we compare the sha1 of the "baz" entry to what's in the index.
So what's where your I/O comes from: accessing each of the tree objects.
And that fact that it isn't just "compare the HEAD and index sha1s" is
that the index is stored as a list of flat files.
That being said, we do have an index extension to store the tree sha1 of
whole directories (i.e., we populate it when we write a whole tree or
subtree into the index from the object db, and it becomes invalidated
when a file becomes modified). This optimization is used by things like
"git commit" to avoid having to recreate the same sub-trees over and
over when creating tree objects from the index. But we could also use it
here to avoid having to even read the sub-tree objects from the object
db.
No, it's ext4 and the disk Seagate Barracuda 7200.12 500GB, as it
reads on the cover :)
But IMO faster disk won't help with this - times will be smaller, but
you'll still have to read the same data, so the subdir times will be
just 2x faster than whole repo, won't it? So maybe in my case it will
go down to e.g. 2s on subdir, but for someone with larger repository
it will still be 10s...
Sure. But a certain amount of I/O is going to be unavoidable to get the
answer to your question. So you will never be able to achieve the
warm-cache case. I'm not saying we can't improve (e.g., I think the
index extension thing I mentioned above is a promising approach). But we
have to be realistic about what will make things faster; if I/O is your
problem, faster disk is one possible solution (especially because some
of this is related to seeking and latency, an SSD is a nice improvement
for cold-cache times).
-Peff