Hi *,
In cold cache "git rev-list origin/master --not --all" is slow
reading many files:
cold cache:
$ /usr/bin/time git rev-list origin/master --not --all
0.03user 0.02system 0:04.57elapsed 1%CPU (0avgtext+0avgdata 0maxresident)k
77848inputs+0outputs (410major+1798minor)pagefaults 0swaps
hot cache:
$ /usr/bin/time git rev-list origin/master --not --all
0.01user 0.00system 0:00.06elapsed 31%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+2207minor)pagefaults 0swaps
I think that, in this particular case (when the arguments are the tips
of some of the branches), this should not read that many files.
Moreover, this is used in "git fetch" (git rev-list --quiet --objects
<list_of_remote_sha1> --not --all) to detect if all the objects are
reachable from the local repository. When nothing has changed in the
remote repository (so refs/<remote>/* has all the remote refs) the
"git fetch" could be almost instantaneous (even in coldcache), but
currently it is not because of the above.
Thanks,
Santi
On Wed, 5 Nov 2008, Santi Béjar wrote:
In cold cache "git rev-list origin/master --not --all" is slow
reading many files:
Hmm. It sounds like you possibly don't have packed refs.
Have you done "git gc" on that thing lately? What does "strace" say?
Linus
On Wed, Nov 5, 2008 at 6:37 PM, Linus Torvalds
[off-list ref] wrote:
On Wed, 5 Nov 2008, Santi Béjar wrote:
quoted
In cold cache "git rev-list origin/master --not --all" is slow
reading many files:
Sorry, s/files/data/
Hmm. It sounds like you possibly don't have packed refs.
They are packed up to v2.6.27.
Have you done "git gc" on that thing lately? What does "strace" say?
Linus
It is a recently cloned linux-2.6 repo, with 10 or so packs and 70
loose objects. It spends a good fraction in:
brk(0x8318000) = 0x8318000
and
mmap2(NULL, 33554432, PROT_READ, MAP_PRIVATE, 4, 0x2000) = 0xb2227000
(strace attatched).
I should have given more info:
It is an old computer (Pentium 4 2.5 GHz)
and the repo is on an external USB drive.
On Wed, Nov 5, 2008 at 6:32 PM, Junio C Hamano [off-list ref] wrote:
"Santi Béjar" [off-list ref] writes:
quoted
In cold cache "git rev-list origin/master --not --all" is slow
reading many files:
cold cache:
$ /usr/bin/time git rev-list origin/master --not --all
0.03user 0.02system 0:04.57elapsed 1%CPU (0avgtext+0avgdata 0maxresident)k
77848inputs+0outputs (410major+1798minor)pagefaults 0swaps
hot cache:
$ /usr/bin/time git rev-list origin/master --not --all
0.01user 0.00system 0:00.06elapsed 31%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+2207minor)pagefaults 0swaps
I think that, in this particular case (when the arguments are the tips
of some of the branches), this should not read that many files.
What kind of "many files" are you making git read? Do you have too many
unpacked refs? Too many loose objects?
See above.
quoted
... When nothing has changed in the remote repository (so
refs/<remote>/* has all the remote refs) the "git fetch" could be almost
instantaneous (even in coldcache),...
You at least need to read:
- what "--all" refs point at; to find this out, you need to read all
unpacked refs files, and one packed-refs file;
- commit objects that these refs point at; to cull refs that do not point
at committish and dereference tag objects that point at commit, you
need to read these objects (either loose objects or in packs);
- commit objects on the ancestry graph starting from the commit pointed
at by origin/master and the commits from "--all" refs, until your
traversal from origin/master hit one of the ancestors of "--all" refs.
Yes, in the general case it is, but in this case we can bypass the
checking of the --all refs after checking if all the given refs are
equal to some of the --all refs.
For me the main thing is the slow "git fetch" when downloading
nothing. I think it was/is faster without the quickfetch path (at
least in the coldcache case).
Maybe we can just check if the new fetched refs are already the tips
of the corresponding remote branch.
Santi
On Wed, 5 Nov 2008, Santi Béjar wrote:
quoted
Hmm. It sounds like you possibly don't have packed refs.
They are packed up to v2.6.27.
Yeah, your strace isn't at all horrible. You don't open very many files at
all, and you don't have any big directories.
The biggest cost when things are cold-cache is probably the seeking from
just opening all those index files.
It is an old computer (Pentium 4 2.5 GHz) and the repo is on an external
USB drive.
There should be basically no CPU spent on that load, so your computer is
fine. But I think the issue is the dog-slow IO on the USB drive,
especially since there are multiple pack-files and thus index files.
Your strace would be more interesting with "-Ttt", but much of the cost is
likely in the page faulting of the mmap'ed data, and none of that would
show up in the trace, except indirectly (ie just looking at the times
between the system calls).
Yes, in the general case it is, but in this case we can bypass the
checking of the --all refs after checking if all the given refs are
equal to some of the --all refs.
I don't think we'll actually walk anything, because all commits will end
up being negative.
But we'll look up the objects for even the negative commits, yes. So we're
doing several "unnecessary" object lookups, and in that sense we could
make this much faster by not even bothering to look them up.
But we do that to validate that the refs are _valid_, so in that sense the
object lookup is not "unnecessary" at all. Oh, and we need to peel them to
see if they are tag objects, in order to mark the _commit_ uninteresting
if the object itself was uninteresting.
So in practice we do end up having to pretty much parse them all.
We could do some crazy special case for the empty set, but it would be
better to see if you can improve performance with a slow disk some other,
less hacky, way. If you use a USB stick to move between machines, maybe
you can make sure that it's fully packed (ie a single index file) before
moving it to the USB stick? That would likely help quite a bit.
Linus