Linus Torvalds [off-list ref] writes:
Umm. See my earlier numbers. For "git checkout" with cold cache, the
*bulk* of the time is actually the ".gitignore" file lookups, so if you
see a three-second improvement out of 17s, it may not look spectacular,
but considering that probably 10s of those 17s were something *else* going
on, I suspect that if you really did just a plain "git checkout", you
actually *do* have a spectacular improvement of roughly 7s -> 4s!
I am hoping that "probably 10s of those 17s" can actually be measured
with the patch I sent out last night. Has anybody took a look at it?
Partitioning the pack data by object type shifts the tradeoffs from the
current "the data in the same tree are mostly together, except commits
are treated differently because rev walk is done quite often" layout.
Because we do not ever look at blob objects while pruning the history
(unless the -Spickaxe option is used, I think), partitioned layout would
optimize ancestry walking even more than the current packfile layout.
On the other hand, any operation that wants to look at the contents are
penalized. A two-tree diff that inspects the contents (e.g. fuzzy
renames and pickaxe) needs to read from the tree section to find which
blob to compare with which other blob, and and then needs to seek to the
blob section to actually read the contents, while the current layout
tends to group both trees and blobs that belong to the same tree
together. It is natural that blame is penalized by the new layout,
mostly because it needs to grab two blobs to compare from parent-child
pair, but also because it needs to find two-tree diffs for parent-child
pair it traverses whenever it needs to follow across renames (that is,
when it sees there is no corresponding path in the parent). I would
expect to see similar slowdown from grep which wants to inspect blobs
that are in the same tree.
When I do archaeology, I think I often run blame first to see which
change made the block of text into the current shape first, and then run
a path limited "git log -p" either starting or ending at that revision.
In that workflow, the initial blame may get slower with the new layout,
but I suspect it would help by speeding up the latter "git log -p" step.
On Thu, 29 Nov 2007, Junio C Hamano wrote:
I am hoping that "probably 10s of those 17s" can actually be measured
with the patch I sent out last night. Has anybody took a look at it?
Sorry, I missed it. But I just did timings.
Your patch helps
git read-tree -m -u --exclude-per-directory=.gitignore HEAD HEAD
timings enormously, and it's now down to 3s for me (which is the same
speed as it is without any per-directory-excludes). That's a big
improvement from the ~10s I see without your patch (I've repacked my
tree, I have to admit that I don't even know if it's the new or the old
older, but I can state that 7s for me was just those .gitignore files).
Sadly, the full "git checkout" itself is not actually improved, due to the
git update-index --refresh
there, which will end up populating the whole directory cache anyway.
I wonder why I didn't see that as the expensive operation when I timed
"git checkout". Probably because I narrowed down on the "git read-tree" as
the operation that actually accesses the pack-file and the object
directory, while the "git update-index" never touches the actual objects.
Anyway, I think your patch is great. It just doesn't help the full case of
a "git checkout", only the read-tree portion of it ;(
As to partitioning the data according to types:
When I do archaeology, I think I often run blame first to see which
change made the block of text into the current shape first, and then run
a path limited "git log -p" either starting or ending at that revision.
In that workflow, the initial blame may get slower with the new layout,
but I suspect it would help by speeding up the latter "git log -p" step.
I really cannot convince myself one way or the other. I have a suspicion
that sometimes it helps to have objects (regardless of type) close to each
other, and sometimes it helps to have the trees packed densely. A lot of
operations *do* work on both blobs and trees (a *raw* diff doesn't, but
they are fairly rare), so this is not at all clear-cut like the commit
case.
So sorting the commits together is a no-brainer, since a lot of really
important ops only look at them. But blobs and trees? The numbers
certainly go both ways, and I suspect we are probably better off not
messing with the sort order unless we have some unambiguous real results.
Oh, well. I was hoping that I'd have a number of cases that showed good
improvements, with perhaps the bulk of it not showing much difference at
all. But while I saw the good improvements, the very first try at "git
blame" also showed quite worse numbers, so I think we should consider it
an interesting idea, but probably shelve it.
Linus