Re: pack operation is thrashing my server

On Thu, 14 Aug 2008, Linus Torvalds wrote:

On Wed, 13 Aug 2008, Nicolas Pitre wrote:

quoted

A much better solution would consist of finding just _why_ object 
enumeration is so slow.  This is indeed my biggest grip with git 
performance at the moment.

|nico@xanadu:linux-2.6> time git rev-list --objects --all > /dev/null
|
|real    0m21.742s
|user    0m21.379s
|sys     0m0.360s

That's way too long for 1030198 objects (roughly 48k objects/sec).

Why do you think that's horribly slow?

Call it gut feeling.  Or 60% CPU wasted in zlib.

Doing a rev-list of all objects is a fairly rare operation, but even if 
you want to clone/repack all of your archives the whole time, please 
realize that listing objects is _not_ a simple operation. It opens up and 
parses every single tree in the whole history. That's a _lot_ of data to 
unpack.

I disagree.  Well, right _now_ it is not a simple operation.  But if you 
remember, I'm one of the co-investigator of the pack v4 format which 
goal is to make history and tree walking much much cheaper, while making 
their packed representation denser too.  Even with early prototypes of 
the format with the overhead of converting objects back into the current 
format on the fly in unpack_entry() the object enumeration was _faster_ 
than current git.

So this might just be what was needed to bring back some incentive 
behind the pack v4 effort.


Nicolas