Re: pack operation is thrashing my server

Nicolas Pitre [off-list ref] wrote:

On Wed, 13 Aug 2008, Shawn O. Pearce wrote:

quoted

Doing the object
enumeration is pointless as a security measure.

It is good for network bandwidth efficiency as I mentioned.

The network bandwidth efficiency is the most valid argument for
the enumeration.

quoted

I'm too busy to write a pack concat implementation proposal

A much better solution would consist of finding just _why_ object 
enumeration is so slow.  This is indeed my biggest grip with git 
performance at the moment.

...

|nico@xanadu:gcc> time git rev-list --objects --all > /dev/null
|
|real    1m51.591s
|user    1m50.757s
|sys     0m0.810s

That's for 1267993 objects, or about 11400 objects/sec.

Clearly something is not scaling here.

Yikes.  Last time I was looking at this sort of thing I think we
spent around 60% of our time dealing with inflating, patching and
parsing commit and tree objects.  pack v4's formatting spawned
out of that particular point, but we never really finished that.
Its been years so I can't trust my memory enough to say pack v4 is
the solution to this, without redoing the profiling.  But I think
that is what one would find.

Though the decreasing objects/sec rate with increased total number
of objects suggets the object hash isn't scaling.

-- 
Shawn.