Re: pack operation is thrashing my server

On Wed, 13 Aug 2008, Shawn O. Pearce wrote:

Nicolas Pitre [off-list ref] wrote:

quoted

Well, we are talking about 50MB which is not that bad.

I think we're closer to 100MB here due to the extra overheads
I just alluded to above, and which weren't in your 104 byte
per object figure.

Sure.  That should still be workable on a machine with 256MB of RAM.

quoted

However there is a point where we should be realistic and just admit 
that you need a sufficiently big machine if you have huge repositories 
to deal with.  Git should be fine serving pull requests with relatively 
little memory usage, but anything else such as the initial repack simply 
require enough RAM to be effective.

Yea.  But it would also be nice to be able to just concat packs
together.  Especially if the repository in question is an open source
one and everything published is already known to be in the wild,
as say it is also available over dumb HTTP.  Yea, I know people
like the 'security feature' of the packer not including objects
which aren't reachable.

It is not only that, even if it is a point I consider important.  If you 
end up with 10 packs, it is likely that a base object in each of those 
packs could simply be a delta against a single common base object, and 
therefore the amount of data to transfer might be up to 10 times higher 
than necessary.

But how many times has Linus published something to his linux-2.6
tree that he didn't mean to publish and had to rewind?  I think
that may be "never".  Yet how many times per day does his tree get
cloned from scratch?

That's not a good argument.  Linus is a very disciplined git users, 
probably more than average.  We should not use that example to paper 
over technical issues.

This is also true for many internal corporate repositories.
Users probably have full read access to the object database anyway,
and maybe even have direct write access to it.  Doing the object
enumeration there is pointless as a security measure.

It is good for network bandwidth efficiency as I mentioned.

I'm too busy to write a pack concat implementation proposal, so
I'll just shutup now.  But it wouldn't be hard if someone wanted
to improve at least the initial clone serving case.

A much better solution would consist of finding just _why_ object 
enumeration is so slow.  This is indeed my biggest grip with git 
performance at the moment.

|nico@xanadu:linux-2.6> time git rev-list --objects --all > /dev/null
|
|real    0m21.742s
|user    0m21.379s
|sys     0m0.360s

That's way too long for 1030198 objects (roughly 48k objects/sec).  And 
it gets even worse with the gcc repository:

|nico@xanadu:gcc> time git rev-list --objects --all > /dev/null
|
|real    1m51.591s
|user    1m50.757s
|sys     0m0.810s

That's for 1267993 objects, or about 11400 objects/sec.

Clearly something is not scaling here.


Nicolas