Re: Repacking many disconnected blobs

On Wed, 14 Jun 2006, Keith Packard wrote:

Ok, sounds like shuffling isn't necessary; the only benefit packing
gains me is to reduce the size of each directory in the object store;

There's actually a secondary benefit to packing that turned out to be much 
bigger from a performance standpoint: the size benefit coupled with the 
fact that it's all in one file ends up meaning that accessing packed 
objects is _much_ faster than accessing individual files.

The Linux system call overhead is one of the lowest ones out there, but 
it's still much bigger than just a function call, and doing a full 
pathname walk and open/close is bigger yet. In contrast, if you access 
lots of objects and they are all in a pack, you only end up doing one mmap 
and a page fault for each 4kB entry, and that's it.

So packing has a large performance benefit outside of the actual disk use 
one, and to some degree that performance benefit is then further magnified 
by good locality (ie you get more effective objects per page fault), but 
in your case that locality issue is secondary.

I assume that you never actually end up looking at the _contents_ of the 
objects any more ever afterwards, because in a very real sense you're 
really interested in the SHA1 names, right? All the latter phases of 
parsecvs will just use the SHA1 names directly, and never actually even 
open the data (packed or not).

So in that sense, you only care about the disksize and a much improved 
directory walk from fewer files (until the repository has actually been 
fully created, at which point a repack will do the right thing).

			Linus