Re: Figured out how to get Mozilla into git

On Fri, 9 Jun 2006, Linus Torvalds wrote:

On Fri, 9 Jun 2006, Nicolas Pitre wrote:

quoted

quoted

So a single 2GB pack is already very much pushing it. It's really really 
hard to map in a 2GB file on a 32-bit platform: your VM is usually 
fragmented enough that it simply isn't practical. In fact, I think the 
limit for _practical_ usage of single packs is probably somewhere in the 
half-gig region, unless you just have 64-bit machines.

Sure, but have we already reached that size?

Not for the Linux repos.

But apparently the mozilla repo ends up being 2GB in git. From Martin:

  >> oh, I went back to a cvsimport that I started a couple days ago.
  >> Completed with no problems...
  >> 
  >> Last commit:
  >> commit 5ecb56b9c4566618fad602a8da656477e4c6447a
  >> Author: wtchang%redhat.com <wtchang%redhat.com>
  >> Date:   Fri Jun 2 17:20:37 2006 +0000
  >> 
  >>    Import NSPR 4.6.2 and NSS 3.11.1
  >> 
  >> mozilla.git$ du -sh .git/
  >> 2.0G    .git/

He also sais:

| git-repack -a -d but it OOMs on my 2GB+2GBswap machine :(

now that was done with _incremental_ repacking (ie his .git directory
won't be just one large pack),

So given the nature of packs, incrementally packing an imported 
repository _might_ cause worse problems since each pack must be self 
referenced by definition.  That means you may end up with multiple 
revisions of the same file distributed amongst as many packs hence none 
of those revisions are ever deltified, and to repack that you currently 
have to mmap all those packs at once.

but I bet that if you were to clone it
(without using the "-l" flag or rsync/http), you'd end up with serious
trouble because of the single-pack limit.

Maybe that single pack would instead be under the 512MB limit?  I'd be 
curious to know.

So we're starting to see archives where single packs are problematic for
a 32-bit architecture. 

Depending on the operation, the single pack might actually be better, 
especially for a full clone where everything gets mapped.  Multiple 
packs will always take more space, which is fine if you don't need 
access to all objects at once since individual packs are small, but the 
whole of them (when repacking or cloning) isn't.


Nicolas