Re: [RFC] Packing large repositories

Linus Torvalds [off-list ref] wrote:

On Wed, 28 Mar 2007, Dana How wrote:

quoted

Of course this is unusable, since object_entry's in an .idx
file have only 32 bits in their offset fields.  I conclude that
for such large projects,  git-repack/git-pack-objects would need
new options to control maximum packfile size.

Either that, or update the index file format. I think that your approach 
of having a size limiter is actually the *better* one, though. 

Nico and I were hoping we could push the index file format change back
until pack v4 was also worthy of merging.  So I had also started work
on an index-pack splitter:

  URL:    git://repo.or.cz/git/fastimport.git
  Branch: sp/splitpack
 
Its far from complete.

Well, it does affect fetching, in that "git index-pack" obviously would 
also need to be taught how to split the resulting indexed packs up into 
multiple smaller ones from one large incoming one. But that shouldn't be 
fundamentally hard either, apart from the inconvenience of having to 
rewrite the object count in the pack headers..

We already do this if its a thin-pack that is being made non-thin.
So its annoying, yea, but we already have one toe in that particular
pond...
 

To avoid that issue, it may be that it's actually better to split things 
up at pack-generation time *even* for the case of --stdout

This is actually a pretty good idea.  index-pack knows when the end
of its current packfile is; if there are 12 more bytes available in
the input stream and it looks like another pack header then it can
just restart itself over again with that next pack.  This is actually
a pretty small change in index-pack, certainly a lot less intrusive
than what I was trying to do in my sp/splitpack topic above.
 
-- 
Shawn.