Re: [RFC] Packing large repositories

On Fri, 30 Mar 2007, Shawn O. Pearce wrote:

Linus Torvalds [off-list ref] wrote:

quoted

On Wed, 28 Mar 2007, Dana How wrote:

quoted

Of course this is unusable, since object_entry's in an .idx
file have only 32 bits in their offset fields.  I conclude that
for such large projects,  git-repack/git-pack-objects would need
new options to control maximum packfile size.

Either that, or update the index file format. I think that your approach 
of having a size limiter is actually the *better* one, though. 

Nico and I were hoping we could push the index file format change back
until pack v4 was also worthy of merging.  So I had also started work
on an index-pack splitter:

  URL:    git://repo.or.cz/git/fastimport.git
  Branch: sp/splitpack
 
Its far from complete.

Well... actually I really think the best solution might simply be a new 
index format right now.  All the preparatory work has been done already 
anyway.

IMHO that's the solution that would require the least work at this 
point, with the least possibility of issues/bugs.

Pack v4 could just use an index v3 which would be almost the same as 
index v2.

Remains only to determine what this new index format should look like.  
I think that the SHA1 table and the offset table should be separate.  
For one it will require less mmap space to binary search through 
the SHA1 table, and it will make things much easier if pack v4 stores 
the SHA1 table itself.


Nicolas