Re: Errors cloning large repo

Jakub Narebski [off-list ref] wrote:

Shawn O. Pearce wrote:

quoted

One thing that you could do is segment the repository into multiple
packfiles yourself, and then clone using rsync or http, rather than
using the native Git protocol.

By the way, it would be nice to have talked about fetch / clone
support for sending (and creating) _multiple_ pack files. Beside
the situation where we must use more than one packfile because
of size limits, it would also help clone as it could send existing
packs and pack only loose objects (trading perhaps some bandwidth
with CPU load on the server; think kernel.org).

I've thought about adding that type of protocol extension on
more than one occasion, but have now convinced myself that it is
completely unnecessary.  Well at least until a project has more
than 2^32-1 objects anyway.

The reason is we can send any size packfile over the network; there
is no index sent so there is no limit on how much data we transfer.
We could easily just dump all existing packfiles as-is (just clip
the header/footers and generate our own for the entire stream)
and then send the loose objects on the end.

The client could easily segment that into multiple packfiles
locally using two rules:

  - if the last object was not a OBJ_COMMIT and this object is
  an OBJ_COMMIT, start a new packfile with this object.

  - if adding this object to the current packfile exceeds my local
  filesize threshold, start a new packfile.

The first rule works because we sort objects by type, and commits
appear at the front of a packfile.  So if you see a non-commit
followed by a commit, that's the packfile boundary that the
server had.

The second rule is just common sense.  But I'm not sure the first
rule is even worthwhile; the server's packfile boundaries have no
real interest for the client.


But Linus has already pointed all of this out (more or less) in a
different fork of this thread.  ;-)

-- 
Shawn.