Re: Errors cloning large repo

I suspect we shouldn't bother with the diffstat for the initial commit. 
Just removing "--root" migth be sufficient.

My problem is git-clone though since for commit it's no big deal
to git commit [a-c]* , or use xargs as a workaround

For git clone I got this

Deltifying 144511 objects.
 100% (144511/144511) done
1625.375MB  (1713 kB/s)       
1729.057MB  (499 kB/s)       
/usr/bin/git-clone: line 321: 24360 File size limit exceededgit-fetch-pack --all -k $quiet "$repo"

again after git repack and don't see how to work around that aside from artifically
splitting the tree at the top or resorting to a tarball on an ftp site.
That 64 bit indexing code you previously mentioned would force me to upgrade git on both ends?
Anywhere I can pull it out from?






 
____________________________________________________________________________________
Food fight? Enjoy some healthy debate 
in the Yahoo! Answers Food & Drink Q&A.
http://answers.yahoo.com/dir/?link=list&sid=396545367

Anton Tropashko [off-list ref] wrote:

again after git repack and don't see how to work around that aside from artifically
splitting the tree at the top or resorting to a tarball on an ftp site.
That 64 bit indexing code you previously mentioned would force me to upgrade git on both ends?
Anywhere I can pull it out from?

I'm shocked you were able to repack an 8.5 GiB repository.
The default git-repack script that we ship assumes you want to
combine everything into one giant packfile; this is what is also
happening during git-clone.  Clearly your system is rejecting this
packfile; and even if the OS allowed us to make that file the
index offsets would all be wrong as they are only 32 bits wide.
The repository becomes corrupt when those overflow.


Troy Telford (with the help of Eric Biederman) recently posted a
patch that attempts to push the index to 64 bits:

  http://thread.gmane.org/gmane.comp.version-control.git/40680/focus=40999

You can try Troy's patch.  Nico and my's 64 bit index work is *not*
ready for anyone to use.  It doesn't exist as a compileable chunk
of code.  ;-)

Just to warn you, I have (re)done some of Troy's changes and Junio
has applied them to the current 'master' branch.  So Troy's patch
would need to be applied to something that is futher back, like
around 2007-02-28 (when Troy sent the patch).  But my changes alone
are not enough to get "64 bit packfiles" working.


As Linus said earlier in this thread; Nico and I are working on
pushing out the packfile limits, just not fast enough for some users
needs apparently (sorry about that!).  Troy's patch was rejected
mainly because it is a file format change that is not backwards
compatible (once you use the 64 bit index, anything accessing that
repository *must* also support that).

Nico and I are working on other file format changes that are
more extensive than just expanding the index out to 64 bits, and
likewise are also not backwards compatible.  To help users manage
the upgrades, we want to do a single file format change in 2007,
not two.  So we are trying to be very sure that what we give Junio
for final application really is the best we can do this year.

Otherwise we would have worked with Troy to help test his patch and
get that into shape for application to main the git.git repository.


One thing that you could do is segment the repository into multiple
packfiles yourself, and then clone using rsync or http, rather than
using the native Git protocol.

For segmenting the repository, you would do something like:

	git rev-list --objects HEAD >S
	# segment S up into several files, e.g. T1, T2, T3
	foreach s in T*
	do
		name=$(git pack-objects tmp <$s)
		touch .git/objects/pack/pack-$name.keep
		mv tmp-$name.pack .git/objects/pack/pack-$name.pack
		mv tmp-$name.idx .git/objects/pack/pack-$name.idx
	done
	git prune-packed

The trick here is to segment S up into enough T1, T2, ... files such
that when packed they each are less than 2 GiB.  You can then clone
this repository by copying the .git directory using more standard
filesystem tools, which is what a clone with rsync or http is
(more or less) doing.

Yes, the above process is horribly tedious and has a some trial
and error involved in terms of selecting the packfile segmenting.
We don't have anything that can automate this right now.

-- 
Shawn.

On Fri, 9 Mar 2007, Anton Tropashko wrote:

My problem is git-clone though since for commit it's no big deal
to git commit [a-c]* , or use xargs as a workaround

Sure, but there were two problems.

The "git commit" problem is trivial, and in no way fundamental. The thing 
that uses tons of memory is literally just eyecandy, to show you *what* 
you're committing.

In fact, by the time it starts using tons of memory, the commit has 
literally already happened. It's just doing statistics afterwards that 
bloats it up.

For git clone I got this

The "git clone" problem is different, in that it's due to the 2GB 
pack-file limit. It's not "fundmentally hard" either, but it's at least 
not just a small tiny silly detail.

In fact, you can just do

	git add .
	git commit -q

and the "-q" flag (or "--quiet") will mean that the diffstat is never 
done, and the commit should be almost instantaneous (all the real work is 
done by the "git add .")

So "git commit" issue really is just a small beauty wart.

Deltifying 144511 objects.
 100% (144511/144511) done
1625.375MB  (1713 kB/s)       
1729.057MB  (499 kB/s)       
/usr/bin/git-clone: line 321: 24360 File size limit exceededgit-fetch-pack --all -k $quiet "$repo"

again after git repack and don't see how to work around that aside from artifically
splitting the tree at the top or resorting to a tarball on an ftp site.

So the "git repack" actually worked for you? It really shouldn't have 
worked.

Is the server side perhaps 64-bit? If so, the limit ends up being 4GB 
instead of 2GB, and your 8.5GB project may actually fit.

If so, we can trivially fix it with the current index file even for a 
32-bit machine. The reason we limit pack-files to 2GB on 32-bit machines 
is purely that we don't use O_LARGEFILE. If we enable O_LARGEFILE, that 
moves the limit up from 31 bits to 32 bits, and it might be enough for 
you. No new data structures for the index necessary at all.

		Linus

On Fri, 9 Mar 2007, Shawn O. Pearce wrote:

I'm shocked you were able to repack an 8.5 GiB repository.

Side note - it would be nice to hear just how big the repository *really* 
is.

For example, if "du -sh" says 8.5GB, it doesn't necessarily mean that 
there really is 8.5GB of data there.

With a normal 4kB blocksize filesystem, and ~150.000 filesystem objects, 
you'd have an average of 300MB of just padding (roughly 2kB per file). 
Depending on the file statistics, it could be even more.

And if it's compressible, it's entirely possible that even without 
much delta compression, it could fit in a pack-file smaller than 4GB. At 
which point a 32-bit index file should work fine, just not with a 32-bit 
off_t.

So this really could be a situation where just small tweaks makes it work 
out for now. We'll need the full 64-bit index eventually for sure, but..

		Linus

Linus Torvalds [off-list ref] wrote:

On Fri, 9 Mar 2007, Shawn O. Pearce wrote:

quoted

I'm shocked you were able to repack an 8.5 GiB repository.

Side note - it would be nice to hear just how big the repository *really* 
is.

For example, if "du -sh" says 8.5GB, it doesn't necessarily mean that 
there really is 8.5GB of data there.

Oh, good point.  Thanks for reminding me of reality.

I'm just so used to not looking at repository size unless the
repository has been fully repacked first.  So I somehow just read
this thread has Anton having 8.5 GiB worth of *packed* data (where
filesystem wastage in the tail block is minimal) and not 8.5 GiB
of loose objects.

Its very likely this did fit in just under 4 GiB of packed data,
but as you said, without O_LARGEFILE we can't work with it.
 

So this really could be a situation where just small tweaks makes it work 
out for now. We'll need the full 64-bit index eventually for sure, but..

Yes.  ;-)

-- 
Shawn.

Shawn O. Pearce wrote:

One thing that you could do is segment the repository into multiple
packfiles yourself, and then clone using rsync or http, rather than
using the native Git protocol.

By the way, it would be nice to have talked about fetch / clone
support for sending (and creating) _multiple_ pack files. Beside
the situation where we must use more than one packfile because
of size limits, it would also help clone as it could send existing
packs and pack only loose objects (trading perhaps some bandwidth
with CPU load on the server; think kernel.org).

-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

hoi :)

On Sat, Mar 10, 2007 at 01:01:44AM -0500, Shawn O. Pearce wrote:

Its very likely this did fit in just under 4 GiB of packed data,
but as you said, without O_LARGEFILE we can't work with it.

but newer git version can cope with it:

-r--r--r-- 1 martin martin 3847536413 18. Feb 10:36 pack-ffe867679d673ea5fbfa598b28aca1e58528b8cd.pack

-- 
Martin Waitz

On Sat, 10 Mar 2007, Martin Waitz wrote:

On Sat, Mar 10, 2007 at 01:01:44AM -0500, Shawn O. Pearce wrote:

quoted

Its very likely this did fit in just under 4 GiB of packed data,
but as you said, without O_LARGEFILE we can't work with it.

but newer git version can cope with it:

-r--r--r-- 1 martin martin 3847536413 18. Feb 10:36 pack-ffe867679d673ea5fbfa598b28aca1e58528b8cd.pack

Are you sure you're not just running a 64-bit process?

64-bit processes don't need O_LARGEFILE to process files larger than 2GB, 
since for them, off_t is already 64-bit.

Grepping for O_LARGEFILE shows nothing.

Oh, except we have that 

	#define _FILE_OFFSET_BITS 64

which is just a horrible hack. That's nasty. We should just use 
O_LARGEFILE rather than depend on some internal glibc thing that works 
nowhere else.

		Linus

Jakub Narebski [off-list ref] wrote:

Shawn O. Pearce wrote:

quoted

One thing that you could do is segment the repository into multiple
packfiles yourself, and then clone using rsync or http, rather than
using the native Git protocol.

By the way, it would be nice to have talked about fetch / clone
support for sending (and creating) _multiple_ pack files. Beside
the situation where we must use more than one packfile because
of size limits, it would also help clone as it could send existing
packs and pack only loose objects (trading perhaps some bandwidth
with CPU load on the server; think kernel.org).

I've thought about adding that type of protocol extension on
more than one occasion, but have now convinced myself that it is
completely unnecessary.  Well at least until a project has more
than 2^32-1 objects anyway.

The reason is we can send any size packfile over the network; there
is no index sent so there is no limit on how much data we transfer.
We could easily just dump all existing packfiles as-is (just clip
the header/footers and generate our own for the entire stream)
and then send the loose objects on the end.

The client could easily segment that into multiple packfiles
locally using two rules:

  - if the last object was not a OBJ_COMMIT and this object is
  an OBJ_COMMIT, start a new packfile with this object.

  - if adding this object to the current packfile exceeds my local
  filesize threshold, start a new packfile.

The first rule works because we sort objects by type, and commits
appear at the front of a packfile.  So if you see a non-commit
followed by a commit, that's the packfile boundary that the
server had.

The second rule is just common sense.  But I'm not sure the first
rule is even worthwhile; the server's packfile boundaries have no
real interest for the client.


But Linus has already pointed all of this out (more or less) in a
different fork of this thread.  ;-)

-- 
Shawn.

hoi :)

On Sat, Mar 10, 2007 at 02:46:35PM -0800, Linus Torvalds wrote:

Are you sure you're not just running a 64-bit process?

pretty sure, yes :-)

64-bit processes don't need O_LARGEFILE to process files larger than 2GB, 
since for them, off_t is already 64-bit.

but O_LARGEFILE is a Linux-only thing, right?

Oh, except we have that 

	#define _FILE_OFFSET_BITS 64

which is just a horrible hack. That's nasty. We should just use 
O_LARGEFILE rather than depend on some internal glibc thing that works 
nowhere else.

Well, if I remember correctly the *BSD systems always use 64bit now,
its sad that glibc does not do the same out of the box for Linux.
_FILE_OFFSET_BITS is the documented way to get 64bit file sizes on
glibc, so I think it is the right thing for us (even when that define
is really ugly).

-- 
Martin Waitz

Shawn O. Pearce wrote:

Jakub Narebski [off-list ref] wrote:

quoted

Shawn O. Pearce wrote:

quoted

One thing that you could do is segment the repository into multiple
packfiles yourself, and then clone using rsync or http, rather than
using the native Git protocol.

By the way, it would be nice to have talked about fetch / clone
support for sending (and creating) _multiple_ pack files. Beside
the situation where we must use more than one packfile because
of size limits, it would also help clone as it could send existing
packs and pack only loose objects (trading perhaps some bandwidth
with CPU load on the server; think kernel.org).

I've thought about adding that type of protocol extension on
more than one occasion, but have now convinced myself that it is
completely unnecessary.  Well at least until a project has more
than 2^32-1 objects anyway.

The reason is we can send any size packfile over the network; there
is no index sent so there is no limit on how much data we transfer.
We could easily just dump all existing packfiles as-is (just clip
the header/footers and generate our own for the entire stream)
and then send the loose objects on the end.

But what would happen if server supporting concatenated packfiles
sends such stream to the old client? So I think some kind of protocol
extension, or at least new request / new feature is needed for that.

Wouldn't it be better to pack loose objects into separate pack
(and perhaps save it, if some threshold is crossed, and we have
writing rights to repo), by the way?

The client could easily segment that into multiple packfiles
locally using two rules:

  - if the last object was not a OBJ_COMMIT and this object is
  an OBJ_COMMIT, start a new packfile with this object.

  - if adding this object to the current packfile exceeds my local
  filesize threshold, start a new packfile.

The first rule works because we sort objects by type, and commits
appear at the front of a packfile.  So if you see a non-commit
followed by a commit, that's the packfile boundary that the
server had.

The second rule is just common sense.  But I'm not sure the first
rule is even worthwhile; the server's packfile boundaries have no
real interest for the client.

Without first rule, wouldn't client end with strange packfile?
Or would it have to rewrite a pack?

-- 
Jakub Narebski
Poland

Jakub Narebski [off-list ref] wrote:

But what would happen if server supporting concatenated packfiles
sends such stream to the old client? So I think some kind of protocol
extension, or at least new request / new feature is needed for that.

No, a protocol extension is not required.  The packfile format
is: 12 byte header, objects, 20 byte SHA-1 footer.  When sending
concatenated packfiles to a client the server just needs to:

  - figure out how many objects total will be sent;
  - send its own (new) header with that count;
  - initialize a SHA-1 context and update it with the header;
  - for each packfile to be sent:
    - strip the first 12 bytes of the packfile;
    - send the remaining bytes, except the last 20;
    - update the SHA-1 context with the packfile data;
  - send its own footer with the SHA-1 context.

Very simple.  Even the oldest Git clients (pre multi-ack extension)
would understand that.  That's what's great about the way the
packfile protocol and disk format is organized.  ;-)
 

Wouldn't it be better to pack loose objects into separate pack
(and perhaps save it, if some threshold is crossed, and we have
writing rights to repo), by the way?

Perhaps.  Interesting food for thought, something nobody has tried
to experiment with.  Currently servers pack to update the fetching
client.  That means they may be sending a mixture of already-packed
(older) objects and loose (newer) objects.  But with the new kept
pack thing in receive-pack its more likely that things are already
packed on the server, and not loose.  (I suspect most public open
source users are pushing >100 objects when they do push to their
server.)
 

quoted

The client could easily segment that into multiple packfiles
locally using two rules:

  - if the last object was not a OBJ_COMMIT and this object is
  an OBJ_COMMIT, start a new packfile with this object.

...

Without first rule, wouldn't client end with strange packfile?
Or would it have to rewrite a pack?

Nope.  We don't care about the order of the objects in a packfile.
Never have.  Never will.  Even in pack v4 where we have special
object types that should only appear once in a packfile, they can
appear at any position within the packfile.  MUCH simpler code.

-- 
Shawn.

On Mon, 12 March 2007, Shawn O. Pearce wrote:

Jakub Narebski [off-list ref] wrote:

quoted

But what would happen if server supporting concatenated packfiles
sends such stream to the old client? So I think some kind of protocol
extension, or at least new request / new feature is needed for that.

No, a protocol extension is not required.  The packfile format
is: 12 byte header, objects, 20 byte SHA-1 footer.  When sending
concatenated packfiles to a client the server just needs to:

  - figure out how many objects total will be sent;
  - send its own (new) header with that count;
  - initialize a SHA-1 context and update it with the header;
  - for each packfile to be sent:
    - strip the first 12 bytes of the packfile;
    - send the remaining bytes, except the last 20;
    - update the SHA-1 context with the packfile data;
  - send its own footer with the SHA-1 context.

Very simple.  Even the oldest Git clients (pre multi-ack extension)
would understand that.  That's what's great about the way the
packfile protocol and disk format is organized.  ;-)

It would be a very nice thing to have, if it is backwards compatibile.
It would ease load to server on clone, even if packs are divided into
large tight archive pack and perhaps a few more current packs to make
dumb transport do not neeed to download everything on [incremental]
fetch.

On fetch... perhaps there should be some configuration variable which
would change balance between load and bandwidth used...

And automatic splitting large pack on client side would help if for
example we have huge repository (non-compressable binaries) and client
has smaller filesystem limit on maximum file size than server.

-- 
Jakub Narebski
Poland