git doesn't like big files when pushing

I have a mail archive stored with git, in mbox form, and I made some
changes to a few of the files and checked them back in.

That worked fine, but when I went to push the stuff to my server, I got
the following errors:

$ git push origin
updating 'refs/heads/master'
  from 490badd9bec9ada3a21be275c97fb2a3a390f49e
  to   16be8985abc8a9c89ad2cc8f46a0d8e9786e832f
Generating pack...
Done counting 8 objects.
Deltifying 8 objects.
fatal: Out of memory, malloc failed
fatal: early EOF
unpack unpacker exited with error code
ng refs/heads/master n/a (unpacker error)


And here are the file sizes of the files that were changed:
$ ls -lh linux-usb-devel.save.200*
-rw-r--r-- 1 greg users 41M Jan  6 14:30 linux-usb-devel.save.2001
-rw-r--r-- 1 greg users 80M Jan  6 14:30 linux-usb-devel.save.2002
-rw-r--r-- 1 greg users 74M Jan  6 14:30 linux-usb-devel.save.2003
-rw-r--r-- 1 greg users 99M Jan  6 14:30 linux-usb-devel.save.2004
-rw-r--r-- 1 greg users 89M Jan  6 14:30 linux-usb-devel.save.2005

So, am I just foolish for trying to use git for this?  Should I just go
back to using rsync to back stuff like this up with?

thanks,

greg k-h

On Wed, Mar 01, 2006 at 02:08:02PM -0800, Greg KH wrote:

I have a mail archive stored with git, in mbox form, and I made some
changes to a few of the files and checked them back in.

That worked fine, but when I went to push the stuff to my server, I got
the following errors:

$ git push origin
updating 'refs/heads/master'
  from 490badd9bec9ada3a21be275c97fb2a3a390f49e
  to   16be8985abc8a9c89ad2cc8f46a0d8e9786e832f
Generating pack...
Done counting 8 objects.
Deltifying 8 objects.
fatal: Out of memory, malloc failed
fatal: early EOF
unpack unpacker exited with error code
ng refs/heads/master n/a (unpacker error)

Oh, and I'm using:
	$ git --version
	git version 1.2.3.g8c2f

if that helps or not.

thanks,

greg k-h

On Wed, Mar 01, 2006 at 02:08:40PM -0800, Greg KH wrote:

On Wed, Mar 01, 2006 at 02:08:02PM -0800, Greg KH wrote:

quoted

I have a mail archive stored with git, in mbox form, and I made some
changes to a few of the files and checked them back in.

That worked fine, but when I went to push the stuff to my server, I got
the following errors:

$ git push origin
updating 'refs/heads/master'
  from 490badd9bec9ada3a21be275c97fb2a3a390f49e
  to   16be8985abc8a9c89ad2cc8f46a0d8e9786e832f
Generating pack...
Done counting 8 objects.
Deltifying 8 objects.
fatal: Out of memory, malloc failed
fatal: early EOF
unpack unpacker exited with error code
ng refs/heads/master n/a (unpacker error)

Oh, and I'm using:
	$ git --version
	git version 1.2.3.g8c2f

Hm, 1.2.4.g6177 seems better, it's still trying to pack things, after
about 10 minutes, but at least it isn't dying yet.  I'll let you know if
it finishes properly or not...

thanks,

greg k-h

On Wed, 1 Mar 2006, Greg KH wrote:

That worked fine, but when I went to push the stuff to my server, I got
the following errors:

$ git push origin
updating 'refs/heads/master'
  from 490badd9bec9ada3a21be275c97fb2a3a390f49e
  to   16be8985abc8a9c89ad2cc8f46a0d8e9786e832f
Generating pack...
Done counting 8 objects.
Deltifying 8 objects.
fatal: Out of memory, malloc failed
fatal: early EOF

Gaah. We probably have a memory leak somewhere, and it just normally 
doesn't much matter.

Git does want to keep the "window" of the objects it packs in memory while 
packing (it would be really costly to read them in one at a time, over and 
over again), but it should hopefully not really not need tons more memory 
than that. Since the window is normally 10, and you only have 8 objects, 
it really wants to have all eight in memory, but it shouldn't need a whole 
lot more.

But maybe it's really the case that you can't fit those 8 objects in 
memory. One option (which might also solve some of the performance issues) 
is to make the window be based on object _size_ rather than just be a 
fixed number (ie with an 80MB object, you'd only try a couple of objects 
around it, not the full window).

		Linus

Hi,

Linus Torvalds wrote:

But maybe it's really the case that you can't fit those 8 objects in 
memory. One option (which might also solve some of the performance issues) 
is to make the window be based on object _size_ rather than just be a 
fixed number (ie with an 80MB object, you'd only try a couple of objects 
around it, not the full window).

Well, doesn't the pack process involve keeping objects of similar size 
in memory next to each other? My suspicion is that the system is running 
out of virtual address space because almost every malloc() will attempt 
to get an entirely new block because the one that was just freed is a 
few bytes too small.

    Simon