Re: [FYI] very large text files and their problems.

Seems like i ruined my dovecot config in a recent upgrade - which also
affected my mail... =/

Anyway, it's all fixed now.

from: Nguyen Thai Ngoc Duy <pclouds () gmail ! com>

On Wed, Feb 22, 2012 at 10:49 PM, Ian Kumlien [off-list ref] wrote:

quoted

Hi,

We just saw a interesting issue, git compressed a ~3.4 gb project to

~57 mb.

How big are those files? How many of them? How often do they change?

This is the initial check in, one of the files is a 3.3 gb text file.

quoted

But when we tried to clone it on a big machine we got:

fatal: Out of memory, malloc failed (tried to allocate
18446744072724798634 bytes)

This is already fixed in the 1.7.10 mainline - but it also seems

like

Does 1.7.9 have this problem?

I've tested with 1.7.9.1, haven't downgraded to test with 1.7.9...

quoted

git needs to have atleast the same ammount of memory as the largest
file free... Couldn't this be worked around?

On a (32 bit) machine with 4GB memory - results in:
fatal: Out of memory, malloc failed (tried to allocate 3310214313

bytes)

quoted

(and i see how this could be a problem, but couldn't it be

mitigated? or

quoted

is it bydesign and intended behaviour?)

I think that it's delta resolving that hogs all your memory. If your
files are smaller than 512M, try lower core.bigFileThreshold. The
topic jc/split-blob, which stores a big file are several smaller
pieces, might solve your problem. Unfortunately the topic is not
complete yet.

the problem here is that there is one file that is exactly: 3310214313
bytes, so it should all be one "blob".

split-blob would be really interesting for several reasons though =)

-- 
Duy
--

-- 
Ian Kumlien  -- http://demius.net || http://pomac.netswarm.net