Re: Git chokes on large file

Duy Nguyen [off-list ref] writes:

quoted

    $ git fsck --full --strict
    notice: HEAD points to an unborn branch (master)
    Checking object directories: 100% (256/256), done.
    fatal: Out of memory, malloc failed (tried to allocate 21474836481 bytes)

Back trace for this one
...
Not easy to fix. I started working on converting fsck to use
index-pack code for pack verification. index-pack supports large files
well, so in the end it might fix this (as well as speeding up fsck).
But that work has stalled for a long time.

You need to have enough memory (virtual is fine if you have enough
time) to do fsck.  Some part of index-pack could be refactored into
a common helper function that could be called from fsck, but I think
it would be a lot of work.

quoted

    $ git commit -m Test.
    [master (root-commit) 3df3655] Test.
    fatal: Out of memory, malloc failed (tried to allocate 21474836481 bytes)

I suspect that this one is only because you are letting the status
which involves diff to kick in (and for that you would need to have
enough memory).  As you suggested, it might be a good idea to take
bigfilethreshold account when deciding if we would want to run diff.

From: Junio C Hamano <redacted>

You need to have enough memory (virtual is fine if you have enough
time) to do fsck.  Some part of index-pack could be refactored into
a common helper function that could be called from fsck, but I think
it would be a lot of work.

How much memory is "enough"?  And how is it that fsck is coded so that
the available RAM is a limit on which repositories can be checked?
Did someone forget that RAM may be considerably smaller than the
amount of data a program may have to deal with?

Dale