Re: [PATCH 2/2] index-pack: reduce memory usage when the pack has large blobs

On Sat, Feb 25, 2012 at 08:49:55AM +0700, Nguyen Thai Ngoc Duy wrote:

2012/2/24 Ian Kumlien [off-list ref]:

quoted

Writing objects: 100% (1425/1425), 56.06 MiB | 4.62 MiB/s, done.
Total 1425 (delta 790), reused 1425 (delta 790)
fatal: Out of memory, malloc failed (tried to allocate 3310214315 bytes)
fatal: Out of memory, malloc failed (tried to allocate 3310214315 bytes)
fatal: Out of memory, malloc failed (tried to allocate 3310214315 bytes)
fatal: Out of memory, malloc failed (tried to allocate 3310214315 bytes)
To ../test_data/
 ! [remote rejected] master -> master (missing necessary objects)
 ! [remote rejected] origin/HEAD -> origin/HEAD (missing necessary objects)
 ! [remote rejected] origin/master -> origin/master (missing necessary objects)
error: failed to push some refs to '../test_data/'

So there are additional code paths to look at... =(

I can't say where that came from. Does this help? (Space damaged, may
need manual application)

If not, you might need to apply this to generate coredump, then look
and see where that failed malloc comes from

Actually, i added a backtrace and used addr2line to confirm my
suspicion... which is:
builtin/index-pack.c:414

ie get_data_from_pack... 

It looks to me like, if we are to support this kind of things, we need a
slightly different approach - instead of passing the data around, it
feels like passing a function pointer around would be beneficial.

Looking at the code i see alot of places where this would be a issue,
just the fact that get_data_from_pack is used in several functions that
might do some small operation and then just free it.

I understand and recognize that my "problem" is not what git was
designed for; it was designed for small files, which is very evident in
how it approaches the data... And I'd most definetly have to look alot
closer to this code... =)

-- 
Duy