Re: [PATCH 2/2] index-pack: reduce memory usage when the pack has large blobs

On Fri, Feb 24, 2012 at 07:23:21PM +0700, Nguyễn Thái Ngọc Duy wrote:

This command unpacks every non-delta objects in order to:

1. calculate sha-1
2. do byte-to-byte sha-1 collision test if we happen to have objects
   with the same sha-1
3. validate object content in strict mode

All this requires the entire object to stay in memory, a bad news for
giant blobs. This patch lowers memory consumption by not saving the
object in memory whenever possible, calculating SHA-1 while unpacking
the object.

This patch assumes that the collision test is rarely needed. The
collision test will be done later in second pass if necessary, which
puts the entire object back to memory again (We could even do the
collision test without putting the entire object back in memory, by
comparing as we unpack it).

In strict mode, it always keeps non-blob objects in memory for
validation (blobs do not need data validation). "--strict --verify"
also keeps blobs in memory.

Signed-off-by: Nguyễn Thái Ngọc Duy <redacted>

Actually, nevermind my last report - i had missed a merge :(

And now that i merged that part it seems like it doesn't do much..
(No real output for 2+ minutes)

I think i should reapply the patches again and verify that everything is
correct before reporting any additional progress.

But, this might not be before monday, unfortunately... But *thanks* for
posting the patches!