Re: pack operation is thrashing my server

Andi Kleen [off-list ref] wrote:

quoted

If you have only 256 MB of RAM, I'm afraid the machine dives into swap 
the moment it attempts to process that single 150-MB object during 
repacking.  Objects are always allocated entirely, including the 
deflated and inflated copy at some point.  Making git handle partial 
objects in memory would add complexity all over the map so I don't think 
it'll ever be implemented nor be desirable.

If the access pattern is sequential and not much reuse it might be possible
to madvise() strategically to do prefetch and early unmap of not used
anymore data. I used that successfully in a few programs in the past that did
aggressive mmap on very large files.

We actually do something better where we can.  However parts of
Git assume that it can get back a contiguous block of memory which
contains the entire file content, decompressed.  The data is stored
on disk compressed, so we cannot just mmap the data from disk.

-- 
Shawn.