I'm working with a repository containing a large number of revisions (it
has the full 2.6 kernel history) along with a bunch of large tarfiles of
prebuilt images (basically firmware images). These tarfiles are about 50MB
and are updated frequently.
I'm trying to figure out how to limit memory consumption when fetching from
this repo. The repo lives on a network drive, and during the
"remote: Compressing objects: ..."
phase, git-pack-objects quickly grows to 3 or more GB in size. I've tried
setting pack.windowMemory = 512m, and pack.deltaCacheLimit = 512m. Any
other suggested things to try?
This is running 1.5.4.3.
Thanks,
David
On Mon, 17 Mar 2008, David Brown wrote:
phase, git-pack-objects quickly grows to 3 or more GB in size. I've tried
setting pack.windowMemory = 512m, and pack.deltaCacheLimit = 512m. Any
other suggested things to try?
This *might* be the malloc() fragmentation issue. If you have lots of
large files with deltas in between them, you might well end up with a lot
of the allocations having been 'free()'d, but even if you have "only" one
gig in active use, there might be lots of holes in the allocation patterns
with "free" memory.
Does lowering the limits more aggressively make any difference (ie try
with those limits set to just a few tens of megs?) It will cause much
worse deltas (since now the delta machinery won't work well on those big
tar-files), so you probably don't want to do this for real, but as a test
of whether it's those allocations or soemthing else it would still be
interesting.
But using valgrind (--tool=massif) to show the real allocations might also
give an even better view. Maybe valgrind tells you only 1 gig is used, and
then yes, it would be about some kind of fragmentation issue. Or maybe
valgrind points out some other memory user entirely..
Linus