Re: Git server eats all memory

Avery Pennarun [off-list ref] wrote:

On Wed, Aug 4, 2010 at 10:57 AM, Ivan Kanis

quoted

I am having problem with memory ballooning when receiving object
from the server. The amount of memory used on the server seems to be same
size as the object received.

Git works fine with huge repositories; it does not work fine at all
with very large individual objects in a repository, and it does what
you're experiencing.

Hello Avery,

The largest object is 120M. I didn't describe the problem very
accurately. The memory consumed is the *sum* of the objects
downloaded. 

For example, a repository of 4G consumes 4G of memory at the end of the
receiving objects phase. What is very interesting is the total of
objects downloaded is the same as the memory consumed. That makes me
think there is a link somewhere. Surely it shouldn't consume that much
memory.

I am ready to do whatever to diagnose the problem. I know C pretty well
and am ready to look into it but I am not sure where to start.

Take care,
-- 
Ivan Kanis
http://kanis.fr

Seriousness is the only refuge of the shallow.
    -- Oscar Wilde 

On Thu, Aug 05, 2010 at 08:33:02AM +0200, Ivan Kanis wrote:

I am ready to do whatever to diagnose the problem. I know C pretty well
and am ready to look into it but I am not sure where to start.

It sounds like Git is reading each object into memory to send it to
the client, but doesn't remember to free the memory at the end of
sending the object, so it remains as a memory leak.

I would look in the code for malloc calls that don't have a free call,
or spots where free calls might not be hit.

On Thu, Aug 5, 2010 at 4:33 PM, Ivan Kanis
[off-list ref] wrote:

I am ready to do whatever to diagnose the problem. I know C pretty well
and am ready to look into it but I am not sure where to start.

Try "git pack-objects --all --stdout > /dev/null" on the repo on
server to see if it uses the same amount of memory you saw in cloning.
You can then try debugging that command if it does.
-- 
Duy

On Fri, Aug 6, 2010 at 11:37 AM, Nguyen Thai Ngoc Duy [off-list ref] wrote:

On Thu, Aug 5, 2010 at 4:33 PM, Ivan Kanis
[off-list ref] wrote:

quoted

I am ready to do whatever to diagnose the problem. I know C pretty well
and am ready to look into it but I am not sure where to start.

Try "git pack-objects --all --stdout > /dev/null" on the repo on
server to see if it uses the same amount of memory you saw in cloning.
You can then try debugging that command if it does.

Naah, git pack-objects needs list of commit tips. Try
git for-each-ref|cut -c 1-40|git pack-objects --all --stdout > /dev/null
-- 
Duy

Nguyen Thai Ngoc Duy [off-list ref] writes:

On Fri, Aug 6, 2010 at 11:37 AM, Nguyen Thai Ngoc Duy [off-list ref] wrote:

quoted

On Thu, Aug 5, 2010 at 4:33 PM, Ivan Kanis [off-list ref] wrote:

quoted

quoted

I am ready to do whatever to diagnose the problem. I know C pretty well
and am ready to look into it but I am not sure where to start.

Try "git pack-objects --all --stdout > /dev/null" on the repo on
server to see if it uses the same amount of memory you saw in cloning.
You can then try debugging that command if it does.

Naah, git pack-objects needs list of commit tips. Try
git for-each-ref|cut -c 1-40|git pack-objects --all --stdout > /dev/null

Nitpick: git-for-each-ref has `--format' option, no need for `cut'.

-- 
Jakub Narebski
Poland
ShadeHawk on #git