Re: Git server eats all memory

Nguyen Thai Ngoc Duy [off-list ref] wrote:

Naah, git pack-objects needs list of commit tips. Try
git for-each-ref|cut -c 1-40|git pack-objects --all --stdout > /dev/null

Jared Hance [off-list ref] wrote:

I would look in the code for malloc calls that don't have a free call,
or spots where free calls might not be hit.

Hello Jared and Nguyen,

Thank you Nguyen for your command. I can now reproduce the problem
without needing the network. I have been following Jared lead today on
a potential memory leak. Here is what I found out.

I downloaded the latest release of git 1.7.2.1 and compiled it with
debugging support. I ran valgrind on the command and found two memory
leaks. I put the output at the bottom of the e-mail as it's not very
interesting. I patched one of the leak in pack_objects.c but got the
same problem: over 4G of memory consumption for a 4G repository.

I've come to the conclusion that it's not a memory leak. 

This afternoon I put macro around the following functions: xmalloc
xmallocz, xrealloc, xcalloc and xmmap. It reported the line of code and
size passed in each functions. I then run the result through a script
that totaled the amount used by each bit of code.

Here are the top 3 consumers:

| function | source                     | size in M |
|----------+----------------------------+-----------|
| xrealloc | builtin/pack-objects.c:690 |        86 |
| xmallocz | patch-delta.c:36           |       301 |
| xmmap    | sha1_file.c:772            |      4393 |

I expected the malloc to take 4G but was surprised it didn't. It seems
to be mmap taking all the memory. I am not familiar with that function,
it looks like it's mapping memory to a file... Is it reasonable to mmap
so much memory?

Today I chatted with someone on freenode #git and he reported the same
problem on his 2G repository, I am glad I am not the only one seeing
this ;)

I tried reading the code but it's going over my head. I'll look at is
some more next monday.

If anyone is familiar with the code source of git I would love to have
some insight into this.

Take care,

Ivan Kanis

PS: output of valgrind --leak-check=full

65 bytes in 1 blocks are definitely lost in loss record 4 of 7
   at 0x4C2260E: malloc (vg_replace_malloc.c:207)
   by 0x4C22797: realloc (vg_replace_malloc.c:429)
   by 0x4C600D: xrealloc (wrapper.c:80)
   by 0x4B7939: strbuf_grow (strbuf.c:70)
   by 0x4B80BA: strbuf_addf (strbuf.c:201)
   by 0x4832EF: system_path (exec_cmd.c:37)
   by 0x483411: setup_path (exec_cmd.c:104)
   by 0x404AF2: main (git.c:536)

512 bytes in 1 blocks are definitely lost in loss record 5 of 8
   at 0x4C203E4: calloc (vg_replace_malloc.c:397)
   by 0x4C5F9D: xcalloc (wrapper.c:96)
   by 0x445741: cmd_pack_objects (pack-objects.c:2117)
   by 0x4048EE: handle_internal_command (git.c:270)
   by 0x404B03: main (git.c:470)
-- 
http://kanis.fr

Everything should be made as simple as possible, but not simpler.
    -- Albert Einstein 

On Fri, Aug 06, 2010 at 07:23:17PM +0200, Ivan Kanis wrote:

I expected the malloc to take 4G but was surprised it didn't. It seems
to be mmap taking all the memory. I am not familiar with that function,
it looks like it's mapping memory to a file... Is it reasonable to mmap
so much memory?

AFAIK, Git does not need to mmap the whole pack to memory, but it
is more efficient to mmap the whole pack wherever possible, because
it has a completely random access, so if you store only one sliding
window, you will have to re-read it many times. Besides, mmap size
does not mean that so much physical memory is used. Pages should
be loaded when they are necessary, and if you have more than one
client cloning the same repo, this memory should be shared by them.


Dmitry