'git gc --aggressive' effectively unusable

Note: this is on a different repo from the 'git reflog expire --all' I
reported a bit earlier.

I have a git-svn checkout of a subversion repo which I wanted to compress
as much as possible. 'git gc --aggressive' starts to run fairly well, but
eats more and more memory and gets slower and slower. After it gets to
about 45% or 50% progress slows down noticeably and so far I haven't had
the patience to let it finish (40 minutes is already way too long).

A regular 'git gc' run completes without any problems.

$ du -sh .git/
612M    .git/

Special about this repo is that it contains two huge objects [1], which
could maybe be a factor:
     size    pack  SHA
- packages/po/sublevel4/da.po:
     495661  4654  801cd6451ece536c0ab41f79e09fc52efdf3361f
- packages/arch/powerpc/quik-installer/debian/po/da.po
     149515  1403  83a787b20817dc4d72db052de4055e7a7c9221d7  

Below some output from top and of the progress of the command showing the
problem. Check the change in number of compressed objects against the
timestamps from top.

Cheers,
FJP

[1] Caused by a bug in a script a couple of years back.

$ git gc --aggressive

Counting objects: 843342, done.
Delta compression using up to 2 threads.
Compressing objects:  53% (449663/836424)

top - 22:55:02 up 18 min,  1 user,  load average: 1.83, 1.68, 1.07
Tasks: 161 total,   1 running, 160 sleeping,   0 stopped,   0 zombie
Cpu0  : 91.4%us,  0.7%sy,  0.0%ni,  1.3%id,  6.6%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  : 97.7%us,  0.3%sy,  0.0%ni,  1.3%id,  0.7%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   2034284k total,  2018288k used,    15996k free,    10188k buffers
Swap:  2097148k total,    22612k used,  2074536k free,   449444k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 5861 fjp       20   0 1775m 1.3g 194m S  188 66.7  21:10.89 git


Counting objects: 843342, done.
Delta compression using up to 2 threads.
Compressing objects:  58% (486001/836424)

top - 23:00:12 up 23 min,  1 user,  load average: 1.96, 1.84, 1.30
Tasks: 158 total,   2 running, 156 sleeping,   0 stopped,   0 zombie
Cpu0  : 98.3%us,  0.7%sy,  0.0%ni,  0.7%id,  0.3%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  : 87.4%us,  1.7%sy,  0.0%ni,  0.0%id, 10.6%wa,  0.0%hi,  0.3%si,  0.0%st
Mem:   2034284k total,  2017516k used,    16768k free,     4696k buffers
Swap:  2097148k total,    22572k used,  2074576k free,   336944k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 5861 fjp       20   0 1903m 1.4g 172m S  182 71.4  30:37.58 git


Counting objects: 843342, done.
Delta compression using up to 2 threads.
Compressing objects:  61% (515958/836424)

top - 23:05:56 up 29 min,  1 user,  load average: 1.68, 1.85, 1.48
Tasks: 159 total,   1 running, 158 sleeping,   0 stopped,   0 zombie
Cpu0  : 86.7%us,  1.7%sy,  0.0%ni,  2.0%id,  9.7%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  : 96.7%us,  0.0%sy,  0.0%ni,  0.7%id,  2.7%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   2034284k total,  2018644k used,    15640k free,     2748k buffers
Swap:  2097148k total,    24312k used,  2072836k free,   343256k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 5861 fjp       20   0 1903m 1.4g 189m S  176 72.3  40:29.50 git

The environment is the same as for the reflog problem, but I should have 
added that info anyway. Here it is again.

On Saturday 03 April 2010, Frans Pop wrote:

I have a git-svn checkout of a subversion repo which I wanted to
compress as much as possible. 'git gc --aggressive' starts to run fairly
well, but eats more and more memory and gets slower and slower. After it
gets to about 45% or 50% progress slows down noticeably and so far I
haven't had the patience to let it finish (40 minutes is already way too
long).

I'm seeing this with both git 1.6.6.1 and 1.7.0.3 on the same repo.
Environment:
- Debian amd64/Lenny; Core Duo x86_64 2.6.34-rc3 -> 1.6.6.1
- Debian i386/Sid; chroot on the same machine -> 1.7.0.3

On Saturday 03 April 2010, Frans Pop wrote:

Special about this repo is that it contains two huge objects [1], which
could maybe be a factor:
     size    pack  SHA
- packages/po/sublevel4/da.po:
     495661  4654  801cd6451ece536c0ab41f79e09fc52efdf3361f
- packages/arch/powerpc/quik-installer/debian/po/da.po
     149515  1403  83a787b20817dc4d72db052de4055e7a7c9221d7  

To avoid confusion: these sizes are in kB.

On Fri, Apr 2, 2010 at 16:05, Frans Pop [off-list ref] wrote:

I haven't had the patience to let it finish

There's your problem.

$ git help gc | sed -n /--aggressive$/,+3p
       --aggressive
           Usually git gc runs very quickly while
           providing good disk space utilization
           and performance. This option will
           cause git gc to more aggressively
           optimize the repository at the expense
           of taking much more time. The effects
           of this optimization are persistent, so
           this option only needs to be used
           occasionally; every few hundred
           changesets or so.

Last time I used this option (on Linus's Linux repo), I let the
algorithm do its thing for a couple of hours. Maybe the efficiency
could be vastly improved, but it does finish if you let it.

SIncerely,
Michael Witten

On Sat, Apr 3, 2010 at 15:33, Michael Witten [off-list ref] wrote:

$ git help gc | sed -n /--aggressive$/,+3p

As an aside: I didn't realize I copied that in there; this would
probably be better:

$ git help gc | sed -n /--aggressive$/,/^$/p

On Saturday 03 April 2010, Michael Witten wrote:

On Fri, Apr 2, 2010 at 16:05, Frans Pop [off-list ref] wrote:

quoted

I haven't had the patience to let it finish

There's your problem.

Yes, I had seen that. But there's a difference between taking much more 
time and slowing down to such an extend that it never finishes.

I've tried it today on my linux-2.6 repo as well and the same thing 
happened. At first the progress is not fast but reasonable. When it gets 
to about 45% percent it starts slowing down a lot: from ~1500 objects per 
update of the counters to ~300 objects per update. And who knows what the 
progress is going to be when it reaches 70% or 90%: 10 per update?

With a total of over 2 milion objects in the repository such a low speed is 
simply not going to work, ever. So I maintain that it is effectively 
unusable.

Cheers,
FJP

On Sat, Apr 3, 2010 at 17:23, Frans Pop [off-list ref] wrote:

On Saturday 03 April 2010, Michael Witten wrote:

quoted

On Fri, Apr 2, 2010 at 16:05, Frans Pop [off-list ref] wrote:

quoted

I haven't had the patience to let it finish

...
I've tried it today on my linux-2.6 repo as well and the same thing
happened. At first the progress is not fast but reasonable. When it gets
to about 45% percent it starts slowing down a lot: from ~1500 objects per
update of the counters to ~300 objects per update. And who knows what
the progress is going to be when it reaches 70% or 90%: 10 per update?

With a total of over 2 milion objects in the repository such a low speed is
simply not going to work, ever. So I maintain that it is effectively
unusable.

Well, all I can do is quote myself:

    Last time I used this option (on Linus's Linux repo),
    I let the algorithm do its thing for a couple of hours.
    Maybe the efficiency could be vastly improved, but
    it does finish if you let it.

On Fri, Apr 2, 2010 at 16:12, Frans Pop [off-list ref] wrote:

I'm seeing this with both git 1.6.6.1 and 1.7.0.3 on the same repo.

I think I must have run gc with 1.7.0.2.199.g90a2bf9; perhaps you
could use something like oprofile to figure out where gc is spending
most of its time.

On Sun, 2010-04-04 at 01:23 +0200, Frans Pop wrote:

On Saturday 03 April 2010, Michael Witten wrote:

quoted

On Fri, Apr 2, 2010 at 16:05, Frans Pop [off-list ref] wrote:

quoted

I haven't had the patience to let it finish

There's your problem.

Yes, I had seen that. But there's a difference between taking much more 
time and slowing down to such an extend that it never finishes.

I've tried it today on my linux-2.6 repo as well and the same thing 
happened. At first the progress is not fast but reasonable. When it gets 
to about 45% percent it starts slowing down a lot: from ~1500 objects per 
update of the counters to ~300 objects per update. And who knows what the 
progress is going to be when it reaches 70% or 90%: 10 per update?

With a total of over 2 milion objects in the repository such a low speed is 
simply not going to work, ever. So I maintain that it is effectively 
unusable.

As a data point, when I do gc, I routinely use --aggressive.  It takes a
while here, but not forever.  (I'm a tad short of 2 million objects)

Repo is mainline + next + tip + stable >= 2.6.22 + local branches.

git@marge:..git/linux-2.6> time git gc --aggressive
Counting objects: 1909894, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (1889774/1889774), done.
Writing objects: 100% (1909894/1909894), done.
Total 1909894 (delta 1674098), reused 0 (delta 0)

real    22m24.943s
user    55m33.756s
sys     0m8.149s

git is 1.7.0.3

	-Mike