From: Anthony Muller <hidden> Date: 2021-03-08 21:31:59
What did you do before the bug happened? (Steps to reproduce your issue)
git clone https://github.com/notracking/hosts-blocklists
cd hosts-blocklists
git reflog expire --all --expire=now && git gc --prune=now --aggressive
What did you expect to happen? (Expected behavior)
Running gc on a ~300 MB repo should not take 1 hour 55 minutes when
running gc on a 2.6 GB repo (LLVM) only takes 24 minutes.
What happened instead? (Actual behavior)
Command took 1h 55m to complete on a ~300MB repo and used enough
resources that the machine is almost unusable.
What's different between what you expected and what actually happened?
Compression stage uses the majority of the resources and time. Compression
itself, when compared to something like zlib or lzma, should not take very long.
While more may be happening as objects are compressed, the amount of time
gc takes to compress the objects and the resources it consumed are both
unreasonable.
Memory: RSS = 3451152 KB (3.29 GB), VSZ = 29286272 KB (27.92 GB)
Time: 12902.83s user 8995.41s system 315% cpu 1:55:36.73 total
I've seen this issue with a number of repos and size of the repo does not
determine if this happens. LLVM @ 2.6 GB worked flawlessly, a 900 MB
repo never finished, this 300 MB repo takes forever, and if you test something
like chromium git will just crash.
[System Info]
hardware: 2.9Ghz Quad Core i7
git version:
git version 2.30.0
cpu: x86_64
no commit associated with this build
sizeof-long: 8
sizeof-size_t: 8
shell-path: /bin/sh
uname: Darwin 19.6.0 Darwin Kernel Version 19.6.0: Tue Jan 12 22:13:05 PST 2021; root:xnu-6153.141.16~1/RELEASE_X86_64 x86_64
compiler info: clang: 12.0.0 (clang-1200.0.32.28)
libc info: no libc information available
$SHELL (typically, interactive shell): /usr/local/bin/zsh
On Mon, Mar 8, 2021 at 1:32 PM Anthony Muller [off-list ref] wrote:
What did you do before the bug happened? (Steps to reproduce your issue)
git clone https://github.com/notracking/hosts-blocklists
cd hosts-blocklists
git reflog expire --all --expire=now && git gc --prune=now --aggressive
--aggressive tells git gc to discard all of its existing delta chains
and go find new ones, and to be fairly aggressive in how it looks for
candidates. This is going to be the primary source of the resource
usage you see, as well as the time.
Aggressive GCs are something you do once in a (very great) while. If
you try this without the --aggressive, how does it look?
What did you expect to happen? (Expected behavior)
Running gc on a ~300 MB repo should not take 1 hour 55 minutes when
running gc on a 2.6 GB repo (LLVM) only takes 24 minutes.
What happened instead? (Actual behavior)
Command took 1h 55m to complete on a ~300MB repo and used enough
resources that the machine is almost unusable.
What's different between what you expected and what actually happened?
Compression stage uses the majority of the resources and time. Compression
itself, when compared to something like zlib or lzma, should not take very long.
While more may be happening as objects are compressed, the amount of time
gc takes to compress the objects and the resources it consumed are both
unreasonable.
The compression happening here is delta compression, not simple
compression like zip. Git searches across the repository for similar
objects and stores them as chains with a base object and (essentially)
instructions for converting that base object into another object.
That's significantly more resource-intensive work than zipping some
data.
Memory: RSS = 3451152 KB (3.29 GB), VSZ = 29286272 KB (27.92 GB)
Time: 12902.83s user 8995.41s system 315% cpu 1:55:36.73 total
Git offers several knobs that can be used to influence (though not
necessarily control) its resource usage. On 64-bit Linux the defaults
are 1 thread per logical CPU (so hyperthreaded CPUs use double) and
_unlimited_ memory usage per thread. You might want to investigate
some options like pack.threads and pack.windowmemory to apply some
constraints.
I've seen this issue with a number of repos and size of the repo does not
determine if this happens. LLVM @ 2.6 GB worked flawlessly, a 900 MB
repo never finished, this 300 MB repo takes forever, and if you test something
like chromium git will just crash.
[System Info]
hardware: 2.9Ghz Quad Core i7
git version:
git version 2.30.0
cpu: x86_64
no commit associated with this build
sizeof-long: 8
sizeof-size_t: 8
shell-path: /bin/sh
uname: Darwin 19.6.0 Darwin Kernel Version 19.6.0: Tue Jan 12 22:13:05 PST 2021; root:xnu-6153.141.16~1/RELEASE_X86_64 x86_64
compiler info: clang: 12.0.0 (clang-1200.0.32.28)
libc info: no libc information available
$SHELL (typically, interactive shell): /usr/local/bin/zsh
From: brian m. carlson <hidden> Date: 2021-03-08 23:57:48
On 2021-03-08 at 22:29:16, Bryan Turner wrote:
On Mon, Mar 8, 2021 at 1:32 PM Anthony Muller [off-list ref] wrote:
quoted
What did you do before the bug happened? (Steps to reproduce your issue)
git clone https://github.com/notracking/hosts-blocklists
cd hosts-blocklists
git reflog expire --all --expire=now && git gc --prune=now --aggressive
--aggressive tells git gc to discard all of its existing delta chains
and go find new ones, and to be fairly aggressive in how it looks for
candidates. This is going to be the primary source of the resource
usage you see, as well as the time.
Aggressive GCs are something you do once in a (very great) while. If
you try this without the --aggressive, how does it look?
I should point out that this repository is also rather pathologically
structured. Almost every commit is an automatic commit updating the
same five files which are text files ranging from 5 MB to 11 MB.
When you use --aggressive, as Bryan pointed out, you're asking to throw
away all the deltas and try really hard to compute all of them fresh.
That's going to use a lot of memory because you're loading many large
text files into memory. It's also going to use a lot of CPU because
these files do indeed delta extremely well, and since computing deltas
on larger files is more expensive, especially when there are many of
them.
And that's just the blobs. The trees and commits are also going to be
nearly identically structured and will also delta well with virtually
every other similar object of their type. Normally Git sorts by size
which helps pick better candidates, but since these are all going to be
identically sized, the performance is going to suffer.
Now, I have the advantage in this case of being a person who's sometimes
on call for the maintenance of Git repositories and in that capacity,
that this is pathologically structured is obvious to me. But, yeah, I
would definitely not run --aggressive on this repo unless I needed to
and I would not expect it to perform well.
--
brian m. carlson (he/him or they/them)
Houston, Texas, US
From: Anthony Muller <hidden> Date: 2021-03-09 00:14:43
Thank you Brian and Bryan. You both clarified what was happening and now I know what to look for.
I can use a shallow clone for most repos, but there are some I want to keep history for. I don't need a full copy of this repo, but it was a good repo to show the issue I was facing.
Thanks again!
---- On Mon, 08 Mar 2021 23:56:53 +0000 brian m. carlson [off-list ref] wrote ----
> On 2021-03-08 at 22:29:16, Bryan Turner wrote:
> > On Mon, Mar 8, 2021 at 1:32 PM Anthony Muller [off-list ref] wrote:
> > >
> > > What did you do before the bug happened? (Steps to reproduce your issue)
> > >
> > > git clone https://github.com/notracking/hosts-blocklists
> > > cd hosts-blocklists
> > > git reflog expire --all --expire=now && git gc --prune=now --aggressive
> >
> > --aggressive tells git gc to discard all of its existing delta chains
> > and go find new ones, and to be fairly aggressive in how it looks for
> > candidates. This is going to be the primary source of the resource
> > usage you see, as well as the time.
> >
> > Aggressive GCs are something you do once in a (very great) while. If
> > you try this without the --aggressive, how does it look?
>
> I should point out that this repository is also rather pathologically
> structured. Almost every commit is an automatic commit updating the
> same five files which are text files ranging from 5 MB to 11 MB.
>
> When you use --aggressive, as Bryan pointed out, you're asking to throw
> away all the deltas and try really hard to compute all of them fresh.
> That's going to use a lot of memory because you're loading many large
> text files into memory. It's also going to use a lot of CPU because
> these files do indeed delta extremely well, and since computing deltas
> on larger files is more expensive, especially when there are many of
> them.
>
> And that's just the blobs. The trees and commits are also going to be
> nearly identically structured and will also delta well with virtually
> every other similar object of their type. Normally Git sorts by size
> which helps pick better candidates, but since these are all going to be
> identically sized, the performance is going to suffer.
>
> Now, I have the advantage in this case of being a person who's sometimes
> on call for the maintenance of Git repositories and in that capacity,
> that this is pathologically structured is obvious to me. But, yeah, I
> would definitely not run --aggressive on this repo unless I needed to
> and I would not expect it to perform well.
> --
> brian m. carlson (he/him or they/them)
> Houston, Texas, US
>