From: Ken Pratt <hidden> Date: 2016-06-15 22:45:08
Hi,
I'm having memory issues when trying to clone a remote git repository.
I'm running: "git clone git+ssh://user@foo.bar.com/var/git/foo"
The remote repository is bare, and is 180MB in size (says du), with
1824 objects. The remote (VPS) server is running git version 1.5.6.4
on Arch Linux on a x86_64 Opteron with 256MB of dedicated RAM.
The clone command fires off some packing operations that bring the
server to its knees:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21782 kenpratt 20 0 444m 212m 272 D 3 83.0 0:04.98 git-pack-object
The clone also seems to hang forever. Progress stays at 0% for hours,
and it never progresses past compressing the first object.
I've tried very conservative pack settings:
[pack]
threads = 1
windowmemory = 64M
deltacachesize = 1M
deltacachelimit = 1M
[pack]
threads = 1
windowmemory = 16M
deltacachesize = 16M
deltacachelimit = 0
I've tries many variations like those, but nothing seems to help.
A "git repack -a -d" only takes 5 seconds to run on the same
repository on my laptop (a non-bare copy), and seems to peak at ~160MB
of RAM usage.
Any tips/help would be greatly appreciated. This repository is still
small -- it will eventually grow to multiple GB in size, as it is a
mix of small text files and binaries ranging in size from 2MB to
200MB. Is it not feasible to clone repositories of that size that are
hosted on a server with 256MB of RAM?
Thanks!
Ken
From: Martin Langhoff <hidden> Date: 2016-06-15 22:45:08
On Mon, Aug 11, 2008 at 7:47 AM, Ken Pratt [off-list ref] wrote:
A "git repack -a -d" only takes 5 seconds to run on the same
repository on my laptop (a non-bare copy), and seems to peak at ~160MB
of RAM usage.
As a workaround, if you repack on your laptop and rsync the pack+index
to the server, it will work. This can be used to serve huge projects
out of lightweight-ish servers. Yet another workaround is to perform
initial clones via rsync or http.
In your case, I agree that the repo doesn't seem large enough (or to
have large enough objects) to warrant having this problem. But that I
can't help much with myself - pack-machiner experts probably can.
cheers,
m
--
martin.langhoff@gmail.com
martin@laptop.org -- School Server Architect
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
From: Ken Pratt <hidden> Date: 2016-06-15 22:45:08
Thanks for the tips, Martin.
How does git over rsync work? It is unauthenticated, like git over
http? Or authenticated, like git+ssh?
Great ideas though. Unfortunately I don't think I'll be able to use
the repack locally and then upload strategy for this particular
workflow, but the rsync clone approach might do it.
-Ken
On Sun, Aug 10, 2008 at 4:06 PM, Martin Langhoff
[off-list ref] wrote:
On Mon, Aug 11, 2008 at 7:47 AM, Ken Pratt [off-list ref] wrote:
quoted
A "git repack -a -d" only takes 5 seconds to run on the same
repository on my laptop (a non-bare copy), and seems to peak at ~160MB
of RAM usage.
As a workaround, if you repack on your laptop and rsync the pack+index
to the server, it will work. This can be used to serve huge projects
out of lightweight-ish servers. Yet another workaround is to perform
initial clones via rsync or http.
In your case, I agree that the repo doesn't seem large enough (or to
have large enough objects) to warrant having this problem. But that I
can't help much with myself - pack-machiner experts probably can.
cheers,
m
--
martin.langhoff@gmail.com
martin@laptop.org -- School Server Architect
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
From: Martin Langhoff <hidden> Date: 2016-06-15 22:45:08
On Mon, Aug 11, 2008 at 11:12 AM, Ken Pratt [off-list ref] wrote:
Thanks for the tips, Martin.
NP! :-)
How does git over rsync work? It is unauthenticated, like git over
http? Or authenticated, like git+ssh?
I've always used it as rsync+ssh. Not sure about bare rsync.
Great ideas though. Unfortunately I don't think I'll be able to use
the repack locally and then upload strategy for this particular
workflow, but the rsync clone approach might do it.
A few specific versions of git had bad repack cpu/memory usage
patterns, so an update to git might help. In any case, the repack
machinery experts are probably asleep. Give it a bit of time and
smarter answers will probably materialise.
cheers,
m
--
martin.langhoff@gmail.com
martin@laptop.org -- School Server Architect
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
From: Ken Pratt <hidden> Date: 2016-06-15 22:45:08
Sounds good.
quoted
How does git over rsync work? It is unauthenticated, like git over
http? Or authenticated, like git+ssh?
I've always used it as rsync+ssh. Not sure about bare rsync.
Do you use file-level rsync+ssh? Or rsync+ssh with git?
When I try a "git clone rsync+ssh://foo.bar.com/var/git/bar", I get a
"fatal: I don't handle protocol 'rsync+ssh'" error.
I know git supports the rsync protocol, but I don't think installing
an rsync server and using bare rsync will be an option in this case.
Thanks again,
Ken
From: Shawn O. Pearce <hidden> Date: 2016-06-15 22:45:08
Ken Pratt [off-list ref] wrote:
I'm having memory issues when trying to clone a remote git repository.
The remote repository is bare, and is 180MB in size (says du), with
1824 objects. The remote (VPS) server is running git version 1.5.6.4
on Arch Linux on a x86_64 Opteron with 256MB of dedicated RAM.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21782 kenpratt 20 0 444m 212m 272 D 3 83.0 0:04.98 git-pack-object
Well, clearly the server is swapping at this point. 212m resident
for this git-pack-objects process leaves no room available for
anything else. Git is using too much memory for this system.
Have you tried something like this?
[core]
packedGitWindowSize = 16m
packedGitLimit = 64m
[pack]
threads = 1
windowMemory = 64m
deltaCacheSize = 1m
On a 64 bit system packedGitWindowSize and packedGitLimit have very
large thresholds which will cause it to mmap in the entire pack file.
You may need to try even smaller settings than these; 256m physical
memory isn't a lot when dealing with a repository 180m in size.
Especially on a 64 bit system.
--
Shawn.
From: Ken Pratt <hidden> Date: 2016-06-15 22:45:08
Have you tried something like this?
[core]
packedGitWindowSize = 16m
packedGitLimit = 64m
[pack]
threads = 1
windowMemory = 64m
deltaCacheSize = 1m
On a 64 bit system packedGitWindowSize and packedGitLimit have very
large thresholds which will cause it to mmap in the entire pack file.
You may need to try even smaller settings than these; 256m physical
memory isn't a lot when dealing with a repository 180m in size.
Especially on a 64 bit system.
I just went as low as:
[core]
packedGitWindowSize = 1m
packedGitLimit = 4m
[pack]
threads = 1
windowMemory = 4m
deltaCacheSize = 128k
And it didn't make a dent in memory usage. Server is still swapping
within ~10 seconds of starting object compression.
I'm starting to think repacking is just not feasible on a 64-bit
server with 256MB of RAM (which is a very popular configuration in the
VPS market).
Thanks!
Ken
From: Shawn O. Pearce <hidden> Date: 2016-06-15 22:45:08
Ken Pratt [off-list ref] wrote:
I just went as low as:
[core]
packedGitWindowSize = 1m
packedGitLimit = 4m
[pack]
threads = 1
windowMemory = 4m
deltaCacheSize = 128k
And it didn't make a dent in memory usage. Server is still swapping
within ~10 seconds of starting object compression.
I'm starting to think repacking is just not feasible on a 64-bit
server with 256MB of RAM (which is a very popular configuration in the
VPS market).
What is the largest object in that repository? Do you have a
rough guess? You said earlier:
The remote repository is bare, and is 180MB in size (says du), with
1824 objects.
That implies there is at least one really large object in that
repository. The average of 101KB per object is not going to be
a correct figure here as most commits and trees are _very_ tiny.
It must be a large object. Those big objects are going to consume
a lot of memory if they get inflated in memory.
You may very well be right that this particular repository of
yours is simply not packable on a 64 bit system with only 256M.
Packing takes a good chunk of memory as we maintain data about
every single object, plus we need working space to unpack several
objects at once so we can perform diffs to find deltas.
I'm not sure there are any more tunables you can try to tweak to
reduce the memory usage further. The configuration above is pushed
down about as low as it will go. For the most part the code is
pretty good about not exploding memory usage.
You said earlier this was Git 1.5.6.4. I recently fixed a bug in
the code that reads data from packs to prevent it from blowing out
memory usage, but that bug fix was included in 1.5.6.4.
On the up side, packing should only be consuming huge memory like
this when it needs to move loose objects into a pack file. I think
Martin Langhoff suggested packing this on your laptop then using
rsync over SSH to copy the pack file and .idx file to the server, so
the server didn't have to spend time figuring out the deltas itself.
Even though the clone command will fire off git-pack-objects the
pack-objects command will have a lot less work to do if the data
it needs is already stored in existing pack files.
--
Shawn.
On Mon, Aug 11, 2008 at 11:01 AM, Shawn O. Pearce [off-list ref] wrote:
On the up side, packing should only be consuming huge memory like
this when it needs to move loose objects into a pack file. I think
Martin Langhoff suggested packing this on your laptop then using
rsync over SSH to copy the pack file and .idx file to the server, so
the server didn't have to spend time figuring out the deltas itself.
Do you need to also introduce a ".keep" file to get the benefit from
this? I had a repo with some very large objects, and it was killing
my low-memory server *every* time I did "git gc", until I repacked on
another system, created the .keep file, and rsynced it back. Does
that make sense?
Thanks,
Avery
From: Shawn O. Pearce <hidden> Date: 2016-06-15 22:45:08
Avery Pennarun [off-list ref] wrote:
On Mon, Aug 11, 2008 at 11:01 AM, Shawn O. Pearce [off-list ref] wrote:
quoted
On the up side, packing should only be consuming huge memory like
this when it needs to move loose objects into a pack file. I think
Martin Langhoff suggested packing this on your laptop then using
rsync over SSH to copy the pack file and .idx file to the server, so
the server didn't have to spend time figuring out the deltas itself.
Do you need to also introduce a ".keep" file to get the benefit from
this? I had a repo with some very large objects, and it was killing
my low-memory server *every* time I did "git gc", until I repacked on
another system, created the .keep file, and rsynced it back. Does
that make sense?
No, the ".keep" file wouldn't have an impact. Delta reuse (the
feature I was alluding to) works whether or not there is a .keep
file present.
I wonder if your "git gc" was using --aggressive?
--
Shawn.
From: Ken Pratt <hidden> Date: 2016-06-15 22:45:08
What is the largest object in that repository? Do you have a
rough guess? You said earlier:
quoted
The remote repository is bare, and is 180MB in size (says du), with
1824 objects.
That implies there is at least one really large object in that
repository. The average of 101KB per object is not going to be
a correct figure here as most commits and trees are _very_ tiny.
It must be a large object. Those big objects are going to consume
a lot of memory if they get inflated in memory.
Largest object is ~150MB, and there are a couple 5-10MB objects as well.
You said earlier this was Git 1.5.6.4. I recently fixed a bug in
the code that reads data from packs to prevent it from blowing out
memory usage, but that bug fix was included in 1.5.6.4.
I tried upgrading to 1.5.6.5 as well, but that didn't help.
On the up side, packing should only be consuming huge memory like
this when it needs to move loose objects into a pack file. I think
Martin Langhoff suggested packing this on your laptop then using
rsync over SSH to copy the pack file and .idx file to the server, so
the server didn't have to spend time figuring out the deltas itself.
Unfortunately, that will only work as a band-aid solution for my
workflow. I think I'll have to limit the file size in the repository
to something that the server can handle.
From: Jakub Narebski <hidden> Date: 2016-06-15 22:45:09
[...]
If I remember correctly there were on git mailing list some patches by
Dana How which put an upper bound on the size of individual objects
going to pack; objects with size above threshold would be left as
loose object (and shared via network drive).
Unfortunately if I remember correctly they were not accepted in git.
You can try to pack large objects into separate pack, and .keep it,
or try to ressurect the patches from git mailing list archive.
HTH.
--
Jakub Narebski
Poland
ShadeHawk on #git