Hi folks,
when adding files which are larger than available physical memory,
git performs very slow. Perhaps it has to do with git's mmap()ing
the whole file. Is there any way to do it w/o mmap (hoping that
might perform a bit better) ?
cu
--
----------------------------------------------------------------------
Enrico Weigelt, metux IT service -- http://www.metux.de/
phone: +49 36207 519931 email: weigelt@metux.de
mobile: +49 151 27565287 icq: 210169427 skype: nekrad666
----------------------------------------------------------------------
Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------
On Mon, Oct 4, 2010 at 2:20 AM, Enrico Weigelt [off-list ref] wrote:
when adding files which are larger than available physical memory,
git performs very slow. Perhaps it has to do with git's mmap()ing
the whole file. Is there any way to do it w/o mmap (hoping that
might perform a bit better) ?
The mmap() isn't the problem. Its the allocation of a buffer that is
larger than the file in order to hold the result of deflating the file
before it gets written to disk. When the file is bigger than physical
memory, the kernel has to page in parts of the file as well as swap in
and out parts of that allocated buffer to hold the deflated file.
This is a known area in Git where big files aren't handled well.
--
Shawn.
On Mon, Oct 4, 2010 at 2:20 AM, Enrico Weigelt[off-list ref] wrote:
quoted
when adding files which are larger than available physical memory,
git performs very slow. Perhaps it has to do with git's mmap()ing
the whole file. Is there any way to do it w/o mmap (hoping that
might perform a bit better) ?
The mmap() isn't the problem. Its the allocation of a buffer that is
larger than the file in order to hold the result of deflating the file
before it gets written to disk. When the file is bigger than physical
memory, the kernel has to page in parts of the file as well as swap in
and out parts of that allocated buffer to hold the deflated file.
This is a known area in Git where big files aren't handled well.
As a curiosity, I've always done streaming decompression with zlib using
minimal buffer sizes (64k, perhaps). I'm sure there is good reason why
Git doesn't do this (delta application?). Do you know what it is?
Josh
On Mon, Oct 4, 2010 at 11:24 AM, Joshua Jensen
[off-list ref] wrote:
quoted
On Mon, Oct 4, 2010 at 2:20 AM, Enrico Weigelt[off-list ref] wrote:
quoted
when adding files which are larger than available physical memory,
git performs very slow.
The mmap() isn't the problem. Its the allocation of a buffer that is
larger than the file in order to hold the result of deflating the file
before it gets written to disk.
...
quoted
This is a known area in Git where big files aren't handled well.
As a curiosity, I've always done streaming decompression with zlib using
minimal buffer sizes (64k, perhaps). I'm sure there is good reason why Git
doesn't do this (delta application?). Do you know what it is?
Laziness. Git originally assumed it would only be used for smaller
source files written by humans. Its easier to write the code as a
single malloc'd buffer than to stream it. We'd like to fix it, but
its harder than it sounds. Today we copy the file into a buffer
before we deflate and compute the SHA-1 as this prevents us from
getting into a consistency error when the file is modified between
these two stages.
--
Shawn.
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:42
Shawn Pearce wrote:
The mmap() isn't the problem. Its the allocation of a buffer that is
larger than the file in order to hold the result of deflating the file
before it gets written to disk.
Wasn't this already fixed, at least in some cases?
commit 9892bebafe0865d8f4f3f18d60a1cfa2d1447cd7 (tags/v1.7.0.2~11^2~1)
Author: Nicolas Pitre [off-list ref]
Date: Sat Feb 20 23:27:31 2010 -0500
sha1_file: don't malloc the whole compressed result when writing out objects
There is no real advantage to malloc the whole output buffer and
deflate the data in a single pass when writing loose objects. That is
like only 1% faster while using more memory, especially with large
files where memory usage is far more. It is best to deflate and write
the data out in small chunks reusing the same memory instead.
For example, using 'git add' on a few large files averaging 40 MB ...
Before:
21.45user 1.10system 0:22.57elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+828040outputs (0major+142640minor)pagefaults 0swaps
After:
21.50user 1.25system 0:22.76elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+828040outputs (0major+104408minor)pagefaults 0swaps
While the runtime stayed relatively the same, the number of minor page
faults went down significantly.
Signed-off-by: Nicolas Pitre [off-list ref]
Signed-off-by: Junio C Hamano [off-list ref]
On Mon, Oct 4, 2010 at 11:58 AM, Jonathan Nieder [off-list ref] wrote:
Shawn Pearce wrote:
quoted
The mmap() isn't the problem. Its the allocation of a buffer that is
larger than the file in order to hold the result of deflating the file
before it gets written to disk.
Wasn't this already fixed, at least in some cases?
commit 9892bebafe0865d8f4f3f18d60a1cfa2d1447cd7 (tags/v1.7.0.2~11^2~1)
Author: Nicolas Pitre [off-list ref]
Date: Sat Feb 20 23:27:31 2010 -0500
sha1_file: don't malloc the whole compressed result when writing out objects
This change only removes the deflate copy. But due to the SHA-1
consistency issue I alluded to earlier, I think we're still making a
full copy of the file in memory before we SHA-1 it or deflate it. So
Nico halved the memory usage, but we're still using 1x the size of the
file rather than ~2x.
--
Shawn.
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:42
Shawn Pearce wrote:
This change only removes the deflate copy. But due to the SHA-1
consistency issue I alluded to earlier, I think we're still making a
full copy of the file in memory before we SHA-1 it or deflate it.
Hmm, I _think_ we still use mmap for that (which is why 748af44c needs
to compare the sha1 before and after).
But
1) a one-pass calculation would presumably be a little (5%?) faster
2) if there are smudge/clean filters or autocrlf involved, the
cleaned-up file is backed by swap and this all becomes moot.
The mmap() isn't the problem. Its the allocation of a buffer that is
larger than the file in order to hold the result of deflating the file
before it gets written to disk.
When the file is bigger than physical memory, the kernel has to
page in parts of the file as well as swap in and out parts of
that allocated buffer to hold the deflated file.
What are the access pattern of these memory areas ?
Perhaps madvise() could help ?
cu
--
----------------------------------------------------------------------
Enrico Weigelt, metux IT service -- http://www.metux.de/
phone: +49 36207 519931 email: weigelt@metux.de
mobile: +49 151 27565287 icq: 210169427 skype: nekrad666
----------------------------------------------------------------------
Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------
The mmap() isn't the problem. Its the allocation of a buffer that is
larger than the file in order to hold the result of deflating the file
before it gets written to disk.
Wasn't this already fixed, at least in some cases?
commit 9892bebafe0865d8f4f3f18d60a1cfa2d1447cd7 (tags/v1.7.0.2~11^2~1)
Author: Nicolas Pitre [off-list ref]
Date: Sat Feb 20 23:27:31 2010 -0500
I guess I'll have to do a update.
But: latest tag (1.7.3.1) doesnt build:
CC read-cache.o
read-cache.c: In function `fill_stat_cache_info':
read-cache.c:73: structure has no member named `st_ctim'
read-cache.c:74: structure has no member named `st_mtim'
read-cache.c: In function `read_index_from':
read-cache.c:1334: structure has no member named `st_mtim'
read-cache.c: In function `write_index':
read-cache.c:1614: structure has no member named `st_mtim'
make: *** [read-cache.o] Fehler 1
Is my libc too old ?
cu
--
----------------------------------------------------------------------
Enrico Weigelt, metux IT service -- http://www.metux.de/
phone: +49 36207 519931 email: weigelt@metux.de
mobile: +49 151 27565287 icq: 210169427 skype: nekrad666
----------------------------------------------------------------------
Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------
Laziness. Git originally assumed it would only be used for smaller
source files written by humans. Its easier to write the code as a
single malloc'd buffer than to stream it. We'd like to fix it, but
its harder than it sounds. Today we copy the file into a buffer
before we deflate and compute the SHA-1 as this prevents us from
getting into a consistency error when the file is modified between
these two stages.
hmm, perhaps copy it to a temporary file, if it's too large ?
cu
--
----------------------------------------------------------------------
Enrico Weigelt, metux IT service -- http://www.metux.de/
phone: +49 36207 519931 email: weigelt@metux.de
mobile: +49 151 27565287 icq: 210169427 skype: nekrad666
----------------------------------------------------------------------
Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------
On Tue, Oct 5, 2010 at 00:57, Enrico Weigelt [off-list ref] wrote:
* Jonathan Nieder [off-list ref] wrote:
quoted
Shawn Pearce wrote:
quoted
The mmap() isn't the problem. Its the allocation of a buffer that is
larger than the file in order to hold the result of deflating the file
before it gets written to disk.
Wasn't this already fixed, at least in some cases?
commit 9892bebafe0865d8f4f3f18d60a1cfa2d1447cd7 (tags/v1.7.0.2~11^2~1)
Author: Nicolas Pitre [off-list ref]
Date: Sat Feb 20 23:27:31 2010 -0500
I guess I'll have to do a update.
But: latest tag (1.7.3.1) doesnt build:
CC read-cache.o
read-cache.c: In function `fill_stat_cache_info':
read-cache.c:73: structure has no member named `st_ctim'
read-cache.c:74: structure has no member named `st_mtim'
read-cache.c: In function `read_index_from':
read-cache.c:1334: structure has no member named `st_mtim'
read-cache.c: In function `write_index':
read-cache.c:1614: structure has no member named `st_mtim'
make: *** [read-cache.o] Fehler 1
Is my libc too old ?
Those lines are accessing members called st_ctime, i.e. with an "e" at
the end, but your errors just report "st_ctim". What gives?
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:42
Enrico Weigelt wrote:
CC read-cache.o
read-cache.c: In function `fill_stat_cache_info':
read-cache.c:73: structure has no member named `st_ctim'
read-cache.c:74: structure has no member named `st_mtim'
read-cache.c: In function `read_index_from':
read-cache.c:1334: structure has no member named `st_mtim'
read-cache.c: In function `write_index':
read-cache.c:1614: structure has no member named `st_mtim'
make: *** [read-cache.o] Fehler 1
Is my libc too old ?
What platform are you on? You probably need USE_ST_TIMESPEC;
if so, please send a makefile patch so the next person trying
it doesn't need to worry about it.
Also, please don't destroy the cc lists; it makes it hard for
people with some mail setups (e.g., mine) to notice when you've
replied to them.
* Enrico Weigelt [off-list ref] wrote:
<snip>
Found another possible bottleneck: git-commit seems to scan through
a lot of files. Shouldnt it just create a commit object from the
current index and update the head ?
cu
--
----------------------------------------------------------------------
Enrico Weigelt, metux IT service -- http://www.metux.de/
phone: +49 36207 519931 email: weigelt@metux.de
mobile: +49 151 27565287 icq: 210169427 skype: nekrad666
----------------------------------------------------------------------
Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------
On Tue, Oct 5, 2010 at 2:41 PM, Enrico Weigelt [off-list ref] wrote:
* Enrico Weigelt [off-list ref] wrote:
<snip>
Found another possible bottleneck: git-commit seems to scan through
a lot of files. Shouldnt it just create a commit object from the
current index and update the head ?
You mean a lot of stat()? There is no way to avoid that unless you set
assume-unchanged bits. Or you could use
write-tree/commit-tree/update-ref directly.
--
Duy
On Tue, Oct 5, 2010 at 2:16 AM, Jonathan Nieder [off-list ref] wrote:
Shawn Pearce wrote:
quoted
This change only removes the deflate copy. But due to the SHA-1
consistency issue I alluded to earlier, I think we're still making a
full copy of the file in memory before we SHA-1 it or deflate it.
Hmm, I _think_ we still use mmap for that (which is why 748af44c needs
to compare the sha1 before and after).
Just tried valgrind massif on a 200MB file with master. It used ~270kb
heap. I haven't tested but I believe git-checkout will keep the whole
inflated copy in memory. So git-add alone does not help much.
--
Duy
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:42
Enrico Weigelt wrote:
* Jonathan Nieder [off-list ref] wrote:
quoted
What platform are you on?
GNU/Linux. glibc-2.25
Hmm, I've heard of glib 2.25 but never glibc 2.25. :)
$ /lib/libc.so.6 | head -1
GNU C Library (Debian EGLIBC 2.11.2-6) stable release version 2.11.2, by Roland McGrath et al.
From: Nicolas Pitre <nico@fluxnic.net> Date: 2016-06-15 22:49:42
On Tue, 5 Oct 2010, Enrico Weigelt wrote:
* Shawn Pearce [off-list ref] wrote:
quoted
The mmap() isn't the problem. Its the allocation of a buffer that is
larger than the file in order to hold the result of deflating the file
before it gets written to disk.
When the file is bigger than physical memory, the kernel has to
page in parts of the file as well as swap in and out parts of
that allocated buffer to hold the deflated file.
What are the access pattern of these memory areas ?
From: Nicolas Pitre <nico@fluxnic.net> Date: 2016-06-15 22:49:42
On Tue, 5 Oct 2010, Nguyen Thai Ngoc Duy wrote:
On Tue, Oct 5, 2010 at 2:41 PM, Enrico Weigelt [off-list ref] wrote:
quoted
* Enrico Weigelt [off-list ref] wrote:
<snip>
Found another possible bottleneck: git-commit seems to scan through
a lot of files. Shouldnt it just create a commit object from the
current index and update the head ?
You mean a lot of stat()? There is no way to avoid that unless you set
assume-unchanged bits. Or you could use
write-tree/commit-tree/update-ref directly.
Avoiding memory exhaustion is also going to help a lot as the stat()
information will remain cached instead of requiring disk access. Just a
guess given $subject.
Nicolas
From: Nicolas Pitre <nico@fluxnic.net> Date: 2016-06-15 22:49:42
On Mon, 4 Oct 2010, Jonathan Nieder wrote:
Shawn Pearce wrote:
quoted
This change only removes the deflate copy. But due to the SHA-1
consistency issue I alluded to earlier, I think we're still making a
full copy of the file in memory before we SHA-1 it or deflate it.
Hmm, I _think_ we still use mmap for that (which is why 748af44c needs
to compare the sha1 before and after).
But
1) a one-pass calculation would presumably be a little (5%?) faster
You can't do a one-pass calculation. The first one is required to
compute the SHA1 of the file being added, and if that corresponds to an
object that we already have then the operation stops right there as
there is actually nothing to do. The second pass is to deflate the
data, and recompute the SHA1 to make sure what we deflated and written
out is still the same data.
In the case of big files, what we need to do is to stream the file data
in, compute the SHA1 and deflate it, in order to stream it out into a
temporary file, then rename it according to the final SHA1. This would
allow Git to work with big files, but of course it won't be possible to
know if the object corresponding to the file is already known until all
the work has been done, possibly just to throw it away. But normally
big files are the minority.
Nicolas
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:49:42
Nicolas Pitre wrote:
You can't do a one-pass calculation. The first one is required to
compute the SHA1 of the file being added, and if that corresponds to an
object that we already have then the operation stops right there as
there is actually nothing to do.
Ah. Thanks for a reminder.
In the case of big files, what we need to do is to stream the file data
in, compute the SHA1 and deflate it, in order to stream it out into a
temporary file, then rename it according to the final SHA1. This would
allow Git to work with big files, but of course it won't be possible to
know if the object corresponding to the file is already known until all
the work has been done, possibly just to throw it away.
To make sure I understand correctly: are you suggesting that for big
files we should skip the first pass?
I suppose that makes sense: for small files, using a patch application
tool to reach a postimage that matches an existing object is something
git historically needed to expect, but for typical big files:
- once you've computed the SHA1, you've already invested a noticeable
amount of time.
- emailing patches around is difficult, making "git am" etc less important
- hopefully git or zlib can notice when files are uncompressible,
making the deflate not cost so much in that case.
From: Nicolas Pitre <nico@fluxnic.net> Date: 2016-06-15 22:49:42
On Tue, 5 Oct 2010, Jonathan Nieder wrote:
Nicolas Pitre wrote:
quoted
You can't do a one-pass calculation. The first one is required to
compute the SHA1 of the file being added, and if that corresponds to an
object that we already have then the operation stops right there as
there is actually nothing to do.
Ah. Thanks for a reminder.
quoted
In the case of big files, what we need to do is to stream the file data
in, compute the SHA1 and deflate it, in order to stream it out into a
temporary file, then rename it according to the final SHA1. This would
allow Git to work with big files, but of course it won't be possible to
know if the object corresponding to the file is already known until all
the work has been done, possibly just to throw it away.
To make sure I understand correctly: are you suggesting that for big
files we should skip the first pass?
For big files we need a totally separate code path to process the file
data in small chunks at 'git add' time, using a loop containing
read()+SHA1sum()+deflate()+write(). Then, if the SHA1 matches an
existing object we delete the temporary output file, otherwise we rename
it as a valid object. No CRLF, no smudge filters, no diff, no deltas,
just plain storage of huge objects, based on the value of
core.bigFileThreshold config option.
Same thing on the checkout path: a simple loop to
read()+inflate()+write() in small chunks.
That's the only sane way to kinda support big files with Git.
I suppose that makes sense: for small files, using a patch application
tool to reach a postimage that matches an existing object is something
git historically needed to expect, but for typical big files:
- once you've computed the SHA1, you've already invested a noticeable
amount of time.
- emailing patches around is difficult, making "git am" etc less important
- hopefully git or zlib can notice when files are uncompressible,
making the deflate not cost so much in that case.
Emailing is out of the question. We're talking file sizes in the
hundreds of megabytes and above here. So yes, simply computing the SHA1
is a significant cost, given that you are going to trash your page cache
in the process already, so better pay the price of deflating it at the
same time even if it turns out to be unnecessary.
Nicolas
The mmap() isn't the problem. Its the allocation of a buffer that is
larger than the file in order to hold the result of deflating the file
before it gets written to disk.
When the file is bigger than physical memory, the kernel has to
page in parts of the file as well as swap in and out parts of
that allocated buffer to hold the deflated file.
What are the access pattern of these memory areas ?
Perfectly linear.
In this case, I wonder why my machine goes into thrashing so easily
(P3 w/ 256MB ram). Seems the mmu/paging code doesnt recognize that
the previously-used pages can be kicked-off quickly ;-o
Perhaps I should talk to the kernel folks.
quoted
Perhaps madvise() could help ?
Perhaps.
hmm, so we should try it ;-p
where'd be the right place to add it ?
cu
--
----------------------------------------------------------------------
Enrico Weigelt, metux IT service -- http://www.metux.de/
phone: +49 36207 519931 email: weigelt@metux.de
mobile: +49 151 27565287 icq: 210169427 skype: nekrad666
----------------------------------------------------------------------
Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------