Junio C Hamano [off-list ref] writes:
Nicolas Pitre [off-list ref] writes:
quoted
On Fri, 7 Dec 2007, Jon Smirl wrote:
quoted
Starting with a 2GB pack of the same data my process size only grew to
3GB with 2GB of mmaps.
Which is quite reasonable, even if the same issue might still be there.
So the problem seems to be related to the pack access code and not the
repack code. And it must have something to do with the number of deltas
being replayed. And because the repack is attempting delta compression
roughly from newest to oldest, and because old objects are typically in
a deeper delta chain, then this might explain the logarithmic slowdown.
So something must be wrong with the delta cache in sha1_file.c somehow.
I was reaching the same conclusion but haven't managed to spot anything
blatantly wrong in that area. Will need to dig more.
Does this problem have correlation with the use of threads? Do you see
the same bloat with or without THREADED_DELTA_SEARCH defined?
On 12/9/07, Junio C Hamano [off-list ref] wrote:
Junio C Hamano [off-list ref] writes:
quoted
Nicolas Pitre [off-list ref] writes:
quoted
On Fri, 7 Dec 2007, Jon Smirl wrote:
quoted
Starting with a 2GB pack of the same data my process size only grew to
3GB with 2GB of mmaps.
Which is quite reasonable, even if the same issue might still be there.
So the problem seems to be related to the pack access code and not the
repack code. And it must have something to do with the number of deltas
being replayed. And because the repack is attempting delta compression
roughly from newest to oldest, and because old objects are typically in
a deeper delta chain, then this might explain the logarithmic slowdown.
So something must be wrong with the delta cache in sha1_file.c somehow.
I was reaching the same conclusion but haven't managed to spot anything
blatantly wrong in that area. Will need to dig more.
Does this problem have correlation with the use of threads? Do you see
the same bloat with or without THREADED_DELTA_SEARCH defined?
I just started a non-threaded one. It will be four or five hours
before it finishes.
--
Jon Smirl
jonsmirl@gmail.com
On 12/9/07, Junio C Hamano [off-list ref] wrote:
Junio C Hamano [off-list ref] writes:
quoted
Nicolas Pitre [off-list ref] writes:
quoted
On Fri, 7 Dec 2007, Jon Smirl wrote:
quoted
Starting with a 2GB pack of the same data my process size only grew to
3GB with 2GB of mmaps.
Which is quite reasonable, even if the same issue might still be there.
So the problem seems to be related to the pack access code and not the
repack code. And it must have something to do with the number of deltas
being replayed. And because the repack is attempting delta compression
roughly from newest to oldest, and because old objects are typically in
a deeper delta chain, then this might explain the logarithmic slowdown.
So something must be wrong with the delta cache in sha1_file.c somehow.
I was reaching the same conclusion but haven't managed to spot anything
blatantly wrong in that area. Will need to dig more.
Does this problem have correlation with the use of threads? Do you see
the same bloat with or without THREADED_DELTA_SEARCH defined?
Something else seems to be wrong.
With threading turned off, 5000 CPU seconds and 13% done.
With threading turned on, threads = 1, 5000 CPU seconds, 13%
With threading turned on, threads = 2, 180 CPU seconds, 13%
With threading turned on, threads = 4, 150 CPU seconds, 13%
This can't be right, four cores are not 40x one core. So maybe the
observed logarithmic slow down is because the percent complete is
being reported wrong in the threaded case. If that's the case we may
be looking in the wrong place for problems.
The times are only approximate, I'm using the CPU for other things.
--
Jon Smirl
jonsmirl@gmail.com
On Sun, 9 Dec 2007, Jon Smirl wrote:
On 12/9/07, Junio C Hamano [off-list ref] wrote:
quoted
Junio C Hamano [off-list ref] writes:
quoted
Nicolas Pitre [off-list ref] writes:
quoted
On Fri, 7 Dec 2007, Jon Smirl wrote:
quoted
Starting with a 2GB pack of the same data my process size only grew to
3GB with 2GB of mmaps.
Which is quite reasonable, even if the same issue might still be there.
So the problem seems to be related to the pack access code and not the
repack code. And it must have something to do with the number of deltas
being replayed. And because the repack is attempting delta compression
roughly from newest to oldest, and because old objects are typically in
a deeper delta chain, then this might explain the logarithmic slowdown.
So something must be wrong with the delta cache in sha1_file.c somehow.
I was reaching the same conclusion but haven't managed to spot anything
blatantly wrong in that area. Will need to dig more.
Does this problem have correlation with the use of threads? Do you see
the same bloat with or without THREADED_DELTA_SEARCH defined?
Something else seems to be wrong.
With threading turned off, 5000 CPU seconds and 13% done.
With threading turned on, threads = 1, 5000 CPU seconds, 13%
With threading turned on, threads = 2, 180 CPU seconds, 13%
With threading turned on, threads = 4, 150 CPU seconds, 13%
This can't be right, four cores are not 40x one core.
It may be right. The object list to apply delta compression on doesn't
necessarily require a uniform amount of cycles throughout. When using
multiple threads, the list is broken in parts for each thread, and later
parts might end up being simply much easier to process, therefore
changing the percentage figure.
So maybe the observed logarithmic slow down is because the percent
complete is being reported wrong in the threaded case. If that's the
case we may be looking in the wrong place for problems.
I really doubt it.
Nicolas