From: Eric Wong <hidden> Date: 2016-06-15 22:42:29
Jon Smirl [off-list ref] wrote:
I'm going back to cvsimport tomorrow. My svn import that had been
running for five days got killed this morning when the city decided to
move the telephone pole that provides my electricty.
Some oprofile data, this doesn't make a lot of sense to me. Why is it
in libcypto so much?
The sha1 calculation is done in libcrypto, afaik.
Anybody want to see how my latest patches to git-svn (and using SVN perl
libraries) stacks up against the mozilla repo? Speedwise, I don't
expect git-svn to be too different than git-svnimport, but it should use
much less memory (I'll probably port the hacks to git-svnimport, too).
I'll see about freeing up one of my machines to test the mozilla repo.
Unfortunately, all of my hardware is a few years old and not extremely
fast.
--
Eric Wong
From: Jon Smirl <hidden> Date: 2016-06-15 22:42:29
On 6/13/06, Eric Wong [off-list ref] wrote:
Jon Smirl [off-list ref] wrote:
quoted
I'm going back to cvsimport tomorrow. My svn import that had been
running for five days got killed this morning when the city decided to
move the telephone pole that provides my electricty.
Some oprofile data, this doesn't make a lot of sense to me. Why is it
in libcypto so much?
The sha1 calculation is done in libcrypto, afaik.
That make sense, but it's eating up 14% of my CPU in a long sample.
Anybody want to see how my latest patches to git-svn (and using SVN perl
libraries) stacks up against the mozilla repo? Speedwise, I don't
expect git-svn to be too different than git-svnimport, but it should use
much less memory (I'll probably port the hacks to git-svnimport, too).
Can svnimport be rewritten to avoid calling fork? If I am reading the
oprofiles correctly that fork is very expensive especially when the
svnimport task grows to 600MB.
I have an import running but post your code when it is ready and I can
try it on the next run. They always seem to fail so there will
probably be another run.
I'll see about freeing up one of my machines to test the mozilla repo.
Unfortunately, all of my hardware is a few years old and not extremely
fast.
--
Eric Wong
From: Eric Wong <hidden> Date: 2016-06-15 22:42:29
Linus: I hope I'm right on [1] (the stuff about fork).
Jon Smirl [off-list ref] wrote:
On 6/13/06, Eric Wong [off-list ref] wrote:
quoted
Jon Smirl [off-list ref] wrote:
quoted
I'm going back to cvsimport tomorrow. My svn import that had been
running for five days got killed this morning when the city decided to
move the telephone pole that provides my electricty.
Some oprofile data, this doesn't make a lot of sense to me. Why is it
in libcypto so much?
The sha1 calculation is done in libcrypto, afaik.
That make sense, but it's eating up 14% of my CPU in a long sample.
quoted
Anybody want to see how my latest patches to git-svn (and using SVN perl
libraries) stacks up against the mozilla repo? Speedwise, I don't
expect git-svn to be too different than git-svnimport, but it should use
much less memory (I'll probably port the hacks to git-svnimport, too).
Can svnimport be rewritten to avoid calling fork? If I am reading the
oprofiles correctly that fork is very expensive especially when the
svnimport task grows to 600MB.
I think the problem is the process growing to 600MB, and not the fork :)
git-svn avoids process growth pretty well from my tests with the gcc
repo.
See the fetch_lib() function in this patch on how I avoid process
growth by _using_ fork():
Subject: [PATCH 12/13] git-svn: add support for Perl SVN::* libraries
([off-list ref])
Perl processes (at least on my machines (5.8.x, Linux x86) don't like to
release memory back to the OS when they're done using it (although it
can reuse the memory within the process itself). This is why SVN::Pool
isn't very effective in many cases.
fork() will only duplicate memory for the pages that are changed by the
child, not the entire process[1]. So I fork children that run temporarily
to avoid accumulating memory usage inside the process.
This technique should probably be added to git-svnimport as well.
I have an import running but post your code when it is ready and I can
try it on the next run. They always seem to fail so there will
probably be another run.
I've posted a two series of patches the past few days that have yet
to be merged by Junio:
Subject: [PATCH] git-svn: bug fixes (some resends)
[off-list ref]
Subject: [PATCH 0/13] git-svn: better branch support, SVN:: lib usage, feature additions
[off-list ref]
--
Eric Wong
From: Ryan Anderson <hidden> Date: 2016-06-15 22:42:29
On Tue, Jun 13, 2006 at 07:01:08PM -0700, Eric Wong wrote:
Anybody want to see how my latest patches to git-svn (and using SVN perl
libraries) stacks up against the mozilla repo? Speedwise, I don't
expect git-svn to be too different than git-svnimport, but it should use
much less memory (I'll probably port the hacks to git-svnimport, too).
I've got access to a pretty good machine to run this on - where can I
grab the svn repo from?
(I can just grab the CVS one and convert it, first, as well, just point
me at that, if that's got more bandwidth.)
--
Ryan Anderson
sometimes Pug Majere
From: Jon Smirl <hidden> Date: 2016-06-15 22:42:29
On 6/14/06, Ryan Anderson [off-list ref] wrote:
On Tue, Jun 13, 2006 at 07:01:08PM -0700, Eric Wong wrote:
quoted
Anybody want to see how my latest patches to git-svn (and using SVN perl
libraries) stacks up against the mozilla repo? Speedwise, I don't
expect git-svn to be too different than git-svnimport, but it should use
much less memory (I'll probably port the hacks to git-svnimport, too).
I've got access to a pretty good machine to run this on - where can I
grab the svn repo from?
(I can just grab the CVS one and convert it, first, as well, just point
me at that, if that's got more bandwidth.)
rsync -az cvs-mirror.mozilla.org::mozilla ~/mozilla/cvs-mirror
It took about three days for my machine to convert that cvs to svn.
I have the converted repo local but it is 8.2GB and I have 256kb up.
There is no real purpose in converting mozilla cvs to svn to git other
than to test the tools. My last attempt at svn to git ran five days
before I lost power. Towards the end it was getting significantly slow
implying some kind of n squared problem in the import process. The
idea was to see if cvsimport and svnimport both end up with the same
output.
I am going to use git-cvsimport on the mozilla repo but that tool
needs to 2GB+ physical RAM to run. I ordered 2GB more and it will be
here tomorrow. I have just been playing with the svn conversion while
I wait five days for my 2nd day air package to show up.
From: Jon Smirl <hidden> Date: 2016-06-15 22:42:29
Stats after 18 hours into git-svnimport. Process is now stuck in the
kernel 64% of the time. All of the kernel time is in page management.
Perl svnimport process is 290MB now.
My top candidates for causing the problem are the fork in the perl
code or the execing of a million tiny git processes.
The key low level git functions could be made into a library to avoid
the need to exec them continuously. The svn functions are libraries
and they hardly show up.
606218 2.4143 /usr/local/bin/git-update-index
127170 0.5065 /usr/local/bin/git-write-tree
81153 0.3232 /usr/local/bin/git-read-tree
13065 0.0520 /usr/local/bin/git-ls-files
2624 0.0105 /usr/local/bin/git-hash-object
754 0.0030 /usr/local/bin/git-commit-tree
462 0.0018 /usr/local/bin/git-ls-tree
398 0.0016 /usr/local/bin/git-rev-parse
versus
102784 0.3641 /usr/lib/libsvn_subr-1.so.0.0.0
70235 0.2488 /usr/lib/libsvn_fs_fs-1.so.0.0.0
67081 0.2376 /usr/lib/libsvn_delta-1.so.0.0.0
848 0.0030 /usr/lib/libsvn_swig_perl-1.so.0.0.0
512 0.0018 /usr/lib/libsvn_ra_local-1.so.0.0.0
350 0.0012 /usr/lib/libsvn_fs-1.so.0.0.0
222 7.9e-04 /usr/lib/libsvn_repos-1.so.0.0.0
124 4.4e-04 /usr/lib/libsvn_ra-1.so.0.0.0
------------------------------------------------------------------------------------------------------------
4093890 64.3711 /home/good/vmlinux
906014 14.2459 /lib/libcrypto.so.0.9.8a
435744 6.8515 /lib/libc-2.4.so
158325 2.4895 /usr/lib/libz.so.1.2.3
139995 2.2012 /usr/local/bin/git-update-index
75322 1.1843 /nvidia
64349 1.0118 /usr/bin/oprofiled
52825 0.8306 /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so
51930 0.8165 /usr/lib/libapr-1.so.0.2.2
42771 0.6725 /usr/local/bin/git-read-tree
37774 0.5939 /lib/ld-2.4.so
34761 0.5466 /usr/local/bin/git-write-tree
29560 0.4648 /usr/lib/libsvn_subr-1.so.0.0.0
28210 0.4436 /usr/lib/libaprutil-1.so.0.2.2
-----------------------------------------------------------------------------------------------------------------
2471826 32.8741 copy_page_range
375260 18.2903 unmap_vmas
574208 7.6367 release_pages
572189 7.6098 page_remove_rmap
233367 3.1037 free_pages_and_swap_cache
191051 2.5409 get_page_from_freelist
169058 2.2484 unlock_page
162027 2.1549 vm_normal_page
155691 2.0706 swap_info_get
136324 1.8130 swap_duplicate
119227 1.5857 page_fault
99729 1.3263 page_waitqueue
49288 0.6555 remove_exclusive_swap_page
39611 0.5268 do_wp_page
39142 0.5206 __wake_up_bit
34384 0.4573 __copy_from_user_ll
31111 0.4138 __handle_mm_fault
29990 0.3989 find_get_page
29682 0.3948 do_page_fault
--
Jon Smirl
jonsmirl@gmail.com
From: Jakub Narebski <hidden> Date: 2016-06-15 22:42:29
Jon Smirl wrote:
Stats after 18 hours into git-svnimport. Process is now stuck in the
kernel 64% of the time. All of the kernel time is in page management.
Perl svnimport process is 290MB now.
My top candidates for causing the problem are the fork in the perl
code or the execing of a million tiny git processes.
The key low level git functions could be made into a library to avoid
the need to exec them continuously. The svn functions are libraries
and they hardly show up.
There is ongoing effort to translate git functions into builtins.
Still you would need to translate git-svnimport Perl code into C,
or somehow access git library from Perl.
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git