From: Jon Smirl <hidden> Date: 2016-06-15 22:42:29
I'm still working on importing Mozilla CVS. I'm at the phase now where
all of the changeset have been identified. The scripts are pulling the
changesets one at a time out of CVS and putting them into git. I've
been running this phase for 2 days now on a 3GB machine and it still
isn't finished.
I am spending over 40% of the time in the kernel. This looks to be
caused from forks and starting small tasks, is that the correct
interpretation? Is the number of process that have been run recorded
any where? 1.4% of the time is spend in the dynamic linker.
Checking with oprofile I see this:
18262372 41.0441 /home/good/vmlinux
5465741 12.2841 /usr/bin/cvs
4374336 9.8312 /lib/libc-2.4.so
3627709 8.1532 /lib/libcrypto.so.0.9.8a
2494610 5.6066 /usr/bin/oprofiled
2471238 5.5540 /usr/lib/libz.so.1.2.3
945349 2.1246 /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so
933646 2.0983 /usr/local/bin/git-read-tree
758776 1.7053 /usr/local/bin/git-write-tree
642502 1.4440 /lib/ld-2.4.so
472903 1.0628 /nvidia
379254 0.8524 /usr/local/bin/git-pack-objects
and breaking down the kernel number:
3467889 18.9893 copy_page_range
2190416 11.9941 unmap_vmas
1156011 6.3300 page_fault
887794 4.8613 release_pages
860853 4.7138 page_remove_rmap
633243 3.4675 get_page_from_freelist
398773 2.1836 do_wp_page
344422 1.8860 __mutex_lock_slowpath
280070 1.5336 __handle_mm_fault
241713 1.3236 do_page_fault
238398 1.3054 __d_lookup
236654 1.2959 vm_normal_page
--
Jon Smirl
jonsmirl@gmail.com
I am spending over 40% of the time in the kernel. This looks to be
caused from forks and starting small tasks, is that the correct
interpretation?
Yes. Your kernel profile is all for stuff related to setting up and
tearing down process space (well, __mutex_lock_slowpath at 1.88% and
__d_lookup at 1.3% is not, but every single one before that does seem to
be about fork/exec/exit).
I think it's both the CVS server that continually forks/exits (it doesn't
actually do a exec at all - it seem sto be using fork/exit as a way to
control its memory usage - knowing that the OS will free all the temporary
memory on exit - I think the newer CVS development trees don't do this,
but that also seems to be why they leak memory like mad and eventually run
out ;).
AND it's git-cvsimport forking and exec'ing git helper processes.
So that process overhead is expected.
What I would _not_ have expected is:
933646 2.0983 /usr/local/bin/git-read-tree
I don't see why git-read-tree is so hot for you. We should never need to
read a tree when we're importing something, unless there are tons of
branches and we switch back and forth between them.
I guess mozilla really does use a fair number of branches?
Martin sent out a patch (that I don't think has been merged yet) to avoid
the git-read-tree overhead when switching branches. Look for an email with
a subject like "cvsimport: keep one index per branch during import", I
suspect that would speed up the git part a lot.
(It will also avoid a few fork/exec's, but you'll still have most of them,
so I don't think you'll see any really _fundamental_ changes to this, but
the git-read-tree overhead should be basically gone, and some of the
libz.so pressure would also be gone with it. It should also avoid
rewriting the index file, so you'd get lower disk pressure, but it looks
like none of your problems are really due to IO, so again, that probably
won't make much of a difference for you).
Linus
From: Jon Smirl <hidden> Date: 2016-06-15 22:42:29
On 6/16/06, Linus Torvalds [off-list ref] wrote:
On Fri, 16 Jun 2006, Jon Smirl wrote:
quoted
I am spending over 40% of the time in the kernel. This looks to be
caused from forks and starting small tasks, is that the correct
interpretation?
Yes. Your kernel profile is all for stuff related to setting up and
tearing down process space (well, __mutex_lock_slowpath at 1.88% and
__d_lookup at 1.3% is not, but every single one before that does seem to
be about fork/exec/exit).
I think it's both the CVS server that continually forks/exits (it doesn't
actually do a exec at all - it seem sto be using fork/exit as a way to
control its memory usage - knowing that the OS will free all the temporary
memory on exit - I think the newer CVS development trees don't do this,
but that also seems to be why they leak memory like mad and eventually run
out ;).
I am using cvs-1.11.21-3.2
I can try running their development tree.
AND it's git-cvsimport forking and exec'ing git helper processes.
Is it worthwhile to make a library version of these? Svn has lib
versions and they barely show up in oprofile. cvsimport is only using
4-5 low level git funtions.
So that process overhead is expected.
What I would _not_ have expected is:
quoted
933646 2.0983 /usr/local/bin/git-read-tree
I don't see why git-read-tree is so hot for you. We should never need to
read a tree when we're importing something, unless there are tons of
branches and we switch back and forth between them.
I guess mozilla really does use a fair number of branches?
Is 1,800 a lot?
Martin sent out a patch (that I don't think has been merged yet) to avoid
the git-read-tree overhead when switching branches. Look for an email with
a subject like "cvsimport: keep one index per branch during import", I
suspect that would speed up the git part a lot.
I'll check this out
(It will also avoid a few fork/exec's, but you'll still have most of them,
so I don't think you'll see any really _fundamental_ changes to this, but
the git-read-tree overhead should be basically gone, and some of the
libz.so pressure would also be gone with it. It should also avoid
rewriting the index file, so you'd get lower disk pressure, but it looks
like none of your problems are really due to IO, so again, that probably
won't make much of a difference for you).
I have been CPU bound for two days, disk activity is minor.
git-cvsimport is 250MB and I have 2GB of disk cache.
After looking at this process for about a week it doesn't look like
processing chronologically is the best strategy. cvsps can quickly
work out the changesets, 15 minutes. Then it might be better to walk
the CVS files one at a time generating git IDs for each revision. Next
use the IDs and changeset info to build the git trees. Finally pack
everything. This strategy would minimize the work load on the CVS
files (adding all those delta to get random revs).
Can git build a repository in this manner? If this is feasible it may
be possible to do all of this in a single pass over the CVS tree by
modifying cvsps.
--
Jon Smirl
jonsmirl@gmail.com
I am using cvs-1.11.21-3.2
I can try running their development tree.
No, don't. We already know that 1.12 leaks memory and makes the cvsimport
not work at all.
quoted
AND it's git-cvsimport forking and exec'ing git helper processes.
Is it worthwhile to make a library version of these? Svn has lib
versions and they barely show up in oprofile. cvsimport is only using
4-5 low level git funtions.
Eventually, I think that's where we'll get. We're already at the stage
where most of the core could just be written as a library.
quoted
I guess mozilla really does use a fair number of branches?
Is 1,800 a lot?
Yeah. Although even just two is enough, if you just alternate committing
on them ;)
So it's actually not number of branches, it's more about frequency of
the branch changing in the cvsps output. And yes, you could probably
improve performance by sorting the changesets differently, but Martin's
change to use separate index files should make it all pretty moot.
Linus
From: Jon Smirl <hidden> Date: 2016-06-15 22:42:29
Is it a crazy idea to read the cvs files, compute an sha1 on each
expanded delta and then write the delta straight into a pack file? Are
the cvs and git delta formats the same? What about CVS's forward and
reverse delta use? While this is going on, track the
branches/changsets in memory and then finish up by writing these trees
into the pack file too. This should take no more ram than cvsps needs
currently.
This leaves the packfile is a non-optimal format but a repack should
fix that, right?
--
Jon Smirl
jonsmirl@gmail.com
From: Keith Packard <keithp@keithp.com> Date: 2016-06-15 22:42:29
On Fri, 2006-06-16 at 13:00 -0400, Jon Smirl wrote:
Is it a crazy idea to read the cvs files, compute an sha1 on each
expanded delta and then write the delta straight into a pack file? Are
the cvs and git delta formats the same? What about CVS's forward and
reverse delta use?
At this point, merging blobs into packs isn't a significant part of the
computational cost. parsecvs is spending all of its time in the
quadratic traversal of the diff chains; fixing that to emit all of the
versions in a single pass should speed up that part of the conversion
process dramatically.
While this is going on, track the
branches/changsets in memory and then finish up by writing these trees
into the pack file too. This should take no more ram than cvsps needs
currently.
cvsps drops too much state on the floor making branch point and branch
contents inaccurate. What I'm hoping is that I can figure out a way to
discard most of the per-version information by computing tree objects in
reverse order, saving only the tree sha1 and other per-commit info, then
stitch the commits together using that, without needing the full
per-file data.
--
keith.packard@intel.com
From: Jon Smirl <hidden> Date: 2016-06-15 22:42:29
On 6/16/06, Keith Packard [off-list ref] wrote:
On Fri, 2006-06-16 at 13:00 -0400, Jon Smirl wrote:
quoted
Is it a crazy idea to read the cvs files, compute an sha1 on each
expanded delta and then write the delta straight into a pack file? Are
the cvs and git delta formats the same? What about CVS's forward and
reverse delta use?
At this point, merging blobs into packs isn't a significant part of the
computational cost. parsecvs is spending all of its time in the
quadratic traversal of the diff chains; fixing that to emit all of the
versions in a single pass should speed up that part of the conversion
process dramatically.
That's not true for the state I am in. cvsps can compute the changeset
tree in 15 minutes, cvs2svn can compute their version in a couple of
hours. cvs2svn builds a much better tree.
I've been extracting versions from cvs and adding them to git now for
2.5 days and the process still isn't finished. It is completely CPU
bound. It's just a loop of cvs co, add it to git, make tree, commit,
etc.
quoted
While this is going on, track the
branches/changsets in memory and then finish up by writing these trees
into the pack file too. This should take no more ram than cvsps needs
currently.
cvsps drops too much state on the floor making branch point and branch
contents inaccurate. What I'm hoping is that I can figure out a way to
discard most of the per-version information by computing tree objects in
reverse order, saving only the tree sha1 and other per-commit info, then
stitch the commits together using that, without needing the full
per-file data.
I agree cvsps is dropping a lot. My screen is full of "Skipping
#CVSPS_NO_BRANCH" and
"Skipping SpiderMonkey140_NES40Rtm_Branch" and "Skipping
SpiderMonkey140_BRANCH" etc.
What about the cvs2svn algorithm described in the attachment? A ram
based version could be faster. Compression could be acheived by
switching from using the full path to a version to the sha1 for it.
--
Jon Smirl
jonsmirl@gmail.com
From: Keith Packard <keithp@keithp.com> Date: 2016-06-15 22:42:29
On Fri, 2006-06-16 at 13:44 -0400, Jon Smirl wrote:
I've been extracting versions from cvs and adding them to git now for
2.5 days and the process still isn't finished. It is completely CPU
bound. It's just a loop of cvs co, add it to git, make tree, commit,
etc.
To do all of mozilla using parsecvs (even with the quadratic algorithm)
takes about three hours on annarchy.freedesktop.org (two dual-core
Opteron with 4GB memory), including all conversion to packs. The pack
time is a tiny fraction of that.
What about the cvs2svn algorithm described in the attachment? A ram
based version could be faster. Compression could be acheived by
switching from using the full path to a version to the sha1 for it.
Yes, parsecvs currently keeps everything in memory when doing the tree
conversion, which means it grows to a huge size to compute the full tree
of revisions. Computing git tree objects from the top down, then
computing commit objects from the bottom up should allow us to free most
of that during the full branch history computation process. I'm starting
a rewrite of parsecvs to try this approach and see how well it works.
If you've looked at the parsecvs source code, you'll notice it's a mess
at present; I started by attempting to do pair-wise tree merges in a
mistaken attempt to convert a linear term to log. Hacking that code into
its present form should be viewed more as a demonstration of how the
overall process can work, not as an optimal expression of the algorithm.
--
keith.packard@intel.com
From: Nicolas Pitre <hidden> Date: 2016-06-15 22:42:29
On Fri, 16 Jun 2006, Jon Smirl wrote:
On 6/16/06, Keith Packard [off-list ref] wrote:
quoted
On Fri, 2006-06-16 at 13:00 -0400, Jon Smirl wrote:
quoted
Is it a crazy idea to read the cvs files, compute an sha1 on each
expanded delta and then write the delta straight into a pack file? Are
the cvs and git delta formats the same? What about CVS's forward and
reverse delta use?
At this point, merging blobs into packs isn't a significant part of the
computational cost. parsecvs is spending all of its time in the
quadratic traversal of the diff chains; fixing that to emit all of the
versions in a single pass should speed up that part of the conversion
process dramatically.
That's not true for the state I am in. cvsps can compute the changeset
tree in 15 minutes, cvs2svn can compute their version in a couple of
hours. cvs2svn builds a much better tree.
Is it a crazy idea to read the cvs files, compute an sha1 on each
expanded delta and then write the delta straight into a pack file? Are
the cvs and git delta formats the same? What about CVS's forward and
reverse delta use? While this is going on, track the
branches/changsets in memory and then finish up by writing these trees
into the pack file too. This should take no more ram than cvsps needs
currently.
What you want is parsecvs, which does it much more like that.
Linus