I hate irc.
I'm reading the irc logs, and seeing that people have problems, but (a) it
was while I was asleep and (b) irc use doesn't encourage people to
actually explain what the problems _are_, so I have no clue.
So now I know that "spyderous" has problems importing some 1GB gentoo CVS
archive, but that's pretty much it. Grr.
Are people afraid to post to git@vger.kernel.org, or what?
I saw that people tried to suggest posting to the git mailing list, but
can any of you who are active on irc be a bit more forceful? And perhaps
we don't make this mailing list address well enough known?
As far as I'm aware, the git mailing list isn't closed, so people should
be able to post here without even subscribing. I can well understand that
you might not want to subscribe and prefer to look ove rthe list through
some archive setup (the way I look at the irc logs), and maybe we should
just make the git mailing list address more obvious.
Right now, the "community" page at http://git.or.cz/community.html doesn't
even mention the git mailing list address directly, it just tells you how
you can subscribe and read the archives.
Can we perhaps fix that, and the people who are active on irc please also
make it clear to people that if they have some real problems that don't
get an immediate answer, the git mailing list ends up where a lot of
people can actually look more closely at it.. And tell them what the
address is.
Linus
On Sat, May 20, 2006 at 10:26:22AM -0700, Linus Torvalds wrote:
I'm reading the irc logs, and seeing that people have problems, but (a) it
was while I was asleep and (b) irc use doesn't encourage people to
actually explain what the problems _are_, so I have no clue.
So now I know that "spyderous" has problems importing some 1GB gentoo CVS
archive, but that's pretty much it. Grr.
FWIW, I have mentionned a problem that may be the same, under
Message-ID [off-list ref], that was on January
7th. Namely, when importing a repository with very large files over
pserver or ssh, timeouts can occur and prevent the import from
working. But, as you said, it's not easy to get precise info from the
logs :)
Best regards,
--
Yann Dirson [off-list ref] |
Debian-related: [off-list ref] | Support Debian GNU/Linux:
| Freedom, Power, Stability, Gratis
http://ydirson.free.fr/ | Check <http://www.debian.org/>
On Sat, May 20, 2006 at 10:26:22AM -0700, Linus Torvalds wrote:
quoted
I'm reading the irc logs, and seeing that people have problems, but (a) it
was while I was asleep and (b) irc use doesn't encourage people to
actually explain what the problems _are_, so I have no clue.
So now I know that "spyderous" has problems importing some 1GB gentoo CVS
archive, but that's pretty much it. Grr.
Hi all,
I just subscribed and this post is the only one I've got from the
thread, so I'm responding to it instead of the original. Gentoo's an
IRC-based community, so I tend to try IRC first for any problems I have
and fall back to the list later if I can't get things figured out.
Here's a rough summary:
Our main repo is actually a bit over 2G (2103621223) now that I check,
but it's not very complex. There's actually just one branch, and I don't
think anyone would care if we lost the history from it because it's a
release branch from a few years ago.
Somebody else tried importing it with git-cvsimport, but he said he hit
some kind of problem and recalled that it was a cvsps segfault. Sounds
about right, since I've never gotten cvsps to run successfully on the
whole repo either.
I tried with parsecvs, but it runs into OOM even on a machine with 4G
RAM after reading in all the ,v files, presumably while it's building
some huge tree of changesets in memory. Keith Packard's suggested that
there are ways to reduce parsecvs's memory use, because it retains the
full tree in memory for each revision rather than just the files that
actually changed. But my C skills are pretty weak; I'm an OK reader but
not much of a writer yet.
Thanks,
Donnie
Our main repo is actually a bit over 2G (2103621223) now that I check,
but it's not very complex. There's actually just one branch, and I don't
think anyone would care if we lost the history from it because it's a
release branch from a few years ago.
Can you point to it? I'm not a CVS user, but I've played with cvsps before
(to get it to work), and I'm a humanitarian - rescuing people from CVS is
to me not just a good idea, it's a moral imperative.
Linus
Our main repo is actually a bit over 2G (2103621223) now that I check,
but it's not very complex. There's actually just one branch, and I don't
think anyone would care if we lost the history from it because it's a
release branch from a few years ago.
Can you point to it? I'm not a CVS user, but I've played with cvsps before
(to get it to work), and I'm a humanitarian - rescuing people from CVS is
to me not just a good idea, it's a moral imperative.
I don't want to post the link publicly for a few reasons, including the
huge amount of bandwidth it would suck up for lots of people to download
it. I've sent it to you off-list, and if anyone else would also like it,
please drop me a note.
Thanks,
Donnie
Somebody else tried importing it with git-cvsimport, but he said he hit
some kind of problem and recalled that it was a cvsps segfault. Sounds
about right, since I've never gotten cvsps to run successfully on the
whole repo either.
Much to my surprise, a cvsps run I started earlier has just finished
without segfaulting. But attempts to actually run cvsps (e.g., cvsps -a
spyderous) spit thousands of warnings of "WARNING: revision 1.1.1.1 of
file $FILENAME on unnamed branch".
Thanks,
Donnie
I don't want to post the link publicly for a few reasons, including the
huge amount of bandwidth it would suck up for lots of people to download
it. I've sent it to you off-list, and if anyone else would also like it,
please drop me a note.
Ok. It's still converting (that's a big archive), but it has passed the
cvsps stage without errors for me, and the conversion so far seems ok. But
it has only gotten to
Author: vapier <vapier> 2002-09-23 12:32:42
Changed GPL to GPL-2 in LICENSE and updated SRC_URI to use mirror:
so it has converted only slightly more than the first two years of
history in the roughly 30 minutes I've let it run. So it will take several
hours.
The reason it works for me is likely simply the fact that I had a few
patches to my cvsps already. I'm appending the stupid patches, I'm not
guaranteeing that they are correct at all, although the three _committed_
patches are almost certainly correct (and the last uncommitted one is
almost certainly totally broken). The patches are against clean cvsps 2.1.
Also, when I say "the conversion so far seems ok", I obviously don't
actually know what the hell the archive is supposed to look like, so I can
only say that the end result seems not totally insane.
To do a good conversion, you'll want to make sure that you have a author
name conversion file. See the "-A" flag in "git help cvsimport" (if you
have the man-pages installed).
Linus
---
commit 534120d9a47062eecd7b53fd7ac0b70d97feb4fd
Author: Linus Torvalds [off-list ref]
Date: Wed Mar 22 11:20:59 2006 -0800
Increase log-length limit to 64kB
Yeah, it should be dynamic. I'm lazy.
---
cvsps_types.h | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
commit 3d1ebcef6b4f9f6c9064efd64da4dd30d93c3c96
Author: Linus Torvalds [off-list ref]
Date: Wed Mar 22 17:20:20 2006 -0800
Fix branch ancestor calculation
Not having any ancestor at all means that any valid ancestor (even of
"depth 0") is fine.
Signed-off-by: Linus Torvalds [off-list ref]
---
cvsps.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
@@ -2357,9 +2357,16 @@ static int revision_affects_branch(CvsFistaticintcount_dots(constchar*p){intdots=0;+intlen=strlen(p);-while(*p)-if(*p++=='.')+while(len>2){+if(memcmp(p+len-2,".1",2))+break;+len-=2;+}++while(len)+if(p[--len]=='.')dots++;returndots;
@@ -2613,7 +2620,7 @@ static void determine_branch_ancestor(Pa/* HACK: we sometimes pretend to derive from the import branch. *justdon'tdothat.thisistheeasiestwaytoprevent...*/-d2=(strcmp(rev->rev,"1.1.1.1")==0)?0:count_dots(rev->rev);+d2=count_dots(rev->rev);if(d2>d1)head_ps->ancestor_branch=rev->branch;
FWIW, I have mentionned a problem that may be the same, under
Message-ID [off-list ref], that was on January
7th. Namely, when importing a repository with very large files over
pserver or ssh, timeouts can occur and prevent the import from
working. But, as you said, it's not easy to get precise info from the
logs :)
For big repositories, you really shouldn't use pserver or ssh anyway. You
should try really really hard to just get a local copy, and do it that
way. It's going to be tons faster, and will avoid a lot of the problems,
including network timeouts etc.
Linus
Ok. It's still converting (that's a big archive), but it has passed the
cvsps stage without errors for me, and the conversion so far seems ok. But
it has only gotten to
Author: vapier <vapier> 2002-09-23 12:32:42
Changed GPL to GPL-2 in LICENSE and updated SRC_URI to use mirror:
so it has converted only slightly more than the first two years of
history in the roughly 30 minutes I've let it run. So it will take several
hours.
Btw, trying this import (which got interrupted by a thunderstorm and one
of our first power failures in a long time - just a few seconds, but
enough to power off everything but my laptops) it became very obvious that
"git cvsimport" really _really_ should re-pack the archive every once in a
while.
The old "repack every month or so" approach doesn't work that well when
you try to import several years of history in a few hours.
Now, you can just repack after the whole thing is done (it will probably
take no more than ~15 minutes or so), but it would probably be best if the
import script itself decided to repack every once in a while just to avoid
wasting a lot of diskspace _during_ the import itself.
So this isn't so much a correctness issue as a "avoid wasting time and
space" issue, but still..
Linus
Ok. It's still converting (that's a big archive), but it has passed the
cvsps stage without errors for me, and the conversion so far seems ok. But
it has only gotten to
Author: vapier <vapier> 2002-09-23 12:32:42
Changed GPL to GPL-2 in LICENSE and updated SRC_URI to use mirror:
so it has converted only slightly more than the first two years of
history in the roughly 30 minutes I've let it run. So it will take several
hours.
Btw, trying this import (which got interrupted by a thunderstorm and one
of our first power failures in a long time - just a few seconds, but
enough to power off everything but my laptops) it became very obvious that
"git cvsimport" really _really_ should re-pack the archive every once in a
while.
Fortunately the storms haven't been that bad down in Corvallis. cvsps
also worked fine for me, but git-cvsimport broke in the middle. The
command I'm using is 'git-cvsimport -P ../gentoo.cvsps -k -d
/media/scm_comparison -A ~/dev/Authors -v gentoo-x86 | tee cvsimport.log'
Here's the last bits:
Fetching gnome-base/gnome-applets/gnome-applets-1.4.0.4-r1.ebuild v 1.5
Update gnome-base/gnome-applets/gnome-applets-1.4.0.4-r1.ebuild: 947 bytes
Fetching gnome-base/gnome-applets/gnome-applets-1.4.0.4-r2.ebuild v 1.3
Update gnome-base/gnome-applets/gnome-applets-1.4.0.4-r2.ebuild: 977 bytes
Fetching gnome-base/gnome-applets/gnome-applets-2.0.0-r1.ebuild v 1.2
Update gnome-base/gnome-applets/gnome-applets-2.0.0-r1.ebuild: 2704 bytes
Fetching gnome-base/gnome-applets/gnome-applets-2.0.0.ebuild v 1.2
Update gnome-base/gnome-applets/gnome-applets-2.0.0.ebuild: 3031 bytes
Tree ID 4d19a84efce2de9cfb42ac0397e0036bbed2ad65
Parent ID ecb78bbe30369a76e2599d0d17de8fe922dca211
Committed patch 14615 (origin 2002-07-16 20:13:15)
Commit ID 4dd2179e0c1369e07cd268fb5c8b150c3a2a1094
Delete net-fs/openafs/openafs-1.2.2-r6.ebuild
Delete net-fs/openafs/files/digest-openafs-1.2.2-r6
Tree ID bfc7320883983655d7d2ea2c6d04f85b45365ce1
Parent ID 4dd2179e0c1369e07cd268fb5c8b150c3a2a1094
Committed patch 14616 (origin 2002-07-16 20:15:15)
Commit ID 7a36de9c4c9b93337ed789ae2341cad3d0991c6d
Unknown: error Cannot allocate memory
Fetching profiles/package.mask v 1.992
cat: write error: Broken pipe
Thanks,
Donnie
Fortunately the storms haven't been that bad down in Corvallis. cvsps
also worked fine for me, but git-cvsimport broke in the middle.
Hmm. It's actually possible that it did that for me too - I had put the
cvsimport in an xterm and forgotten about it, and just assumed that the
power failure was what broke it. But maybe it had broken down before that
happened - I just don't have any logs left ;)
Here's the last bits:
[ snip snip ]
Commit ID 7a36de9c4c9b93337ed789ae2341cad3d0991c6d
Unknown: error Cannot allocate memory
Fetching profiles/package.mask v 1.992
cat: write error: Broken pipe
Hmm. I don't actually know perl, and my original "cvsimport" script was
actually this funny C program that generated a shell script to do the
import. That worked fine, and had no memory leaks, but it was a truly
hacky thing of horrible beauty. Or rather, it _would_ have been that, if
it had had any beauty to be horrible about. But at least I would have been
able to debug it.
But the perl one I can't parse any more. That said, the whole "Unknown:"
printout seems to come from the subroutine "_line()", which just reads a
line from the cvs server.
Did you do a "top" at any time just before this all happened? It _sounds_
like it might actually be a memory leak on the CVS server side, and the
problem may (or may not) be due to the optimization that keeps a single
long-running CVS server instance for the whole process.
I wouldn't be in the least surprised if that ends up triggering a slow
leak in CVS itself, and then CVS runs out of memory.
That would likely have been obvious in any "top" output just before the
failure.
Smurf, Martin, Dscho.. Any ideas? My old script just ran RCS directly on
the files, and had no issues like that. I'll happily admit that my old
script generator thing was horrible, but it was a lot easier to debug than
the smarter perl script that uses a CVS server connection..
Linus
From: Martin Langhoff <hidden> Date: 2016-06-15 22:42:27
On 5/22/06, Linus Torvalds [off-list ref] wrote:
I wouldn't be in the least surprised if that ends up triggering a slow
leak in CVS itself, and then CVS runs out of memory.
I'm dying to try this out myself after work. I don't discard that
cvsimport might be stuffing data in an array that grows forever. In
any case you'll hear from me soon.
martin
Did you do a "top" at any time just before this all happened? It _sounds_
like it might actually be a memory leak on the CVS server side, and the
problem may (or may not) be due to the optimization that keeps a single
long-running CVS server instance for the whole process.
No. =\ I just started the thing running in a screen session and came
back a few hours later to find it like that.
Thanks,
Donnie
From: Martin Langhoff <hidden> Date: 2016-06-15 22:42:27
On 5/22/06, Linus Torvalds [off-list ref] wrote:
Did you do a "top" at any time just before this all happened? It _sounds_
like it might actually be a memory leak on the CVS server side, and the
problem may (or may not) be due to the optimization that keeps a single
long-running CVS server instance for the whole process.
Running a few tests right now. Looks like cvs (Debian/etch 1.12.9-13)
itself is not leaking any memory. The Perl (Debian/etch
5.8.7-something and now 5.8.8-4) process OTOH is visibly allocating
memory. Starts off at 4MB and gets up to ~17MB by the time it has done
6K commits.
I am trying to figure out whether the leak is in the script or in the
Perl implementation, using PadWalk, Devel::Leak and friends. If the
leak is here, I can't see it (yet).
I wouldn't be in the least surprised if that ends up triggering a slow
leak in CVS itself, and then CVS runs out of memory.
Or a slow leak in Perl? The 5.8.8 release notes do talk about some
leaks being fixed, but this 5.8.8 isn't making a difference.
Working on it.
martin
Or a slow leak in Perl? The 5.8.8 release notes do talk about some
leaks being fixed, but this 5.8.8 isn't making a difference.
Working on it.
Thanks. Looking at what I did convert, that horrid gentoo CVS tree is
interesting. The resulting (partial) git history has 93413 commits and
850,000+ objects total, all in a totally linear history.
And that's just up to April 2004, so the full tree is probably a million
objects.
The good news is that git seems to handle that size repo no problem at
all. The repack did indeed take a long while, but it packed it all down to
a 189MB pack-file (and 20MB pack index).
Considering that the bzip2'd tar-file of the CVS history was 157MB, and
the actual CVS footprint was about 1.6GB, if git stays at under a quarter
gigabyte for the whole archive once converted (which sounds likely,
counting indexing), git would basically cut down the disk usage for a live
repo by a factor of 7 or so.
_And_ I can do a "git log origin > /dev/null" in about 2.4 seconds. Take
that, CVS.
Linus
From: Martin Langhoff <hidden> Date: 2016-06-15 22:42:27
Signed-off-by: Martin Langhoff <redacted>
---
This is ugly, but while I work on cleaning up the leak
that seems to be somewhere in the commit() sub, we may
as well set up a workaround.
I am not 100% happy woth including this in git.git.
In any case, I hope we can revert it soon.
---
git-cvsimport.perl | 8 ++++++--
1 files changed, 6 insertions(+), 2 deletions(-)
64ea3c83d8cd176ee972055bd1d11f398655dad8
From: Martin Langhoff <hidden> Date: 2016-06-15 22:42:27
On 5/22/06, Linus Torvalds [off-list ref] wrote:
On Mon, 22 May 2006, Martin Langhoff wrote:
quoted
Or a slow leak in Perl? The 5.8.8 release notes do talk about some
leaks being fixed, but this 5.8.8 isn't making a difference.
Working on it.
Thanks. Looking at what I did convert, that horrid gentoo CVS tree is
interesting. The resulting (partial) git history has 93413 commits and
850,000+ objects total, all in a totally linear history.
Ok, so there's 3 patches posted that should help narrow down the
problem. There's a new -L <imit> so that Donnie can get his stuff done
by running it in a while(true) loop. Not proud of it, but hey.
And there are two patches that I suspect may fix the leak. After
applying them, the cvsimport process grows up to ~13MB and then tapers
off, at least as far as my patience has gotten me. It's late on this
side of the globe so I'll look at the results tomorrow morning.
(BTW, I typo-ed Linus' address in the git-send-email invocation. Will
resend to him separately)
I'll also prep a patch as Linus suggests to do auto-repacking while
the import runs so we don't eat up the harddisk.
git would basically cut down the disk usage for a live
repo by a factor of 7 or so.
_And_ I can do a "git log origin > /dev/null" in about 2.4 seconds. Take
that, CVS.
And there are two patches that I suspect may fix the leak. After
applying them, the cvsimport process grows up to ~13MB and then tapers
off, at least as far as my patience has gotten me. It's late on this
side of the globe so I'll look at the results tomorrow morning.
Ok, initial results are promising. git-cvsimport appears to be still
slowly growing, but it's at 40M (ie pretty tiny, considering that cvsps
grew to 800+MB on this archive) and growth seems to actually be slowing.
My conversion is only up to September 2002, but if it doesn't suddenly hit
some huge growth spurt, I wouldn't expect it to run out of memory. The CVS
server process itself is tiny, and doesn't seem to grow at all.
As to packing, it doing something like
while :
do
sleep 30
#
# repack roughly every 25600 objects
#
n=$(ls .git/objects/00 2> /dev/null | wc -l)
if [ $n -gt 100 ]; then
git repack -a
#
# Stupid sleep to make sure that nobody is still
# using any unpacked objects after the pack got
# generated
#
sleep 10
git prune-packed
fi
done
or similar (the above is totally untested - I've just done it by hand a
few times) should work. It's perfectly ok to repack the archive even while
the cvsimport script is adding more data and changing it.
Linus
From: Jakub Narebski <hidden> Date: 2016-06-15 22:42:27
Linus Torvalds wrote:
git repack -a
#
# Stupid sleep to make sure that nobody is still
# using any unpacked objects after the pack got
# generated
#
sleep 10
git prune-packed
Is it really necessary (on Linux at least)? Git boast it's atomicity...
--
Jakub Narebski
Warsaw, Poland
git repack -a
#
# Stupid sleep to make sure that nobody is still
# using any unpacked objects after the pack got
# generated
#
sleep 10
git prune-packed
Is it really necessary (on Linux at least)? Git boast it's atomicity...
I don't think it's necessary in practice.
But people _should_ realize that removing objects is very very special.
Whether it's done by "git prune-packed" or "git prune", that's a very
dangerous operations. "git prune" a lot more so than "git prune-packed",
of course (in fact, you should _never_ run "git prune" on a repository
that is active - you _will_ corrupt it)-
Doing "git prune-packed" _should_ be mostly safe on UNIX, since the
objects all exist in packs, and anybody who already opened an object will
keep the fd open, and not even notice that the name is gone. However,
there is at least one race:
object lookup "git repack -a -d"
============= ==================
- a process does its object
database setup. No new pack-file
yet.
- mv tmp-packfile active-packfile
- git prune-packed
- the process looks up the object,
and doesn't look in the pack-file
because it didn't see the pack-file.
So it tries to look up an object,
fails, and errors out.
It's not a fatal error (just re-try)
but it could break something like a
cvsimport
Now, in PRACTICE, I doubt you'd ever hit this. But the fact is, pruning
your repository (whether prune-packed or a full prune) is _the_ special
operation. It's something that removes a filesystem representation of an
object that is otherwise immutable.
Linus
But people _should_ realize that removing objects is very very special.
Just a similar question: is there any reason not tu run git
repack/prune-packed as cron job? I would think of something like this
for every night:
- git prune-packed (remove objects packed last time)
- check how many objects git-count-objects counts, if it are not enough
abort
- git repack
git repack -a -d is probably a bad idea, I guess, because a program
could try to open them after they were deleted. Is there any way to
delete unnecessary packs (those which would repack -a -d delete)?
Making it possible to do a git repack -a and delete those packs the
next night?
Or a slow leak in Perl? The 5.8.8 release notes do talk about some
leaks being fixed, but this 5.8.8 isn't making a difference.
Working on it.
Thanks. Looking at what I did convert, that horrid gentoo CVS tree is
interesting. The resulting (partial) git history has 93413 commits and
850,000+ objects total, all in a totally linear history.
Ok, so there's 3 patches posted that should help narrow down the
problem. There's a new -L <imit> so that Donnie can get his stuff done
by running it in a while(true) loop. Not proud of it, but hey.
And there are two patches that I suspect may fix the leak. After
applying them, the cvsimport process grows up to ~13MB and then tapers
off, at least as far as my patience has gotten me. It's late on this
side of the globe so I'll look at the results tomorrow morning.
OK, I started a new run without -L, and I'm watching it in top right
now. The cvsimport seems to be doing alright, but the cvs server process
sucks about another megabyte of virtual every 4-5 seconds. This is a bit
concerning since I don't have any swap. Shortly after it hit 670M, I got
"Cannot allocate memory" again. I've got a gig of RAM, and around 300M
was resident in various processes at the time.
So it seems the problem is in cvs itself. I will try another run with -L
now.
Thanks,
Donnie
OK, I started a new run without -L, and I'm watching it in top right
now. The cvsimport seems to be doing alright, but the cvs server process
sucks about another megabyte of virtual every 4-5 seconds. This is a bit
concerning since I don't have any swap. Shortly after it hit 670M, I got
"Cannot allocate memory" again. I've got a gig of RAM, and around 300M
was resident in various processes at the time.
Hmm. My cvs server doesn't really grow at all. It's at 13M RSS.
What version of cvs are you running?
[torvalds@g5 ~]$ cvs --version
Concurrent Versions System (CVS) 1.11.21 (client/server)
maybe that matters.
(but my import is only up to Jun 22, 2003 so far).
Linus
From: Martin Langhoff <hidden> Date: 2016-06-15 22:42:27
On 5/23/06, Donnie Berkholz [off-list ref] wrote:
So it seems the problem is in cvs itself. I will try another run with -L
now.
What version of cvs are you using? Perhaps trying a different one?
The dev machine where I am running the import is a slug! It's still
working on it, only gotten to 7700 commits, with the cvsimport process
stable at 28MB RAM and cvs stable at 4MB.
cheers,
martin
From: Martin Langhoff <hidden> Date: 2016-06-15 22:42:27
On 5/23/06, Linus Torvalds [off-list ref] wrote:
Ok, initial results are promising. git-cvsimport appears to be still
slowly growing, but it's at 40M (ie pretty tiny, considering that cvsps
grew to 800+MB on this archive) and growth seems to actually be slowing.
That's great news. The cvs archive seems to have large commits every
once in a while, so I suspect the residual memory growth may be
related to those. Or to a smaller leak I haven't nailed.
My test box is bloody slow it seems. I'll try and get hold of a faster
machine to run this if I can.
As to packing, it doing something like
Given that we are running batch, it is safe and simple to stop the
import, repack, prune-packed, and keep going. Don't think we'll win
any races by running it in parallel ;-)
cheers,
martin
Hmm. My cvs server doesn't really grow at all. It's at 13M RSS.
Yeah, that's the thing. RSS stayed about the same (according to top),
but virtual just kept growing.
What version of cvs are you running?
[torvalds@g5 ~]$ cvs --version
Concurrent Versions System (CVS) 1.11.21 (client/server)
Concurrent Versions System (CVS) 1.12.12 (client/server)
Looks like there's a .13 out but the zlib interaction is badly broken
(-z >=1) so my system didn't get upgraded. I'll try it anyway after the
-L run finishes.
Thanks,
Donnie
The dev machine where I am running the import is a slug! It's still
working on it, only gotten to 7700 commits, with the cvsimport process
stable at 28MB RAM and cvs stable at 4MB.
I have to say, that cvsimport script really does do horrible things. It's
basically a fork/exec/exit benchmark, as far as I can tell. Running
oprofile on the thing, the top offenders are (ignore the 45% idle thing:
it's just because this was run on a dual-cpu system, so since it's almost
completely single-threaded you get ~50% idle by default).
3117654 45.8708 vmlinux vmlinux .power4_idle
802313 11.8046 vmlinux vmlinux .unmap_vmas
632913 9.3122 vmlinux vmlinux .copy_page_range
150359 2.2123 vmlinux vmlinux .release_pages
131330 1.9323 vmlinux vmlinux .vm_normal_page
117836 1.7337 libperl.so libperl.so (no symbols)
74098 1.0902 libgklayout.so libgklayout.so (no symbols)
54680 0.8045 vmlinux vmlinux .free_pages_and_swap_cache
54300 0.7989 libfb.so libfb.so (no symbols)
49052 0.7217 vmlinux vmlinux .copy_4K_page
46559 0.6850 libc-2.4.so libc-2.4.so getc
42677 0.6279 vmlinux vmlinux .page_remove_rmap
41133 0.6052 libc-2.4.so libc-2.4.so ferror
..
those kernel functions are all about process create/exit, and COW faulting
after the fork.
Now, this is on ppc, so process creation is likely slower (idiotic PPC VM
page table hashes), but Linux is actually very good at doing this, and the
fact that process create/exit is so high is a very big sign that the
script just ends up executing a _ton_ of small simple processes that do
almost nothing.
I wonder why those "git-update-index" calls seem to be (assuming I read
the perl correctly) done only a few files at a time. We can do a hundreds
in one go, but it seems to want to do just ten files or something at the
same time. Although since most commits should hopefully just modify a
couple of files, that probably isn't a big deal.
That thing would probably be an order of magnitude faster if written to
use the git library interfaces directly. Of course, the CVS part is
probably a big overhead, so it might not help much (I would not be
surprised at all if a number of the fork/exec/exit things are due to the
CVS server starting RCS or something, not due to git-cvsimport itself)
Linus
OK, I started a new run without -L, and I'm watching it in top right
now.
Tried a run with -L 1024 and it broke in just a couple of minutes:
Fetching
sys-kernel/linux/files/2.4.0.8/linux-2.4.0-ac8-reiserfs-3.6.25-nfs.diff.gz
v 1.1
New
sys-kernel/linux/files/2.4.0.8/linux-2.4.0-ac8-reiserfs-3.6.25-nfs.diff.gz:
6367 bytes
Tree ID 457f629df10e70a5ef430f431eca27ed02a83d46
Parent ID 0541d8b54a02df3be50d529497236556c6862a4c
Committed patch 1024 (origin 2001-01-13 00:29:39)
Commit ID ba9d995d12a37502a851e198b67e141623f79544
DONE; creating master branch
cat: write error: Broken pipe
Thanks,
Donnie
Hmm. My cvs server doesn't really grow at all. It's at 13M RSS.
Yeah, that's the thing. RSS stayed about the same (according to top),
but virtual just kept growing.
Not for me. The virtual size is certainly bigger than RSS, but not by a
huge amount. So this might be a regression in CVS, since you seem to have
a newer version than I do.
The latest stable CVS release is 1.11.21, I think: you seem to be running
the "development" version (1.12.x).
Linus
Of course, the CVS part is probably a big overhead, so it might not help
much (I would not be surprised at all if a number of the fork/exec/exit
things are due to the CVS server starting RCS or something, not due to
git-cvsimport itself)
Ahh. stracing the CVS server seems to imply that it forks off a subprocess
for every command. It doesn't actually execute any external program, but
just does a fork + muck around in the ,v files + exit.
Maybe one of the changes in the 1.12.x versions is to not do that, which
might explain why Donnie seems to see much better performance, but also
sees all the memory leakage?
Linus
I wonder why those "git-update-index" calls seem to be (assuming I read
the perl correctly) done only a few files at a time. We can do a hundreds
in one go, but it seems to want to do just ten files or something at the
same time.
No, fifty.
I simply was too lazy to count the actual filenames' lengths. ;-)
That thing would probably be an order of magnitude faster if written to
use the git library interfaces directly. Of course, the CVS part is
probably a big overhead, so it might not help much
The beast *was* mainly written to do this remotely...
--
Matthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf@smurf.noris.de
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
- -
The worst form of inequality is to try to make unequal things equal.
-- Aristotle
The beast *was* mainly written to do this remotely...
I don't think the remote usability is valid, except for some really small
repositories. The fact that it takes hours even when the CVS server is
local doesn't bode well for doing it remotely for any but the most trivial
things.
I really think it would be better to have local use be the optimized case,
with remote being the "it's _possible_" case.
Linus
From: Martin Langhoff <hidden> Date: 2016-06-15 22:42:27
On 5/23/06, Linus Torvalds [off-list ref] wrote:
I don't think the remote usability is valid, except for some really small
repositories. The fact that it takes hours even when the CVS server is
local doesn't bode well for doing it remotely for any but the most trivial
things.
I really don't think that using the local cvs binary is a problem at
all. In my experience, the thing is fairly fast and optimized when you
ask it to perform file-oriented questions and that's all we do,
really.
If you want to try it, you'll see that local checkouts of large trees
(like this gentoo one) are fairly fast. Not as fast as GIT itself, but
good enough. I think Donnie has hit a bug with a bad version of cvs,
but other than that, my experience with it is that it is fairly well
behaved -- even if the tool is bad, ubiquity has lead to resiliency
over the years.
I really think it would be better to have local use be the optimized case,
with remote being the "it's _possible_" case.
Agreed, but I think we won't see much benefit in direct parsing. And
we'll have to take the hit of double-implementation.
In any case, we have it already -- parsecvs does it quite well (modulo
memory leaks!) and I've used it several times in conjunction with
cvsimport. Just perform the initial import with parsecvs and then
'track' the remote project with cvsimport.
The problem is that they lead to slightly different trees. So their
output is not consistent, and I don't think that'll be easy to fix.
cheers,
martin
From: Martin Langhoff <hidden> Date: 2016-06-15 22:42:27
On 5/23/06, Martin Langhoff [off-list ref] wrote:
The problem is that they lead to slightly different trees.
Sorry! s/trees/histories/ there. The trees are (or should!) be the
same, and tree differences should be addressed as bugs. Differences in
how history is parsed are unavoidable right now.
martin
I really don't think that using the local cvs binary is a problem at
all. In my experience, the thing is fairly fast and optimized when you
ask it to perform file-oriented questions and that's all we do,
really.
Fair enough. My worry was mainly that the cvs server was doing something
stupid, but I suspect most of the fork/exec's are probably from the
cvsimport perl script itself.
In any case, we have it already -- parsecvs does it quite well (modulo
memory leaks!) and I've used it several times in conjunction with
cvsimport. Just perform the initial import with parsecvs and then
'track' the remote project with cvsimport.
I didn't get parsecvs working when I tried it a long time ago, and Donnie
reported that it ran out of memory, so I didn't even really consider it.
I'd love for it to work well, and it may be reasonable to do really big
imports on multi-gigabyte 64-bit machines (after all, they aren't _hard_
to find any more, and you only need to do it once).
That said, it still seems pretty stupid to require that much memory just
to import from CVS.
Linus
This stupid patch on top of yours seems to make git happier. It's
disgusting, I know, but it just repacks things every kilo-commit.
I actually think that I found a real ext3 performance bug from trying to
determine why git sometimes slows down ridiculously when the tree has been
allowed to go too long without a repack.
Linus
---
From: Martin Langhoff (CatalystIT) <hidden> Date: 2016-06-15 22:42:27
Linus Torvalds wrote:
This stupid patch on top of yours seems to make git happier. It's
disgusting, I know, but it just repacks things every kilo-commit.
I actually think that I found a real ext3 performance bug from trying to
determine why git sometimes slows down ridiculously when the tree has been
allowed to go too long without a repack.
Acked (in case anyone cares for such an obvious one), and thanks! I
thought of doing that last night together with that exact patch, but I
was focussing on the leak.
cheers,
m
--
-----------------------------------------------------------------------
Martin @ Catalyst .Net .NZ Ltd, PO Box 11-053, Manners St, Wellington
WEB: http://catalyst.net.nz/ PHYS: Level 2, 150-154 Willis St
OFFICE: +64(4)916-7224 MOB: +64(21)364-017
Make things as simple as possible, but no simpler - Einstein
-----------------------------------------------------------------------
On Mon, May 22, 2006 at 07:28:37PM -0700, Linus Torvalds wrote:
This stupid patch on top of yours seems to make git happier. It's
disgusting, I know, but it just repacks things every kilo-commit.
I actually think that I found a real ext3 performance bug from trying to
determine why git sometimes slows down ridiculously when the tree has been
allowed to go too long without a repack.
Do you have dir_index (the hashed btree) feature enabled by any chance?
- Ted
On Mon, May 22, 2006 at 07:28:37PM -0700, Linus Torvalds wrote:
quoted
I actually think that I found a real ext3 performance bug from trying to
determine why git sometimes slows down ridiculously when the tree has been
allowed to go too long without a repack.
Do you have dir_index (the hashed btree) feature enabled by any chance?
No, and I know I probably should, since it would hopefully help git usage.
But my problem actually happens even with moderately sized directories:
they were just 40kB or so in size, and the problem isn't high system CPU
usage, but tons of extra IO. I ran things on a machine with 2GB of RAM,
and as far as I could tell, the working set _should_ have fit into memory,
but CPU utilization was consistently in the 1% range.
Now, it's possible that I'm just wrong, and it really didn't fit in
memory, but I I _suspect_ that the issue is that ext3 directory handling
still uses the "buffer_head" thing rather than the page cache, and that we
simply don't LRU the memory appropriately so we don't let the memory
pressure expand the buffer cache.
Now, using buffer cache in this day and age is insane and horrible
(there's a reason I suspect the LRU doesn't work that well: the buffer
heads aren't supposed to be used as a cache, and people are supposed to
use the page cache for it these days), but Andrew tells me that the whole
JBD thing basically requires it. Whatever.
Now, repacking obviously hides it entirely (because then the load becomes
entirely a page-cache load, and the kernel does _that_ beautifully), but
I'm a bit bummed that I think I hit an ext3 braindamage.
So an unpacked git archive on ext3 (but not ext2, I believe: ext2 should
use the page cache for directories) ends up being very buffer-cache
intensive. And the buffer cache is basically deprecated..
Linus
PS. I'll see if I can figure out the problem, and maybe the good news is
that I'll be able to just fix a real kernel performance issue. Still,
there's a _reason_ we tried to get away from the buffer heads as a caching
entity..
So an unpacked git archive on ext3 (but not ext2, I believe: ext2 should
use the page cache for directories) ends up being very buffer-cache
intensive. And the buffer cache is basically deprecated..
A few notes: I'm not 100% sure things really fit in the 2GB (*), so it
really may be IO limited, and the low CPU use is just because that machine
also happens to have a dang fast next-gen Intel CPU that hasn't even been
released yet.
Also, I do realize that hashed directories should actually decrease the
buffer cache pressure too, just because we wouldn't need to read all of
the directory for a lookup.
Linus
(*) cvsps itself grows to 1.6GB of the 2GB and while that memory should be
largely idle, the problem may simply be that we don't swap it out eagerly
enough. Allowing filesystem metadata to swap out processes is something
we've tuned against, because it tends to result in horrible interactive
behaviour after a nightly "updatedb" run. So it's entirely possible that
this is all normal..
From: Jakub Narebski <hidden> Date: 2016-06-15 22:42:27
Linus Torvalds wrote:
[...] people _should_ realize that removing objects is very very special.
Whether it's done by "git prune-packed" or "git prune", that's a very
dangerous operations. "git prune" a lot more so than "git prune-packed",
of course (in fact, you should _never_ run "git prune" on a repository
that is active - you _will_ corrupt it)-
Would it be possible to make 'git prune' command repository corruption safe,
even if some information might be lost (like 'git add')? Or do _corruption_
mean some recoverable only information is lost? Not always one can use "one
repository per developer" workflow.
One of the solution would be to to use reader/writer lock (filesystem
semaphore), with each command modyfying repository performing locking, and
git-prune waiting on lock until noone is accessing repository. Of course
the problem is with OS and filesystems which does not support locking, and
with stale locks...
Second solution would be to [optionally] wait until no process is accessing
repository, copy repository in some safe place, [optionally] calculate
checksum, prune, [optionally] check if the repository was modified
meanwhile and either abort or repeat, and finally copy pruned repository
back.
--
Jakub Narebski
Warsaw, Poland
From: Martin Langhoff <hidden> Date: 2016-06-15 22:42:27
On 5/23/06, Linus Torvalds [off-list ref] wrote:
This stupid patch on top of yours seems to make git happier. It's
disgusting, I know, but it just repacks things every kilo-commit.
Call me slow, but I am still running and rerunning that gentoo import. ;-)
The current import has reached ~200K commits, and .git is 450MB, while
the checked out tree is 230MB (680MB with .git). At this stage, git
repack -a -d is too memory hungry. I just had to kill it after it took
2GB and was still growing fast. So I have dropped the -a in my test
import for the time being.
Other than that, we are doing ~6K commits per hour on a 2GB 2GHz Opteron.
cheers,
martin
Call me slow, but I am still running and rerunning that gentoo import. ;-)
I'm doing it too, just for fun.
Of course, since I'm doing this on a machine that basically has a laptop
disk, the "just for fun" part is a bit sad. It's waiting for disk about
25% of the time ;/
And it's slow as hell. I really wish we could do better on the CVS import
front.
The current import has reached ~200K commits, and .git is 450MB, while
the checked out tree is 230MB (680MB with .git). At this stage, git
repack -a -d is too memory hungry.
I've got 2GB in that puppy, and "repack -a -d" is fine for me. I'm not
quite up to 200k commits yet (I'm at 160k), but the repacking is certainly
faster than the rest of the import.. Gaah.
It's "git-rev-list --objects" that is the memory sucker for me, the
packing itself doesn't seem to be too bad.
The biggest cost seems to be git-write-tree, which is about 0.225 seconds
for me on that tree on that machine. Which _should_ mean that we could do
4 commits a second, but that sure as hell ain't how it works out. It seems
to do about 1.71 commits a second for me on that tree, which is pretty
damn pitiful. Some cvs overhead, and probably some other git overhead too.
(That's a 2GHz Merom, so the fact that you get ~6k commits per hour on
your 2GHz Opteron is about the same speed - I suspect you're also at least
partly limited by disk, our numbers seem to match pretty well).
200k commits at 6k commits per hour is about a day and a half (plus the
occasional packing load). Taking that long to import a CVS archive is
horrible. But I guess it _is_ several years of work, and I guess you
really have to do it only once, but still.
Linus
From: Jakub Narebski <hidden> Date: 2016-06-15 22:42:27
Linus Torvalds wrote:
200k commits at 6k commits per hour is about a day and a half (plus the
occasional packing load). Taking that long to import a CVS archive is
horrible. But I guess it _is_ several years of work, and I guess you
really have to do it only once, but still.
And how parsecvs (which as far as I remember didn't have incremental mode)
compares wrt speed to git-cvsimport? It is supposed to be faster...
--
Jakub Narebski
Warsaw, Poland
From: Martin Langhoff <hidden> Date: 2016-06-15 22:42:27
On 5/26/06, Linus Torvalds [off-list ref] wrote:
I'm doing it too, just for fun.
Well, it's good to not be so alone in our definition of fun ;-)
Of course, since I'm doing this on a machine that basically has a laptop
disk, the "just for fun" part is a bit sad. It's waiting for disk about
25% of the time ;/
Ouch.
And it's slow as hell. I really wish we could do better on the CVS import
front.
Me too. However, I don't think the perl part is so costly anymore.
It's down to waiting on IO. git-write-tree is also prominently there.
It takes a lot of memory in some writes -- I had thought it'd be
cheaper as it takes one tree object at the time...
I also have a trivial patch that I haven't posted yet, that runs cvsps
to a tempfile, and then reads the file. Serialising the tasks means
that we don't carry around cvsps' memory footprint during the import
itself.
...
It's "git-rev-list --objects" that is the memory sucker for me, the
packing itself doesn't seem to be too bad.
No, you're right, it's git-rev-list that gets called during the
repack. But it was pushing everything it could to swap. Once it didn't
fit in memory, it hit a brick wall :(
The biggest cost seems to be git-write-tree, which is about 0.225 seconds
for me on that tree on that machine. Which _should_ mean that we could do
4 commits a second, but that sure as hell ain't how it works out. It seems
to do about 1.71 commits a second for me on that tree, which is pretty
damn pitiful. Some cvs overhead, and probably some other git overhead too.
Well, we _have_ to fetch the file. I guess you are thinking of
extracting if frrom the RCS ,v file directly? One tihng that I found
that seemed to speed things up a bit was to declare TMPDIR to be a
directory in the same partition.
(That's a 2GHz Merom, so the fact that you get ~6k commits per hour on
your 2GHz Opteron is about the same speed - I suspect you're also at least
partly limited by disk, our numbers seem to match pretty well).
Yup. This is _very_ diskbound.
200k commits at 6k commits per hour is about a day and a half (plus the
occasional packing load). Taking that long to import a CVS archive is
horrible. But I guess it _is_ several years of work, and I guess you
really have to do it only once, but still.
The latest stable CVS release is 1.11.21, I think: you seem to be running
the "development" version (1.12.x).
Backed down to the 1.11 series, things seem to be going fine so far.
Finally hit an OOM sometime in the past day (yep, a week later) =\. Not
sure whether it was cvsimport or cvs. Anyone else had more luck?
Thanks,
Donnie
From: Martin Langhoff <hidden> Date: 2016-06-15 22:42:27
On 5/30/06, Donnie Berkholz [off-list ref] wrote:
Donnie Berkholz wrote:
quoted
Linus Torvalds wrote:
quoted
The latest stable CVS release is 1.11.21, I think: you seem to be running
the "development" version (1.12.x).
Backed down to the 1.11 series, things seem to be going fine so far.
Finally hit an OOM sometime in the past day (yep, a week later) =\. Not
sure whether it was cvsimport or cvs. Anyone else had more luck?
It seemed like it had finished on the machine I was running it, and I
assumed it was alright in yours too. Looking closer it only made it
till April 2004 -- but it may have been killed by a sysadmin, the
captured log talks about 'signal 9', I have no idea what the OOM
sends.
It had done 285070 of 343822 patchsets.
Have you dropped the -a from the git-repack invocation? That should
help. Try also Linus' patch for git-rev-list. The other thing hurting
us is that the commits are _huge_. I wonder how you guys were managing
this with CVS. Now _this_ explains why cvsimport grows humongous.
I'll try to rework the commit loop so that we don't need to hold all
the filenames in memory. It seems to be choking with the commits after
April 2004. But that will have to wait till tonight.
cheers,
martin
Finally hit an OOM sometime in the past day (yep, a week later) =\. Not
sure whether it was cvsimport or cvs. Anyone else had more luck?
It seemed like it had finished on the machine I was running it, and I
assumed it was alright in yours too. Looking closer it only made it
till April 2004 -- but it may have been killed by a sysadmin, the
captured log talks about 'signal 9', I have no idea what the OOM
sends.
Looking closer, I see that the memory suckers do appear to be git, from
dmesg:
Out of Memory: Kill process 17230 (git-repack) score 97207 and children.
Out of memory: Killed process 17231 (git-rev-list).
Just ends like this:
Tree ID 2cc632e5e1d3a430a2cc891bf33c4a12f19a4d0e
Parent ID ad92d7073a52458e0581633bbd8ccbbec838d9e6
Committed patch 249100 (origin 2005-08-20 05:05:58)
Commit ID 28941f00d714f57ab49f1fd725d1c3ce8a5d0b93
Fetching sys-kernel/ck-sources/ChangeLog v 1.113
Update sys-kernel/ck-sources/ChangeLog: 25425 bytes
Fetching sys-kernel/ck-sources/Manifest v 1.164
Update sys-kernel/ck-sources/Manifest: 252 bytes
Delete sys-kernel/ck-sources/ck-sources-2.6.12_p5-r1.ebuild
Fetching sys-kernel/ck-sources/ck-sources-2.6.12_p6.ebuild v 1.1
New sys-kernel/ck-sources/ck-sources-2.6.12_p6.ebuild: 1438 bytes
Delete sys-kernel/ck-sources/files/digest-ck-sources-2.6.12_p5-r1
Fetching sys-kernel/ck-sources/files/digest-ck-sources-2.6.12_p6 v 1.1
New sys-kernel/ck-sources/files/digest-ck-sources-2.6.12_p6: 279 bytes
Can't fork at /usr/bin/git-cvsimport line 592, <CVS> line 3810053.
cat: write error: Broken pipe
It had done 285070 of 343822 patchsets.
Have you dropped the -a from the git-repack invocation? That should
help. Try also Linus' patch for git-rev-list. The other thing hurting
us is that the commits are _huge_. I wonder how you guys were managing
this with CVS. Now _this_ explains why cvsimport grows humongous.
I wasn't running with a version that did repacks; I just suspended the
cvsimport a couple of times and ran a repack manually.
I'll try to rework the commit loop so that we don't need to hold all
the filenames in memory. It seems to be choking with the commits after
April 2004. But that will have to wait till tonight.
From: Martin Langhoff <hidden> Date: 2016-06-15 22:42:27
On 5/30/06, Donnie Berkholz [off-list ref] wrote:
Looking closer, I see that the memory suckers do appear to be git, from
dmesg:
Out of Memory: Kill process 17230 (git-repack) score 97207 and children.
Out of memory: Killed process 17231 (git-rev-list).
That would mean that you do have Linus' patch then. Grep cvsimport for
repack and remove the -a -- and consider using his recent patch to
rev-list.
My dmesg talks about an earlier cvs segfault. Nasty tree you have here
-- it's breaking all sorts of things... and teaching us a thing or two
about the import process.
Hmmm? How can you be at patch 249100 and still be a good year ahead of
me? Have you told cvsps to cut off old history?
Another thing I found is that this import uses a lot of $TMPDIR, so if
your TMPDIR is small, you'll hit all sorts of problems.
cheers,
martin
Looking closer, I see that the memory suckers do appear to be git, from
dmesg:
Out of Memory: Kill process 17230 (git-repack) score 97207 and children.
Out of memory: Killed process 17231 (git-rev-list).
Sounds like you had the "git repack -a -d" thing in your cvsimport.
The current git rev-list should use only about a third of the memory of
the one you used, so hopefully you could just update your git version, and
then continue with the "git cvsimport" without having to start all over.
Linus
Looking closer, I see that the memory suckers do appear to be git, from
dmesg:
Out of Memory: Kill process 17230 (git-repack) score 97207 and children.
Out of memory: Killed process 17231 (git-rev-list).
That would mean that you do have Linus' patch then. Grep cvsimport for
repack and remove the -a -- and consider using his recent patch to
rev-list.
You certainly would think so, and I did as well, but available evidence
indicates otherwise. I'm not sure how the repack got in there.
donnie@supernova ~ $ type git-cvsimport
git-cvsimport is /usr/bin/git-cvsimport
donnie@supernova ~ $ grep repack /usr/bin/git-cvsimport
donnie@supernova ~ $
All I can think of is that I somehow OOM'd when I manually ran a repack
and didn't notice it. But that should've at least made me unable to
resume the cvsimport process, which happily kept chugging along later on.
My dmesg talks about an earlier cvs segfault. Nasty tree you have here
-- it's breaking all sorts of things... and teaching us a thing or two
about the import process.
From: Martin Langhoff <hidden> Date: 2016-06-15 22:42:28
On 5/30/06, Donnie Berkholz [off-list ref] wrote:
All I can think of is that I somehow OOM'd when I manually ran a repack
and didn't notice it. But that should've at least made me unable to
resume the cvsimport process, which happily kept chugging along later on.
Sounds likely -- and cvsimport restarts gracefully, though you might want to do
git checkout HEAD
to get a usable checkout if the very first import failed. However, the
default head is master, and what you want to look at is origin or
whatever you passed as your -o parameter. I use cvshead normally, so I
do
git log cvshead
quoted
My dmesg talks about an earlier cvs segfault. Nasty tree you have here
-- it's breaking all sorts of things... and teaching us a thing or two
about the import process.
Hmmm? How can you be at patch 249100 and still be a good year ahead of
me? Have you told cvsps to cut off old history?
Nope. I ran the exact cvsps flags you posted earlier to create it.
Oh, that was an earlier PEBKAK at my end: I did git log HEAD instead
of git log cvshead. My import is now at 293145 (cvshead +0000
2005-12-25 12:24:42) which looks promising.
cheers,
martin
Heh. I think you should enable caching in your apache config.
And maybe we should make that part of the gitweb docs. Without a caching
web-server, gitweb is pretty slow, but it caches _beautifully_.
That gentoo repo has a lot of "duplicate" commits that cvsps will mark as
two separate commits because there's one commit for the files, and one
commit for whatever the "Manifest" file is. I wonder if those commits
should generally be merged or something.
That said, things like that are most easily fixed as a git->git update
(along with adding name translation), which can avoid re-writing the
trees.
Linus
Heh. I think you should enable caching in your apache config.
I know I should -- but I'm hoping to find the time to rework gitweb a
bit to actually work fast instead. It bothers me that it is so slow on
a basically idle machine, and where I can perform the corresponding
git operations in the commandline in a blink.
And caching is great for really busy sites (aka kernel.org) but
git.catalyst.net.nz only serves a handful of small repos for a small
group of people, and is 99% idle. Should blaze through this stuff.
That gentoo repo has a lot of "duplicate" commits that cvsps will mark as
two separate commits because there's one commit for the files, and one
commit for whatever the "Manifest" file is. I wonder if those commits
should generally be merged or something.
That said, things like that are most easily fixed as a git->git update
(along with adding name translation), which can avoid re-writing the
trees.
Yep, large projects often have good reasons to run custom imports,
merging certain commits, rewriting log messages (like the X.org guys
were doing). It can be done at the cvsimport stage or later -- I think
Pasky has a rewritehistory tool hidden somewhere in Cogito, but I
haven't used it.
cheers,
martin
That gentoo repo has a lot of "duplicate" commits that cvsps will mark as
two separate commits because there's one commit for the files, and one
commit for whatever the "Manifest" file is. I wonder if those commits
should generally be merged or something.
That said, things like that are most easily fixed as a git->git update
(along with adding name translation), which can avoid re-writing the
trees.
Yep, large projects often have good reasons to run custom imports,
merging certain commits, rewriting log messages (like the X.org guys
were doing). It can be done at the cvsimport stage or later -- I think
Pasky has a rewritehistory tool hidden somewhere in Cogito, but I
haven't used it.
We've got a guy who got a Summer of Code project to work on CVS
migration, so this could be something along his lines.
Thanks,
Donnie
From: Alec Warner <hidden> Date: 2016-06-15 22:42:28
Martin Langhoff wrote:
On 5/31/06, Donnie Berkholz [off-list ref] wrote:
quoted
We've got a guy who got a Summer of Code project to work on CVS
migration, so this could be something along his lines.
He'll want a fast box to wrangle with this repo ;-)
martin
I have a dual opteron with 4gb of ram "on loan" from work :)
It still dies though, using git cvsimport or parsecvs.
I talked to Keith Packard about adding support to parsecvs for recording
the actual changed changesets, but I haven't yet started on implementing
that since he isn't using cvsps in parsecvs.
I also haven't had a chance to look at the git-cvsimport sources yet,
was hoping to get to that later this week.
From: Martin Langhoff <hidden> Date: 2016-06-15 22:42:28
On 6/1/06, Alec Warner [off-list ref] wrote:
I have a dual opteron with 4gb of ram "on loan" from work :)
It still dies though, using git cvsimport or parsecvs.
The machine I am running this is more constrained than that, and it
doesn't die. It just takes maybe 30hs. Make sure it's not a bad cvs
binary you got there (latest from gentoo seems to leak memory).
And if it's still dying... give us some more details ;-)
cheers,
martin
From: Alec Warner <hidden> Date: 2016-06-15 22:42:28
Martin Langhoff wrote:
On 6/1/06, Alec Warner [off-list ref] wrote:
quoted
I have a dual opteron with 4gb of ram "on loan" from work :)
It still dies though, using git cvsimport or parsecvs.
The machine I am running this is more constrained than that, and it
doesn't die. It just takes maybe 30hs. Make sure it's not a bad cvs
binary you got there (latest from gentoo seems to leak memory).
And if it's still dying... give us some more details ;-)
cheers,
martin
After reading the whole thread on this, I've using a git checkout of
git, cvsps-2.1 and cvs-1.11.12, running overnight in verbose mode with
screen. Hopefully will have a repo in the morning ;)
From: Martin Langhoff <hidden> Date: 2016-06-15 22:42:28
On 6/1/06, Alec Warner [off-list ref] wrote:
After reading the whole thread on this, I've using a git checkout of
git, cvsps-2.1 and cvs-1.11.12, running overnight in verbose mode with
screen. Hopefully will have a repo in the morning ;)
Good stuff. I am rerunning it to prove (and bench) a complete an
uninterrupted import. So far it's done 4hs 30m, footprint grown to
207MB, 49750 commits. So I think it will be done in approx 30hs on
this single-cpu opteron.
Most commits are small, but there is a handful that are downright
massive -- and we hold all the file list in memory, which I think
explains (most of) the memory growth. I've looked into avoiding
holding the whole filelist in memory, but it involves rewriting the
cvsps output parsing loop, which is better left for a rainy day, with
a test case that doesn't take 30hs to resolve.
cheers,
martin
From: Alec Warner <hidden> Date: 2016-06-15 22:42:28
Martin Langhoff wrote:
On 6/1/06, Alec Warner [off-list ref] wrote:
quoted
After reading the whole thread on this, I've using a git checkout of
git, cvsps-2.1 and cvs-1.11.12, running overnight in verbose mode with
screen. Hopefully will have a repo in the morning ;)
Good stuff. I am rerunning it to prove (and bench) a complete an
uninterrupted import. So far it's done 4hs 30m, footprint grown to
207MB, 49750 commits. So I think it will be done in approx 30hs on
this single-cpu opteron.
Most commits are small, but there is a handful that are downright
massive -- and we hold all the file list in memory, which I think
explains (most of) the memory growth. I've looked into avoiding
holding the whole filelist in memory, but it involves rewriting the
cvsps output parsing loop, which is better left for a rainy day, with
a test case that doesn't take 30hs to resolve.
Ok the box this was running on had issues, so I switched to using
pearl.amd64.dev.gentoo.org, a dual core amd64 X2 4600+ with 4 gigs of
ram and plenty of disk. The "problem" now is just converstion time...30
hours and I'm into 2004-09-17...but it's been in 2004 all day, seems
like most of the commits are in the last three years. Are there
architectural issues with doing this in parallel?
Since the repository commits are all in cvs, it should be possible to do
the work in parallel, since you know what all the commits touch. The
concern would be ordering of nodes in the tree; you'd end up building a
bunch of subtrees and patching them together?
-Alec Warner
From: Martin Langhoff <hidden> Date: 2016-06-15 22:42:28
On 6/5/06, Alec Warner [off-list ref] wrote:
Ok the box this was running on had issues, so I switched to using
pearl.amd64.dev.gentoo.org, a dual core amd64 X2 4600+ with 4 gigs of
ram and plenty of disk. The "problem" now is just converstion time...30
hours and I'm into 2004-09-17...but it's been in 2004 all day, seems
like most of the commits are in the last three years. Are there
architectural issues with doing this in parallel?
I don't think you can do this in parallel. What I would do is remove
the -a from the git-repack invocation. It does hurt import times quite
a bit -- just do a git-repack -a -d when it's done.
And... having said that, there is still a memory leak somehow,
somewhere. It's been evading me for 2 weeks now, so I feel an idiot
now. Not too bad in general, but it shows clearly in the gentoo and
mozilla imports.
Since the repository commits are all in cvs, it should be possible to do
the work in parallel, since you know what all the commits touch. The
concern would be ordering of nodes in the tree; you'd end up building a
bunch of subtrees and patching them together?
Well... parsecvs does a bit of this but in sequential fashion... it
imports all the files first, and then runs through the history
building the tree+commits in order, committing them. It saves a lot of
time in the file imports by parsing the RCS file directly. The
downside is that it must keep a filename+version=>sha1 mapping --
which I think is why parsecvs won't fit in memory until it's changed
to store it on disk somehow ;-)
You are forced to do it in a sequence because cvsps only tells you
about the files added/removed/changed in a commit -- you need the
ancestor to have a view of what the whole tree looked like. The only
room for parallelism I see is to fork off new processes to work on
branches in parallel.
martin
From: Alec Warner <hidden> Date: 2016-06-15 22:42:28
Martin Langhoff wrote:
On 6/5/06, Alec Warner [off-list ref] wrote:
quoted
Ok the box this was running on had issues, so I switched to using
pearl.amd64.dev.gentoo.org, a dual core amd64 X2 4600+ with 4 gigs of
ram and plenty of disk. The "problem" now is just converstion time...30
hours and I'm into 2004-09-17...but it's been in 2004 all day, seems
like most of the commits are in the last three years. Are there
architectural issues with doing this in parallel?
I don't think you can do this in parallel. What I would do is remove
the -a from the git-repack invocation. It does hurt import times quite
a bit -- just do a git-repack -a -d when it's done.
Only repack at the end then? disk space isn't an issue here so I'll give
that a shot.
And... having said that, there is still a memory leak somehow,
somewhere. It's been evading me for 2 weeks now, so I feel an idiot
now. Not too bad in general, but it shows clearly in the gentoo and
mozilla imports.
30565 antarus 17 0 470m 456m 1640 S 14 11.6 234:23.38
git-cvsimport
30566 antarus 16 0 6753m 147m 752 S 7 3.7 120:27.06 cvs
I'm on cvs-1.11.12 and the git version of git
You are forced to do it in a sequence because cvsps only tells you
about the files added/removed/changed in a commit -- you need the
ancestor to have a view of what the whole tree looked like. The only
room for parallelism I see is to fork off new processes to work on
branches in parallel.
Not helpful in the Gentoo case, since we only have one branch; minus an
accident when a dev branched gentoo-x86 a while back ;)
I'll keep chugging on this one; it won't be the final import as I
haven't used the complete Authors file, so I will try the repacking
optimization next time I do an import.
-Alec Warner
From: Martin Langhoff <hidden> Date: 2016-06-15 22:42:28
On 6/5/06, Alec Warner [off-list ref] wrote:
quoted
I don't think you can do this in parallel. What I would do is remove
the -a from the git-repack invocation. It does hurt import times quite
a bit -- just do a git-repack -a -d when it's done.
Only repack at the end then? disk space isn't an issue here so I'll give
that a shot.
Not exactly -- by removing the -a from the git-repack invocation what
you get is cheap "partial" packing rather than a full repack. This is
somewhat inefficient disk-wise, perhaps by 10% or so. But full repacks
get more and more expensive as the repo grows.
So you don't need to run git-repack -a -d at the end, but it will be a
good measure to see how compact the packing gets.
quoted
And... having said that, there is still a memory leak somehow,
somewhere. It's been evading me for 2 weeks now, so I feel an idiot
now. Not too bad in general, but it shows clearly in the gentoo and
mozilla imports.
30565 antarus 17 0 470m 456m 1640 S 14 11.6 234:23.38
git-cvsimport
30566 antarus 16 0 6753m 147m 752 S 7 3.7 120:27.06 cvs
I'm on cvs-1.11.12 and the git version of git
Yep, I see roughly the same. It grows slowly and I don't know why :(
I'll keep chugging on this one; it won't be the final import as I
haven't used the complete Authors file, so I will try the repacking
optimization next time I do an import.
Cool. If it dies for any reason, just do
git-update-ref refs/heads/master refs/heads/origin
git-update-ref HEAD origin
git-checkout
You only need to do this the first time -- after that, the core heads
are set. Rerun the script and it will pick up where it left. If it
dies again, just do git-checkout to see the latest files.
(Above, replace origin with your -o option if you are using it. I
normally use -o cvshead.)
martin
On Sun, 04 Jun 2006 22:36:44 -0400
Alec Warner [off-list ref] wrote:
I'll keep chugging on this one; it won't be the final import as I
haven't used the complete Authors file, so I will try the repacking
optimization next time I do an import.
Hi Alec,
You may want to go back and do another import for other reasons, but if
the only reason is to fix up the author information it would be _much_
faster to simply rewrite the git commit history. Cogito has something
called "cg-admin-rewritehist" which should do what you need and there
are other scripts floating around specificially for rewriting just the
author information.
HTH,
Sean