From: Kelly F. Hickel <hidden> Date: 2016-06-15 22:46:54
Hi all,
We're converting out of CVS after 10 years... The cvs2git
conversion takes around 4-5 days, and there doesn't seem to be any way
to speed that up. So, our current plan is to take the tip of the
branches that we need for the next week, and import each of those
branches into its own clean git repo (top-skim). Then we'll work out of
those repos while the conversion is in process (presumably creating
branches in those repos as needed). Once the conversion is finished,
we'll need to get all the work done in the top-skimmed repos merged into
the converted repo so that we end up with little to no developer down
time, and all of our history, pre and post conversion in one repo.
I'm testing all of this in advance, of course, and the tricky
part at the moment is how to "stitch" the commits from the top-skim
repos back onto the converted repo when the conversion is done. The
file content of the initial commit for the skimmed repo is identical to
the last commit for the respective branch in the converted repo, but the
SHA1s are different, presumably because the history of the content is
different.
Stated another way, I have two repositories, "new" and "old",
where the files in the initial commit on branch "B1" in "new" have
exactly the same content as the last commit on branch "B1" in "old".
There also exist various branches in "new" based on "B1". I'd like to
merge all the commits from "new" into "old", but the SHA1s are
different, presumably because the history leading up to those points are
different.
Other than using manually format-patch on every branch in new,
then applying the patches (presumably with regular old patch, since the
ancestor commit IDs won't match), is there any "good" way to merge "new"
into "old"?
Thanks,
--
Kelly F. Hickel
Senior Product Architect
MQSoftware, Inc.
952-345-8677 Office
952-345-8721 Fax
kfh@mqsoftware.com
www.mqsoftware.com
Certified IBM SOA Specialty
Your Full Service Provider for IBM WebSphere
Learn more at www.mqsoftware.com
On Fri, Jun 5, 2009 at 12:30 PM, Kelly F. Hickel [off-list ref] wrote:
Stated another way, I have two repositories, "new" and "old",
where the files in the initial commit on branch "B1" in "new" have
exactly the same content as the last commit on branch "B1" in "old".
There also exist various branches in "new" based on "B1". I'd like to
merge all the commits from "new" into "old", but the SHA1s are
different, presumably because the history leading up to those points are
different.
Other than using manually format-patch on every branch in new,
then applying the patches (presumably with regular old patch, since the
ancestor commit IDs won't match), is there any "good" way to merge "new"
into "old"?
The usual replacement for "manually using format-patch" is to use "git
rebase." It does pretty much exactly what you're describing, assuming
you don't do too many complicated merges in the meantime.
Another option is to use the .git/info/grafts file. Here's a brief
intro: http://git.or.cz/gitwiki/GraftPoint
You'd use that to pretend the parent of your top-skimmed branch is
actually the equivalent commit in your new branch. Then could run
"git filter-branch" to make the graft permanent, and get all your
users to switch to the new repository.
Or you could skip the filter-branch stuff and keep the really hold
history somewhere else, available for use if someone installs the
graft in their local repo. This would lead to a smaller repository in
the general case. (I gather that's what the Linux kernel does for
per-2.6.11 versions.)
Have fun,
Avery
From: Kelly F. Hickel <hidden> Date: 2016-06-15 22:46:54
-----Original Message-----
From: Rostislav Svoboda [mailto:rostislav.svoboda@gmail.com]
Sent: Friday, June 05, 2009 11:54 AM
To: Kelly F. Hickel
Cc: git@vger.kernel.org
Subject: Re: Best way to merge two repos with same content, different
history
On Fri, Jun 5, 2009 at 18:30, Kelly F. Hickel[off-list ref]
wrote:
quoted
We're converting out of CVS after 10 years... The cvs2git
conversion takes around 4-5 days, and there doesn't seem to be any
way
quoted
to speed that up.
try google:
"cvs2git migration - cloning CVS repository"
Bost
Bost,
Thanks, but I'm already working with a local copy of the CVS repo. I've corresponded with Michael Haggerty about the time this takes, and there just doesn't seem to be any way to improve the speed, without making some fairly drastic changes to cvs2git.
Thanks,
Kelly
From: Kelly F. Hickel <hidden> Date: 2016-06-15 22:46:54
-----Original Message-----
From: Avery Pennarun [mailto:apenwarr@gmail.com]
Sent: Friday, June 05, 2009 12:01 PM
To: Kelly F. Hickel
Cc: git@vger.kernel.org
Subject: Re: Best way to merge two repos with same content, different
history
On Fri, Jun 5, 2009 at 12:30 PM, Kelly F. Hickel [off-list ref]
wrote:
quoted
Stated another way, I have two repositories, "new" and "old",
where the files in the initial commit on branch "B1" in "new" have
exactly the same content as the last commit on branch "B1" in "old".
There also exist various branches in "new" based on "B1". I'd like
to
quoted
merge all the commits from "new" into "old", but the SHA1s are
different, presumably because the history leading up to those points
are
quoted
different.
Other than using manually format-patch on every branch in new,
then applying the patches (presumably with regular old patch, since
the
quoted
ancestor commit IDs won't match), is there any "good" way to merge
"new"
quoted
into "old"?
The usual replacement for "manually using format-patch" is to use "git
rebase." It does pretty much exactly what you're describing, assuming
you don't do too many complicated merges in the meantime.
Another option is to use the .git/info/grafts file. Here's a brief
intro: http://git.or.cz/gitwiki/GraftPoint
You'd use that to pretend the parent of your top-skimmed branch is
actually the equivalent commit in your new branch. Then could run
"git filter-branch" to make the graft permanent, and get all your
users to switch to the new repository.
Or you could skip the filter-branch stuff and keep the really hold
history somewhere else, available for use if someone installs the
graft in their local repo. This would lead to a smaller repository in
the general case. (I gather that's what the Linux kernel does for
per-2.6.11 versions.)
Have fun,
Avery
Thanks Avery,
This appears to be just what I was looking for! I'll fiddle with it a bit to see if I can convince it to work for me.
Thanks,
Kelly
From: Markus Heidelberg <hidden> Date: 2016-06-15 22:46:54
Kelly F. Hickel, 05.06.2009:
Other than using manually format-patch on every branch in new,
then applying the patches (presumably with regular old patch, since the
ancestor commit IDs won't match), is there any "good" way to merge "new"
into "old"?
If rebasing 'new' on top of 'old' isn't an option, then you could try:
$ git checkout new
$ git merge -s ours old
It's the other way round (not merging 'new' into 'old', but vice versa),
but there is now merge strategy "theirs".
Markus
On Fri, Jun 5, 2009 at 19:10, Kelly F. Hickel[off-list ref] wrote:
Bost,
Thanks, but I'm already working with a local copy of the CVS repo. I've corresponded with Michael Haggerty about the time this takes, and there just doesn't seem to be any way to improve the speed, without making some fairly drastic changes to cvs2git.
uhm, see below
Bost
On Tue, Feb 3, 2009 at 14:55, Johannes
Schindelin[off-list ref] wrote:
quoted
If you do not have filesystem access to your CVS repository, you might
be able to clone it using CVSSuck [2,3].
A substantially faster option would be to go with cvsclone:
http://samba.org/ftp/tridge/rtc/cvsclone.l
(in my case, cvsclone was not only faster, but it actually worked, too,
which is more than I could say of CVSSuck).
From: Robin H. Johnson <hidden> Date: 2016-06-15 22:46:54
On Fri, Jun 05, 2009 at 12:10:30PM -0500, Kelly F. Hickel wrote:
Bost,
Thanks, but I'm already working with a local copy of the CVS repo.
I've corresponded with Michael Haggerty about the time this takes,
and there just doesn't seem to be any way to improve the speed,
without making some fairly drastic changes to cvs2git.
I've been working with mhagger lately as it also pertains to the Gentoo
conversion. We've made some very good progress.
A couple of comments in that regard:
- Make really sure your box is not short of RAM. Throw some measurement
tools onto there to see it. A couple of GiB is worthwhile. After we
found this early on, and switched boxes, we dropped from our initial
multiple days to 20 hours.
- His latest ExternalBlobGenerator code (_NOT_ available in SVN yet)
reduced our pass1 time from 36204 seconds to 1598 seconds, with
a potential to be much faster now, as parallelization of part of that
is now trivial.
- pass9 is still the remaining large time-eater for us. I've started to
look at it, but I haven't made any actual developments yet.
Would you mind posting your cvs2svn stats like these?
http://archives.gentoo.org/gentoo-scm/msg_b69b2f6ecee0ec7bb402d31b372b945b.xml
--
Robin Hugh Johnson
Gentoo Linux Developer & Infra Guy
E-Mail : robbat2@gentoo.org
GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85
From: Kelly F. Hickel <hidden> Date: 2016-06-15 22:46:54
-----Original Message-----
From: git-owner@vger.kernel.org [mailto:git-owner@vger.kernel.org] On
Behalf Of Robin H. Johnson
Sent: Friday, June 05, 2009 1:46 PM
To: Git Mailing List
Subject: Re: Best way to merge two repos with same content,
differenthistory
On Fri, Jun 05, 2009 at 12:10:30PM -0500, Kelly F. Hickel wrote:
quoted
Bost,
Thanks, but I'm already working with a local copy of the CVS
repo.
quoted
I've corresponded with Michael Haggerty about the time this
takes,
quoted
and there just doesn't seem to be any way to improve the speed,
without making some fairly drastic changes to cvs2git.
I've been working with mhagger lately as it also pertains to the
Gentoo
conversion. We've made some very good progress.
A couple of comments in that regard:
- Make really sure your box is not short of RAM. Throw some
measurement
tools onto there to see it. A couple of GiB is worthwhile. After we
found this early on, and switched boxes, we dropped from our initial
multiple days to 20 hours.
- His latest ExternalBlobGenerator code (_NOT_ available in SVN yet)
reduced our pass1 time from 36204 seconds to 1598 seconds, with
a potential to be much faster now, as parallelization of part of
that
is now trivial.
- pass9 is still the remaining large time-eater for us. I've started
to
look at it, but I haven't made any actual developments yet.
Would you mind posting your cvs2svn stats like these?
http://archives.gentoo.org/gentoo-
scm/msg_b69b2f6ecee0ec7bb402d31b372b945b.xml
--
Robin Hugh Johnson
Gentoo Linux Developer & Infra Guy
E-Mail : robbat2@gentoo.org
GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85
Robin,
That's all good news, I have an 8 way box with 32gb of ram
running a 64 bit Linux, a box with 4 gb of ram panics during the
conversion.
My conversion data is below...
Thanks,
Kelly
cvs2svn Statistics:
------------------
Total CVS Files: 18488
Total CVS Revisions: 225208
Total CVS Branches: 15203751
Total CVS Tags: 39079236
Total Unique Tags: 11453
Total Unique Branches: 4364
CVS Repos Size in KB: 3355895
Total SVN Commits: 49967
First Revision Date: Mon Nov 8 02:26:51 1999
Last Revision Date: Wed Apr 22 17:59:44 2009
------------------
Timings (seconds):
------------------
251546 pass1 CollectRevsPass
4 pass2 CleanMetadataPass
142 pass3 CollateSymbolsPass
53491 pass4 FilterSymbolsPass
4 pass5 SortRevisionSummaryPass
163 pass6 SortSymbolSummaryPass
8825 pass7 InitializeChangesetsPass
418 pass8 BreakRevisionChangesetCyclesPass
418 pass9 RevisionTopologicalSortPass
4256 pass10 BreakSymbolChangesetCyclesPass
4914 pass11 BreakAllChangesetCyclesPass
4575 pass12 TopologicalSortPass
3111 pass13 CreateRevsPass
270 pass14 SortSymbolsPass
154 pass15 IndexSymbolsPass
5517 pass16 OutputPass
337808 total
251783.89user 80800.42system 93:50:11elapsed 98%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (3major+3132264023minor)pagefaults 0swaps
git-fast-import statistics:
---------------------------------------------------------------------
Alloc'd objects: 415000
Total objects: 410079 ( 2628078 duplicates )
blobs : 152002 ( 68715 duplicates 135125 deltas)
trees : 213636 ( 2559363 duplicates 164052 deltas)
commits: 44441 ( 0 duplicates 0 deltas)
tags : 0 ( 0 duplicates 0 deltas)
Total branches: 15822 ( 6184 loads )
marks: 1073741824 ( 265158 unique )
atoms: 14807
Memory total: 32402 KiB
pools: 16192 KiB
objects: 16210 KiB
---------------------------------------------------------------------
pack_report: getpagesize() = 4096
pack_report: core.packedGitWindowSize = 1073741824
pack_report: core.packedGitLimit = 8589934592
pack_report: pack_used_ctr = 311525
pack_report: pack_mmap_calls = 13303
pack_report: pack_open_windows = 1 / 1
pack_report: pack_mapped = 403041230 / 403041230
---------------------------------------------------------------------
git repack -a -d -f --depth=4000 --window=4000 && git pack-refs --all
Counting objects: 409458, done.5/119582)
Compressing objects: 100% (119582/119582), done.
Writing objects: 100% (128713/128713), done.
Total 128713 (delta 82330), reused 0 (delta 0)
Compressing objects: 100% (384171/384171), done.
Writing objects: 100% (409458/409458), done.
Total 409458 (delta 309214), reused 0 (delta 0)
From: Robin H. Johnson <hidden> Date: 2016-06-15 22:46:54
On Fri, Jun 05, 2009 at 02:06:25PM -0500, Kelly F. Hickel wrote:
Robin,
That's all good news, I have an 8 way box with 32gb of ram
running a 64 bit Linux, a box with 4 gb of ram panics during the
conversion.
Thanks for your data.
For comparison, our conversion box is also 8-way, but only 16GiB RAM.
I'm surprised at how long pass1 is for you, especially since you've got
a lot less CVS Files and CVS Revisions than the Gentoo repo (I do deduce
that your individual revisions are larger, averaging at 15KiB vs. our
711 bytes).
I think there's something odd in the total CVS branches/tags count
however, as the counts there imply an average of 67 branches and 173
tags per CVS revision. You might want to dig into that part manually and
see about it (not sure of your Python skills). That would probably cut
down both your pass1 and pass4 times significantly.
Hopefully mhagger will get the external blob stuff committed soon, I was
working on validating it's results.
In doing so discovered a testcase where RCSRevisionReader and
CVSRevisionReader gave different output themselves, the latter (which is
documented as more accurate otherwise) missing the contents of an entire
file. It's on the cvs2svn-dev mailing list now. Tracing that first,
thereafter comparing it to the new Git side.
Did those extreme depth/window values actually help size much? The
Gentoo ones actually didn't improve significantly over depth=window=50.
--
Robin Hugh Johnson
Gentoo Linux Developer & Infra Guy
E-Mail : robbat2@gentoo.org
GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85
From: Kelly F. Hickel <hidden> Date: 2016-06-15 22:46:54
-----Original Message-----
From: git-owner@vger.kernel.org [mailto:git-owner@vger.kernel.org] On
Behalf Of Robin H. Johnson
Sent: Friday, June 05, 2009 3:02 PM
To: Git Mailing List
Subject: Re: Best way to merge two repos with same content,
differenthistory
On Fri, Jun 05, 2009 at 02:06:25PM -0500, Kelly F. Hickel wrote:
quoted
Robin,
That's all good news, I have an 8 way box with 32gb of ram
running a
quoted
64 bit Linux, a box with 4 gb of ram panics during the conversion.
Thanks for your data.
For comparison, our conversion box is also 8-way, but only 16GiB RAM.
I'm surprised at how long pass1 is for you, especially since you've
got
a lot less CVS Files and CVS Revisions than the Gentoo repo (I do
deduce
that your individual revisions are larger, averaging at 15KiB vs. our
711 bytes).
I think there's something odd in the total CVS branches/tags count
however, as the counts there imply an average of 67 branches and 173
tags per CVS revision. You might want to dig into that part manually
and
see about it (not sure of your Python skills). That would probably cut
down both your pass1 and pass4 times significantly.
Robin, I'm not much with python, so haven't dug into the code much at
all. The numbers are high, although we do create a lot of branches (had
to contribute a fix a year or two to CVS to get the branching time down
from the 2.5 hours it was taking). At one point I carefully examined
the symbol file that cvs2git was outputting and convinced myself that it
was doing the right thing, but that was awhile ago.
Hopefully mhagger will get the external blob stuff committed soon, I
was
working on validating it's results.
In doing so discovered a testcase where RCSRevisionReader and
CVSRevisionReader gave different output themselves, the latter (which
is
documented as more accurate otherwise) missing the contents of an
entire
file. It's on the cvs2svn-dev mailing list now. Tracing that first,
thereafter comparing it to the new Git side.
quoted
git repack -a -d -f --depth=4000 --window=4000 && git pack-refs
--all
Did those extreme depth/window values actually help size much? The
Gentoo ones actually didn't improve significantly over
depth=window=50.
I know that they were still (apparently) improving after the 200 mark,
it took long enough at 200 that I just decided to crank the numbers way
up and let it run over the weekend.
--
Robin Hugh Johnson
Gentoo Linux Developer & Infra Guy
E-Mail : robbat2@gentoo.org
GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85
I'll be looking forward to a newer faster cvs2git, although I did just
get the graft idea working, so not sure if we'll wait that long or not
(would be nice not to have to muck around with it though).
Thanks,
Kelly
From: Michael Haggerty <hidden> Date: 2016-06-15 22:46:58
Robin H. Johnson wrote:
[Regarding cvs2git] In doing so discovered a testcase where RCSRevisionReader and
CVSRevisionReader gave different output themselves, the latter (which is
documented as more accurate otherwise) missing the contents of an entire
file. It's on the cvs2svn-dev mailing list now. Tracing that first,
thereafter comparing it to the new Git side.
In case anybody is following this, the issue that Robin found was in
release 2.2.0 of cvs2svn/cvs2git, but had already been fixed in the
current trunk version...
Conclusion 1: Please use the trunk version of cvs2git, not release
2.2.0. Trunk is usually the most stable version available, especially
for 2git conversions.
Conclusion 2: I've really got to get around to making a new release :-/
Michael