From: Michael Haggerty <hidden> Date: 2016-06-15 22:43:25
[I am CCing this response to the mailing lists.]
Guilhem Bonnefille wrote:
On 8/1/07, Michael Haggerty [off-list ref] wrote:
quoted
I am the maintainer of cvs2svn[1], which is a program for one-time
conversions from CVS to Subversion. cvs2svn is very robust against the
many peculiarities of CVS and can convert just about every CVS
repository we have ever seen.
I'm not extremely familiar with cvsps, and I don't really want to get
into a "my-tool-is-better-than-your-tool" kind of argument. Instead I
will mention that the goals of the two projects are somewhat different:
cvs2svn is meant for one-time conversions from CVS, and therefore aims
for maximum conversion accuracy, robustness even in the presence of some
kinds of CVS repository corruption, intelligent translation of CVS
idioms to the idioms of a modern SCM, and scalability to large
repositories (by using on-disk databases instead of RAM for intermediate
data). Conversion speed is not a primary goal of cvs2svn, and
incremental conversions are not supported at all. cvs2svn requires
filesystem access to the CVS repository (it parses the RCS files directly).
cvsps is not a conversion tool at all, though it is used by other
conversion tools to generate the changesets. It appears (I hope I am
not misinterpreting things) to emphasize speed and incremental
operation, for example attempting to make changesets consistent from one
run to the next, even if the CVS repository has been changed prudently
between runs. cvsps does not appear to attempt to create atomic branch
and tag creation commits or handle CVS's special vendorbranch behavior.
cvsps operates via the CVS protocol; you don't need filesystem access
to the CVS repository.
I can also point you to a list of cvs2svn features, which includes a
list of some of the CVS quirks that it knows how to handle:
http://cvs2svn.tigris.org/cvs2svn.html#features
cvs2svn includes a large suite of perverse CVS repositories that we use
for testing. Many of them are derived from real-life CVS repositories
that people have had problems with. It would be very interesting to see
how other conversion tools handle these repositories, but I don't expect
to have time to do so in the near future.
Michael
From: Martin Langhoff <hidden> Date: 2016-06-15 22:43:25
On 8/3/07, Michael Haggerty [off-list ref] wrote:
cvsps is not a conversion tool at all, though it is used by other
conversion tools to generate the changesets. It appears (I hope I am
not misinterpreting things) to emphasize speed and incremental
operation, for example attempting to make changesets consistent from one
run to the next, even if the CVS repository has been changed prudently
between runs. cvsps does not appear to attempt to create atomic branch
and tag creation commits or handle CVS's special vendorbranch behavior.
cvsps operates via the CVS protocol; you don't need filesystem access
to the CVS repository.
100% in agreement. And though I can't claim to be happy with cvsps, in
many scenarios it is mighty useful, in spite of its significant warts.
The "does incrementals" is hugely important these days, as lots of
people use git to run "vendor branches" of upstream projects that use
CVS.
To me, that's *the* killer-app feature of git. Of course, others see
different aspects of git as their deal-maker. But I'm sure I'm not
alone on this. Surely enough, others have written git-svn which
accomplishes this and more for those tracking SVN upstreams.
Is there any way we can run tweak cvs2svn to run incrementals, even if
not as fast as cvsps/git-cvsimport? The "do it remotely" part can be
worked around in most cases.
cheers,
martin
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:43:26
Hi,
On Fri, 3 Aug 2007, Martin Langhoff wrote:
On 8/3/07, Michael Haggerty [off-list ref] wrote:
quoted
cvsps is not a conversion tool at all, though it is used by other
conversion tools to generate the changesets. It appears (I hope I am
not misinterpreting things) to emphasize speed and incremental
operation, for example attempting to make changesets consistent from one
run to the next, even if the CVS repository has been changed prudently
between runs. cvsps does not appear to attempt to create atomic branch
and tag creation commits or handle CVS's special vendorbranch behavior.
cvsps operates via the CVS protocol; you don't need filesystem access
to the CVS repository.
100% in agreement. And though I can't claim to be happy with cvsps, in
many scenarios it is mighty useful, in spite of its significant warts.
The "does incrementals" is hugely important these days, as lots of
people use git to run "vendor branches" of upstream projects that use
CVS.
Me too: 100% agreement. A couple of people seem to be content to proclaim
that their incomplete solutions are better, but in the end of the day,
they are as bad as the programs they purport to replace: incomplete.
For the moment, I help myself with tracking the different branches
individually, but there, really, git-cvsimport is as good as the other
"solutions", with the further advantage that they are actually hackable,
and not closed to everybody outside a very small community.
So I look forward to testing cvs2svn(git-branch) this weekend.
Ciao,
Dscho
On Aug 3, 2007, at 6:03 AM, Johannes Schindelin wrote:
On Fri, 3 Aug 2007, Martin Langhoff wrote:
quoted
On 8/3/07, Michael Haggerty [off-list ref] wrote:
quoted
cvsps is not a conversion tool at all, though it is used by other
conversion tools to generate the changesets. It appears (I hope
I am
not misinterpreting things) to emphasize speed and incremental
operation, for example attempting to make changesets consistent
from one
run to the next, even if the CVS repository has been changed
prudently
between runs. cvsps does not appear to attempt to create atomic
branch
and tag creation commits or handle CVS's special vendorbranch
behavior.
cvsps operates via the CVS protocol; you don't need filesystem
access
to the CVS repository.
100% in agreement. And though I can't claim to be happy with
cvsps, in
many scenarios it is mighty useful, in spite of its significant
warts.
The "does incrementals" is hugely important these days, as lots of
people use git to run "vendor branches" of upstream projects that use
CVS.
Me too: 100% agreement. A couple of people seem to be content to
proclaim
that their incomplete solutions are better, but in the end of the day,
they are as bad as the programs they purport to replace: incomplete.
For the moment, I help myself with tracking the different branches
individually, but there, really, git-cvsimport is as good as the other
"solutions", with the further advantage that they are actually
hackable,
and not closed to everybody outside a very small community.
I just want to add a warning. You should be suspicious of branched
imported
using git-cvsimport (which is based on cvsps). If the time the branch is
created differs from the time of the first commit to the branch git-
cvsimport
may get the branching point wrong. This introduces a race condition.
Someone
may have committed changes to a file that is later changed on the
branch. At
that point the history of the imported branch is broken and git reports
_wrong_ changesets.
I ran into this issue and abandoned the use of git-cvsimport. It's
too dangerous
for me. The testcase in [1] illustrates the problem. I still strongly
believe
the warning should be stated in *BOLD* in the documentation.
I'm not saying git-cvsimport is useless. But you should be suspicious
about
the result of the import, especially if you plan to rely on
changesets derived
from the imported repo, for example if you plan to do cherry-picking
or merging
in git; or if you plan to blame people for their stupid changes based
on what
you see in gitk (almost happend to me ;).
Steffen
[1] http://marc.info/?l=git&m=118260312708709&w=2
On Aug 3, 2007, at 1:08 AM, Martin Langhoff wrote:
Is there any way we can run tweak cvs2svn to run incrementals, even if
not as fast as cvsps/git-cvsimport? The "do it remotely" part can be
worked around in most cases.
What I currently do with parsecvs is to run complete imports again
on the repo. For 'normal' changes to cvs the old import can be fast
forwarded to the new import. However, if you add or remove files or
tweak revision in another abnormal way (cvs admin) this might fail.
In this case I manually search the last common commit and rebase
new commits to the old, already imported branch. I need to do this if
I already publishes the imported branch. Otherwise I can as well just
reset to the newly imported branch and rebase my work on top of it.
Some careful validation (git diff-*) is included in my workflow.
A complete run of parsecvs is fine for me because it is so fast. I run
git-filter-branch afterwards anyway to cleanup some commit messages
and author information. This takes most of the time, because it spawns
off tons of sub processes.
I'd not recommend my approach for incremental imports every hour, but
you can run it every day (although I do less often). You only need to
validate the final result (fast forward or not). The rest can be fully
automated by some shell scripting.
Steffen
From: Michael Haggerty <hidden> Date: 2016-06-15 22:43:26
Martin Langhoff wrote:
Is there any way we can run tweak cvs2svn to run incrementals, even if
not as fast as cvsps/git-cvsimport? The "do it remotely" part can be
worked around in most cases.
I don't see any fundamental reason why not, but I think it would be a
significant amount of work. There are two main issues:
1. With CVS, it is possible to change things retroactively, such as
changing which version of a file is included in a tag, or adding a new
file to a tag, or changing whether a file is text vs. binary. And many
people copy and/or rename files within the CVS repository itself (to get
around CVS's inability to rename a file). This makes it look like the
file has *always* existed under the new name and *never* existed under
the old name. An incremental conversion tool would have to look
carefully for such changes and either handle them properly or complain
loudly and abort.
2. cvs2svn uses a lot of repository-wide information to make decisions
about how to group CVSItems into changesets, and a lot of these
decisions are based on heuristics. Incremental conversion would require
that the decisions made in one cvs2svn run are recorded and treated as
unalterable in subsequent runs.
This hasn't been a priority in the Subversion world, because, frankly,
what reason would a person have to stick with CVS instead of switching
to Subversion, given that (1) they are intentionally so similar in
workflow, an (2) there is no significant competition from other
centralized SCMs? But of course until the distributed SCM playing field
has been thinned out a bit, people will probably be reluctant to commit
to one or the other.
I don't expect to have time to implement incremental conversions in
cvs2svn in the near future. (I'd much rather work on output back ends
to other distributed SCMs.) But if any volunteers step forward (hint,
hint) I would be happy to help them get started and answer their
questions. I think that cvs2svn is quite hackable now, so the learning
curve is hopefully much less frightening than when I started on the
project :-)
Michael
Hello Michael,
I will explain a scenario (we are passing thru this right now)
1) you have 10 years worth of cvs data.
2) We want to move to svn.
3) The repository move should be in such a way that the development does
not get hampered for any 1 work day.
4) We have atleast 4 major modules in cvs which takes about 30 - 40
hours each for conversion currently.
5) With increamental conversions we can do a few things ...
A) Keep the downtime for hard cutoff minimal
B) try out the svn move for other auxillary tools that are
needed by the SCM process.
C) Do some meaningful testing and validation with simulated live
moves of changes from cvs to svn before the actual move on a day to day
basis.
Hopefuly this would substantiate the request \ need for increamental
moves. Or if someone out there has a better suggestion for such
scenario's please point me in the right direction.
Regards,
Rajesh
-----Original Message-----
From: Michael Haggerty [mailto:mhagger@alum.mit.edu]
Sent: Friday, August 03, 2007 1:36 AM
To: Martin Langhoff
Cc: Guilhem Bonnefille; git@vger.kernel.org; users@cvs2svn.tigris.org
Subject: Re: cvs2svn conversion directly to git ready for
experimentation
Martin Langhoff wrote:
Is there any way we can run tweak cvs2svn to run incrementals, even if
not as fast as cvsps/git-cvsimport? The "do it remotely" part can be
worked around in most cases.
I don't see any fundamental reason why not, but I think it would be a
significant amount of work. There are two main issues:
1. With CVS, it is possible to change things retroactively, such as
changing which version of a file is included in a tag, or adding a new
file to a tag, or changing whether a file is text vs. binary. And many
people copy and/or rename files within the CVS repository itself (to get
around CVS's inability to rename a file). This makes it look like the
file has *always* existed under the new name and *never* existed under
the old name. An incremental conversion tool would have to look
carefully for such changes and either handle them properly or complain
loudly and abort.
2. cvs2svn uses a lot of repository-wide information to make decisions
about how to group CVSItems into changesets, and a lot of these
decisions are based on heuristics. Incremental conversion would require
that the decisions made in one cvs2svn run are recorded and treated as
unalterable in subsequent runs.
This hasn't been a priority in the Subversion world, because, frankly,
what reason would a person have to stick with CVS instead of switching
to Subversion, given that (1) they are intentionally so similar in
workflow, an (2) there is no significant competition from other
centralized SCMs? But of course until the distributed SCM playing field
has been thinned out a bit, people will probably be reluctant to commit
to one or the other.
I don't expect to have time to implement incremental conversions in
cvs2svn in the near future. (I'd much rather work on output back ends
to other distributed SCMs.) But if any volunteers step forward (hint,
hint) I would be happy to help them get started and answer their
questions. I think that cvs2svn is quite hackable now, so the learning
curve is hopefully much less frightening than when I started on the
project :-)
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cvs2svn.tigris.org
For additional commands, e-mail: users-help@cvs2svn.tigris.org
From: Jon Smirl <hidden> Date: 2016-06-15 22:43:26
On 8/3/07, Patwardhan, Rajesh [off-list ref] wrote:
Hello Michael,
I will explain a scenario (we are passing thru this right now)
1) you have 10 years worth of cvs data.
2) We want to move to svn.
3) The repository move should be in such a way that the development does
not get hampered for any 1 work day.
4) We have atleast 4 major modules in cvs which takes about 30 - 40
hours each for conversion currently.
There are known ways (that haven't been implemented) to get the 40 hr
number down to 1/2 hour. Would that be a better approach than doing
incremental imports?
5) With increamental conversions we can do a few things ...
A) Keep the downtime for hard cutoff minimal
B) try out the svn move for other auxillary tools that are
needed by the SCM process.
C) Do some meaningful testing and validation with simulated live
moves of changes from cvs to svn before the actual move on a day to day
basis.
Hopefuly this would substantiate the request \ need for increamental
moves. Or if someone out there has a better suggestion for such
scenario's please point me in the right direction.
Regards,
Rajesh
-----Original Message-----
From: Michael Haggerty [mailto:mhagger@alum.mit.edu]
Sent: Friday, August 03, 2007 1:36 AM
To: Martin Langhoff
Cc: Guilhem Bonnefille; git@vger.kernel.org; users@cvs2svn.tigris.org
Subject: Re: cvs2svn conversion directly to git ready for
experimentation
Martin Langhoff wrote:
quoted
Is there any way we can run tweak cvs2svn to run incrementals, even if
quoted
not as fast as cvsps/git-cvsimport? The "do it remotely" part can be
worked around in most cases.
I don't see any fundamental reason why not, but I think it would be a
significant amount of work. There are two main issues:
1. With CVS, it is possible to change things retroactively, such as
changing which version of a file is included in a tag, or adding a new
file to a tag, or changing whether a file is text vs. binary. And many
people copy and/or rename files within the CVS repository itself (to get
around CVS's inability to rename a file). This makes it look like the
file has *always* existed under the new name and *never* existed under
the old name. An incremental conversion tool would have to look
carefully for such changes and either handle them properly or complain
loudly and abort.
2. cvs2svn uses a lot of repository-wide information to make decisions
about how to group CVSItems into changesets, and a lot of these
decisions are based on heuristics. Incremental conversion would require
that the decisions made in one cvs2svn run are recorded and treated as
unalterable in subsequent runs.
This hasn't been a priority in the Subversion world, because, frankly,
what reason would a person have to stick with CVS instead of switching
to Subversion, given that (1) they are intentionally so similar in
workflow, an (2) there is no significant competition from other
centralized SCMs? But of course until the distributed SCM playing field
has been thinned out a bit, people will probably be reluctant to commit
to one or the other.
I don't expect to have time to implement incremental conversions in
cvs2svn in the near future. (I'd much rather work on output back ends
to other distributed SCMs.) But if any volunteers step forward (hint,
hint) I would be happy to help them get started and answer their
questions. I think that cvs2svn is quite hackable now, so the learning
curve is hopefully much less frightening than when I started on the
project :-)
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cvs2svn.tigris.org
For additional commands, e-mail: users-help@cvs2svn.tigris.org
-
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Thank you very much for the email.
Yes if the time for conversion can be brought down to 1/2 hour then it
would be really great.
We could do a automated cvs2svn everyday for testing and that way
maximum lag between cvs and test svn repo would be 1 day.
Please do let me know when available.
Regards,
Rajesh
-----Original Message-----
From: Jon Smirl [mailto:jonsmirl@gmail.com]
Sent: Friday, August 03, 2007 8:41 AM
To: Patwardhan, Rajesh
Cc: Michael Haggerty; Martin Langhoff; Guilhem Bonnefille;
git@vger.kernel.org; users@cvs2svn.tigris.org
Subject: Re: Re: cvs2svn conversion directly to git ready for
experimentation
On 8/3/07, Patwardhan, Rajesh [off-list ref] wrote:
Hello Michael,
I will explain a scenario (we are passing thru this right now)
1) you have 10 years worth of cvs data.
2) We want to move to svn.
3) The repository move should be in such a way that the development
does not get hampered for any 1 work day.
4) We have atleast 4 major modules in cvs which takes about 30 - 40
hours each for conversion currently.
There are known ways (that haven't been implemented) to get the 40 hr
number down to 1/2 hour. Would that be a better approach than doing
incremental imports?
5) With increamental conversions we can do a few things ...
A) Keep the downtime for hard cutoff minimal
B) try out the svn move for other auxillary tools that are
needed by the SCM process.
C) Do some meaningful testing and validation with simulated
live moves of changes from cvs to svn before the actual move on a day
to day basis.
Hopefuly this would substantiate the request \ need for increamental
moves. Or if someone out there has a better suggestion for such
scenario's please point me in the right direction.
Regards,
Rajesh
-----Original Message-----
From: Michael Haggerty [mailto:mhagger@alum.mit.edu]
Sent: Friday, August 03, 2007 1:36 AM
To: Martin Langhoff
Cc: Guilhem Bonnefille; git@vger.kernel.org; users@cvs2svn.tigris.org
Subject: Re: cvs2svn conversion directly to git ready for
experimentation
Martin Langhoff wrote:
quoted
Is there any way we can run tweak cvs2svn to run incrementals, even
if
quoted
not as fast as cvsps/git-cvsimport? The "do it remotely" part can be
quoted
worked around in most cases.
I don't see any fundamental reason why not, but I think it would be a
significant amount of work. There are two main issues:
1. With CVS, it is possible to change things retroactively, such as
changing which version of a file is included in a tag, or adding a new
file to a tag, or changing whether a file is text vs. binary. And
many people copy and/or rename files within the CVS repository itself
(to get around CVS's inability to rename a file). This makes it look
like the file has *always* existed under the new name and *never*
existed under the old name. An incremental conversion tool would have
to look carefully for such changes and either handle them properly or
complain loudly and abort.
2. cvs2svn uses a lot of repository-wide information to make decisions
about how to group CVSItems into changesets, and a lot of these
decisions are based on heuristics. Incremental conversion would
require that the decisions made in one cvs2svn run are recorded and
treated as unalterable in subsequent runs.
This hasn't been a priority in the Subversion world, because, frankly,
what reason would a person have to stick with CVS instead of switching
to Subversion, given that (1) they are intentionally so similar in
workflow, an (2) there is no significant competition from other
centralized SCMs? But of course until the distributed SCM playing
field has been thinned out a bit, people will probably be reluctant to
commit to one or the other.
I don't expect to have time to implement incremental conversions in
cvs2svn in the near future. (I'd much rather work on output back ends
to other distributed SCMs.) But if any volunteers step forward (hint,
hint) I would be happy to help them get started and answer their
questions. I think that cvs2svn is quite hackable now, so the
learning curve is hopefully much less frightening than when I started
on the project :-)
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cvs2svn.tigris.org
For additional commands, e-mail: users-help@cvs2svn.tigris.org
-
To unsubscribe from this list: send the line "unsubscribe git" in the
body of a message to majordomo@vger.kernel.org More majordomo info at
From: Michael Haggerty <hidden> Date: 2016-06-15 22:43:26
[I set followup-to users@cvs2svn.tigris.org, since this has nothing to
do with git.]
Jon Smirl wrote:
On 8/3/07, Patwardhan, Rajesh [off-list ref] wrote:
quoted
Hello Michael,
I will explain a scenario (we are passing thru this right now)
1) you have 10 years worth of cvs data.
2) We want to move to svn.
3) The repository move should be in such a way that the development does
not get hampered for any 1 work day.
4) We have atleast 4 major modules in cvs which takes about 30 - 40
hours each for conversion currently.
There are known ways (that haven't been implemented) to get the 40 hr
number down to 1/2 hour. Would that be a better approach than doing
incremental imports?
Jon, I would like very much to hear how you propose to get an 60-fold
speed increase in cvs2svn. I've never heard of any plausible way to
accomplish anything even close to this.
Please note that the user wants to convert to Subversion, not git. But
even converting to git, I don't think that such speeds are possible
without massive changes that would include processing everything in RAM
and switching large parts of cvs2svn from Python to a compiled language.
Michael
From: Jon Smirl <hidden> Date: 2016-06-15 22:43:26
On 8/3/07, Michael Haggerty [off-list ref] wrote:
[I set followup-to users@cvs2svn.tigris.org, since this has nothing to
do with git.]
Jon Smirl wrote:
quoted
On 8/3/07, Patwardhan, Rajesh [off-list ref] wrote:
quoted
Hello Michael,
I will explain a scenario (we are passing thru this right now)
1) you have 10 years worth of cvs data.
2) We want to move to svn.
3) The repository move should be in such a way that the development does
not get hampered for any 1 work day.
4) We have atleast 4 major modules in cvs which takes about 30 - 40
hours each for conversion currently.
There are known ways (that haven't been implemented) to get the 40 hr
number down to 1/2 hour. Would that be a better approach than doing
incremental imports?
Jon, I would like very much to hear how you propose to get an 60-fold
speed increase in cvs2svn. I've never heard of any plausible way to
accomplish anything even close to this.
Please note that the user wants to convert to Subversion, not git. But
even converting to git, I don't think that such speeds are possible
without massive changes that would include processing everything in RAM
and switching large parts of cvs2svn from Python to a compiled language.
Make a bulk importer for SVN like git-fastimport. I measured some SVN
imports and the bulk of the time was spent forking off SVN. Before
git-fast import it would have taken git two weeks to import Mozilla
CVS.
From: Jon Smirl <hidden> Date: 2016-06-15 22:43:26
On 8/3/07, Jon Smirl [off-list ref] wrote:
Make a bulk importer for SVN like git-fastimport. I measured some SVN
imports and the bulk of the time was spent forking off SVN. Before
git-fast import it would have taken git two weeks to import Mozilla
CVS.
And add a CVS parser to cvs2svn. Use the one I posted or write it again.
Fork is not a very fast operation, millions of forks take a week to run.
In the cvs2git code I did there was one process running cvs2svn and it
parsed the CVS files internally. A second process ran git-fastimport.
Nothing else was forked.
When I first started we were forking both git and cvs. When I ran
oprofile on it 95% of the CPU time was being spent in the kernel.
Linus helped me figure out what was going on. It was the overhead of
page table copies associated with millions of forks that was taking so
long. The solution is to eliminate the forks.
My first try with forks for both cvs and git took about a week to
import Mozilla CVS. After all the forks were eliminated I could import
Mozilla CVS in four hours.