Re: Converting to Git using svn-fe (Was: Speeding up the initial git-svn fetch)

Hi Stephen,

[sorry about the delayed reply; was ill]

Stephen Bash writes:

Okay, here we go!  I've uploaded the applicable scripts to 
   https://gist.github.com/f6902cb4e3534f07ba48

If you (or anyone) finds I describe something here that isn't on
github, let me know and I'll add to it.  I did a cursory pass
through the scripts to remove a lot of the specific-to-our-repo
stuff, so I'm not even sure these scripts will run as is...  But
most errors should be pretty minor (typos in variable names, etc)
the overall procedure is unchanged.  (And please be gentle, these
are not anything approaching production-ready)

Maybe you'd like to fork off the repository we're working on and add
the scripts there for convinience? We'll start working on the mapping
as soon as svn-fe3 is finished.

As always, these scripts come with ABSOLUTELY NO WARRANTEE, use at
your own risk, your mileage may vary, etc.

Converting to Git using svn-fe
------------------------------
Most people who have tried using git-svn to convert a medium to
large Subversion repository have found it's a slow process.  When I
asked the Git mailing list about this problem in June 2010, I was
pointed to David Barr's svn-dump-fast-export tool:
   http://github.com/barrbrain/svn-dump-fast-export
svn-fe (as the executable is called) converts an entire svn
repository to git very quickly (our repository took about 20
minutes), but the entire svn file system is one branch.  I developed
the following process to reproduce the svn history in git.

So you used the version that supports dumpfile v2 that's merged into
git.git `master`.

Initial Thoughts
----------------
1) Our SVN repository was approximately 20k commits, about 7k files
   in HEAD, a little less than 400 tags, and about 100-150 branches.
   It was organized /trunk/project rather than /project/trunk.
   Branches were /branches/branchName where the branchName directory
   was a copy of the entire trunk (so /branches/branchName/project
   is what a user would checkout).  This does affect the scripts,
   but I think it should be relatively easy to modify (no guarantee
   though).

This is a relatively simple scenario. When building the actual mapper,
we must also take into account moves like /project -> /trunk/project.

2) Our SVN repository originated from cvs2svn, so there are some
   artifacts from that conversion that affect this conversion.

3) I make very little use of Git.pm because while I was developing I
   ran into a bunch of problems with it (none of which I remember
   now).  Instead I make use of perl's system call to send commands
   to Git (where possible I avoid invoking the shell, see perldoc -f
   exec).  I don't want to imply Git.pm doesn't work, but at the
   time it didn't work for me (and I was more focused on making my
   scripts work than improving Git.pm. Sorry!).

I see that you've invoked a lot of Git porcelain command in your
scripts. We must factor these out eventually.

4) The vast majority of our history was before SVN introduced
   merge-info, so I made no attempt to capture SVN merges in Git.
   Rather I kept all branch heads, but moved most of them to a
   "hidden" namespace (see hideFromGit.pl for details).  This does
   mean for a couple merges post-conversion I've had to add
   temporary grafts to make the merge work, but I haven't bothered
   making those grafts permanent (hopefully this isn't a problem?)

5) I performed this entire process using a local mirror of our SVN
   repository in about 4 hours.  It is mostly automated, but does
   require some human monitoring (maybe I'm just paranoid).  Since
   svn-fe runs off a SVN dump file, creating the local mirror was a
   trivial additional step.

The mirroring process is painfully slow :(

6) To keep what follows a *little* shorter, I'm going to assume you
   can read Perl to extract the details of what's going on.  I'll
   try to keep the prose to a high level...

Extracting SVN's History
------------------------
First we want to understand SVN's branching/tagging history.  Modify
buildSVNTree.pl as necessary, then run
   perl buildSVNTree.pl > svnBranches.txt

This looks like the heart of the mapper. You've used the libsvn Perl
bindings to mine data from a local mirror of the repository -- this is
not the way to go. Our current approach is to first get everything
into Git-land and then perform the mapping.

Fundamentally, we get our dumpfile parser (svn-fe) to dump all the
data to a Git object store blindly and then post-process (see
db-svn-filter-root). I'm currently investigating if the dumpfile
parser knows something more that we need to note down before
performing the actual mapping.

Also, since we're aiming for a two-way mapping, it's going to be
significantly more challenging: we will need a mapping function that
can be inverted perfectly.

buildSVNTree.pl does the following steps:
1. Traverses the SVN history chronologically looking for copies.

Unnecessary: the object store doesn't care about being told about
copies explicitly. Then again, we can't produce the exact same
Copyfrom information when we export it back to a dumpfile. The aim of
the two-way mapping:

SVN repository 1 -> dumpfile -> Git repository
Git repository -> dumpfile' -> SVN repository 2

I don't care if dumpfile and dumpfile' don't match. The SVN
repositories must, that's all.

2. Records the source path/rev and destination path/rev for (most)
   copies (see script for details)

Unnecessary again.

3. Once all copies are collected, further filters copies based on:
   * source path is a directory
   * source and destination are not in trunk
   * source and destination are not in the same branch or tag
   * source path is not /vendor (an artifact of cvs2svn)

Interesting. I can't figure out why this is necessary yet (I'll
probably find out later in the email).

4. Checks that source path is "shortest" path from it's rev (protect
   against subdirectories that get added in the same commit)

5. Checks the source and destination paths match globs for expected
   paths (non-matching copies that make it this far are printed to
   STDERR)

Easier in Git-land.

6. Creates a Git branch name for destination (note that svn tags are
   closer to git branches than git tags)

In future, we should give users the flexibility to choose - I'm
thinking of a language for specifying the mapping. And this
information should be persistent: maybe put it in a Git note?

7. Search history for the last commit that actually changed the source path
8. Find a parent path from the source path (mostly recurse up the SVN tree to a known branch)
9. Use the parent path to determine the parent git branch name
10. Record parent/child relationships
11. Dump output to STDOUT (which you should redirect to a file for later use)

Simplified in Git-land.

I did run into one place where two SVN branches had the same name
but different SVN paths (it's complicated).  In this case I just
manually edited the git branch name in svnBranches.txt.  As long as
you do that before continuing, everything should be okay.

Another interesting point. I'm thinking of a tool that assumes some
layout and produces something similar to svnBranches.txt that
end-users can edit before the actual mapping occurs.

There's also some logic in buildSVNTree to determine if a branch/tag
is deleted in the SVN head.  That information is used by
hideFromGit.

It'll be in the revision history in Git anyway- it doesn't require
special handling.

Create the Single Branch Git Repo
---------------------------------
Use svn-fe for what it's designed:
1. svnadmin dump /path/to/svn/repo > svn-dump.txt
2. git init /path/to/initial/git/repo
3. cd /path/to/initial/git/repo
4. cat /path/to/svn-dump.txt | svn-fe svnRepoName | git fast-import

svnRepoName in step 4 can be anything you want, but it has to be
specified so that svn-fe appends the git-svn style "git-svn-id:
svnRepoName@svnRevNum svnRepoUUID" line to each commit message.
This line is required later to map SVN revs to Git commits.

Create Git Branches and Tags
----------------------------
Now comes the next script, filterBranch.pl.  filterBranch will
create Git branches and tags out of the single branch repo by
creating a ton of clones and filtering each one.  While it's doing
this, it also changes the SVN user names to proper Git user IDs
(name + email).  fetchSVNNames.pl can be used to get all the svn
users, then you can edit $authorScript in filterBranch to modify
names appropriately ($authorScript is a git-filter-branch
--env-filter, so it gets eval'ed by git).  Per the git-filter-branch
manpage, you'll want to create/use a RAM disk for temporary files
(see $tempdir).  And you'll need to set various paths like
$parentRepo (this is the repo created in step 2 above), etc.

Hm, we need to maintain a persistent author map too -- another note
object? Also, as Tom's earlier notes point, we need a persistent
timestamp mapping too because Git's concept of commit timestamp is
different from SVN's concept.

Then the script should be (?) relatively automated:
   perl filterBranch.pl svnBranches.txt

The fancy logic here is probably figuring out which Git refs go to
which Git commit, but I'll leave that as an exercise to the
reader...  Ah, I should probably mention: svn-fe can produce "empty"
commits, and filterBranch does nothing to remove them.  By "empty" I
mean there will be a commit object without any content changes.  So
creating a branch/tag in SVN creates a commit, but doesn't change
content.  That commit will be part of the new Git history.
Similarly, filterBranch will create git tags from svn tags, but they
point to one of these "empty" commits rather than the branch they
are tagged from.  It's not very git-ish, but it seems to work...

Oh, I didn't realize that fast-import allows the creation of empty
commits. We should probably fix this?

filterBranch is probably the longest step of the process; there's a
lot of filtering going on.  It will be very verbose on STDOUT, so I
recommend tee'ing to a file or a terminal with infinite scroll back.
It also involves a lot of disk hits (somewhat reduced if $tempdir is
a RAM disk), and potentially a lot of space (it will create a git
repo for every branch/tag in your subversion history).  For our
repository this step took about 1.5-2 hours IIRC.

Wow, this really brute-force.

Create SVN/Git Revmaps
----------------------
Next step is to create a map that goes from SVN rev to Git commit
object.  genRevmap.pl and genJointRevmap.pl will be helpful here:
1. cd $cleanDir (from filterBranch)
2. find . -type d -name "*.git" -exec genRevmap.pl '{}' svnRepoName
   destDir ';'
3. cd destDir
4. find . -name "*.revmap" -exec grep . '{}' + | genJointRevMap.pl >
   jointRevmap.revmap

genRevmap will respect the directory hierarchy created by
filterBranch, and destDir must have a similar structure (doesn't
require the individual Git repos, but any directory that contains a
git repo must exist in destDir).  genJointRevMap takes individual
revmaps and creates a big revmap for all the repositories.  These
scripts aren't doing any real magic, just parsing the Git log
messages for commit ID and the git-svn-id line to get the SVN rev
the commit corresponds to.  Note that SVN rev to Git commit can be
one to many!  (genRevmap just lists the same rev twice if it has
more than one git commit associated with it, genJointRevMap flags
those revs specially and lists all commit IDs on a single line).

Unless there's a one-to-one mapping between Git revisions and SVN
revisions, a two-way bridge will become very difficult to build. Can
you think of any scenarios where a one-to-one mapping doesn't make
sense?

Assembling the Final Git Repo
-----------------------------
Now we need to combine all the small git repos into one repo that
represents the SVN history.  Similar to filterBranch, you'll need to
edit paths in repoFusion.pl to make sure it finds everything.  Then
simply:
   perl repoFusion.pl svnBranches.txt jointRevmap.revmap

At a high level, repoFusion:
1. Clones the trunk repository, this will become the new master branch
2. Performs a git-fetch on every other repository created by
   filterBranch to retrieve the git branch/tags contained there
3. Creates grafts to match up git branches with their parents using
   the revmap
4. If manual grafts are required, it will pause so the user can edit
   the grafts file (search for '*', the message there might be a
   little cryptic, but using svn log and git log in combination,
   hopefully you can figure out what the correct SHA is to insert)
5. Runs filter-branch one more time to make the grafts permanent.

This is a bit faster than filterBranch, but still takes on the order
of an hour for our repository.  It also produces a lot of stuff on
STDOUT, but I think it's a little easier on the disk.  At the end of
the filter branch, I found it useful to scan the output for refs
that weren't updated...  That usually indicates a graft didn't get
created correctly (although due to SVN conventions, it's unlikely
the master ref will change) At this point it's also possible to get
some branch/tag name clashes (I did), so those may require clean up.

Grafts and filter-branch. db-svn-filter-root does this more elegantly.

Hiding 'Deleted' Branches
-------------------------
hideFromGit.pl will use the svnBranches.txt file to move any git
refs associated with deleted SVN paths to refs/hidden in the new
repository.  This keeps the objects associated with those refs from
getting garbage collected, but hides them from most user commands.
This is entirely a personal preference.  (Just like the other
scripts, you'll probably have to edit the paths in the script
itself)

Hm. You didn't include the history of deleted branches in the main
repository. Why? Does it make sense to provide the user an option to
exclude some (deleted) branches in the SVN history? It'll make the
two-way mapping extremely difficult.

'Validating' the Conversion
---------------------------
gitValidation.pl is a script I wrote to randomly select revs from
SVN and try to compare the SVN diffs to the Git diffs.  It uses
git-patch-id to compute a SHA of the changes in each repository, and
reports if something doesn't match up.  It's not particularly
polished, and does find "errors" in our Git repo, but after
investigating all the discrepancies I'm pretty happy that nothing
vital is wrong.

Until the Git -> SVN bridge is complete, I don't suppose we can do
much better than this.

Closing Thoughts
----------------
Do I have any?  This is quite the brain dump, so I'm sure I've been
incomplete and probably somewhat confusing...  I'm happy to answer
questions as I can, but again, this is entirely based on my
experience with our local repo.  YMMV!

Thanks for the interesting and insightful read :)
Do discuss a little more with us on #git-devel or otherwise.

-- Ram