Re: Trying to sync two svn repositories with git-svn (repost)

Thanks for your great explanations, Avery!

On Wed, Apr 29, 2009 at 02:13:29PM -0400, Avery Pennarun wrote:

On Wed, Apr 29, 2009 at 12:01 PM, Josef Wolf [off-list ref] wrote:

quoted

On Tue, Apr 28, 2009 at 11:19:51PM -0400, Avery Pennarun wrote:

quoted

Okay, I think I'm following you.  And I think the difficulty of your
solution will depend on how important it is to cherry-pick each
individual commit from each repo vs. just merging everything as a
batch.

I've already done the cherry-picking.

So you're saying that from now on, *all* changes from *both* branches
need to be integrated in both directions?

Exactly.  Those three commands:

  git diff first-svn        second-svn  # this should be the "private" diff
  git diff first-svn/trunk  first-svn   # what my cherry-picking has changed
                                        # (and waits for push) in first-svn
  git diff second-svn/trunk second-svn  # what my cherry-picking has changed
                                        # (and waits for push) in second-svn

show me _exactly_ what I want them to be.  The manual synchronizations
which were done in the past are resolved now.  But I can't find the way
how to put the result of this cherry-picking back into the svn repositories

If so, you're done with cherry-picking.  If not, you're not.

Yeah, I think I'm done with it.  That's why I tried to "git merge -s ours"
to tell git about the good news ;-)

quoted

 Basically, I've done this:

 # first, move patches from second-svn to first-svn
 git checkout first-svn
 git svn rebase
 git cherry-pick sha1 # repeat as needed
 git merge -s ours second-svn

 # Now, the other way around
 git checkout second-svn
 git svn rebase
 git cherry-pick sha1 # repeat as needed
 git merge -s ours first-svn

The first git-svn-rebase after the merge causes all the (already picked)
commits from the other branch to be pulled into the current branch.
Adding the --no-ff option does not help.  Omitting the cherry-picking
does not help, either.

Hmm, I don't see any 'git svn dcommit' in there.  The steps I listed
referred to dcommit, but explicitly left out calls to 'git svn
rebase'.

Ah!  I thought I _have_ to "git svn rebase" before I dcommit, like I need
to "svn update" before I can do "svn commit".

I think it's likely that your problems stem from this. The git svn
documentation refers to the 'git svn rebase' operation a lot, but it's
only really useful for one thing: linearizing history to make it look
(to svn) like git was never involved. This is handy for people who
want to use git at work without their boss knowing about it, but it
*loses information* and will mess up future merges.

OK, I'll try without rebase.

In general, 'git svn rebase' should be avoided for all the same
reasons that 'git rebase' should be avoided.  They're both great when
used carefully, but they shouldn't be your main day-to-day activity.

Unfortunately, all the howto's I could find recommend exactly that:
git-svn-rebase for getting commits from svn and dcommit for sending
commits to svn.

Unfortunately git svn encourages you to use rebase in your day-to-day
activity... but the workflow I'm talking about actually avoids this
problem completely.  What you want most of the time is really just
'git svn fetch'. and 'git svn dcommit'.

I think I was also a bit too offhand in my previous email when
expanding my suggestion to work with multiple svn hosts.  The clearest
way to do this is with three branches:

  - 1 remote branch: git-svn-1
  - 1 remote branch: git-svn-2
  - 1 local branch: master

I will try this one.  But this will take a while, since my
cherry-picking was done criss-cross.  Thus, I need to "rebase"
the cherries now to get them onto a single branch.  Is there
a simple way to do that or do I have to redo the cherry-picking from
scratch?

So the steps are something like this.  (Again, WARNING: I'm not
running these as I type them, so I could be screwing up just about
anything.)

Getting started:

  git checkout master
  ... Use 'git svn fetch' to update git-svn-1 and git-svn-2 ...
  ... git merge/cherry-pick what you want from git-svn-1 and
git-svn-2.  ALWAYS use --no-ff if using git merge
  git merge --no-ff -s ours git-svn-1
  git merge --no-ff -s ours git-svn-2
    # now master has everything from both svn repositories

I guess I need

   git checkout git-svn-1; git svn dcommit
   git checkout git-svn-2; git svn dcommit

at this point to push the synchronization work to the svn repositories?

quoted

From now on:

  # Update git-svn-1 with the latest master
  git checkout git-svn-1
    # since git-svn-1 is a remote branch, you now have a detached HEAD
  git merge --no-ff master
  git svn dcommit

  # Update git-svn-2 with the latest master
  git checkout git-svn-2
    # since git-svn-2 is a remote branch, you now have a *different*
detached HEAD
  git merge --no-ff master
  git svn dcommit

  # Update master with the latest svn
  git checkout master
    # HEAD is now attached to master
  git merge --no-ff git-svn-1
  git merge --no-ff git-svn-2
    # no need for '-s ours' in the above merge, as no rebasing means
no merge history was lost

Looks reasonable.

quoted

quoted

At Versabanq, we're using git for a bunch of stuff including our
autobuilder (http://github.com/apenwarr/gitbuilder) and my own

Interesting project.  One question: the README mentions that the
gitbuilder as capable to update itself.  But I can not actually see
this functionality in the scripts.  Is that just a typo or am I missing
something?

I guess you're reading the line that says, "Now that your gitbuilder
is working, you probably want to have it continue
to update itself automatically."

Exactly.

This is actually talking about
*running* itself automatically, as in "upating the build results to
the latest copy of your project."  I can see how it's a very unclear
word to use there.  Thanks for the feedback.

OK..

quoted

quoted

As long as you "git config merge.summary true" (to make the merge
commit list all the commits it's merging)

How does this option influence the merge operation?  Or is this meant
to provide additional information to the person who does the next merge?

When you *merge* (as opposed to rebase or cherry-pick) into an svn
branch, you only create a *single* svn commit that contains *all* the
changes.  The above config setting just makes the merge commit contain
a list of all the commits it contains.

But git will not use this information in any way, AFAIK.  So this information
is only for the person who will do the next merge?

quoted

quoted

Now, your problem is a little more complex, because it sounds like
people are checking in two types of things on both sides: private
things and public things.  So if you want *only* the private things,

I want both.  The difference is that I (usually) want to pull the public
things unmodified, while I want to generalize/localize the private things.
So when merging the private part, I would not want to pick the specific
entries.  But I still want to pick the _structure_ (possibly removing or
modifying the localized entries).

If you're going to be mangling things so thoroughly, then you might
just have to resort to cherry-picking everything one by one from one
branch to the other.  It doesn't sound very fun, but if other people
are being so uncooperative by mixing public and private stuff in their
repositories, there's no way I can see to automate it anyhow.

The people are not uncooperative.  It is just that there's no way to
completely separate the public and private content.  For example, the
private part of my apache config looks like this (somewhat simplified):

    &set_conf (apache => {
        servername => "my.host.org",
        davlock => "/m/a/dav/lock",
        confdir => "/m/a/etc/apache",
        httpdir => "/m/b/httpd",
        docroot => "/m/b/httpd/htdocs",
        svndir  => "/m/b/repos/svn",
        gitdir  => "/m/b/repos/git",
        vhosts => {
            "*:80"=> {
                downloads => {
                    "my-debian-repos"=>["/m/b/lib/my-debian",
                },
            },
            "*:443"=> {
                docroot => "/m/b/httpd/htdocs",
                downloads => {
                    "kdb"      => ["/m/b/lib/kdb/SRC",   "Kdb download"],
                    "kdbdemo"  => ["/m/b/lib/kdbdemo",   "Kdbdemo download"],
                    "pictures" => ["/m/b/Pictures",      "Pictures"],
                },
                cgis => {
                    "svn"  => ["/m/l/svn/cgi", "Svn repos administration"],
                    "misc" => ["/m/a/cgi", "Misc cgi scripts"],
                },
                svn => { # FIXME: svn, repos
                    "ab"            => ["Ab repository"],
                    "misc"          => ["Subversion Repository"],
                    "pmisc"         => ["Private Subversion Repository"],
                },
                git => {
                    "test" => ["Git test repos"],
                },
                revproxies => {
                    "/test/"   => ["http://localhost:3000/railstest/"],
                    "/foobar/" => ["http://foo.bar.org:8001/"],
                }
            },
        },
    });

From this information, the public part knows how to generate the apache
config.  Other people are not interested which locations are defined
here,  But they _are_ interested in the layout how this (perl) structure 
looks like.  So I need to strip my localizations and provide the
structure as a template for other people to fill in.  Of course, this
leads to more conflicts in the future every time the layout is changed.

If you're using cherry-pick for everything, there's no reason to use
tricks like 'merge -s ours'.  Just leave out the merging entirely and
don't pretend that what you're doing is merging; it isn't.  (You still
don't need 'git svn rebase' for anything.  Just checkout the branch
you want to change, cherry-pick stuff into it, and 'git svn dcommit'
if appropriate.)

But then I have to do the book-keeping (what was already picked in which
direction) by myself?

If the situation ever changes, you can always do one last 'merge -s
ours' and mark the histories as combined.  Then future merges will
bring in any future changes.

OK