Re: Trying to sync two svn repositories with git-svn (repost)

On Wed, Apr 29, 2009 at 10:07:31PM -0400, Avery Pennarun wrote:

On Wed, Apr 29, 2009 at 6:37 PM, Josef Wolf [off-list ref] wrote:

quoted

On Wed, Apr 29, 2009 at 02:13:29PM -0400, Avery Pennarun wrote:

quoted

So you're saying that from now on, *all* changes from *both* branches
need to be integrated in both directions?

Exactly.  Those three commands:

 git diff first-svn        second-svn  # this should be the "private" diff
 git diff first-svn/trunk  first-svn   # what my cherry-picking has changed
                                       # (and waits for push) in first-svn
 git diff second-svn/trunk second-svn  # what my cherry-picking has changed
                                       # (and waits for push) in second-svn

show me _exactly_ what I want them to be.  The manual synchronizations
which were done in the past are resolved now.  But I can't find the way
how to put the result of this cherry-picking back into the svn repositories

Okay, I think perhaps you're missing something that took me a long
time to figure out about git-svn, but once I understood it, my life
went a lot more smoothly.

Basically, 'git svn fetch' updates just a *remote* branch.  (Remote
branches are in .git/refs/remotes/* and you can't change them
yourself, because you can't attach your HEAD to them.  If you try to
check them out, you get the latest revision in that branch, but your
HEAD becomes detached.)

I'm not actually sure which of the above branches you're referring to
is remote and which is local.  Let's guess that first-svn/trunk is
remote, and first-svn is a local copy of it, onto which you
cherry-picked some extra patches.

Yeah, your guess is correct:

  first-svn/trunk   remote branch of the fist svn repos
  second-svn/trunk  remote branch of the second svn repos
  first-svn         local branch where first-svn/trunk is to be tracked
  second-svn        local branch where second-svn/trunk is to be tracked

What I'm suggesting is that you think of your local branch (first-svn
in this case, I guess) as *not* an svn branch at all.  You never do
any git-svn operations directly on this branch.  In fact, rename it to
master so you aren't tempted to get yourself into trouble.

I see.  But currently, I have _two_ local branches and not only one, as
you suggested in your last post.

Now, you've merged from first-svn/trunk and cherry-picked some extra
stuff onto this branch, right?  Good.  Now you want to *merge* this
branch into first-svn/trunk, producing just *one* new commit, and
dcommit that into svn.

   git checkout first-svn/trunk
      # detaches the HEAD
   git merge master
      # produces a merge commit on the detached HEAD
   git svn dcommit
      # produces a *different* commit object on the first-svn/trunk branch
      # ...and moves HEAD to it.

The newly-produced commit tells git that first-svn/trunk is now
up-to-date with master.  Note that the interim merge commit (produced
by 'git merge') is never shared with *anyone*, so it's perfectly okay
that we replace it with the next command.

Finally, this seems to work.  Luckily, I've done the cherry-picking
not directly, but instead wrote a perl-script with instructions which
commits to pick and how to resolve the conflicts for specific commits.
I use copies of the repositories for this, so I can re-play the scenario
until the result fits the expectations.

With your explanation, I finally arrived at the _first_ work flow
that is able to push the results of the cherry-picking back to the
svn repositories.  Having the branches described above, it goes
like this:

  # first, retrieve what is available
  #
  git svn fetch first-svn
  git svn fetch second-svn

  # cherry-pick from second-svn to first-svn
  #
  git svn checkout first-svn
  git cherry-pick sha1-from-second-svn # repeat as needed
  git checkout first-svn/trunk
  git merge --no-ff first-svn
  git diff first-svn/trunk first-svn >changes.diff
  git svn dcommit

  # now do the same the other way around
  #
  git svn checkout second-svn
  git cherry-pick sha1-from-first-svn # repeat as needed
  git checkout second-svn/trunk
  git merge --no-ff second-svn
  git diff second-svn/trunk second-svn >changes.diff
  git svn dcommit

But I am still somewhat confused:

  git log -1 first-svn/trunk

says "Merge branch first-svn into HEAD".  But this does not reflect
what I've actually done: I've picked _from_ second-svn and committed
that _to_ first-svn.

What you probably thought you should do, given that the existing
git-svn documentation says to do it, is more like this:

   # WRONG
   git checkout first-svn
   git cherry-pick some stuff
   git merge [perhaps -s ours] second-svn/trunk
   git svn dcommit

Almost... In addition, I was trying to "git svn rebase" before the
dcommit

But the above will *change* every single commit you put on first-svn,
because dcommit needs to *regenerate* all the commits after putting
them into svn and getting them back again.  This is essentially a
rebase, and disrupts any merges you might have made from this branch
to another one.  Next time you merge, you'll get a zillion conflicts.

quoted

Ah!  I thought I _have_ to "git svn rebase" before I dcommit, like I need
to "svn update" before I can do "svn commit".

This is true and yet not true.  The reason I don't have to ever use
'git svn rebase' is that 'git svn fetch' updates my first-svn/trunk
branch, and then I quickly do a merge-then-dcommit on that branch.  If
I was to do a 'git svn rebase' first, nothing would happen, because
svn doesn't change.

This is important, since 'git svn dcommit' actually *does* do a 'git
svn rebase' for you automatically, trying to be helpful.

What would happen if somebody else creates a new commit just after I
"git svn fetch" but before I dcommit?  Guess, svn will not accept this
commit, because it is based on an outdated revision.  How would I
get out from this situation?

quoted

quoted

In general, 'git svn rebase' should be avoided for all the same
reasons that 'git rebase' should be avoided.  They're both great when
used carefully, but they shouldn't be your main day-to-day activity.

Unfortunately, all the howto's I could find recommend exactly that:
git-svn-rebase for getting commits from svn and dcommit for sending
commits to svn.

Yeah, they're trying to keep things simple, at the cost of preventing
you from doing anything complicated.  I'm not smart enough to do both,
so I'm just making things complicated for you here ;)

At least I could finally submit the cherries to the svn repositories.
That's a big step forward, although I still don't fully understand all
the details.

As it happens, I wrote the git-svn chapter for the
very-nearly-available new O'Reilly book "Version Control with Git."  I
gave the complicated solution there too.

Interesting.  Do you have any information when it will be available?

I'm eagerly awaiting the
giant flames from people who actually wrote git-svn (and its
documentation) and therefore are highly qualified to disagree with me.

quoted

quoted

  - 1 remote branch: git-svn-1
  - 1 remote branch: git-svn-2
  - 1 local branch: master

I will try this one.  But this will take a while, since my
cherry-picking was done criss-cross.  Thus, I need to "rebase"
the cherries now to get them onto a single branch.  Is there
a simple way to do that or do I have to redo the cherry-picking from
scratch?

No no!  Stop rebasing!

You have a branch that looks the way you want, right?

Ummm, no.. I have _two_ branches:

  first-svn:  contains the cherries that I picked from second-svn. This
              branch looks the way first-svn/trunk should be
  second-svn: contains the cherries that I picked from first-svn. This
              looks the way second-svn/trunk should be

Don't I need to rebase at least one of them if I want to "merge" those
two branches into a single one?

I have a hard time to adopt my mental model to the one-branch method for
some reason.  OTOH, I can easily understand the multiple-branch method:
for every remote branch, I have a local branch on which I collect the
commits that should go to this remote.

That means
you're 99% of the way there.  You just have to convince git that this
branch and the svn branch are related to each other in the way they
actually are.

To do that, you just need to do is make a single merge commit on your
svn remote branch that looks the way you want and merges from your
existing branch, then do a single 'git svn dcommit'.  Here's one way
(assuming you want to make svn look like your new local branch):

   git checkout my-local-branch
   git merge -s ours svn-branch
   git checkout svn-branch
   git merge --no-ff my-local-branch
   git svn dcommit

(If the occasionally-suggested '-s theirs' merge strategy existed, you
could just do the last three steps using git merge -s theirs.)

quoted

quoted

quoted

quoted

As long as you "git config merge.summary true" (to make the merge
commit list all the commits it's merging)

When you *merge* (as opposed to rebase or cherry-pick) into an svn
branch, you only create a *single* svn commit that contains *all* the
changes.  The above config setting just makes the merge commit contain
a list of all the commits it contains.

But git will not use this information in any way, AFAIK.  So this information
is only for the person who will do the next merge?

In fact, it *only* affects the svn log.  Otherwise svn log ends up
with a useless commit that says "Merged from commit (giant hex
string)", and you can't actually do anything with the giant hex string
because svn doesn't know what it is.

OK.  I just noticed the list is limited to 22 entries.  Can this be
configured somehow to contain the complete list?

quoted

The people are not uncooperative.  It is just that there's no way to
completely separate the public and private content.

There is, if you're willing to do it.  The usual way is two have two
branches: public and private.

Well, my plan was to have one (generic) public repository that contains
templates instead of the localized information.  Separating the
repositories is a security measure here.

Whether separate repositories or only different branches, conflicts
_are_ to be expected in this area.  

Whenever you make a change that you want to be public, you commit it
on the public branch, then merge (git merge or svn merge, it doesn't
matter) from public to private.  If you want to make a private change,
you just commit it directly to private.

This way, you will always have the two sets of changes isolated, you
never have to cherry-pick anything, and "git diff public private" is
always a sensible thing to do.

Yeah, I see.  OTOH, I'd rather avoid doing two steps in one go.
Currently, I have a hard time to get _one_ branch in sync.  I'll come
back to the multiple-branch-thing as soon as I have mastered the
one-branch-thing ;-)

(In fact, when I do this, I often don't share the private branch with
anyone at all, which means it's safe to rebase.  That means I can keep
a clean set of patches against the public branch, and sort and
rearrange or share them whenever I feel like it.  This is useful in
some cases.  Rebasing isn't *always* bad :))

quoted

quoted

If you're using cherry-pick for everything, there's no reason to use
tricks like 'merge -s ours'.  Just leave out the merging entirely and
don't pretend that what you're doing is merging; it isn't.  (You still
don't need 'git svn rebase' for anything.  Just checkout the branch
you want to change, cherry-pick stuff into it, and 'git svn dcommit'
if appropriate.)

But then I have to do the book-keeping (what was already picked in which
direction) by myself?

On branch b, 'git merge x' will always merge all the changes from the
most recent merge of x into b (which might be a "-s ours" merge if you
want), up to the tip of x.  So if you don't commit any *new* private
stuff to x, you can use merge.  If you're intermixing the changes,
you'll need to use cherry-pick.  git won't attempt to track the
cherry-picks for you (like eg. svnmerge will).