cherry picking and merge

Cherry picking doesn’t work as well as it should.  I was testing on git version 1.7.9.5.

Put in a line in a file, call it:

first version

then cherry pick this into your branch. Then update on master and transform that into:

second version

then, merge that branch back to master.  Death in the form of conflicts.

In gcc land, I do this sort of thing all the time, and I need a merging subsystem to actually keep track of things.  I can manage this will diff and patch and it works flawlessly.  The point of using something better than diff and patch is for it to be better than diff and patch.

I’d like for merge to merge in the work that has yet to be merged.  Not that, plus blindly try and apply or reapply cherry picked items.

On Thu, Jul 31, 2014 at 05:58:17PM -0700, Mike Stump wrote:

Cherry picking doesn’t work as well as it should.  I was testing on
git version 1.7.9.5.

Put in a line in a file, call it:

first version

then cherry pick this into your branch. Then update on master and transform that into:

second version

then, merge that branch back to master.  Death in the form of conflicts.

In gcc land, I do this sort of thing all the time, and I need a
merging subsystem to actually keep track of things.  I can manage this
will diff and patch and it works flawlessly.  The point of using
something better than diff and patch is for it to be better than diff
and patch.

I’d like for merge to merge in the work that has yet to be merged.
Not that, plus blindly try and apply or reapply cherry picked items.

You're not the first person to be surprised by the way merge works.
From the git-merge manpage:

  [This behavior] occurs because only the heads and the merge base are
  considered when performing a merge, not the individual commits.

(That was added after 1.7.9.5.)

If you want the behavior of applying multiple patches in a row, you want
to use git rebase, not git merge.  Since rebase re-applies the patches
of each of your commits on top of another branch, the identical change
won't cause conflicts.

-- 
brian m. carlson / brian with sandals: Houston, Texas, US
+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187

W dniu 2014-08-01 04:43, brian m. carlson pisze:

On Thu, Jul 31, 2014 at 05:58:17PM -0700, Mike Stump wrote:

quoted

Cherry picking doesn’t work as well as it should.  I was testing on
git version 1.7.9.5.

Put in a line in a file, call it:

first version

then cherry pick this into your branch. Then update on master and transform that into:

second version

then, merge that branch back to master.  Death in the form of conflicts.

In gcc land, I do this sort of thing all the time, and I need a
merging subsystem to actually keep track of things.  I can manage this
will diff and patch and it works flawlessly.  The point of using
something better than diff and patch is for it to be better than diff
and patch.

I’d like for merge to merge in the work that has yet to be merged.
Not that, plus blindly try and apply or reapply cherry picked items.

Note that you should try to avoid cherry-picking, as they do not
leave trace in the graph of revisions.

For example if you are creating a bugfix, instead of putting it
directly on maint, and then cherry-picking to master, it is better
to create a separate feature branch for this fix (based at an early
version), and then merge said branch into maint, then into master.
It is described in blog post by Junio Hamano (which I cannot find now).

You're not the first person to be surprised by the way merge works.
 From the git-merge manpage:

   [This behavior] occurs because only the heads and the merge base are
   considered when performing a merge, not the individual commits.

(That was added after 1.7.9.5.)

If you want the behavior of applying multiple patches in a row, you want
to use git rebase, not git merge.  Since rebase re-applies the patches
of each of your commits on top of another branch, the identical change
won't cause conflicts.

There is also git-imerge, third party tool that is intended to help
merging changes (and make it possible to do it in incremental way).

HTH
-- 
Jakub Narębski

On Jul 31, 2014, at 7:43 PM, brian m. carlson [off-list ref] wrote:

You're not the first person to be surprised by the way merge works.

I’m not the first, because the merge command is broken.  Once fixed, I would be happy to be the last.  Until then, the bug remains unfixed.  I’m sending the mail to petition for this bug to be fixed.

From the git-merge manage:

 [This behavior] occurs because only the heads and the merge base are
 considered when performing a merge, not the individual commits.

From:

google(”git merge”):

git-merge - Join two or more development histories together

Either, the command should do as documented, or be fixed.  In that reference, there is no mention that merge will not merge.  There is no mention that merge isn’t the command I want to merge, but that I should use rebase.

Further, google(“git rebase”) says:

There is no difference in the end product of the integration,

Clearly, this is a lie.  There is a difference.

Now, about rebase:

Do not rebase commits that you have pushed to a public repository.

If you follow that guideline, you’ll be fine. If you don’t, people will hate you, and you’ll be scorned by friends and family.

Since everything I do goes up and down into repositories and I don’t want my friends and family to scorn me, rebase isn’t the command I want to use.

I want to use the simple, it works, named for the operation I want to perform, merge.  I’m a simple user, and the simple command I want to work.  You can name the old merge command, merge-mostly or merge-fast and the new one can be called merge.

(That was added after 1.7.9.5.)

I don’t want bugs documented, I want them fixed.  I’m not reporting a doc bug, the doc is correct.

If you want the behavior of applying multiple patches in a row, you want
to use git rebase, not git merge.  Since rebase re-applies the patches
of each of your commits on top of another branch, the identical change
won't cause conflicts.

But, I don’t want the series of patches, I just want a simple,  merged feature X on trunk single commit that merge does.

Given branch B, master M, and cherry picked C, what I want merged is B-(M+C), not B-M.  The problem with B-M, is that when you do B += C (aka cherry pick from master onto your branch), then M += B-M (merge your branch into master), that C is then replicated.  This replication is wrong, always wrong, never right, incorrect, broken.  This is the bug I want fixed.

On Aug 1, 2014, at 9:27 AM, Jakub Narębski [off-list ref] wrote:

Note that you should try to avoid cherry-picking, as they do not
leave trace in the graph of revisions.

Fine, then I want a new command to merge in a change into my branch from another branch and I want merge to account for the motion and not duplicate it when I merge that branch back into master.  Funny thing is, cherry and merge seem to be documented mostly to do exactly what I want.

For example if you are creating a bugfix, instead of putting it
directly on maint, and then cherry-picking to master, it is better
to create a separate feature branch for this fix

You’re assuming that I’m the author of master, I’m not, I’m merely a contributor.  This tail doesn’t wag that dog.  What that means is that I cannot change the world to work around a simple bug in git.

There is also git-imerge, third party tool that is intended to help
merging changes (and make it possible to do it in incremental way).

Then remove git merge and replace it with git-imerge.  :-)  Anyway, I read that, and I can see some beauty of that that might be nice in complex merges.  The problem is, I want git merge to work.


I was curious if svn handles this better the same or worse, and it did it just fine.  I know that a while ago, svn could not handle this, it would do what git does currently.  Apparently they figured out it was a bug and fixed it.  Have you guys figured out it is a bug yet?  The first step in solving a problem, is admitting you have a problem.

From: "Mike Stump" <redacted>

On Aug 1, 2014, at 9:27 AM, Jakub Narębski [off-list ref] wrote:

quoted

Note that you should try to avoid cherry-picking, as they do not
leave trace in the graph of revisions.

Fine, then I want a new command to merge in a change into my branch 
from another branch and I want merge to account for the motion and not 
duplicate it when I merge that branch back into master.  Funny thing 
is, cherry and merge seem to be documented mostly to do exactly what I 
want.

quoted

For example if you are creating a bugfix, instead of putting it
directly on maint, and then cherry-picking to master, it is better
to create a separate feature branch for this fix

You’re assuming that I’m the author of master, I’m not, I’m merely a 
contributor.  This tail doesn’t wag that dog.  What that means is that 
I cannot change the world to work around a simple bug in git.

quoted

There is also git-imerge, third party tool that is intended to help
merging changes (and make it possible to do it in incremental way).

Then remove git merge and replace it with git-imerge.  :-)  Anyway, I 
read that, and I can see some beauty of that that might be nice in 
complex merges.  The problem is, I want git merge to work.


I was curious if svn handles this better the same or worse, and it did 
it just fine.  I know that a while ago, svn could not handle this, it 
would do what git does currently.  Apparently they figured out it was 
a bug and fixed it.  Have you guys figured out it is a bug yet?  The 
first step in solving a problem, is admitting you have a problem.

--
But that goes both ways, and is a philosophical issue about what is to 
be expected in various cases. For some central control use styles, the 
ideas behind _distributed_ version control are anathema and (Git) just 
grinds away at the policies that are expected.

That said, Git doesn't claim to be perfect (and can't because of the 
'relativity' that comes with being distributed - truth has to give way 
to a web of trust). Also the artefacts that Git validates are at a 
different level of abstraction i.e. the whole project as a commit, 
rather than just a few/one file at a time.

In your example (when generalised) the problem is deciding when, in the 
change sequence, the cherry pick is to be backed out, especially if 
there are conflicts in the change sequence that would need fixing 
anyway, and in a long change sequence that would be a lot of conflict 
fix-ups, hence the current choice of getting the merge conflicts all 
resolved in the one go.

The alternate case, mentioned/implied by Brian, is to use a rebase 
(probably after duplicating the branch so as to retain the original if 
required) so as to see each patch/changeset being applied, and doing 
any/many conflit resolutions as they appear, before finally doing any 
merge of the new line of development back into the mainline (which again 
presumes your earlier resolutions don't cause more conflicts on that 
merge). But do note that I've hidden the problem of deciding where the 
rebase start point should be, relative to the merge point, because 
that's actually where the original problem is hidden (which bits merge 
with what!)

git-imerge is a visual tool to show which bits merge cleanly with what 
between two change sequences.

Selecting a compatible workflow is a problem of usage, rather than a 
problem in Git. If Git has a problem, it's that it has too many ways of 
doing things, leaving most of us with too much rope entangled round our 
neck.
--
Philip

On Thursday, July 31, 2014, Mike Stump [off-list ref] wrote:

Cherry picking doesn’t work as well as it should.  I was testing on git version 1.7.9.5.

Put in a line in a file, call it:

first version

then cherry pick this into your branch. Then update on master and transform that into:

second version

then, merge that branch back to master.  Death in the form of conflicts.

The problem is that cherry-picked commits lack the metadata that git
merge could use to avoid this spurious conflict report.  The reflog
has the metadata in question, but there's no guarantee that that
reflog will be available where you do the merge.  (IMO this is another
reason to want branches as objects, so such ancillary information can
be recorded somewhere, but in a way that can get dropped if desired
and without changing commit hashes, but I digress.)

If you always rebase your commits on top of the upstream, then this
problem goes away.  You can't always rebase your commits on top of the
upstream though, but wherever possible it's the best course of action
for this and other reasons.

Nico
--

Hi Mike,

Mike Stump wrote:

Cherry picking doesn’t work as well as it should.  I was testing on
git version 1.7.9.5.

Put in a line in a file, call it:

first version

then cherry pick this into your branch. Then update on master and
transform that into:

second version

then, merge that branch back to master.  Death in the form of conflicts.

Do you mean that "git merge" should be aware of what changes you have
already cherry-picked?

It isn't, and that's deliberate ("git merge" is designed to be simple
as possible, though no more simple than that).  This way, if on a side
branch someone makes a change that would conflict with "master" and
then backs it out, then the branch can still merge cleanly.

Generally people do one of the following:

 * Use a merge-centric workflow.  Don't cherry-pick "forward" but
   merge instead.  (Do use cherry-pick for backports when you forgot
   to commit a fix on top of the oldest supported branch that would
   need it.)  The gitworkflows(7) manpage has more details on how
   this works.

 * Use a cherry-pick-centric workflow.  Never merge.  Notice when
   you're trying to apply a patch you already applied and skip it.
   (Others in the thread have covered this workflow a little.)

Even in those workflows, it's possible to have conflicts due to
genuinely conflicting changes, even with no cherry-pick involved.  I
find the '[merge] conflictstyle = diff3' setting (see git-config(1))
and git-rerere(1) to be helpful in making that less painful.

Hope that helps,
Jonathan

On 08/01/2014 10:48 AM, Mike Stump wrote:

quoted

There is also git-imerge, third party tool that is intended to help
merging changes (and make it possible to do it in incremental way).

Then remove git merge and replace it with git-imerge.  :-)  Anyway, I read that, and I can see some beauty of that that might be nice in complex merges.  The problem is, I want git merge to work.

Git merge has a notion of discrete "merge strategies".  The default,
"recursive" merge strategy isn't completely oblivious to history; in the
event that the two branches don't have a single merge bases, it performs
3-way merges (strangely enough) recursively, with the merge bases of the
branch you're trying to merge until it completes.  In general, this
works pretty well.  Some systems even simpler than that (eg, github's
green merge button) work acceptably as well.

There's no particular reason that you couldn't implement a merge
strategy which works more like SVN's approach, which essentially does an
internal rebase and then commits the result.  The advantages of a rebase
in this situation is that you get to eliminate changes which don't need
to be applied, either because (in SVN's case), it had some
metadata/hearsay information that told it that it could skip that
change, or (in git's case), because it found content/facts that the
change already was applied on one side.

However, there are corresponding disadvantages to this strategy.  It's
just as easy to contrive a situation where this "internal rebasing"
doesn't do the right thing, even without cheating by getting the
metadata wrong.  And besides, there's already a way to do this: do an
actual rebase.  You could also do a rebase, and then if, say, the
original branch you're rebasing is published and you don't want to
rewrite, then you can easily enough use squash merging, merge -s ours,
etc to make it look like the strategy you wanted was a built-in git
merge strategy.  Or, in the spirit of open source, you could contribute
the code required to make 'imerge' a built-in strategy.

I was curious if svn handles this better the same or worse, and it did it just fine.  I know that a while ago, svn could not handle this, it would do what git does currently.  Apparently they figured out it was a bug and fixed it.  Have you guys figured out it is a bug yet?  The first step in solving a problem, is admitting you have a problem.

So, I have to chuckle when I read this indignant comment.  There's a
funny story to the "while ago" you refer to.  This refers to the time
period during which SVN was relevant; about versions 1.4 and earlier
(being generous).  Back in those days, SVN projects for the most part
avoided merging, because it was so problematic and not tracked at all. 
As one core SVN developer said to me, they found "teams collaborate more
closely if they're all working on the same branch".  Sure, you could do
it, and I even know of a few communities who did, but by and large, it
was avoided.  Then, the new wave of version control systems including
Git, bzr and Mercurial were cropping up, and their merges were actually
good enough that you could practically use them.

The SVN core team had to keep pace to match.  So, in 1.5 the "merge
tracking" system, previously only supplied as a "contrib" script, became
core.  This is ironic, because the version control system which SVN
imitated poorly--Perforce--had a very sophisticated, if
over-complicated, merge tracking system which was also based on
metadata.  Per-branch, per-patch, per-file entries for whether or not a
patch had been "integrated" into the target branch.  I can only guess
that the reason they didn't implement this in the original SVN version
was that it was something of a pain point for users in Perforce. 
Possibly something to do with the way that Perforce would store double
entries for each merge (yes: two rows in a relational store, one
representing the mirror image of the other), and differentiated between
many different forms of "integrated" (ie, 2 rows and 4 states instead
of, say, a single bit).  So the underlying data model wasn't as simple
as it could have been, and this was reflected in the difficult to use
command-line tools.  Plus, they were using BerkeleyDB for metadata
instead of the relational ISAM library, and debugging a rabbit's nest of
merge record as Perforce used would have been a nightmare.  They didn't
go there.  And besides, they found that often, detecting patches as
already applied based on content, like 'patch' did, worked.

Prior to 1.5, the Perl community developed SVK, an offline version of
SVN, and this had a far simpler model for merge tracking, more similar
to git's: just tracking whole-branch merges rather than individual
files, patches, and branches.  SVN eventually added two separate ways of
tracking merges: either a per-file, per-branch, per-commit or a
per-branch, per-commit model.

Anyway, I'm not sure where I'm going with this, but I guess a little
extra perspective would be useful!

Sam

Jonathan Nieder wrote:

Do you mean that "git merge" should be aware of what changes you have
already cherry-picked?

It isn't, and that's deliberate

That said, when today's "git merge" fails to resolve conflicts, it's
easily possible that we could do better at resolving the merge by
walking through both sides and understanding what happened.

The detailed history lets you

   i) Present conflicts in an easier to resolve way.

      "Patch #1 which tries to do X conflicted with patch #2 which
      tries to do Y; please reconcile them" can be less painful to
      deal with than "Something in this pile conflicted with something
      in that pile".

  ii) Break a seeming conflict into pieces that can be automatically
      resolved more easily.

      X vs X'+Y may conflict where X' is a cherry-pick of X, if X and
      Y touch the same code.  Meanwhile if we're lucky then X vs X'
      will not conflict because they make the same change, and Y can
      apply on top.

 iii) Handle cherry-picked changes in a *different* way.  For example,
      if patch X was applied on one side and applied and then reverted
      on the other side, this could show up as a conflict.  After all,
      the two sides don't agree on whether patch X is a good change or
      not.

These features have corresponding downsides:

   i') (Speaking from experience of using git-imerge) Too many tiny
       conflicts can sometimes be more painful to resolve than all the
       conflicts at once.  When X, Y, Z, and W had various conflicts,
       how to reconcile X and Y alone or Z and W alone are academic
       questions that don't actually need to be answered to produce
       the merge result.

  ii') This kind of clean, broken-down merge can produce a "clean"
       but wrong result.

       For example, if the following sequence of events occured:

         1. Build fancy new feature X on "master".

	 2. Cherry-pick X to the bugfixes-only branch "maint".
	    Whoops.
	 3. Correct the mistake: revert X on "maint".  Now "maint"
	    is bugfixes-only again!

	 4. Merge "maint" to "master".

       Then a naive, 3-way merge will notice there is no change
       on "maint" since it was last merged to master and the
       merge will bring in no change (good).

       And on the other hand a one-patch-at-a-time merge would
       try to apply X (with no effect, since it's already applied)
       and then try to apply the revert of X.  The net effect would
       be to revert X from "master" (bad)!

 iii') See (ii').

git-imerge from https://github.com/mhagger/git-imerge can help with
(i) and (ii) but not (iii).

Hoping that clarifies,
Jonathan

On Fri, Aug 1, 2014 at 3:50 PM, Jonathan Nieder [off-list ref] wrote:

Jonathan Nieder wrote:

quoted

Do you mean that "git merge" should be aware of what changes you have
already cherry-picked?

It isn't, and that's deliberate

That said, when today's "git merge" fails to resolve conflicts, it's
easily possible that we could do better at resolving the merge by
walking through both sides and understanding what happened.

It would help if cherry-pick history where recorded somewhere (beyond
the reflog)...

Cherry-picks should record two parents, like merges.

(Of course, it does no good to know about an unreachable parent, when
a commit with two parents is pushed to a repo that doesn't have one of
those parents, which can happen when topic branches aren't pushed
upstream.)

Nico
--

On Aug 1, 2014, at 12:22 PM, Nico Williams [off-list ref] wrote:

If you always rebase

I can’t use rebase unless you make rebase work with multiple users and pushing pulling.

On Aug 1, 2014, at 11:57 AM, Philip Oakley [off-list ref] wrote:

But that goes both ways, and is a philosophical issue about what is to be expected in various cases.

The problem is, users expect merge to merge.  There isn’t a user that expects it to scramble the source code, because the command is called merge, not scramble.  That word has semantics that were not invented by your project.  You cannot change the semantic of the word.  Merge has a nice mathematical definition.  Merge branch into master means, place into into master the work from that branch.  git already does this 99% correct, it is missing one corner case.

This is not a philosophical issue.  It is a definitional one.

For some central control use styles, the ideas behind _distributed_ version control are anathema and (Git) just grinds away at the policies that are expected.

This is irrelevant to the issue at hand.

That said, Git doesn't claim to be perfect

Again, irrelevant.

(and can't because

Do you mean, and can’t be?  If so, you are wrong in the case at hand.  svn is an existence proof that you are wrong.

of the 'relativity' that comes with being distributed - truth has to give way to a web of trust). Also the artefacts that Git validates are at a different level of abstraction i.e. the whole project as a commit, rather than just a few/one file at a time.

Ah, so that gives me an idea.  [ pause ] If we try the cherry-pick as retroactively creating a feature branch, cherrying into that, then merge unconditionally so that no change happens that into trunk (thus killing those conflicts), and then git merge that feature branch into branch then it all works perfectly.  See, another existence proof that you are wrong, this time with git itself.

It was 13 lines of code, so, apparently, it is possible and easy to do, in git.  Now, we just want the cherry-pick to create a temporary cherry branch, cherry the pick into it, merge and drop into trunk and merge into branch…

I tested with the below and it worked just fine.  Things to clean up, we want the meta data on the cherry on the merge commit, but, you get the idea.

branch=b
master=master
base=$(git merge-base $branch $master)
cherry="$1"

git checkout -b cherry-$branch $base
git cherry-pick "$cherry"
git checkout $master
git merge -s ours cherry-$branch
git checkout $branch
git merge cherry-$branch
git branch -d cherry-$branch
git cherry-pick --strategy=ours --allow-empty "$cherry"
git commit --allow-empty

I tested that with two cherries with further changes on master to ensure that it works for more than a single one, no problem.  Wow, even tried a merge of master back into b, and it worked just fine, no conflicts, yet, all the code was jammed up together nicely.

So, if you wish to continue your position, please explain why it can’t get this better, given the existence proof above of it working better in git.

In your example (when generalized)

I’m not interested in other bugs that I didn’t state, in this email.  I don’t care about those.  Please don’t detract from fixing this issue, because you can identify other things that might not be perfect.  We attain perfection one step at a time.

the problem is deciding when, in the change sequence, the cherry pick is to be backed out, especially if there are conflicts in the change sequence that would need fixing anyway, and in a long change sequence that would be a lot of conflict fix-ups, hence the current choice of getting the merge conflicts all resolved in the one go.

I have two possible conflict fixups in the above.  In my case (I have a specific patch in gcc-land i wanted to cherry), those fixups were trivial (no conflicts).  When they are trivial, I don’t care much that there were two of them.  When non-trivial, well, I’m resigned to the idea that I have to explain what is going on.

Selecting a compatible workflow is a problem of usage,

Not when the workflow is mandated on you to work around trivial little bugs that can be fixed but for which the author’s don't even comprehend the bug.

rather than a problem in Git.

On Fri, Aug 1, 2014 at 5:13 PM, Mike Stump [off-list ref] wrote:

On Aug 1, 2014, at 12:22 PM, Nico Williams [off-list ref] wrote:

quoted

If you always rebase

I can’t use rebase unless you make rebase work with multiple users and pushing pulling.

That works now, and I do it all the time.  Have a single repo ("the
truth"), always rebase local commits on top of the latest upstream,
always do fast-forward pushes.  Done.

On Aug 1, 2014, at 1:02 PM, Jonathan Nieder [off-list ref] wrote:

Do you mean that "git merge" should be aware of what changes you have
already cherry-picked?

Yes, it amounts to that.

It isn't, and that's deliberate

Deliberate bugs are still bugs.  In time, users will either wear you down until you fix it, or they will move on to other systems that work better.

("git merge" is designed to be simple as possible, though no more simple than that).

I sketched a solution that retains a simple git merge…

This way, if on a side branch someone makes a change that would conflict with "master" and
then backs it out, then the branch can still merge cleanly.

Yeah, my solution doesn’t impinge upon that working nicely.  In it, I make cherry use scratch branches to record meta information so that the existing git merge just works.  git cherry is free to do the same.  Having a git cherry that fully interoperates with git merge, is a feature.

Even in those workflows, it's possible to have conflicts due to

genuinely conflicting changes, even with no cherry-pick involved.  I
find the '[merge] conflictstyle = diff3' setting (see git-config(1))
and git-rerere(1) to be helpful in making that less painful.

I think those two should be the default, but it is easy enough to turn them on that it doesn’t matter much.  In my environment, I have both on.

Mike Stump wrote:

On Aug 1, 2014, at 1:02 PM, Jonathan Nieder [off-list ref] wrote:

quoted

It isn't, and that's deliberate

Deliberate bugs are still bugs.

Yes, you and I disagree about what the behavior should be.

I actively prefer the current behavior over the one you proposed,
unless I'm misunderstanding the one you proposed.  As I understand it,
there would be no way to undo the mistake of cherry-picking a change
that didn't belong on "maint".

You said that 3-way merge doesn't fit your idea of what a merge is.
It does fit mine.

I'm even slightly against there being an option for the 'git cherry'
based thing.  I think it would be a support burden.  But if someone
else wants to do the work of implementing it, I wouldn't stop them
(and would instead just help make sure the documentation is crystal
clear).

Hoping that clarifies,
Jonathan

On Aug 1, 2014, at 1:12 PM, Sam Vilain [off-list ref] wrote:

Git merge has a notion of discrete "merge strategies”.

There's no particular reason that you couldn't implement a merge
strategy which works more like SVN's approach, which essentially does an
internal rebase and then commits the result.

Well, being a simple user, wanting to do a simple thing, I want the default strategy to just work.  The expert that wants it to work faster, but less well, well, they can use a merge -s faster, or cherry-pick -s faster.

However, there are corresponding disadvantages to this strategy.  It's
just as easy to contrive a situation where this "internal rebasing"
doesn't do the right thing, even without cheating by getting the
metadata wrong.

I sketched a solution using branches…  A large portion of the failures that happen when a cherry is remerged are gone.  I feel that benefit easily swamps the problem you sketched.

And besides, there's already a way to do this: do an actual rebase.

One problem is that rebase doesn’t work with co-workers nicely…  The other is that it isn’t spelled merge.  I am a simple user.

quoted

I was curious if svn handles this better the same or worse, and it did it just fine.  I know that a while ago, svn could not handle this, it would do what git does currently.  Apparently they figured out it was a bug and fixed it.  Have you guys figured out it is a bug yet?  The first step in solving a problem, is admitting you have a problem.

So, I have to chuckle when I read this indignant comment.

:-)  Yeah, a chuckle, good.  Actually, no anger is involved.  I’d just like for git to work better in this regard.

On Fri, Aug 1, 2014 at 6:06 PM, Mike Stump [off-list ref] wrote:

On Aug 1, 2014, at 1:12 PM, Sam Vilain [off-list ref] wrote:

quoted

Git merge has a notion of discrete "merge strategies”.

quoted

There's no particular reason that you couldn't implement a merge
strategy which works more like SVN's approach, which essentially does an
internal rebase and then commits the result.

Well, being a simple user, wanting to do a simple thing, I want the default strategy to just work.  [...]

Different users want different defaults.  You can't always have the
one you want.

As for rebase, I still don't understand why it doesn't work for you.
You didn't really explain.  Suppose we're right and it's the right
solution for you, then you might be ecstatic, but you gotta try it
first.

My workflow is rebase-heavy.  It's long been so, and it was so before
git happened.  The only case where I can imagine not using a
rebase-heavy workflow is where I have to track multiple forked
upstreams and so I want to merge each into my branch.

If tracking multiple forked upstreams is not your case and yet rebase
can't work for you then I'd like to understand why.  Please help me
understand your use case.  OTOH, if your use case is amenable to
rebase, then I highly recommend that you try it.

(I find that many users are allergic to rebasing.  Many people have
told me that rebase is lying, that history must be immutable, and so
on, all ignoring that: git users don't rebase published branches, and
that other VCSes tend to squash (therefore lose) history anyways when
pushing merges upstream.  But this all seems theological rather than
rational.  It's true that I dislike merge commits, but that's a
different story; I'm not allergic to merging after all.)

Nico
--

On Aug 1, 2014, at 1:50 PM, Jonathan Nieder [off-list ref] wrote:

      And on the other hand a one-patch-at-a-time merge would
      try to apply X (with no effect, since it's already applied)
      and then try to apply the revert of X.  The net effect would
      be to revert X from "master" (bad)!

Yeah, I appreciate that.  I know the type of user that would do that, and I understand why you would want to do that and even do that by default.

However, as an expert user, I don’t need that particular type of hand holding given the cost of more conflicts and would like an option to let me to choose to have fewer conflicts when merging.

On Fri, 2014-08-01 at 18:40 -0500, Nico Williams wrote:

On Fri, Aug 1, 2014 at 6:06 PM, Mike Stump [off-list ref] wrote:

quoted

On Aug 1, 2014, at 1:12 PM, Sam Vilain [off-list ref] wrote:

quoted

Git merge has a notion of discrete "merge strategies”.

quoted

There's no particular reason that you couldn't implement a merge
strategy which works more like SVN's approach, which essentially does an
internal rebase and then commits the result.

Well, being a simple user, wanting to do a simple thing, I want the default strategy to just work.  [...]

Different users want different defaults.  You can't always have the
one you want.

Or to put it another way: one man's bug is another man's feature.

As for rebase, I still don't understand why it doesn't work for you.
You didn't really explain.  Suppose we're right and it's the right
solution for you, then you might be ecstatic, but you gotta try it
first.
...
Nico
--

Data point:

We've been using a rebase-centric workflow for a while at my current
employer. It's simple and generally straightforward for new development
on master.

However we need to maintain multiple 'released' version branches which
receive hotfixes and (sadly) features from later development, and merges
make it much easier to visualise which releases have received specific
fixes/shinies than cherry-picks do.

In a hybrid merge/rebase workflow it is convenient to have the option of
merges which yield rebase-like output. We have seen awkward merges
outside of master, but I mostly see that as an indication that we
shouldn't be mixing workflows so much, ie. hotfixing one thing with a
cherry-pick and another with a merge (usually only happens when the
branch predates our use of merges).

TL;DR: The option of a rebase-like merge would be a nice feature, but
the default system does not seem so onerous.

Alex

From: "Mike Stump" <redacted>
Sent: Friday, August 01, 2014 11:10 PM

On Aug 1, 2014, at 11:57 AM, Philip Oakley [off-list ref] 
wrote:

quoted

But that goes both ways, and is a philosophical issue about what is 
to be expected in various cases.

The problem is, users expect merge to merge.  There isn’t a user that 
expects it to scramble the source code, because the command is called 
merge, not scramble.

Unfortunately we are back at the problem af what 'merge' means. Git uses 
a snapshot model, while most other version control systems uses a 
changeset model (which has been used since before the Titanic was built 
for various reasons [1]) for their storage. Thus most VCS users see 
merge as the addition of a delta "A + delta -> B", while Git sees merge 
as the union of snapshots "A + G -> Q".

The cherry-pick and rebase methods determine the change deltas between 
adjacent snaphots (i.e. patches) and then apply that to a different 
snapshot. In those cases we are not merging snapshots, rather applying 
patches (note my changed use of 'merge').

That word has semantics that were not invented by your project.  You 
cannot change the semantic of the word.  Merge has a nice mathematical 
definition.  Merge branch into master means, place into into master the 
work from that branch.  git already does this 99% correct, it is 
missing one corner case.

This is not a philosophical issue.  It is a definitional one.

quoted

For some central control use styles, the ideas behind _distributed_ 
version control are anathema and (Git) just grinds away at the 
policies that are expected.

This is irrelevant to the issue at hand.

quoted

That said, Git doesn't claim to be perfect

Again, irrelevant.

quoted

(and can't because

Do you mean, and can’t be?  If so, you are wrong in the case at hand. 
svn is an existence proof that you are wrong.

quoted

of the 'relativity' that comes with being distributed - truth has to 
give way to a web of trust). Also the artefacts that Git validates 
are at a different level of abstraction i.e. the whole project as a 
commit, rather than just a few/one file at a time.

<snipped remainder because of time limitations>

--
Philip

[1] Way back when blueprints really were blue, and real drawing were on 
kaolin & linen drawing sheets with indian ink, the 'master' drawings 
were the most prized items, that if damaged would stop production, so 
there were many layers of drawing office controls to avoid touching it 
(e.g. tracers to copy drawings) and keep the master drawings pristine.

From that, the ideas of Change requests, Change orders, Approved changes 
etc. became the norm. It was the changes that were recorded. These 
processes are still in place today in most engineering companies and the 
military, in fact anyone with real artefacts. These techniques were 
copied into the computing world when it was young.

The big change has been that Computing has reduced the cost of 
production (duplication, replication) to zero, so now the problem is in 
trying to validate and verify which alledged copy is actually the 
'master' - especially in a fully distributed environment, such as the 
many different Linux distributions and applications. It was into this 
gap that Git steps, with the use of the sha1 to both validate the 
history chain and verify any specific snapshot. It doesn't get hung up 
on what the change was, that can be determined easily after the fact by 
a simple diff between adjacent snapshots, though having lots of small 
changes and crisp commit messages does help.

That's the way I view it anyway. (When I first started work, they still 
had blue prints, but had moved to microfiche and melinex (polyester) 
drawing sheets, and the 8088 / IBM PC was new and powerful! I still work 
in engineering where the old VC model is ubiquitous)

From: "Mike Stump" <redacted>
Sent: Friday, August 01, 2014 11:10 PM
(part 2)

On Aug 1, 2014, at 11:57 AM, Philip Oakley [off-list ref] 
wrote:

quoted

For some central control use styles, the ideas behind _distributed_ 
version control are anathema and (Git) just grinds away at the 
policies that are expected.

...

quoted

of the 'relativity' that comes with being distributed - truth has to 
give way to a web of trust). Also the artefacts that Git validates 
are at a different level of abstraction i.e. the whole project as a 
commit, rather than just a few/one file at a time.

Ah, so that gives me an idea.  [ pause ] If we try the cherry-pick as 
retroactively creating a feature branch, cherrying into that, then 
merge unconditionally so that no change happens that into trunk (thus 
killing those conflicts), and then git merge that feature branch into 
branch then it all works perfectly.  See, another existence proof that 
you are wrong, this time with git itself.

It was 13 lines of code, so, apparently, it is possible and easy to 
do, in git.  Now, we just want the cherry-pick to create a temporary 
cherry branch, cherry the pick into it, merge and drop into trunk and 
merge into branch…

I tested with the below and it worked just fine.  Things to clean up, 
we want the meta data on the cherry on the merge commit, but, you get 
the idea.

I've annotated some of the bits to make sure we are on the same 
wavelength as to what this does...

branch=b
master=master
base=$(git merge-base $branch $master)

cherry="$1"  # not quite sure where this commit is located relative to 
either $branch or $master

# create a new branch, starting at base, for our cherry picked commit

git checkout -b cherry-$branch $base
git cherry-pick "$cherry" # which also commits onto our cherry pick 
barnch

git checkout $master
git merge -s ours cherry-$branch # "mark/remember" the cherry branch, 
its fix and it's base point, but don't actualy use it here on $master

git checkout $branch
git merge cherry-$branch # bring the 'fix' into $branch

git branch -d cherry-$branch # remove the fix branch that started at 
$base - branches are ephemeral anyway.

# still on $branch, which already has the change merged in (Git style) ?

git cherry-pick --strategy=ours --allow-empty "$cherry" # check its 
all already included?
git commit --allow-empty

Does my annotation match your understanding? It wasn't clear to me where 
$1 had been hiding previously, nor why the common fix didn't use a 
"merge -s ours cherry-$branch" in both cases - that maybe my 
misunderstanding about how your workflow goes.

I tested that with two cherries with further changes on master to 
ensure that it works for more than a single one, no problem.  Wow, 
even tried a merge of master back into b, and it worked just fine, no 
conflicts, yet, all the code was jammed up together nicely.

So, if you wish to continue your position, please explain why it can’t 
get this better, given the existence proof above of it working better 
in git.

...

I have two possible conflict fixups in the above.  In my case (I have 
a specific patch in gcc-land i wanted to cherry), those fixups were 
trivial (no conflicts).  When they are trivial, I don’t care much that 
there were two of them.  When non-trivial, well, I’m resigned to the 
idea that I have to explain what is going on.

quoted

Selecting a compatible workflow is a problem of usage,

Not when the workflow is mandated on you to work around trivial little 
bugs that can be fixed but for which the author’s don't even 
comprehend the bug.

quoted

rather than a problem in Git.

--

Philip. 

W dniu 2014-08-01 22:55, Nico Williams pisze:

On Fri, Aug 1, 2014 at 3:50 PM, Jonathan Nieder [off-list ref] wrote:

quoted

Jonathan Nieder wrote:

quoted

Do you mean that "git merge" should be aware of what changes you have
already cherry-picked?

It isn't, and that's deliberate

That said, when today's "git merge" fails to resolve conflicts, it's
easily possible that we could do better at resolving the merge by
walking through both sides and understanding what happened.

It would help if cherry-pick history where recorded somewhere (beyond
the reflog)...

Cherry-picks should record two parents, like merges.

(Of course, it does no good to know about an unreachable parent, when
a commit with two parents is pushed to a repo that doesn't have one of
those parents, which can happen when topic branches aren't pushed
upstream.)

There was (long time ago) a long thread about idea of adding some
kind of 'weak' references (links), 'weakparent' that can be 
automatically used by Git but do not pollute the commit message,
and do not affect reachability calculations.  Ultimately it went
nowhere (as you can see) - there were many problems.

For example: how it would work for reverts and rebases?

-- 
Jakub Narębski

On Wed, Aug 6, 2014 at 10:58 AM, Jakub Narębski [off-list ref] wrote:

W dniu 2014-08-01 22:55, Nico Williams pisze:

quoted

It would help if cherry-pick history where recorded somewhere (beyond
the reflog)...

Cherry-picks should record two parents, like merges.

(Of course, it does no good to know about an unreachable parent, when
a commit with two parents is pushed to a repo that doesn't have one of
those parents, which can happen when topic branches aren't pushed
upstream.)

There was (long time ago) a long thread about idea of adding some
kind of 'weak' references (links), 'weakparent' that can be automatically
used by Git but do not pollute the commit message,
and do not affect reachability calculations.  Ultimately it went
nowhere (as you can see) - there were many problems.

My proposal was to put this sort of ancillary history info in a
"branch object" (and that branches should be objects).  This would
have a number of benefits, not the least of which is that at push time
you can drop such ancillary history without having to alter the
commits being pushed.

For example: how it would work for reverts and rebases?

Reverts upstream?  The revert should record the commit hash of the
commit it reverts (but file-level reverts lose), so that this could be
noticed.

Rebases upstream?  Well, that shouldn't happen, but if it does then
you must rebase --onto and any cherry-picks of upstream rebased
commits lose their ties to those (but this can be detected).

In general recording more metadata (assuming there's not privacy
issues to consider) can't hurt.  Using it might, but having the option
to can also help.

Nico
--

On Aug 1, 2014, at 4:40 PM, Nico Williams [off-list ref] wrote:

As for rebase, I still don't understand why it doesn't work for you.

http://git-scm.com/docs/git-rebase says:

  Rebasing (or any other form of rewriting) a branch that others have based work on is a bad idea

If you read stack-overflow, you will discover a ton of people that like doing this, and they get hammered because of it.  My use case fits exactly (as near as I can tell) the hammer case.

If you make a different command that isn’t guaranteed to screw me and my co-workers over, and tell us to use that, then I’d be happy to use it.  Bet the farm that it won’t bite you, just to be bitten isn’t what I want to try and recover from.

You didn't really explain.

If you say it will never bit you and then fix all the documentation to not say it will bite you…  I’d be happy to contemplate it again.  Now, I found the stack-overflow commentary first, and all the horrors of it, and all the nuances.  I carefully read what people were doing, how what I wanted to related to what they were doing, and it sure felt like I was in the, don’t go there camp.

Suppose we're right and it's the right solution for you, then you might be ecstatic, but you gotta try it
first.

So, I like to know if I’m driving off a cliff, before I do.  I’m the type of person that would rather know were the road goes, and merely avoid driving off the cliff.  When stack-overflow is littered with the bodies of people that thought it would be fun, I tend to just say, that’s not for me.

The only case where I can imagine not using a
rebase-heavy workflow is where I have to track multiple forked
upstreams and so I want to merge each into my branch.

So, sounds like I fit that use case and rebase could be my friend.  How do I square what you said and:

  Rebasing (or any other form of rewriting) a branch that others have based work on is a bad idea

?

I want all old refs in old emails to work.  I want all refs in bugzilla to work.  I want to see the original dates of all the work.  I want git blame to report those artifacts in email and bugzilla.  I have coworkers that I push to, pull from (through a single sharing point, we call the master tree).  We work on gcc, we pull git gcc down to a local copy, then merge it into our tree.  I want to cherry pick changes from upstream.  I do work and push to our master, I pull work of coworkers from the master, my coworkers do the same.  Isn’t this the canonical open source use case?

(I find that many users are allergic to rebasing.  Many people have
told me that rebase is lying, that history must be immutable, and so
on, all ignoring that: git users don't rebase published branches,

So, when I push, and someone else pulls, is that published?  I thought it was.

On Wed, Aug 06, 2014 at 12:11:16PM -0700, Mike Stump wrote:

On Aug 1, 2014, at 4:40 PM, Nico Williams [off-list ref] wrote:

quoted

As for rebase, I still don't understand why it doesn't work for you.

http://git-scm.com/docs/git-rebase says:

  Rebasing (or any other form of rewriting) a branch that others have
  based work on is a bad idea

If you read stack-overflow, you will discover a ton of people that
like doing this, and they get hammered because of it.  My use case
fits exactly (as near as I can tell) the hammer case.

It's not a good idea to rebase a branch in a repo that others pull from.

There's nothing wrong with rebasing your patches on that same branch in
your clone as long as the end result is a fast forward merge when you
push.

    $ git clone https://..../blah
    $ cd blah
    $ <do some work>
    $ git commit ...
    $ git fetch origin
--->$ git rebase origin/master
    $ git push origin master

There's NOTHING wrong with that rebase.  It's perfectly safe because
it's only in your *private* clone.

(If later you publish that clone and expect people to track your master
branch, then rebase becomes problematic, but that's not something most
people ever do with their clones.)

The only use-case I've seen where a rebase-based workflow doesn't work
is where you have multiple upstreams that you're following.  I.e., the
upstream forked and you want to take some commits from one, some from
the other, or otherwise keep a merge of both.

(Also, if an upstream is ever rebased you can usually recover on the
downstream side by rebasing with the --onto option, so it's not the end
of the world.)

Now, I found the stack-overflow commentary first, and all the horrors
of it, and all the nuances.  I carefully read what people were doing,
how what I wanted to related to what they were doing, and it sure felt
like I was in the, don’t go there camp.

A lot of people rant about rebase.  They're wrong.  They've distorted
your perspective.

So, I like to know if I’m driving off a cliff, before I do.  I’m the
[...]

There's just two simple rules to follow and you'll be safe:

1) NEVER git push -f (--force) to a "published" repo/branch.

   The upstream should enforce this with a receive hook.

2) NEVER work directly in a published repo.  Instead work in a private
   clone.  To help make sure of this, never publish a non-bare repo
   (bare == has no workspace; non-bare == has a workspace).

If you ever do a rebase that produces results you're unhappy with you
can undo that rebase like so:

 - use git reflog to find the branch's previous HEAD commit
 - reset the branch to point to that commit

It really helps to think of git as a pile of commits arranged in a
Merkle has tree.  Branches and tags are just symbolic names for specific
commits.  Rebase builds a new line of commits in the tree then it
changes the symbolic branch name's HEAD to point to the head of that new
line of commits, BUT NOTHING IS LOST in the pile of commits that is the
repo, not until you git-prune(1) to remove commits not reachable from
symbolic names (branches and tags).

quoted

The only case where I can imagine not using a
rebase-heavy workflow is where I have to track multiple forked
upstreams and so I want to merge each into my branch.

So, sounds like I fit that use case and rebase could be my friend.

Excellent.

How do I square what you said and:

  Rebasing (or any other form of rewriting) a branch that others have
  based work on is a bad idea

?

See above.

I want all old refs in old emails to work.  I want all refs in

They will if you stick to the two rules I mention above.

bugzilla to work.  I want to see the original dates of all the work.

Ditto.

I want git blame to report those artifacts in email and bugzilla.  I
have coworkers that I push to, pull from (through a single sharing
point, we call the master tree).  We work on gcc, we pull git gcc down
to a local copy, then merge it into our tree.  I want to cherry pick
changes from upstream.  I do work and push to our master, I pull work
of coworkers from the master, my coworkers do the same.  Isn’t this
the canonical open source use case?

That means that you have/maintain an intermediate upstream, yes?

This is a bit trickier since once in a while that intermediate upstream
and everyone downstream of it has to catch up with the real upstream.

Here you have two options:

 - the intermediate diverges from the real upstream, and then you
   merge/cherry-pick from the upstream as needed
   
   The intermediate's maintainer must still merge/rebase/cherry-pick
   from the intermediate branch and onto a branch of the upstream in
   order to push to the upstream.

or

 - the intermediate occasionally rebases onto the upstream, and then the
   repos downstream of the intermediate must also rebase with --onto.

   In this case the intermediate's maintainer must tell the downstreams
   what rebase command to execute.

   This makes it easier to push from the intermediate to the upstream:
   the intermediate's commits should always be on top of the upstream
   (at rebase time) so it's easy to push them.

The latter is the workflow we used at Sun for decades for large
projects.  We followed that workflow long before git, first with
Teamware, then later with Mercurial (even though Mercurial technically
had no support for rebasing at that time; we just made it work).

(We always left symbolic names for the pre-rebase branch HEADs, mind
you, to make life easier for everyone.)

quoted

(I find that many users are allergic to rebasing.  Many people have
told me that rebase is lying, that history must be immutable, and so
on, all ignoring that: git users don't rebase published branches,

So, when I push, and someone else pulls, is that published?  I thought
it was.

Yes.  You shouldn't push -f.  As long as you don't there's no problem.

Nico
-- 

On Wed, Aug 06, 2014 at 02:44:59PM -0500, Nico Williams wrote:

That means that you have/maintain an intermediate upstream, yes?

This is a bit trickier since once in a while that intermediate upstream
and everyone downstream of it has to catch up with the real upstream.

Here you have two options:

 - the intermediate diverges from the real upstream, and then you
   merge/cherry-pick from the upstream as needed
   
   The intermediate's maintainer must still merge/rebase/cherry-pick
   from the intermediate branch and onto a branch of the upstream in
   order to push to the upstream.

I should add something important here.

Rebasing makes life easier for the intermediate maintainer, and for any
upstream maintainer who has to merge "pull requests" or patches sent in
email.  Rebasing puts the onus for merging on the contributor, exactly
where it belongs!

(Granted, for an e-mail based workflow one's patches might have made for
a fast-forward merge when sent but not when the upstream gets to them.
With long enough latency this gets painful.  Which is why I don't
recommend an e-mail based commit integration workflow.)

Nico
-- 

On Fri, 2014-08-01 at 09:56 -0700, Mike Stump wrote:

Since everything I do goes up and down into repositories and I don’t want my friends and family to scorn me, rebase isn’t the command I want to use.

You completely mis-understand what "published" means. Published history
is history from which other people can pull right now.

That means it has to be in a publicly addressable repository (ie: just
like the remote that you are pulling from as upstream).

rebasing commits which are already in the upstream is bad. Rebasing
commits which you have created locally is NOT bad. These commits would
not be published until you do a push.

This is the fundamental issue with rebase, and it is infact easy to
avoid mis-using, especially if you don't publish changes. The key is
that a commit isn't published until it's something someone else can
depend on.

Doing "git pull --rebase" essentially doesn't ever get you into trouble.

Regards,
Jake

On Thu, 2014-08-21 at 17:36 +0000, Keller, Jacob E wrote:

On Fri, 2014-08-01 at 09:56 -0700, Mike Stump wrote:

quoted

Since everything I do goes up and down into repositories and I don’t want my friends and family to scorn me, rebase isn’t the command I want to use.

You completely mis-understand what "published" means. Published history
is history from which other people can pull right now.

That means it has to be in a publicly addressable repository (ie: just
like the remote that you are pulling from as upstream).

rebasing commits which are already in the upstream is bad. Rebasing
commits which you have created locally is NOT bad. These commits would
not be published until you do a push.

This is the fundamental issue with rebase, and it is infact easy to
avoid mis-using, especially if you don't publish changes. The key is
that a commit isn't published until it's something someone else can
depend on.

Doing "git pull --rebase" essentially doesn't ever get you into trouble.

Regards,
Jake
�{.n�+�������+%��lzwm��b�맲��r��z��{ay�

ʇڙ�,j��f���h���z�
�w���
���j:+v���w�j�m��������zZ+�����ݢj"��!�i

Pardon me. You can actually ignore this post. I read through more of the
thread, and actually realize I completely misunderstood what your issue
was, and why rebase might not work.

Regards,
Jake