top-git, idea .topbase replacing refs/top-bases, implementation issues and questions..

introduction:

topgit (tg) is a tool keeping track of patches on top of branches so that you
can work on those patches before comitting them upstream.

So you can do
  tg create dep1 dep2
to make your new topic branch depend on the two branches dep1, dep2.

tg first merges dep1 dep2 into a base. On top of this base you start
comitting your changes. when following upstream a new base is created by
merging the old base with dep1, then dep2 again. The new base is merged
into your topic branch.

This requires keeping track of the base. That has been done by adding an
extra references to .git/refs/top-bases/TG_NAME.
problems: You have to push the branch and the base to the remote
location. If you clone a top git related branch you have to get the
bases as well (tg remote --populate does this for you).

Uwe Kleine-König (current maintainer?) has proposed no longer tracking
the topgit branch base using a ref but adding antoher .top* file .topbase
containing the hash of the base. Neither pushing nor pulling causes
trouble because the base is always contained in the topic branch. So no
need to extra push/ pull contents of that branch. Another benefit of
this change is that you can easily make a top git branch out of an
ordinary branch by adding those 3 files (.topbase, .topmsg, .topdeps).

I don't see any problems implementing this except one. I'm not sure in
which way this has been adressed in the past by topgit:


Have a look at the following illustration:
http://mawercer.de/~marc/tg.svg (32kb).

Imagine two independant clones of a topgit branch (red, green).
Now users run tg update at slightly different points in time. So both
will get a different new base (B2 and B3) of the clone topgit branch.
Now red merges the topic branch (merge 1). Then a third user (blue)
merges that merge (merge 2). If a merge strategy "ours" is used the way
its done for the other .top* files the blue user's base won't know about
B3.

First merge we have a conflict:

If B3 and B2 are different a merge conflict occurs:
.topbase:
<<<<
hash1
====
hash2

quoted

quoted

quoted

Now the two bases should be merged to a new base (Bnew).
That new base should be written to .topbase and everything is fine
again?

Is there a way to specify such a merge action:
extract both hashes (easy)
merge them and put the resulting hash back into .topbase.
Of course this merge action must handle merge conflicts in a way which
is similar to what tg already does. tg starts a subshell and asking the
user to fix the conflicts.

Even if there is a nice way to implement this merge action this doesn't
seem to solve all trouble?
The Bnew base is neither part of the red nor the green topic branch.
Thus it can't be pushed somewhere by pushing red or green only
containing the ref in .topbase?

So in the end:
Have I missed a point ?

So are there more  advantages having a self contained top git branch
compared to beeing able to merge bases?

Maybe merging bases doesn't occur that often in practise?

So if we don't want to give up beeing able to merge bases I 
till want to see a git push branch command also pushing the base.
Currently you can do $git push which will push *all* bases and branches.

Probably I've missed some small details while explaining this idea and
talking about its issues. So please do ask.

Sincerly
Marc Weber

Hi Marc,

I supplement and correct your description a bit, not to be pedantic but
to give the other readers a better overview.

On Mon, Apr 20, 2009 at 10:50:23PM +0200, Marc Weber wrote:

introduction:

topgit (tg) is a tool keeping track of patches on top of branches so that you
can work on those patches before comitting them upstream.

And IMHO more importantly you can update your branches after feedback
from upstream without rebasing.

So you can do
  tg create dep1 dep2

This should read
	tg create TG_NAME dep1 dep2

to make your new topic branch depend on the two branches dep1, dep2.

tg first merges dep1 dep2 into a base. On top of this base you start
comitting your changes. when following upstream a new base is created by
merging the old base with dep1, then dep2 again. The new base is merged
into your topic branch.

The patch represented by TG_NAME is always the diff between the base and
the topic branch.
 

This requires keeping track of the base. That has been done by adding an
extra references to .git/refs/top-bases/TG_NAME.
problems: You have to push the branch and the base to the remote
location. If you clone a top git related branch you have to get the
bases as well (tg remote --populate does this for you).

tg remote adds some config to your repo such that a later git fetch gets
the top-bases, too.

Uwe Kleine-König (current maintainer?

yes

                                     ) has proposed no longer tracking
the topgit branch base using a ref but adding antoher .top* file .topbase
containing the hash of the base.

I think for now using both is a good start.  There are too many places
in topgit that depend on the topbase branch.

                                 Neither pushing nor pulling causes
trouble because the base is always contained in the topic branch. So no
need to extra push/ pull contents of that branch. Another benefit of
this change is that you can easily make a top git branch out of an
ordinary branch by adding those 3 files (.topbase, .topmsg, .topdeps).

hhm, not adding .topbase but another (git) branch isn't much harder, is
it?
 

I don't see any problems implementing this except one. I'm not sure in
which way this has been adressed in the past by topgit:


Have a look at the following illustration:
http://mawercer.de/~marc/tg.svg (32kb).

Either I don't interpret your image correctly or it's not exact.
B1 is a base?  Where is the corresponding topic-branch that both cloners
should have in common?  Further it's not clear to me if merge1 and
merge2 are bases or topic branches.

Let me try an ascii-history with slightly different names ...

Imagine two independant clones of a topgit branch (red, green).

This is the state cloned by red and green:

	                            T.origin
	                           /
	dep1.1 --- dep1.2--B.origin
	                  /
	dep2.1 --- dep2.2-

where B.something is a topbase, T.thesamesomething is the corresponding
topic branch.

Now users run tg update at slightly different points in time. So both
will get a different new base (B2 and B3) of the clone topgit branch.

Then it looks as follows (simplified to only a single dependency dep1
and assuming they get different commits of dep1, not only different
times, as suggested by your image).

	                      ,------------------------T.green
	                     /                          /
	             ,-------------------------------B.green
	            /      /                        /
	           /   T.origin----T.red           /
	          /   /           /               /
	        B.origin-------B.red             /
	      /               /                 /
	dep1.2----------dep1.3----------------dep1.4

so my B.red is your B2, my B.green is your B3.

Now red merges the topic branch (merge 1).

The image suggests red merges green's work.

OK, I fail to make a picture for that, anyhow then we have:

	B.red2 = B.red + B.green
	T.red2 = T.red + T.green + B.red2

where + stands for a merge[1][2].

And obviously T.red2:.topbase has to contain $(git rev-parse B.red2)

                                           Then a third user (blue)
merges that merge (merge 2).

OK, blue having T.origin merges T.red2, then we get:

	B.blue = B.origin + B.red2
	T.blue = T.origin + T.red2 + B.blue

                             If a merge strategy "ours" is used the way
its done for the other .top* files the blue user's base won't know about
B3.

"ours" is wrong for .topbase.  You have to merge to get the right
content.

First merge we have a conflict:

If B3 and B2 are different a merge conflict occurs:
.topbase:
<<<<
hash1
====
hash2

quoted

quoted

quoted

quoted

I think it's not helpful to thing about merge conflicts, you simply need
to handle .topbase explicitly.

Now the two bases should be merged to a new base (Bnew).
That new base should be written to .topbase and everything is fine
again?

Is there a way to specify such a merge action:
extract both hashes (easy)
merge them and put the resulting hash back into .topbase.
Of course this merge action must handle merge conflicts in a way which
is similar to what tg already does. tg starts a subshell and asking the
user to fix the conflicts.

Actually I don't like this subshelling.  git does it better, it saves
the state somewhere in .git and exits.  I'm sure that people who's shell
is != /bin/sh don't like that we spawn a sh.
 

Even if there is a nice way to implement this merge action this doesn't
seem to solve all trouble?

I don't see any trouble, but maybe it's me who missed something.

The Bnew base is neither part of the red nor the green topic branch.
Thus it can't be pushed somewhere by pushing red or green only
containing the ref in .topbase?

I fail to understand that.  only blue has Bnew, so of course neither red
nor green can push it. ???

So in the end:
Have I missed a point ?

So are there more  advantages having a self contained top git branch
compared to beeing able to merge bases?

You can have both.

Maybe merging bases doesn't occur that often in practise?

you need it whenever you merge a remote.  There is no way around it.

So if we don't want to give up beeing able to merge bases I 
till want to see a git push branch command also pushing the base.
Currently you can do $git push which will push *all* bases and branches.

Alternatively you can push the branch+base explicitly to have more
control.  OK, not very comfortable, though you could wrap that in a tg
push command.

Best regards
Uwe

[1] I have not thought deeply about which is the best of the various
possibilies to update T when merging, here are some of them:

	1) T.mynew = (T.myold + T.tomerge) + B.mynew
	2) T.mynew = T.myold + (T.tomerge + B.mynew)
	3) T.mynew = (T.myold + B.mynew) + T.tomerge
	4) T.mynew = T.myold + (B.mynew + T.tomerge)
	5) T.mynew = T.myold + T.tomerge

5) is out because then T.mynew doesn't contain B.mynew, but it only
misses a single merge, namely B.myold + B.tomerge.  Both bases as such
are already included in T.myold + T.tomerge.  So this is only a "would
be nice".
Ad hoc I'd choose 3) but this needs some more thinking.
Just from reading the code, I'd say topgit currently does 4)

[2] and note, that + has to behave differently in different situations
for the .top* files.  The following merges can happen

	a) merge a base into a topic
	b) merge two bases
	c) merge two topics

and depending on this the following low-level merge drivers should be
used for .topmsg and .topdeps:

	a): ours
	b): don't care (delete?)
	c): text

It would be nice if the topbases didn't contain any .top* files, this
only has some overhead if a patch has single (tgish) dependency because
then you cannot just use this branch as base.

-- 
Pengutronix e.K.                              | Uwe Kleine-König            |
Industrial Linux Solutions                    | http://www.pengutronix.de/  |

Hi Uwe,

I came up with this proposal for a rewrite for tg-update.sh now:
I hope this fits your ideas about tg update as well..

Note that second "for each dependency [1]" below allowing handling updates
against multiple remote repos.

I hope that its easy to understand this short documentation draft.

Sincerly
Marc Weber

# documentation:

# Uwe Klein Koenig suggests keeping updating with remote branches
# which corresponds to "git pull" in default git separate from
# updating branches with deps (normal branhces and/or tg branches)
# 
# Because of [3] my implementation of tg update starts at the bottom going up
# first updating with remotes then with deps by default.
# However you can choose to only update with remotes
# or with deps only using --remotes-only or --local-only
# I hope this satisfies everyone.

# Providing a subshellless update experience can only be done by saving
# some state the way git-rebase does. STATE_DIR will be used to save this state
#
# What's saved in STATE_DIR ?
#
# $STATE_DIR/$BRANCHES_VISITING_ORDER is a list of dependency names [1] which is generated
# by traversing the dependency tree of branches form bottom up
#
#  Example:
#     A    B  C   D
#      \  /    \ /
#       t/1   t/2
#         \   /
#       t/working
#
#  will result in
#
#     A
#     B
#     t/1
#     C
#     D
#     t/2
#     t/working

# Then the first line is picked and the specified branch is updated.
# updating is done in this order [3]:
#
#   first:
#   if not --local-only
#      foreach remote location:
#        - merge base with remote tg branch (thereby also including the remote base)
#          update the base ref [2]
#        - update tg [*]
#    
#   then:
#   if not --remote-only
#      for each dependency [1]
#         - merge base with dep [2]
#      - update tg [*]
#    
#     [*]: merge new base with topgit branch updating the topgit ref.
#          After this step the base is always contained in the current branch
#
# Similar to $STATE_DIR/BRANCHES_VISITING_ORDER a file $STATE_DIR/$BRANCH_UPDATE_TASKS is
# created containing the subtasks of a branch update.
# 
# Example file with additional comment lines, two remote locations:
#
#   # first:
#   merge_base_with: $REMOTE_HASH
#   update_tg
#   merge_base_with: $REMOTE_HASH2
#   update_tg
#
#   # then:
#   merge_base_with: $DEP1_HASH
#   update_tg
#   merge_base_with: $DEP2_HASH
#   update_tg
#   merge_base_with: $DEP3_HASH
#   update_tg



# So the overall implementation is easy and looks like this:
#
# if !exists $STATE_DIR/BRANCHES_VISITING_ORDER
#   then create file, goto else
#   else pop_or_continue
#
# pop_or_continue:
# if !exists $STATE_DIR/$BRANCH_UPDATE_TASKS
#   then create file, goto else
#   else pop_or_continue_subtask
#
#  pop_or_continue_subtask:
#  ...

# [1] note for users who knew the old code: not including remotes
#
# [2] This means creating a commit to the topic branch changing the .topbase file only
#
# [3] I think the following statements make sense:
#     *) You want to update against remote first. Or you want to have a way to do so.
#        Why? The remote may already have done some of the dirty merge work for you.
#        If you resolving the same conflicts first by updating with deps and then merge
#        with remote -> hell!
#     *) You don't want to run tg update multiple times. Thus the update strategy has
#        to start at the bottom and go up to the current branch



# state dir and filenames:
STATE_DIR=.git/tg-update-in-progress 
BRANCHES_VISITING_ORDER=branches-visiting-order
BRANCH_UPDATE_TASKS=branch-update-tasks

## Parse options

if [ -n "$1" ]; then
        echo "Usage: tg [--local-only] [--remote-only] update" >&2
        exit 1
fi