Notes on Subproject Support

This is still a draft/WIP, but "release early" is a good
discipline, so...

-- >8 --

Notes on Subproject Support
===========================
Junio C Hamano [off-list ref]
v1.0 January 22, 2006

Scenario
--------

The examples in the following discussion show how this proposal
plans to help this:

. A project to build an embedded Linux appliance "gadget" is
  maintained with git.

. The project uses linux-2.6 kernel as its subcomponent.  It
  starts from a particular version of the mainline kernel, but
  adds its own code and build infrastructure to fit the
  appliance's needs.

. The working tree of the project is laid out this way:
+
------------
 Makefile       - Builds the whole thing.
 linux-2.6/     - The kernel, perhaps modified for the project.
 appliance/     - Applications that run on the appliance, and
                  other bits.
------------

. The project is willing to maintain its own changes out of tree
  of the Linux kernel project, but would want to be able to feed
  the changes upstream, and incorporate upstream changes to its
  own tree, taking advantage of the fact that both itself and
  the Linux kernel project are version controlled with git.

The idea here is to:

. Keep `linux-2.6/` part as an independent project.  The work by
  the project on the kernel part can be naturally exchanged with
  the other kernel developers this way.  Specifically, a tree
  object contained in commit objects belonging to this project
  does *not* have linux-2.6/ directory at the top.

. Keep the `appliance/` part as another independent project.
  Applications are supposed to be more or less independent from
  the kernel version, but some other bits might be tied to a
  specific kernel version.  Again, a tree object contained in
  commit objects belonging to this project does *not* have
  appliance/ directory at the top.

. Have another project that combines the whole thing together,
  so that the project can keep track of which versions of the
  parts are built together.

We will call the project that binds things together the
'toplevel project'.  Other projects that hold `linux-2.6/` part
and `appliance/` part are called 'subprojects'.

Notice that `Makefile` at the top is part of the toplevel
project in this example, but it is not necessary.  We could
instead have the appliance subproject include this file.  In
such a setup, the appliance subproject would have had `Makefile`
and `appliance/` directory at the toplevel.


Setting up
----------

Let's say we have been working on the appliance software,
independently version controlled with git.  Also the kernel part
has been version controlled separately, like this:
------------
$ ls -dF current/*/.git current/*
current/Makefile    current/appliance/.git/  current/linux-2.6/.git/
current/appliance/  current/linux-2.6/
------------

Now we would want to get a combined project.  First we would
clone from these repositories (which is not strictly needed --
we could use `$GIT_ALTERNATE_OBJECT_DIRECTORIES` instead):

------------
$ mkdir combined && cd combined
$ cp ../current/Makefile .
$ git init-db
$ mkdir -p .git/refs/subs/{kernel,gadget}/{heads,tags}
$ git clone-pack ../current/linux-2.6/ master | read kernel_commit junk
$ git clone-pack ../current/appliance/ master | read gadget_commit junk
------------

We will introduce a new command to set up a combined project:

------------
$ git bind-projects \
	$kernel_commit linux-2.6/ \
	$gadget_commit appliance/
------------

This would do an equivalent of:

------------
$ git read-tree --prefix=linux-2.6/ $kernel_commit
$ git read-tree --prefix=appliance/ $gadget_commit
------------
[NOTE]
============
Earlier outlines sent to the git mailing list talked
about `$GIT_DIR/bind` to record what subproject are bound to
which subtree in the curent working tree and index.  This
proposal instead records that information in the index file
when `--prefix=linux-2.6/` is given to `read-tree`.

Also note that in this round of proposal, there is no separate
branches that keep track of heads of subprojects.
============

Let's not forget to add the `Makefile`, and check the whole
thing out from the index file.
------------
$ git add Makefile
$ git checkout-index -f -u -q -a
------------

Now our directory should be identical with the `current`
directory.  After making sure of that, we should be able to
commit the whole thing:

------------
$ diff -x .git -r ../current ../combined
$ git commit -m 'Initial toplevel project commit'
------------

Which should create a new commit object that records what is in
the index file as its tree, with `bind` lines to record which
subproject commit objects are bound at what subdirectory, and
updates the `$GIT_DIR/refs/heads/master`.  Such a commit object
might look like this:
------------
tree 04803b09c300c8325258ccf2744115acc4c57067
bind 5b2bcc7b2d546c636f79490655b3347acc91d17f linux-2.6/
bind 0bdd79af62e8621359af08f0afca0ce977348ac7 appliance/
author Junio C Hamano [off-list ref] 1137965565 -0800
committer Junio C Hamano [off-list ref] 1137965565 -0800

Initial toplevel project commit
------------


Making further commits
----------------------

The easiest case is when you updated the Makefile without
changing anything in the subprojects.  In such a case, we just
need to create a new commmit object that records the new tree
with the current `HEAD` as its parent, and with the same set of
`bind` lines.

When we have changes to the subproject part, we would make a
separate commit to the subproject part and then record the whole
thing by making a commit to the toplevel project.  The user
interaction might go this way:
------------
$ git commit
error: you have changes to the subproject bound at linux-2.6/.
$ git commit --subproject linux-2.6/
$ git commit
------------

With the new `--subproject` option, the directory structure
rooted at `linux-2.6/` part is written out as a tree, and a new
commit object that records that tree object with the commit
bound to that portion of the tree (`5b2bcc7b` in the above
example) as its parent is created.  Then the final `git commit`
would record the whole tree with updated `bind` line for the
`linux-2.6/` part.


Checking out
------------

After cloning such a toplevel project, `git clone` without `-n`
option would check out the working tree.  This is done by
reading the tree object recorded in the commit object (which
records the whole thing), and adding the information from the
"bind" line to the index file.

------------
$ cd ..
$ git clone -n combined cloned ;# clone the one we created earlier
$ cd cloned
$ git checkout
------------

This round of proposal does not maintain separate branch heads
for subprojects.  The bound commits and their subdirectories
are recorded in the index file from the commit object, so there
is no need to do anything other than updating the index and the
working tree.


Switching branches
------------------

Along with the traditional two-way merge by `read-tree -m -u`,
we would need to look at:

. `bind` lines in the current `HEAD` commit.

. `bind` lines in the commit we are switching to.

. subproject binding information in the index file.

to make sure we do sensible things.

Just like until very recently we did not allow switching
branches when two-way merge would lose local changes, we can
start by refusing to switch branches when the subprojects bound
in the index do not match what is recorded in the `HEAD` commit.

Because in this round of the proposal we do not use the
`$GIT_DIR/bind` file nor separate branches to keep track of
heads of the subprojects, there is nothing else other than the
working tree and the index file that needs to be updated when
switching branches.


Merging
-------

Merging two branches of the toplevel projects can use the
traditional merging mechanism mostly unchanged.  The merge base
computation can be done using the `parent` ancestry information
taken from the two toplevel project branch heads being merged,
and merging of the whole tree can be done with a three-way merge
of the whole tree using the merge base and two head commits.
For reasons described later, we would not merge the subproject
parts of the trees during this step, though.

When the two branch heads use different versions of subproject,
things get a bit tricky.  First, let's forget for a moment about
the case where they bind the same project at different location.
We would refuse if they do not have the same number of `bind`
lines that bind something at the same subdirectories.

------------
$ git merge 'Merge in a side branch' HEAD side
error: the merged heads have subprojects bound at different places.
 ours:
	linux-2.6/
	appliance/
 theirs:
	kernel/
	gadget/
	manual/
------------

Such renaming can be handled by first moving the bind points in
our branch, and redoing the merge (this is a rare operation
anyway).  It might go like this:

------------
$ git bind-projects \
	$kernel_commit kernel/ \
	$gadget_commit gadget/
$ git commit -m 'Prepare for merge with side branch'
$ git merge 'Merge in a side branch' HEAD side
error: the merged heads have subprojects bound at different places.
 ours:
	kernel/
	gadget/
 theirs:
	kernel/
	gadget/
	manual/
------------

Their branch added another subproject, so this did not work (or
it could be the other way around -- we might have been the one
with `manual/` subproject while they didn't).  This suggests
that we may want an option to `git merge` to allow taking a
union of subprojects.  Again, this is a rare operation, and
always taking a union would have created a toplevel project that
had both `kernel/` and `linux-2.6/` bound to the same Linux
kernel project from possibly different vintage, so it would be
prudent to require the set of bound subprojects to exactly match
and give the user an option to take a union.

------------
$ git merge --union-subprojects 'Merge in a side branch HEAD side
error: the subproject at `kernel/` needs to be merged first.
------------

Here, the version of the Linux kernel project in the `side`
branch was different from what our branch had on our `bind`
line.  On what kind of difference should we give this error?
Initially, I think we could require one is the fast forward of
the other (ours might be ahead of theirs, or the other way
around), and take the descendant.

Or we could do an independent merge of subprojects heads, using
the `parent` ancestry of the bound subproject heads to find
their merge-base and doing a three-way merge.  This would leave
the merge result in the subproject part of the working tree and
the index.

[NOTE]
This is the reason we did not do the whole-tree three way merge
earlier.  The subproject commit bound to the merge base commit
used for the toplevel project may not be the merge base between
the subproject commits bound to the two toplevel project
commits.

So let's deal with the case to merge only a subproject part into
our tree first.


Merging subprojects
-------------------

An operation of more practical importance is to be able to merge
in changes done outside to the projects bound to our toplevel
project.

------------
$ git pull --subproject=kernel/ git://git.kernel.org/.../linux-2.6/
------------

might do:

. fetch the current `HEAD` commit from Linus.
. find the subproject commit bound at kernel/ subtree.
. perform the usual three-way merge of these two commits, in
  `kernel/` part of the working tree.

After that, `git commit \--subproject` option would be needed to
make a commit.

[NOTE]
This suggests that we would need to have something similar to
`MERGE_HEAD` for merging the subproject part.

On Sun, 22 Jan 2006, Junio C Hamano wrote:

Also note that in this round of proposal, there is no separate
branches that keep track of heads of subprojects.

Interesting; I think it may become useful to allow for such heads, but we 
can deal with that when it arises. (e.g., maybe you want to use topic 
branches in the kernel development you do in the linux-2.6/ subdirectory 
of your superproject working tree; so long as the core isn't using refs 
for its own purposes, this is up to the user to keep straight and we can 
help later when we have usage notes)

============

Let's not forget to add the `Makefile`, and check the whole
thing out from the index file.
------------
$ git add Makefile

Maybe bind-projects should be "add-projects", to match "add", which has a 
similar effect at the user level?

$ git checkout-index -f -u -q -a
------------

Now our directory should be identical with the `current`
directory.  After making sure of that, we should be able to
commit the whole thing:

------------
$ diff -x .git -r ../current ../combined
$ git commit -m 'Initial toplevel project commit'
------------

Which should create a new commit object that records what is in
the index file as its tree, with `bind` lines to record which
subproject commit objects are bound at what subdirectory, and
updates the `$GIT_DIR/refs/heads/master`.  Such a commit object
might look like this:
------------
tree 04803b09c300c8325258ccf2744115acc4c57067

Does this tree include trees for the bound projects?

bind 5b2bcc7b2d546c636f79490655b3347acc91d17f linux-2.6/
bind 0bdd79af62e8621359af08f0afca0ce977348ac7 appliance/
author Junio C Hamano [off-list ref] 1137965565 -0800
committer Junio C Hamano [off-list ref] 1137965565 -0800

Initial toplevel project commit
------------


Making further commits
----------------------

The easiest case is when you updated the Makefile without
changing anything in the subprojects.  In such a case, we just
need to create a new commmit object that records the new tree
with the current `HEAD` as its parent, and with the same set of
`bind` lines.

When we have changes to the subproject part, we would make a
separate commit to the subproject part and then record the whole
thing by making a commit to the toplevel project.  The user
interaction might go this way:
------------
$ git commit
error: you have changes to the subproject bound at linux-2.6/.
$ git commit --subproject linux-2.6/
$ git commit
------------

I think "cd linux-2.6 && git commit" should work for the subproject, too, 
but that can be a later enhancement.

With the new `--subproject` option, the directory structure
rooted at `linux-2.6/` part is written out as a tree, and a new
commit object that records that tree object with the commit
bound to that portion of the tree (`5b2bcc7b` in the above
example) as its parent is created.

And the commit is written to the index, in the special slot for the 
subproject, replacing its parent, I assume.

Switching branches
------------------

Along with the traditional two-way merge by `read-tree -m -u`,
we would need to look at:

. `bind` lines in the current `HEAD` commit.

. `bind` lines in the commit we are switching to.

. subproject binding information in the index file.

to make sure we do sensible things.

This is one place I think storing the bindings in the commit is awkward. 
read-tree deals in trees (hence the name), but will need information from 
the commit.

I think it should be possible to hide the existance of subtrees in an 
add-on to the struct tree API such that code that doesn't handle it 
specifically doesn't see a difference, similarly to how the index file can 
be handled. (parse_tree would fill out the structure as if the subproject 
were a tree instead of a commit, assuming that the structure it's 
pretending to be is the full tree, but there would be an additional 
field for the commit if it's a subproject, until we've gone through 
everything to make it work with subprojects).

I'm hoping to kill off the other tree object parser, which is only used by 
ls-tree and diff-index at this point, but my workstation's home directory 
hard drive seems to have gotten weirdly messed up at the hardware level 
(and seems to have lost a lot of the contents of unused storage, or 
something), so this may take a little while. At that point, whatever 
special things we do in tree objects can be handled automatically with 
changes only to a single location.

	-Daniel
*This .sig left intentionally blank*

Daniel Barkalow [off-list ref] writes:

quoted

tree 04803b09c300c8325258ccf2744115acc4c57067

Does this tree include trees for the bound projects?

Yes, this part has not been changed from earlier thoughts.

quoted

bind 5b2bcc7b2d546c636f79490655b3347acc91d17f linux-2.6/
bind 0bdd79af62e8621359af08f0afca0ce977348ac7 appliance/
author Junio C Hamano [off-list ref] 1137965565 -0800
committer Junio C Hamano [off-list ref] 1137965565 -0800

The tree 04803b...  tree has everything.  If you run git-ls-tree
on 04803b... would have a tree object recorded at linux-2.6, and
it is the same as the tree associated with the commit 5b2bcc...

I think "cd linux-2.6 && git commit" should work for the subproject, too, 
but that can be a later enhancement.

It's just a matter of Porcelain scripting, so that is probably
true.  However I do not want people to get too used to it and
expect "cd Documentation && git commit" to work in git.git
repository.

quoted

With the new `--subproject` option, the directory structure
rooted at `linux-2.6/` part is written out as a tree, and a new
commit object that records that tree object with the commit
bound to that portion of the tree (`5b2bcc7b` in the above
example) as its parent is created.

And the commit is written to the index, in the special slot for the 
subproject, replacing its parent, I assume.

Yes.  It would probably be done with `update-index --bind` to
update the bound subproject commit there.

quoted

Switching branches
------------------

Along with the traditional two-way merge by `read-tree -m -u`,
we would need to look at:

. `bind` lines in the current `HEAD` commit.

. `bind` lines in the commit we are switching to.

. subproject binding information in the index file.

to make sure we do sensible things.

This is one place I think storing the bindings in the commit is awkward. 
read-tree deals in trees (hence the name), but will need information from 
the commit.

That's why it is 'along with'.  Dealing with binding information
can be done between commits and index without bothering tree
objects.  read-tree would not have to deal with it, and I think
keeping it that way is probably a good idea.

In other words, I think the design so far does not require us to
touch tree objects at all, and I'd be happy if we do not have to.

One reason I started the bound commit approach was exactly
because I only needed to muck with commit objects and did not
have to touch trees and blobs; after trying to implement the
core level for "gitlink", which I ended up touching quite a lot
and have abandoned for now.

Here is an update to the still-WIP draft.

-- >8 --
Separate role of read-tree and update-index cleaner

The previous draft prematurely merged what read-tree --prefix
does with what update-index --bind would do.  Keep them separate
for now until we know what the common patterns would be.

Introduce 'update-index --unbind'.  We would probably need a new
command that extracts bind information out of index when we
start writing Porcelainish, but it is not specified yet.

Attempt to clarify what the "merging into subproject part" would
do a bit.  "git pull --subproject=" is fetch + merge, just like
the current subproject-unaware 'git pull' is.

---

diff --git a/Subpro.txt b/Subpro.txt
index 4036e71..837cab8 100644
--- a/Subpro.txt
+++ b/Subpro.txt

@@ -95,19 +95,22 @@ $ git bind-projects \
 	$gadget_commit appliance/
 ------------
 
-This would do an equivalent of:
+This would probably do an equivalent of:
 
 ------------
+$ rm -f "$GIT_DIR/index"
 $ git read-tree --prefix=linux-2.6/ $kernel_commit
 $ git read-tree --prefix=appliance/ $gadget_commit
+$ git update-index --bind linux-2.6/ $kernel_commit
+$ git update-index --bind appliance/ $gadget_commit
 ------------
 [NOTE]
 ============
 Earlier outlines sent to the git mailing list talked
 about `$GIT_DIR/bind` to record what subproject are bound to
-which subtree in the curent working tree and index.  This
+which subtree in the current working tree and index.  This
 proposal instead records that information in the index file
-when `--prefix=linux-2.6/` is given to `read-tree`.
+with `update-index --bind` command.
 
 Also note that in this round of proposal, there is no separate
 branches that keep track of heads of subprojects.

@@ -258,9 +261,11 @@ our branch, and redoing the merge (this 
 anyway).  It might go like this:
 
 ------------
-$ git bind-projects \
-	$kernel_commit kernel/ \
-	$gadget_commit gadget/
+$ git reset
+$ git update-index --unbind linux-2.6/
+$ git update-index --unbind appliance/
+$ git update-index --bind $kernel_commit kernel/
+$ git update-index --bind $gadget_commit gadget/
 $ git commit -m 'Prepare for merge with side branch'
 $ git merge 'Merge in a side branch' HEAD side
 error: the merged heads have subprojects bound at different places.

@@ -336,7 +341,55 @@ make a commit.
 
 [NOTE]
 This suggests that we would need to have something similar to
-`MERGE_HEAD` for merging the subproject part.
+`MERGE_HEAD` for merging the subproject part.  In the case of
+merging two toplevel project commits, we probably can read the
+`bind` lines from the `MERGE_HEAD` commit and either our `HEAD`
+commit or our index file.  Further, we probably would require
+that the latter two must match, just as we currently require the
+index file matches our `HEAD` commit before `git merge`.
 
+Just like the current `pull = fetch + merge` semantics, the
+subproject aware version `git pull \--subproject=frotz` would be
+a `git fetch \--subproject=frotz` followed by a `git merge
+\--subproject=frotz`.  So the above would be:
 
+. Fetch the head.
++
+------------
+$ git fetch --subproject=kernel/ git://git.kernel.org/.../linux-2.6/
+------------
++
+which would do:
+. fetch the commit chain from the remote repository.
+. write something like this to `FETCH_HEAD`:
++
+------------
+3ee68c4...\tfor-merge-into kernel/\tbranch 'master' of git://.../linux-2.6
+------------
+
+. Run `git merge`.
++
+------------
+$ git merge --subproject=kernel/ \
+    'Merge git://.../linux-2.6 into kernel/' HEAD 3ee68c4...
+------------
+
+. In case it does not cleanly automerge, `git merge` would write
+the necessary information for a later `git commit` to use in
+`MERGE_HEAD`.  It may look like this:
++
+------------
+3ee68c4af3fd7228c1be63254b9f884614f9ebb2	kernel/
+------------
+
+With this, a later invocation of `git commit` to record the
+result of hand resolving would be able to notice that:
+
+. We should be first resolving `kernel/` subproject.
+. The remote `HEAD` is `3ee68c4...` commit.
+. The merge message is `Merge git://.../linux-2.6 into kernel/`.
 
+and make a merge commit, and register that resulting commit in
+the index file using `update-index --bind` instead of updating
+*any* branch head (remember, we do not use separate branches to
+keep track of subproject heads anymore).

This proposal doesn't seem to cator for the event when a directory is
renamed or moved to a different location, or am I missing something?

- Martin -
On Sun, Jan 22, 2006 at 05:35:14PM -0800, Junio C Hamano wrote:

This is still a draft/WIP, but "release early" is a good
discipline, so...

-- >8 --

Notes on Subproject Support
===========================
Junio C Hamano [off-list ref]
v1.0 January 22, 2006

Scenario
--------

The examples in the following discussion show how this proposal
plans to help this:

. A project to build an embedded Linux appliance "gadget" is
  maintained with git.

. The project uses linux-2.6 kernel as its subcomponent.  It
  starts from a particular version of the mainline kernel, but
  adds its own code and build infrastructure to fit the
  appliance's needs.

. The working tree of the project is laid out this way:
+
------------
 Makefile       - Builds the whole thing.
 linux-2.6/     - The kernel, perhaps modified for the project.
 appliance/     - Applications that run on the appliance, and
                  other bits.
------------

. The project is willing to maintain its own changes out of tree
  of the Linux kernel project, but would want to be able to feed
  the changes upstream, and incorporate upstream changes to its
  own tree, taking advantage of the fact that both itself and
  the Linux kernel project are version controlled with git.

The idea here is to:

. Keep `linux-2.6/` part as an independent project.  The work by
  the project on the kernel part can be naturally exchanged with
  the other kernel developers this way.  Specifically, a tree
  object contained in commit objects belonging to this project
  does *not* have linux-2.6/ directory at the top.

. Keep the `appliance/` part as another independent project.
  Applications are supposed to be more or less independent from
  the kernel version, but some other bits might be tied to a
  specific kernel version.  Again, a tree object contained in
  commit objects belonging to this project does *not* have
  appliance/ directory at the top.

. Have another project that combines the whole thing together,
  so that the project can keep track of which versions of the
  parts are built together.

We will call the project that binds things together the
'toplevel project'.  Other projects that hold `linux-2.6/` part
and `appliance/` part are called 'subprojects'.

Notice that `Makefile` at the top is part of the toplevel
project in this example, but it is not necessary.  We could
instead have the appliance subproject include this file.  In
such a setup, the appliance subproject would have had `Makefile`
and `appliance/` directory at the toplevel.


Setting up
----------

Let's say we have been working on the appliance software,
independently version controlled with git.  Also the kernel part
has been version controlled separately, like this:
------------
$ ls -dF current/*/.git current/*
current/Makefile    current/appliance/.git/  current/linux-2.6/.git/
current/appliance/  current/linux-2.6/
------------

Now we would want to get a combined project.  First we would
clone from these repositories (which is not strictly needed --
we could use `$GIT_ALTERNATE_OBJECT_DIRECTORIES` instead):

------------
$ mkdir combined && cd combined
$ cp ../current/Makefile .
$ git init-db
$ mkdir -p .git/refs/subs/{kernel,gadget}/{heads,tags}
$ git clone-pack ../current/linux-2.6/ master | read kernel_commit junk
$ git clone-pack ../current/appliance/ master | read gadget_commit junk
------------

We will introduce a new command to set up a combined project:

------------
$ git bind-projects \
	$kernel_commit linux-2.6/ \
	$gadget_commit appliance/
------------

This would do an equivalent of:

------------
$ git read-tree --prefix=linux-2.6/ $kernel_commit
$ git read-tree --prefix=appliance/ $gadget_commit
------------
[NOTE]
============
Earlier outlines sent to the git mailing list talked
about `$GIT_DIR/bind` to record what subproject are bound to
which subtree in the curent working tree and index.  This
proposal instead records that information in the index file
when `--prefix=linux-2.6/` is given to `read-tree`.

Also note that in this round of proposal, there is no separate
branches that keep track of heads of subprojects.
============

Let's not forget to add the `Makefile`, and check the whole
thing out from the index file.
------------
$ git add Makefile
$ git checkout-index -f -u -q -a
------------

Now our directory should be identical with the `current`
directory.  After making sure of that, we should be able to
commit the whole thing:

------------
$ diff -x .git -r ../current ../combined
$ git commit -m 'Initial toplevel project commit'
------------

Which should create a new commit object that records what is in
the index file as its tree, with `bind` lines to record which
subproject commit objects are bound at what subdirectory, and
updates the `$GIT_DIR/refs/heads/master`.  Such a commit object
might look like this:
------------
tree 04803b09c300c8325258ccf2744115acc4c57067
bind 5b2bcc7b2d546c636f79490655b3347acc91d17f linux-2.6/
bind 0bdd79af62e8621359af08f0afca0ce977348ac7 appliance/
author Junio C Hamano [off-list ref] 1137965565 -0800
committer Junio C Hamano [off-list ref] 1137965565 -0800

Initial toplevel project commit
------------


Making further commits
----------------------

The easiest case is when you updated the Makefile without
changing anything in the subprojects.  In such a case, we just
need to create a new commmit object that records the new tree
with the current `HEAD` as its parent, and with the same set of
`bind` lines.

When we have changes to the subproject part, we would make a
separate commit to the subproject part and then record the whole
thing by making a commit to the toplevel project.  The user
interaction might go this way:
------------
$ git commit
error: you have changes to the subproject bound at linux-2.6/.
$ git commit --subproject linux-2.6/
$ git commit
------------

With the new `--subproject` option, the directory structure
rooted at `linux-2.6/` part is written out as a tree, and a new
commit object that records that tree object with the commit
bound to that portion of the tree (`5b2bcc7b` in the above
example) as its parent is created.  Then the final `git commit`
would record the whole tree with updated `bind` line for the
`linux-2.6/` part.


Checking out
------------

After cloning such a toplevel project, `git clone` without `-n`
option would check out the working tree.  This is done by
reading the tree object recorded in the commit object (which
records the whole thing), and adding the information from the
"bind" line to the index file.

------------
$ cd ..
$ git clone -n combined cloned ;# clone the one we created earlier
$ cd cloned
$ git checkout
------------

This round of proposal does not maintain separate branch heads
for subprojects.  The bound commits and their subdirectories
are recorded in the index file from the commit object, so there
is no need to do anything other than updating the index and the
working tree.


Switching branches
------------------

Along with the traditional two-way merge by `read-tree -m -u`,
we would need to look at:

. `bind` lines in the current `HEAD` commit.

. `bind` lines in the commit we are switching to.

. subproject binding information in the index file.

to make sure we do sensible things.

Just like until very recently we did not allow switching
branches when two-way merge would lose local changes, we can
start by refusing to switch branches when the subprojects bound
in the index do not match what is recorded in the `HEAD` commit.

Because in this round of the proposal we do not use the
`$GIT_DIR/bind` file nor separate branches to keep track of
heads of the subprojects, there is nothing else other than the
working tree and the index file that needs to be updated when
switching branches.


Merging
-------

Merging two branches of the toplevel projects can use the
traditional merging mechanism mostly unchanged.  The merge base
computation can be done using the `parent` ancestry information
taken from the two toplevel project branch heads being merged,
and merging of the whole tree can be done with a three-way merge
of the whole tree using the merge base and two head commits.
For reasons described later, we would not merge the subproject
parts of the trees during this step, though.

When the two branch heads use different versions of subproject,
things get a bit tricky.  First, let's forget for a moment about
the case where they bind the same project at different location.
We would refuse if they do not have the same number of `bind`
lines that bind something at the same subdirectories.

------------
$ git merge 'Merge in a side branch' HEAD side
error: the merged heads have subprojects bound at different places.
 ours:
	linux-2.6/
	appliance/
 theirs:
	kernel/
	gadget/
	manual/
------------

Such renaming can be handled by first moving the bind points in
our branch, and redoing the merge (this is a rare operation
anyway).  It might go like this:

------------
$ git bind-projects \
	$kernel_commit kernel/ \
	$gadget_commit gadget/
$ git commit -m 'Prepare for merge with side branch'
$ git merge 'Merge in a side branch' HEAD side
error: the merged heads have subprojects bound at different places.
 ours:
	kernel/
	gadget/
 theirs:
	kernel/
	gadget/
	manual/
------------

Their branch added another subproject, so this did not work (or
it could be the other way around -- we might have been the one
with `manual/` subproject while they didn't).  This suggests
that we may want an option to `git merge` to allow taking a
union of subprojects.  Again, this is a rare operation, and
always taking a union would have created a toplevel project that
had both `kernel/` and `linux-2.6/` bound to the same Linux
kernel project from possibly different vintage, so it would be
prudent to require the set of bound subprojects to exactly match
and give the user an option to take a union.

------------
$ git merge --union-subprojects 'Merge in a side branch HEAD side
error: the subproject at `kernel/` needs to be merged first.
------------

Here, the version of the Linux kernel project in the `side`
branch was different from what our branch had on our `bind`
line.  On what kind of difference should we give this error?
Initially, I think we could require one is the fast forward of
the other (ours might be ahead of theirs, or the other way
around), and take the descendant.

Or we could do an independent merge of subprojects heads, using
the `parent` ancestry of the bound subproject heads to find
their merge-base and doing a three-way merge.  This would leave
the merge result in the subproject part of the working tree and
the index.

[NOTE]
This is the reason we did not do the whole-tree three way merge
earlier.  The subproject commit bound to the merge base commit
used for the toplevel project may not be the merge base between
the subproject commits bound to the two toplevel project
commits.

So let's deal with the case to merge only a subproject part into
our tree first.


Merging subprojects
-------------------

An operation of more practical importance is to be able to merge
in changes done outside to the projects bound to our toplevel
project.

------------
$ git pull --subproject=kernel/ git://git.kernel.org/.../linux-2.6/
------------

might do:

. fetch the current `HEAD` commit from Linus.
. find the subproject commit bound at kernel/ subtree.
. perform the usual three-way merge of these two commits, in
  `kernel/` part of the working tree.

After that, `git commit \--subproject` option would be needed to
make a commit.

[NOTE]
This suggests that we would need to have something similar to
`MERGE_HEAD` for merging the subproject part.


-
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Due to a shortage of devoted followers, the production of great leaders has been discontinued.

On Sun, 22 Jan 2006, Junio C Hamano wrote:

Daniel Barkalow [off-list ref] writes:

quoted

quoted

Switching branches
------------------

Along with the traditional two-way merge by `read-tree -m -u`,
we would need to look at:

. `bind` lines in the current `HEAD` commit.

. `bind` lines in the commit we are switching to.

. subproject binding information in the index file.

to make sure we do sensible things.

This is one place I think storing the bindings in the commit is awkward. 
read-tree deals in trees (hence the name), but will need information from 
the commit.

That's why it is 'along with'.  Dealing with binding information
can be done between commits and index without bothering tree
objects.  read-tree would not have to deal with it, and I think
keeping it that way is probably a good idea.

I think it would be a lot more fragile if switching branches requires 
multiple programs interacting with the index file. If things get 
interrupted after the tree is read but before the bindings are changed, 
the user will probably generate an inconsistant commit or have to deal 
with figuring out what's going on. It is a nice property of the current 
system that the index file never exists under the usual filename without 
being consistant.

In other words, I think the design so far does not require us to
touch tree objects at all, and I'd be happy if we do not have to.

One reason I started the bound commit approach was exactly
because I only needed to muck with commit objects and did not
have to touch trees and blobs; after trying to implement the
core level for "gitlink", which I ended up touching quite a lot
and have abandoned for now.

I think that the same thing that worked with the index file would work for 
minimizing the impact of the changes as far as the code sees. If the 
struct tree parser reported the tree at a path when it found a commit at a 
path, it would work just as if the original tree had used trees (like in 
your proposal).

	-Daniel
*This .sig left intentionally blank*

Daniel Barkalow [off-list ref] writes:

I think it would be a lot more fragile if switching branches requires 
multiple programs interacting with the index file. If things get 
interrupted after the tree is read but before the bindings are changed, 
the user will probably generate an inconsistant commit or have to deal 
with figuring out what's going on. It is a nice property of the current 
system that the index file never exists under the usual filename without 
being consistant.

That is certainly an issue, which we have had already for quite
some time, I am afraid.  We can get interrupted during "switch
branches" flow after read-tree -u -m but before updating HEAD.
We can also get interrupted during "commit" flow after writing
the commit object out before updating the ref pointed at by
HEAD.  No?

If we are truly serious about solving the issue of getting
interrupted in the middle, I suspect we have to take the "index
is a staging area for the next commit" approach I digressed into
last night.  It would involve introducing a git-atomic-checkout
command to replace the current "git-rev-parse, git-read-tree,
then git-symbolic-ref" sequence in the checkout flow.  In the
commit flow, we would need git-commit-index command to replace
the current "git-write-tree, git-commit-tree, then
git-update-ref" sequence.

I am not particularly opposed to that, but I suspect it might be
a moderate amount of work for very little gain.  Continuing with
the digression, the updated index file may contain:

  1. list of <blob path, object name>
  2. list of parent commit object names for the next commit
  3. the name of the local branch to create the next commit on
  4. for each bound path:
     list of parent commit object names for that path.

1. is what we have in the current (version 2) index file.

2. contains:
 - 0 commit in an index file before the initial commit
 - 1 commit in an index file after a fresh checkout and records
   the commit object name we checked out (replaces HEAD+heads/$branch)
 - 2 commits in an index file after 3-way read-tree, or more
   during an Octopus merge (replaces HEAD+MERGE_HEAD)

3. may not be needed, but if we did so, it would replace HEAD.

4. is similar to 2 but for bound subprojects.  Usually we have
   one commit per bound path to record the "bind" line commit we
   read from the commit object after a fresh checkout.  During a
   subproject merge, we would:
   - start out with 1 commit read from the "bind" line;
   - merging in another subproject commit would add that commit;
   - when making a new subproject commit, the recorded commits
     are used as its parents.

On Mon, 23 Jan 2006, Junio C Hamano wrote:

Daniel Barkalow [off-list ref] writes:

quoted

I think it would be a lot more fragile if switching branches requires 
multiple programs interacting with the index file. If things get 
interrupted after the tree is read but before the bindings are changed, 
the user will probably generate an inconsistant commit or have to deal 
with figuring out what's going on. It is a nice property of the current 
system that the index file never exists under the usual filename without 
being consistant.

That is certainly an issue, which we have had already for quite
some time, I am afraid.  We can get interrupted during "switch
branches" flow after read-tree -u -m but before updating HEAD.
We can also get interrupted during "commit" flow after writing
the commit object out before updating the ref pointed at by
HEAD.  No?

The switch branches one is accurate, but I think that, if we get 
interrupted before updating the ref, the index will still be the same, and 
we'll just have a dangling object (which, if we commit the same thing 
again, will be the same object we generate).

I suppose the existing branch switching isn't much less bad than the new 
one would be, though. I sort of worry that rewriting the index file is 
more likely to be interrupted than updating a ref, but that's probably not 
really a significant difference.

If we are truly serious about solving the issue of getting
interrupted in the middle, I suspect we have to take the "index
is a staging area for the next commit" approach I digressed into
last night.  It would involve introducing a git-atomic-checkout
command to replace the current "git-rev-parse, git-read-tree,
then git-symbolic-ref" sequence in the checkout flow. 

Well, if the value of HEAD were in the index file, that would be 
sufficient to prevent anything actually bad from happening in the checkout 
path; if it gets interrupted, the index file's "current commit" field 
would then not match the ref and it would be clear that the system was in 
an intermediate state. (It would appear like if you'd fetched into the 
current branch without it doing the fast-forward.)

In the commit flow, we would need git-commit-index command to replace
the current "git-write-tree, git-commit-tree, then
git-update-ref" sequence.

I don't think there's an issue here, anyway.

I am not particularly opposed to that, but I suspect it might be
a moderate amount of work for very little gain.  Continuing with
the digression, the updated index file may contain:

  1. list of <blob path, object name>
  2. list of parent commit object names for the next commit
  3. the name of the local branch to create the next commit on
  4. for each bound path:
     list of parent commit object names for that path.

1. is what we have in the current (version 2) index file.

2. contains:
 - 0 commit in an index file before the initial commit
 - 1 commit in an index file after a fresh checkout and records
   the commit object name we checked out (replaces HEAD+heads/$branch)
 - 2 commits in an index file after 3-way read-tree, or more
   during an Octopus merge (replaces HEAD+MERGE_HEAD)

3. may not be needed, but if we did so, it would replace HEAD.

I don't think it would be needed; it could certainly be passed in. 
Actually not having HEAD would complicate a lot of programs that use HEAD 
but don't currently read the index (and don't actually care about whether 
you have the branch that you consider current actually checked out).

4. is similar to 2 but for bound subprojects.  Usually we have
   one commit per bound path to record the "bind" line commit we
   read from the commit object after a fresh checkout.  During a
   subproject merge, we would:
   - start out with 1 commit read from the "bind" line;
   - merging in another subproject commit would add that commit;
   - when making a new subproject commit, the recorded commits
     are used as its parents.

Right.

	-Daniel
*This .sig left intentionally blank*