Re: RFC: Subprojects

Linus Torvalds [off-list ref] writes:

On Sat, 14 Jan 2006, Junio C Hamano wrote:

The thing is, if you do the contained projects as "union projects" as you 
suggest, I will bet that it will really really suck, because it ends up 
losing the two positives above.

After a good night's sleep, I agree.  I have not thought things
through and still have a feeling that (feasibilities aside) it
would be interesting if we can do a "union projects" a la "union
mounts" (or translucent filesystem).  But that "interesting"
thing would probably not be very useful in practice.

would actually act like they now act for directories that they don't 
recurse into, ie you'd see something like

	:160000 160000 5eb57670... 3f1a42aa... M	sub-project

and it would be up to higher-level porcelain to recurse.

This I agree with.

The other reason? A lot of the git infrastructure really does only work on 
the "one project" level. The programs work with _one_ index, not two. 
Reading two trees is perfectly possible, but unless you keep them in 
separate stages, you can't separate them afterwards. IOW, trying to be 
recursive really does end up being a big change, for very little gain (and 
for a lot of potential bugs and instability).

Yup.  BTW, I think with a couple of minor tweaking and giving it
the same restriction ("two pluses and one negative") as the
gitlink proposal, the "union" approach would work equally well,
perhaps with a simpler implementation. I'll think aloud about
this at the end.

quoted

Fetching/cloning at the core level is easy.  "git-fetch-pack"
would just need to do one level, but Porcelains need to address
how to actually arrange the subprojects cloning to happen, which
is harder.

...
So only if you actually check it out (which is often in practice the 
second stage of the cloning, of course) do you want to fetch the 
subproject too.

We are in complete agreement here.

I think this one has serious disadvantages:

 - it's much less obvious when there are common names and especially 
   common subdirectories.
 - in _practice_, almost all sub-projects are kept in sub-directories. Are 
   you doing to change the sub-project git tree? How are you going to 
   merge back to the original sub-project?
 - iow, I think this only works for sub-projects that are totally 
   controlled by the top-level project - in which case they might as well 
   just be totally merged into the top level (the way we did with the 
   "tools" project, and largely with "gitk").

Yes, I agree to the above 100%; the serious disadvantages come
from the fact that we do not have clear separation between
subprojects -- which new files belong to what subproject.  I
think re-rooting read-tree and write-tree would help solving
that.  After I wrote the message you are replying to, I came up
with a couple of tweaks.

 - Do the octopus-like thing, but always give subprojects a
   separate directories to work in.

 - Extend "commit" objects for the toplevel project to record
   what subprojects with what head commits are contained at
   which subdirectory.  I wrote in the previous message to make
   subprojects heads parents of aggregate commits, but I think
   that one without "where to" information has a serious
   disadvantage when computing a merge.

In the "embedded linux" example that has "linux-2.6" and
"gcc-4.0" projects as an externally controlled subprojects, and
has all the rest (including the toplevel Makefile) in "master"
branch:

     $ tar xf embed.tar embed && cd embed && git init-db
     $ git add . ;# toplevel Makefile and stuff
     $ git commit -a -m 'embedded repo - initial'

After doing "git-fetch-pack -k git://.../linux-2.6.git/ master"
and "echo $H >.git/refs/heads/kernel" (similar for gcc-4.0) to
set up the branch heads (but we do not have any working tree
files for these subprojects yet):

	$ git bind -m 'Bind kernel and gcc into us' \
        	kernel=linux-2.6 gcc=gcc-4.0

would prepare the subprojects binding (I am just looking for a
better word --- I called it "setup-overlay" in the previous
message).  This would:

 - append the tree object in "kernel" commit object to the
   current index, rerooted at linux-2.6/; similar for "gcc" at
   gcc-4.0/. We may need a new mode and option for read-tree for
   this, or we may not.  Internally this step would be scripted
   in "git bind" wrapper like this:

	git read-tree --bind --prefix=linux-2.6 kernel
	git read-tree --bind --prefix=gcc-4.0 gcc

   and would result in an index file that has these trees
   "mounted" at specified places.  If you look at only the index
   file, you cannot tell this is an overlay, unlike gitlink
   scheme.

 - make a commit that records the tree object (the whole thing
   including the subproject trees), with the initial commit we
   made earlier as the sole parent commit, and additionally
   records the two subproject heads with bind points.  This
   happens in the same "git bind" wrapper, and produces
   something like:

	$ git cat-file commit HEAD
        tree e9de76f2e141824439caa00a65e3b91d05d125c9
        parent bfca932434cc65e7aa90794e7c4d66f75d00b16a
        bind a8fe7257b8427d31cfcca0aa336335bb43689fc9 linux-2.6
        bind b3b2df23226634f42c9646bd7961fbea8b00f914 gcc-4.0
        author Junio C Hamano [off-list ref] 1137205528 -0800
        committer Junio C Hamano [off-list ref] 1137205528 -0800

	Bind kernel and gcc into us.

   "bind" line needs to be taught to fsck-objects.  The format
   is the object name of the commit followed by (c-style quoted)
   subdirectory name.

 - record the branch name vs subproject directory binding in
   $GIT_DIR/ somewhere, say $GIT_DIR/mtab ;-).

	$ cat .git/mtab
	kernel	linux-2.6
        gcc	gcc-4.0

After this, "git checkout-index -f -q -u -a" would populate the
whole thing.  Instead of linux-2.6/.git/HEAD as in gitlink
example, I am using .git/refs/heads/kernel; this would not make
a semantic difference.  One big difference however is I have
only one index file that controls the whole tree, without using
a separate linux-2.6/.git/index.

After mucking with a file in linux-2.6/ subdirectory and nowhere
else, committing the result from the whole tree would work like
this:

 - Look at the current commit and notice the bind for two
   subdirectories; then look them up in $GIT_DIR/mtab to see
   which branches keep track of them.

 - Notice that there are modified paths in the index vs tree
   from the last commit under linux-2.6/ directory.

 - Write out only that part, re-rooted, into a tree.

	git write-tree --prefix=linux-2.6

 - Make a commit to record that tree, with a parent set to the
   "kernel" branch head; update the "kernel" branch head at that
   commit.

 - Make another commit to record the tree made from the whole
   index (obviously linux-2.6 subdirectory would result in the
   same tree object we just committed in the subproject) with
   parent set to .git/HEAD and bind adjusted accordingly; update
   the "HEAD".

Now I have to think about clones and merges but this is getting
too long so I'll leave it to a separate message.

Continuing with the "union" approach...

Junio C Hamano [off-list ref] writes:

 - append the tree object in "kernel" commit object to the
   current index, rerooted at linux-2.6/; similar for "gcc" at
   gcc-4.0/. We may need a new mode and option for read-tree for
   this, or we may not.  Internally this step would be scripted
   in "git bind" wrapper like this:

	git read-tree --bind --prefix=linux-2.6 kernel
	git read-tree --bind --prefix=gcc-4.0 gcc

   and would result in an index file that has these trees
   "mounted" at specified places...

Clarification.  By "mounted", I mean 'without affecting existing
index entries, create index entries from the tree, with all the
paths have "linux-2.6/" prefixed to them'.

 - record the branch name vs subproject directory binding in
   $GIT_DIR/ somewhere, say $GIT_DIR/mtab ;-).

	$ cat .git/mtab
	kernel	linux-2.6
      gcc	gcc-4.0

I now realize this needs to be something like:

	master	kernel=linux-2.6/ gcc=gcc-4.0/

that is, "when on branch master, bind these two heads at these
directories", to allow switching to another branch and switching
back to this branch.  And the file should probably be called
$GIT_DIR/modules, to parallel CVSROOT/modules file.

$ git cat-file commit HEAD
      tree e9de76f2e141824439caa00a65e3b91d05d125c9
      parent bfca932434cc65e7aa90794e7c4d66f75d00b16a
      bind a8fe7257b8427d31cfcca0aa336335bb43689fc9 linux-2.6
      bind b3b2df23226634f42c9646bd7961fbea8b00f914 gcc-4.0
      author Junio C Hamano [off-list ref] 1137205528 -0800
      committer Junio C Hamano [off-list ref] 1137205528 -0800

Bind kernel and gcc into us.
...
Now I have to think about clones and merges but this is getting
too long so I'll leave it to a separate message.

The core-level cloning would just "clone" the objects, treating
"bind" line in the commit just like "parent" to pull necessary
objects.

Checkout would involve the usual read-tree -u which extracts the
tree (which is the whole tree, with files of the subprojects in
it), and notices "bind" lines are there but there are no
matching $GIT_DIR/modules entries for those directories.
Probably it would create $GIT_DIR/refs/heads/bind/a8fe725 for
the linux-2.6 subproject (what the original committer called
"kernel" branch), and similarly for the gcc-4.0 subproject, add
an appropriate entry to $GIT_DIR/modules file.  The user would
then rename the branch names and optionally arrange remotes/
files to update the bound branches appropriately:

	$ mv .git/refs/heads/bind/a8fe725 .git/refs/heads/kernel

Now, let's say this "master" branch is checked out, and somehow
the "kernel" branch gets updated.  That is, the commit recorded
on the "bind" line of the HEAD commit does not match the branch
head that can be found out via $GIT_DIR/modules file.  This will
not happen if you are committing into the "master" branch using
the "commit to subprojects and then to the toplevel project"
mechanism yourself, but it would happen if the "kernel" branch
was moved by "git fetch" fast-forwarding, or if you switched to
the "kernel" branch (which would essentially remove everything
from your tree, and checkout the kernel source at the root
level, not in linux-2.6/ subdirectory), did an upstream merge
yourself, and switched back to the "master" branch.

To keep the problem simpler, let's say we only deal with the
case where "kernel" branch head is a fast-forward of what is on
"bind" in the HEAD commit of "master" branch.  Then "checkout"
needs to notice it, and check out the subdirectory from the
"kernel" branch head (*not* using the object name on "bind"
line).

So the outline of the "checkout" would be like this:

 * Read commit object from new HEAD.

 * For each "bind" line:

   If the subdirectory does not have a corresponding branch,
   create one in $GIT_DIR/refs/heads/bind/; record it in
   $GIT_DIR/modules for the new branch (otherwise leave branch
   as is).

   Make sure the commit recorded on "bind" line is an ancestor
   of the branch head.  Otherwise it is an error and checkout is
   prevented until the "kernel" branch is resolved to be a
   descendant of it.

   Run "read-tree -u --prefix=" to merge in the subtree into the
   index, and update the working tree.

At this point, there may be mismatch between the tree in the
HEAD and the working tree files and index, when subproject
commit recorded on the "bind" line is different from the
corresponding subproject branch head, and "git diff" would show
it.  When making a commit here, the "subproject and then
toplevel" commit scheme I described earlier would record the
current "kernel" branch head on the "bind" line in the new
commit, along with the tree object that contains the tree from
"kernel" branch head commit as a subtree.


About "merge", we should be able to do this:

	$ git checkout master ;# the whole mess
        $ git pull -b kernel git://..torvalds/linux-2.6.git/

that is, 'pull from this URL but into "kernel" branch not to the
current branch'.  Independent of this "subprojects" topic,
merging in a separate temporary directory into non-current
branch is something we have talked about some time ago, and in
this particular case, instead of using a throw-away temporary
directory, we have a pre-made directory to do the merge already,
so let's say that is solved elsewhere first.  Once we have that,
the above "checkout" would be able to integrate the result into
the "master" project.

On Saturday 14 January 2006 21:16, you wrote:

Yes, I agree to the above 100%; the serious disadvantages come
from the fact that we do not have clear separation between
subprojects -- which new files belong to what subproject.  I
...
 - Extend "commit" objects for the toplevel project to record
   what subprojects with what head commits are contained at
   which subdirectory.

The suggested "bind" info in commit objects has the same problem
as the original overlay: if the superproject already has a
subdirectory kernel/, and there is an additional "bind" specification
in commits also for kernel/, what should be done?

So the gitlink object seems to be the only solution if we want to
bind git versions of subprojects into a superproject.

But as this seems to make everything quite complex and not-obvious for
a user, I am with Paskys simple subproject idea.

Josef

Josef Weidendorfer [off-list ref] writes:

The suggested "bind" info in commit objects has the same problem
as the original overlay: if the superproject already has a
subdirectory kernel/, and there is an additional "bind" specification
in commits also for kernel/, what should be done?

So the gitlink object seems to be the only solution if we want to
bind git versions of subprojects into a superproject.

In "pu", I have some of the necessary basic pieces for "bind"
approach, barely enough so that anybody interested could start
prototyping using them as building blocks.  It still has very
rough edges; the missing includes rev-list and fsck-objects, so
you cannot do a send-pack or fetch-pack yet.

Yesterday I was working on "gitlink" approach to have similar
core-side support for prototyping.  I haven't finished it into a
buildable state yet (it is not in "pu"), and I am pessimistic if
I ever will X-<.

I think the updated "bind" thing makes the two approaches
semantically equivalent (i.e. it does not allow an arbitrary
overlayed setup anymore).  We simply do not allow the
conflicting "bind".  So neither is the _only_ solution.  We
probably could make both to work, but the details differ.

 * With "gitlink", the index of containing project never has
   subprojects parts of the tree, which I see it as an advantage
   compared to what "bind" does.  It only has one "gitlink"
   entry per each subproject.  update-index, read-tree,
   ls-files, diff-*, etc. needs to be aware of "gitlink" object.
   Especially tricky is read-tree.  It needs to treat a
   "gitlink" object as a directory for D/F conflict detection
   purposes, but treat it similar to blobs in most other aspects
   (e.g.  results in one entry in the index).  The stat
   information update-index and diff-files uses for quick
   up-to-date check needs to be taught not to worry about the
   stat information of the subdirectory a "gitlink" object
   points at (e.g. if you do a whole-tree build, the timestamp
   of the directory would change, but that does not mean the
   subtree is dirty).  tree/directory traversal code needs to be
   aware of "gitlink" and stop there.  This approach involves
   quite a lot of code changes, mostly because what is in the
   current index never correspond to a directory on the
   filesystem but "gitlink" quacks like a directory.

 * With "bind", the index of containing project keeps the entire
   tree structure, including subproject part.  In fact, there is
   no other separate index for the subproject part.

   An updated write-tree in "pu" can write a tree for only the
   subproject part with "write-tree --prefix=<path>/" from such
   an index file, and read-tree can read with "read-tree
   --prefix=<path>/" to graft a subproject tree on top of the
   current index contents.  Without the --prefix, write-tree
   writes out the whole thing for a commit for the containing
   project, so if somebody cloned that superproject, getting the
   whole tree out in order to "make" is just the matter of doing
   a regular "read-tree && checkout-index".

   We could introduce "bind the rest" to make write-tree write
   out a tree that contains only the containing project part and
   not any of the subproject part (e.g. Makefile, README and
   src/ but not linux-2.6/ nor gcc-4.0/ in the earlier example).
   Essentially the contents of such a tree object would be the
   same as what "gitlink" approach would have had for the
   containing project in the index file, minus "gitlink" entries
   themselves).  This is not so surprising, because the missing
   information "gitlink" approach recorded in the tree object
   itself is expressed on "bind" lines in the commit object with
   this approach.

An advantage with the "bind" approach, from the implementation
point of view, is that none of the "index vs working tree" part
of the core needs to be modified (you would notice that many
issues I had with trying "gitlink" I listed above are "index vs
working tree" issues).  "tree object vs index" part needed to be
enhanced somewhat (e.g. the re-rooting read-tree/write-tree with
the --prefix option) but it was not too painful.

On Mon, 16 Jan 2006, Junio C Hamano wrote:

   We could introduce "bind the rest" to make write-tree write
   out a tree that contains only the containing project part and
   not any of the subproject part (e.g. Makefile, README and
   src/ but not linux-2.6/ nor gcc-4.0/ in the earlier example).
   Essentially the contents of such a tree object would be the
   same as what "gitlink" approach would have had for the
   containing project in the index file, minus "gitlink" entries
   themselves).  This is not so surprising, because the missing
   information "gitlink" approach recorded in the tree object
   itself is expressed on "bind" lines in the commit object with
   this approach.

So why not use the "bind" approach for the "index vs working tree" part, 
but write out "gitlink"-style tree objects? I think putting the info in 
the tree objects in the location the subproject would appear is nicer than 
having tree objects that tell only part of the story, and you don't have 
to worry about commits that stick a subproject on top of something in the 
tree.

In any case, I think it would be good to track where the subprojects are 
in some core state, and probably the right solution is to have special 
index entries for them, in addition to having their contents in the index. 
I'm not seeing a clear way to get from commit objects with "bind" lines to 
an index with the appropriate things read and back otherwise.

One idea I toyed with a while ago for the index/working tree 
implementation is having an index file per bound project, such that each 
project has a completely ordinary index file, and you just need to tell 
checkout-index where to write. This is especially cute because the index 
file for the superproject doesn't need to know about the subprojects at 
all; they're not in that index, and the working tree is just directories 
of untracked files. Not sure if this is a useful idea at this point or 
not.

	-Daniel
*This .sig left intentionally blank*

Dear diary, on Mon, Jan 16, 2006 at 09:49:48PM CET, I got a letter
where Junio C Hamano [off-list ref] said that...

   We could introduce "bind the rest" to make write-tree write
   out a tree that contains only the containing project part and
   not any of the subproject part (e.g. Makefile, README and
   src/ but not linux-2.6/ nor gcc-4.0/ in the earlier example).
   Essentially the contents of such a tree object would be the
   same as what "gitlink" approach would have had for the
   containing project in the index file, minus "gitlink" entries
   themselves).  This is not so surprising, because the missing
   information "gitlink" approach recorded in the tree object
   itself is expressed on "bind" lines in the commit object with
   this approach.

Now, I must have missed the obvious again, but what is the point in
having the write-tree --exclude stuff? My impression (also from your
later mail in this thread) is that now the moment you introduce any
binds, your top-level development changes to "two-tiered" - the
top-level project and the meta-project holding it all together. I'd say
that's pretty confusing and I don't see big gain in this; the simplicity
of the original proposal was a lot more appealing.

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Of the 3 great composers Mozart tells us what it's like to be human,
Beethoven tells us what it's like to be Beethoven and Bach tells us
what it's like to be the universe.  -- Douglas Adams