Re: [PATCH 07/16] git-read-tree: take --submodules option

Alex Riesen [off-list ref] writes:

Sven Verdoolaege, Sat, May 19, 2007 00:08:26 +0200:
...

quoted

The reason for not putting this in shouldn't be that someone doesn't
think it is useful; the reason should be that my code is crap.

The code is not a problem. It can be also discarded because you
implemented something no one wants.

More specifically, at the very high level, what you do and what
people want to happen might share the description (e.g. "this
allows checkout of subprojects", or "this implements clone of
superproject to recurse") but with semantics that may be
different from what people would want (I am not saying that is
the case, as I do not think the current discussion concluded
yet).  The _first_ implementation that goes in the mainline
pretty much sets the _semantics_ so we would need to be extra
careful, all the more so as this is a feature many people seem
to want.

I just meant to say, that even if no one wants your subproject cloning
code, _I_ support your checkout effort and I am asking for it to be
put in.

"First", as the cloning discussion does not seem to be finished (and,
as I said, I am not interested in cloning anyway).

It was partly my fault that I mentioned "clone" example in the
original message, and then let the discussion drifted to a
tangent of the clone topic, namely, how the URL would be
determined to clone the subproject from.

I would agree that checkout is a more fundamental operation, and
I wanted to make that clear in my message, but checkout and
clone has certain chicken-and-egg factor between them.  After
the clone of superproject, checking it out recursively would
need cloning the subprojects.  Also after a clone of
superproject without the recursive behaviour, when the user
explicitly asks a subproject to be checked out, somebody needs
to do a clone before the subject can be checked out.

Having said that, let's throw out an strawman for checkout
proper and then merge.

The user may or may not want to deal with subprojects, and for
something truly large like the KDE case, which is where the
superproject support is really needed, a large On/Off switch
where an option --subproject makes everything checked out and no
subproject is checked out without it is not a usable option.
I've already outlined how the .git/config file can be used to
define which subprojects are of interested so that the Porcelain
layer can decide which ones to recurse into and which ones to
leave alone.  The design is NOT the only possible/sensible one,
and I am sure other people will come up with much nicer
organization, but I would consider that is just the matter of
details.

Now, suppose "git checkout" needs to recurse into one
subdirectory that is to have a subproject.  There are three
cases:

 (1) There is no git repository yet (the plumbing layer already
     makes sure there is a directory, but does not do anything
     else).

 (2) There already is a git repository there, which is the
     correct repository (perhaps determined by .gitmodules and
     .git/config in the superproject, or presense of the commit
     that is recorded in the superproject's index).

 (3) There is a git repository but it is not the correct one.

We've discussed in the other thread about what to do in case
(1) to some degree.

For case (2), I think what should happen there is an equivalent
of this:

	$ commit=$(git-rev-parse :subproject)
        $ cd subproject
	$ git-rev-parse --verify $commit || git fetch || barf
        $ git checkout $commit

That is,

 - figure out what commit should be checked out from
   superproject index;

 - make sure the named commit exists, or fetch to make it exist.

 - go there and check out that commit; this implies two things:

   1. if there are local changes, it will be carried along and we
      checkout the named commit;

   2. the repository's HEAD becomes detached;

It is entirely possible that the repository is the _correct_ one
but not quite up to date, and you haven't fetched $commit.  This
is really a variant of (1) -- before being able to check out,
somebody has to clone the subproject.  Before being able to
check out to update the latest, somebody has to fetch in the
subproject.

If there are local changes, we would not at least lose them.  If
you want to get to a clean slate, you can cd there and perform
"git reset --hard".  If you want to mark that commit in the
subproject, you may want to do "git checkout -b branch" after
the recursive checkout from the superproject detached the HEAD
to the commit.

There is another variant that has already been suggested.  The
superproject tree and index could record 0{40} object name for
the subproject, and say "whatever commit happens to be at the
tip of the branch of subproject" (and most likely that URL and
branch information would come from .gitmodules and confirmed in
the .git/config file).  In such a case, the above outline would
be adjusted _BUT_ I think what would be checked out will not be
the named branch (e.g. refs/heads/master) itself, but the remote
branch that tracks it (e.g. refs/remotes/origin/master).

While I am at it, let me think aloud as to what I _think_ should
happen in a superproject merge.

 - Carry out the tree-level 3-way merge.  If it trivially
   resolves at the tree-level, we are happy.

 - There could be a case where the commit fetched/merged branch
   has and what the current branch has are different.  If one is
   a fast forward of the other, take it.

 - All other cases will leave the superproject index unmerged.

When the merge is cleanly done, the resulting commit is what we
should check out in the subproject directory (if we are
recursing into it, of course).

It is likely that in some cases you would want go to the
subproject directory and merge the commits at the subproject
from our branch and their branch in the superproject's index,
and make the resulting commit as the result of the merge for
that subproject path in the superproject, but I do not think it
is the only valid solution.

Junio C Hamano [off-list ref] wrote:

Now, suppose "git checkout" needs to recurse into one
subdirectory that is to have a subproject.  There are three
cases:

So I've implemented my own Git subproject support in a Java based
tool we use internally.  Its actually driving the Git plumbing (as
JGit isn't complete enough to do the job) but applies quite a bit
to this discussion as it is a working system that implements this
"checkout in superproject and recurse into subproject".

First I don't use the subproject support in the core plumbing,
because that came along from Linus about 2 days after I wrote
this implementation.  Our data file looks like:

	use-component: subproject1 >=df8cfac815...
	use-component: subproject2 >=af9b543820...

Or really anything that is a valid commit-ish, and often these are
actually just annotated tag names.

 (1) There is no git repository yet (the plumbing layer already
     makes sure there is a directory, but does not do anything
     else).

During our build process we scan the root project's data file,
and clone by the relative URL anything we cannot find locally:

	$(git config remote.origin.url)/../component-links/subproject1.git

to get the subproject repository.  We don't require that the
subproject1 directory actually be called subproject1 in the
superproject, its just a recommendation.  That data file is also
our build-system driver and the build system driver is pretty darn
smart about guessing what is going on.  ;-)

You'll notice however that we (more or less) have a very flat
structure.  The component-links directory is really just a set of
symlinks pointing back up a level, as sometimes a component is not
stored in a repository named the component name, but the component
name matters to the build system.

 (2) There already is a git repository there, which is the
     correct repository (perhaps determined by .gitmodules and
     .git/config in the superproject, or presense of the commit
     that is recorded in the superproject's index).

 (3) There is a git repository but it is not the correct one.

For case (2), I think what should happen there is an equivalent
of this:

	$ commit=$(git-rev-parse :subproject)
        $ cd subproject
	$ git-rev-parse --verify $commit || git fetch || barf
        $ git checkout $commit

Yes.  Except we do a few things differently:

 - Only update the subproject if its a strict fast-forward.

 - Abort on a dirty working directory in the subproject or if a merge
 would be required to keep the current commit and the new commit.
 Yes, we don't really support going "backwards".

 - The merge aborting thing is probably wrong for some users,
 but blindly switching to the target commit feels somewhat wrong
 in our own uses.  Sometimes you need the current version of the
 subproject to help you debug an older version of the superproject,
 or sibling subproject.

 - You can't just checkout $commit if you can rev-parse it.
 You need to verify it and its entire reachable object set exists.
 See the local fetch fast-path thing you did recently in e3c6f240fd.

 - We update the user's current branch.  Because we are doing
 a strict fast-forward we're also assuming the user wants the
 current branch to stay correlated to the superproject branch.  Why?
 Most of our users keep the same branch name in all repositories.

-- 
Shawn.

Junio C Hamano, Sat, May 19, 2007 05:59:48 +0200:

 - figure out what commit should be checked out from
   superproject index;

 - make sure the named commit exists, or fetch to make it exist.

What if the fetch is not possible? You can't checkout? What about the
other subprojects, where the checkout succeeded? Will they be reset to
the previuos state?

To me, the fetch sounds pretty dangerous. Maybe the checkout should
be two stage: first - we check all subprojects to be checked out if it
is possible, second - either fail (default) or checkout what possible,
warn the user, leave the incomplete subprojects changed (but not
update the index with them, so that they wont be accidentally
committed).

 - go there and check out that commit; this implies two things:

   1. if there are local changes, it will be carried along and we
      checkout the named commit;

Shouldn't that depend on "-m" option given to git-checkout in
superproject? Sometime the user have to be sure he can checkout
everything as it were, but without breaking the local state (like what
current git-checkout without "-m" does).

   2. the repository's HEAD becomes detached;

Universally agreed upon

On Fri, May 18, 2007 at 08:59:48PM -0700, Junio C Hamano wrote:

Now, suppose "git checkout" needs to recurse into one
subdirectory that is to have a subproject.  There are three
cases:

 (1) There is no git repository yet (the plumbing layer already
     makes sure there is a directory, but does not do anything
     else).

 (2) There already is a git repository there, which is the
     correct repository (perhaps determined by .gitmodules and
     .git/config in the superproject, or presense of the commit
     that is recorded in the superproject's index).

 (3) There is a git repository but it is not the correct one.

We've discussed in the other thread about what to do in case
(1) to some degree.

For case (2), I think what should happen there is an equivalent
of this:

	$ commit=$(git-rev-parse :subproject)
        $ cd subproject
	$ git-rev-parse --verify $commit || git fetch || barf
        $ git checkout $commit

Does everyone agree that we should fetch (possibly after asking
for confirmation from the use) _during_ the checkout ?
I now only fetch submodules during a fetch of the supermodule
(actually, in my current patch set, I only fetch a submodule
the first time I see it, but that's a bug), but if there is
a consensus on this, I can switch to fetching during checkout.

As to the key to use to lookup the URL in the config, right
now I simply use the directory name where it is attached
(which seems like a useful default to me).
I'm not all that convinced that we should store a default URL
in history, so AFAICS, the only thing we need to store is a
mapping between directory names and subproject names.
It has been suggested to do that in .gitattributes.
Is that OK for everyone, or do we really need a separate .gitmodules ?

skimo

Sven Verdoolaege [off-list ref] writes:

Does everyone agree that we should fetch (possibly after asking
for confirmation from the use) _during_ the checkout ?
I now only fetch submodules during a fetch of the supermodule
(actually, in my current patch set, I only fetch a submodule
the first time I see it, but that's a bug), but if there is
a consensus on this, I can switch to fetching during checkout.

I think fetching of subproject during fetch or clone of
superproject would not make much sense.  Making it part of
superproject checkout would probably be the way we will end up
going.  The detail of "which part of the checkout" would need to
be defined, and I tend to agree with Alex that checkout itself
would need to be multi-phased, but I think that is a minor
implementation detail we can discuss after how the overall flows
should look like.

As to the key to use to lookup the URL in the config, right
now I simply use the directory name where it is attached
(which seems like a useful default to me).

I'm not all that convinced that we should store a default URL
in history, so AFAICS, the only thing we need to store is a
mapping between directory names and subproject names.
It has been suggested to do that in .gitattributes.
Is that OK for everyone, or do we really need a separate .gitmodules ?

If your (super)repository is _the_ only repository that knows
about the aggregation it is doing, I do not think you need
anything, as .git/config in the subproject would know where to
get updates from.  Otherwise, Project wide suggestions need to
be there in some machine readable form if you were to allow such
superproject distributed, be it in .gitmodules or
.gitattributes, don't they?

And frankly, I think .gitattributes is a wrong place to store
it, as its semantics is to give attributes to paths that MATCH
the glob.  You can argue that a pattern in .gitattributes can be
written to match only one path, but it still feels conceptually
wrong.  Something more concrete and exact, like the second level
key of .git/config and .gitmodules file format, is preferable.

On Sat, May 19, 2007 at 11:20:12 -0700, Junio C Hamano wrote:

Sven Verdoolaege [off-list ref] writes:

quoted

Does everyone agree that we should fetch (possibly after asking
for confirmation from the use) _during_ the checkout ?
I now only fetch submodules during a fetch of the supermodule
(actually, in my current patch set, I only fetch a submodule
the first time I see it, but that's a bug), but if there is
a consensus on this, I can switch to fetching during checkout.

I think fetching of subproject during fetch or clone of
superproject would not make much sense.  Making it part of
superproject checkout would probably be the way we will end up
going.  The detail of "which part of the checkout" would need to
be defined, and I tend to agree with Alex that checkout itself
would need to be multi-phased, but I think that is a minor
implementation detail we can discuss after how the overall flows
should look like.

IMHO it makes more sense to fetch during fetch of superproject:

 - If you don't fetch the superproject, it won't start refering to
   unavailable commit of subproject. So should only need to fetch subproject
   after fetching superproject.

 - If you fetch from more than one location, you want to fetch subproject
   from location corresponding to where you fetch superproject from.
   
   Let's have a repository of project P with remotes PA and PB. Let it have
   a subproject S with remotes SA and SB.

   Whenever I pull P from PA, it might refer to commit of S, that is only
   available from SA (because that's what PA owner uses). Whenever I pull
   P from PB, it might refer to commit of S, that is only available from SB
   (again because that's what PB owner uses).

   Now checkout does not know, whether I pulled the target revision from PA
   or PB, so:
    - Either it has to fetch both. But say the commit I want is in SB and SA
      contains a lot of new stuff, which will slow the thing down, though
      I don't need it.
    - Or it has to guess by looking whether any heads in remotes/PA or
      remotes/PB are descendants of the commit being checked out. But that
      feels rather hacky.

   I see several options:
    - Fetch will recurse. This should work ok and is IMHO least magic. We can
      also add some way to specify refspecs for the subproject, giving user
      control over what is fetched.
    - Fetch will store a "pending fetch from" note in the subproject and
      checkout, if it does not find the revision, will try fetching from all
      sources pointed to by those notes. There is still a problem with what
      exactly to fetch (user can specify in config).
    - Checkout will ask all subproject repositories whether they have given
      commit and pull the first one that does. This would get the needed
      commit most certainly. It would be slower though, because it would need
      to ask all the repositories whether they have the particular object.
      It also leaves the tracking branches in subproject in somewhat random
      state (maybe both repositories had the commit, so it pulled from the
      other one that user would etc.).

quoted

As to the key to use to lookup the URL in the config, right
now I simply use the directory name where it is attached
(which seems like a useful default to me).

The extra level of indirection has the advantage, that you can describe
moving the same subproject to a different directory.

-- 
						 Jan 'Bulb' Hudec [off-list ref]