Avery Pennarun [off-list ref] writes:
On Mon, Jul 26, 2010 at 09:41:42AM -0700, Linus Torvalds wrote:
quoted
On Mon, Jul 26, 2010 at 9:37 AM, Marc Branchaud [off-list ref] wrote:
quoted
I think I should mention one aspect of what we're doing, which is that a lot
of our submodules are based on external code, and that we occasionally need
to modify or customize some of that code. So it's quite nice for us to
maintain private git mirrors of the external repos, with our own private
branches that contain our modifications. Although we want to get much of our
changes incorporated into the upstream code bases, upstream release cycles
are rarely in sync with ours.
THIS.
This is why I always thought that submodules absolutely have to be
commits, not trees. It's why the git submodule data structures are
done the way they are. Anything that makes the submodule just a tree
is fundamentally broken, I think.
I agree completely. The major failing of git-subtree is that it uses
tree->tree links instead of tree->commit links.
This was necessary only because git fundamentally *mistreats* tree->commit
links: it refuses to push or fetch through them automatically.
I do not think that is so "fundamental" as you seem to think.
Isn't it just the matter of how the default UI of object transfer commands
(like push and fetch) are set up?
Admittedly, the way the default UI is set up is to strongly favor the
early design decision we made back when Linus did his initial "gitlink"
implementation, which is "separate project lives in a separate repository,
and not having to check out any subproject should be the norm for using a
superproject".
Some "recursive" operations have been added to commands for which it makes
sense (e.g. "clone --recursive") by people who cared enough. Even though
there are a few other commands that shouldn't ever learn the recursive
mode (e.g. "commit --recursive -m $msg" would not make sense), there still
are some commands where a similar "--recursive" option would make sense
but haven't learned it (e.g. "push --recursive").
I also consider it merely a lack of UI enhancement that you have to clone
the submodule again (or cannot switch to a clean slate very easily) when
switching between revisions of superproject before and after you add a
submodule, and nothing fundamental.
When switching back in history to lose a recent submodule, the user
experience should be like switching to a revision that didn't have a
directory. You shouldn't be able to lose your change in that directory,
but if the directory is clean, you should be able to lose it. And when
you switch to a more recent revision that has the submodule, you should be
able to get it back (again, if you have a precious file there, the
checkout should barf).
We have added support for having "gitdir: $dir" in a regular file .git
exactly because we wanted to be able to stash away the submodule's .git
directory somewhere inside .git (e.g. .git/modules/<submodulename>) in the
superproject when we do that kind of branch switching, so that we can get
it back when switching back to a revision with the submodule without
having to re-clone (also this presumably would help when you move the
submodule in the superproject tree), but there haven't been further work
to make use of this in "git submodule update" (it probably needs to start
by teaching "git clone" how to make use of "gitdir: $dir", if anybody is
interested).
By the way, I also do not think it is such a bad thing that git-subtree
does not bind commit into its superproject tree while it is working
"natively" (in a "git-subtree" workflow), but allows users to easily split
the history into an exportable shape to upstreams of its submodules when
such an operqation is needed. If you rarely push back to upstreams but
constantly consume their changes, that sounds like a reasonable way to go.
On Tue, Jul 27, 2010 at 4:25 PM, Junio C Hamano [off-list ref] wrote:
Avery Pennarun [off-list ref] writes:
quoted
I agree completely. The major failing of git-subtree is that it uses
tree->tree links instead of tree->commit links.
This was necessary only because git fundamentally *mistreats* tree->commit
links: it refuses to push or fetch through them automatically.
I do not think that is so "fundamental" as you seem to think.
Isn't it just the matter of how the default UI of object transfer commands
(like push and fetch) are set up?
Well, I call it fundamental because there's currently no way to get
the git UI to do otherwise. It's not really just a "default." To
depend on this changing would have prevented me from writing
git-subtree, which is why I didn't depend on it. However, I agree
that it's fixable.
Note that the way git treats a checked-out submodule (as you describe
below) is also very fundamental to how this works. git-subtree
wouldn't have the usability that it does if 'git checkout branchname'
didn't work perfectly will all the subtrees, which it currently does,
but which it wouldn't if I had relied on tree->commit links.
Some "recursive" operations have been added to commands for which it makes
sense (e.g. "clone --recursive") by people who cared enough. Even though
there are a few other commands that shouldn't ever learn the recursive
mode (e.g. "commit --recursive -m $msg" would not make sense), there still
are some commands where a similar "--recursive" option would make sense
but haven't learned it (e.g. "push --recursive").
One problem with this line of reasoning is that "--recursive" is
always an option. But if submodules are ever to be easy to use, I
think it should be the default (or settable as a default using git
config). This would take us a *long* way towards usability (of
course, in addition to adding the missing features, as you mention).
Also, I haven't tried it, but I think 'git gc' will prune away objects
if the only reference to them is a 'commit' link from a tree. This
would be undesirable too.
I also consider it merely a lack of UI enhancement that you have to clone
the submodule again (or cannot switch to a clean slate very easily) when
switching between revisions of superproject before and after you add a
submodule, and nothing fundamental.
I mostly agree with this. There is one problem I don't know how to
solve with this idea, though: what happens when commit A adds a
submodule in modules/mod1, commit B removes it, and then commit C
re-adds the same submodules in modules/mod1-again? Will it reuse the
same submodule .git directory or a new one? Share objects or not?
Share branch names or not? Share .git/config or not?
Unless you have some kind of "unique id" scheme for submodules, this
gets impossible to handle correctly. And the git objects themselves
(trees that link to commits) have nowhere to put such things.
By comparison, simply putting all the stuff related to all the
submodules into the supermodule's repo creates none of these confusing
problems. You could even still choose not to checkout individual
submodules' trees if you wanted.
When switching back in history to lose a recent submodule, the user
experience should be like switching to a revision that didn't have a
directory. You shouldn't be able to lose your change in that directory,
but if the directory is clean, you should be able to lose it. And when
you switch to a more recent revision that has the submodule, you should be
able to get it back (again, if you have a precious file there, the
checkout should barf).
It sounds like you're proposing that we delete the entire submodule's
directory hierarchy when the submodule commit link goes away. Note
that this isn't what happens in the non-submodule case: all the *.o
files, for example, in a deleted subdirectory are not automatically
deleted by git. And I think this is the behaviour we should expect.
With that in mind, the situations where checkout barfs because of a
"precious" file should be the same as they are in normal git: it
should only be a problem if the files in question differ between the
originally-checked-out tree and the newly-checked-out tree.
Apologies if that's what you meant in the first place.
We have added support for having "gitdir: $dir" in a regular file .git
exactly because we wanted to be able to stash away the submodule's .git
directory somewhere inside .git (e.g. .git/modules/<submodulename>) in the
superproject when we do that kind of branch switching, so that we can get
it back when switching back to a revision with the submodule without
having to re-clone (also this presumably would help when you move the
submodule in the superproject tree), but there haven't been further work
to make use of this in "git submodule update" (it probably needs to start
by teaching "git clone" how to make use of "gitdir: $dir", if anybody is
interested).
I guess the real question is: just how much of a "real" repository do
we want a submodule to act like?
Thoughts:
- object store: I think this should just always be shared with the
superproject. There's no reason to separate them that I can see.
- branches: should be a way to simply not worry about branches and
just use what's in the superproject. Other people seem to want to be
able to have a set of branches/tags for their submodule.
- .git/config: entirely shared? entirely separate?
- remotes: I would want my submodules to never do their own
pushing/pulling, and leave that to the supermodule; other people seem
to disagree.
For the particular model I'm proposing, I'm just not sure that *any*
of the features of a separate repo are warranted... and having them
adds a lot of complication. (In the most basic level, you suddenly
need to track .git directories as submodules are added/deleted/moved
around when you checkout different revisions of the superproject, and
there seems to be no way to do that elegantly.)
Have fun,
Avery
Am 27.07.2010 22:57, schrieb Avery Pennarun:
One problem with this line of reasoning is that "--recursive" is
always an option. But if submodules are ever to be easy to use, I
think it should be the default (or settable as a default using git
config). This would take us a *long* way towards usability (of
course, in addition to adding the missing features, as you mention).
And that is exactly what I am currently doing:
- I already teached diff and status to always recurse (and just
sent a patch to add a config option for that behavior, as some
users either can't pay the performance costs or don't want to
see submodules show up as modified just because they contain
untracked files).
- I posted a WIP patch doing recursive checkouts (that is basically
working but I still have to put in the safety checks so that no
modifications to submodules are accidentally discarded unless -f
is used).
- I am working on a recursive fetch too.
And then there is other stuff on my list to be tackled; I try to
fix these issues so that the most annoying problems get solved
first.
Unfortunately that does not proceed as fast as i wished, but
hopefully I can show some progress in the near future. Of course
any help would greatly be appreciated ;-)