Now that we have much better output when displaying diffs of
submodules in git gui and gitk (many thanks to all involved!),
another usability issue shows up: A dirty working directory of
a submodule isn't visible in git gui or gitk.
So you might think a "submodule update" would be ok - as you
see no changes - just too see it fail because the submodules
working directory is dirty.
Or - even worse - you /think/ you committed your changes in
a submodule while you didn't. That can lead to 'interesting'
problems which can be pretty hard to diagnose (like breaking
builds on other peoples machines).
A possible solution could look like this:
AFAICS, git gui and gitk use "git diff-files" both to get the
file names of unstaged local changes and to later display the
actual differences.
If they could tell the diff core to also check the submodule
working directories and to output an extra line - maybe
something like "Submodule <name> contains uncommitted local
changes" - when a submodules working directory is dirty,
git gui and gitk could show the submodules state adequately.
What do you think about this approach?
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:47:58
Hi,
On Sat, 2 Jan 2010, Jens Lehmann wrote:
Now that we have much better output when displaying diffs of submodules
in git gui and gitk (many thanks to all involved!), another usability
issue shows up: A dirty working directory of a submodule isn't visible
in git gui or gitk.
So you might think a "submodule update" would be ok - as you see no
changes - just too see it fail because the submodules working directory
is dirty.
Or - even worse - you /think/ you committed your changes in a submodule
while you didn't. That can lead to 'interesting' problems which can be
pretty hard to diagnose (like breaking builds on other peoples
machines).
A possible solution could look like this:
AFAICS, git gui and gitk use "git diff-files" both to get the file names
of unstaged local changes and to later display the actual differences.
If they could tell the diff core to also check the submodule working
directories and to output an extra line - maybe something like
"Submodule <name> contains uncommitted local changes" - when a
submodules working directory is dirty, git gui and gitk could show the
submodules state adequately.
The real problem is that submodules in the current form are not very well
designed. For example, a submodule being at a different commit than in
the superproject's index is not as fatal as the submodule having changes.
So in the long run, IMHO a proper redesign of the submodules would not
make only a little sense (it does not help, though, that those who
implemented and furthered the current approach over other discussed
approaches do not use submodules themselves -- not even now).
In ths short run, we can paper over the shortcomings of the submodules by
introducing a command line option "--include-submodules" to
update-refresh, diff-files and diff-index, though.
The implementation might be a bit tricky as parts of Git's source code
still use the_index, but at least adding the submodule's object database
is no longer that difficult.
Ciao,
Dscho
The real problem is that submodules in the current form are not very well
designed. For example, a submodule being at a different commit than in
the superproject's index is not as fatal as the submodule having changes.
So in the long run, IMHO a proper redesign of the submodules would not
make only a little sense (it does not help, though, that those who
implemented and furthered the current approach over other discussed
approaches do not use submodules themselves -- not even now).
Do you mean the complete workflow (submodules are links to other git repos)
or the current implementation? Do you have links to other design
approaches/threads? Would be nice if we could take that into account for any
decision.
cheers Heiko
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:47:58
Hi,
On Mon, 4 Jan 2010, Heiko Voigt wrote:
Johannes wrote:
quoted
The real problem is that submodules in the current form are not very
well designed. For example, a submodule being at a different commit
than in the superproject's index is not as fatal as the submodule
having changes.
So in the long run, IMHO a proper redesign of the submodules would not
make only a little sense (it does not help, though, that those who
implemented and furthered the current approach over other discussed
approaches do not use submodules themselves -- not even now).
Do you mean the complete workflow (submodules are links to other git
repos) or the current implementation? Do you have links to other design
approaches/threads? Would be nice if we could take that into account for
any decision.
Unfortunately, I do not have any information about different approaches
except the approach Subversion takes. While Subversion's externals are
not perfect for all applications, for some, they are. So I consider this
a serious shortcoming that Git does not support that workflow (and in
fact, AFAIR Shawn's repo does not use submodules for that exact reason).
But I think that an important precondition to come up with a better design
of the submodules is to have suffered the current implementation in
real-world work using submodules. (Which reminds me very much of the
autocrlf mess.)
Ciao,
Dscho
The real problem is that submodules in the current form are not very well
designed.
IMVHO using the tree sha1 for a submodule seems to be the 'natural' way
to include another git repo. And it gives the reproducibility i expect
from a scm. Or am i missing something?
It looks to me as most shortcomings come from the fact that most git
commands tend to ignore submodules (and if they don't, like git gui and
gitk do now, they e.g. only show certain aspects of their state).
Submodules are in heavy use in our company since last year. Virtually
every patch i submitted for submodules came from that experience and
scratched an itch i or one of my colleagues had (and the situation did
already improve noticeably by the few things we changed). We are still
convinced that using submodules was the right decision. But some work
has still to be done to be able to use them easily and to get rid of
some pitfalls.
In ths short run, we can paper over the shortcomings of the submodules by
introducing a command line option "--include-submodules" to
update-refresh, diff-files and diff-index, though.
Maybe this is the way to go for now (and hopefully we can turn this
option on by default later because we did the right thing ;-).
On 1/4/10, Johannes Schindelin [off-list ref] wrote:
The real problem is that submodules in the current form are not very well
designed. For example, a submodule being at a different commit than in
the superproject's index is not as fatal as the submodule having changes.
So in the long run, IMHO a proper redesign of the submodules would not
make only a little sense (it does not help, though, that those who
implemented and furthered the current approach over other discussed
approaches do not use submodules themselves -- not even now).
In ths short run, we can paper over the shortcomings of the submodules by
introducing a command line option "--include-submodules" to
update-refresh, diff-files and diff-index, though.
Incidentally I was just drafting git-super.sh it see how far it goes.
The goal was to implement some cross-module operations over time. "git
super status", "git super commit" and others could be handy.
--
Duy
On Mon, Jan 4, 2010 at 6:46 AM, Johannes Schindelin
[off-list ref] wrote:
But I think that an important precondition to come up with a better design
of the submodules is to have suffered the current implementation in
real-world work using submodules. (Which reminds me very much of the
autocrlf mess.)
I suffered the current implementation, which is why I wrote
git-subtree :) I'm still suffering, though; git-subtree works much
better for my own use cases, but after some experience with it, I'm
still not totally happy.
For me one big problem comes down to producing accurate output for
'git log'. git submodules assume that the history inside the module
is entirely separate (you need to run multiple 'git log' instances to
see the full history); git-subtree assumes that it's entirely
integrated. In that sense, git-subtree is somewhat more in line with
the core principle of git (we track the history of "the content", not
any particular file or subdir). Unfortunately, it also exposes a
problem with that core principle: taken to its extreme, "the content"
includes all data in the universe. And while git could branch and
merge the universe very efficiently in about O(log n) time, 'git log'
output gets less useful about O(n) with the size of the tree.
Neither git-subtree nor git submodules seem to help with this "log
pollution" problem very much - but I don't know what to do that would
be better.
Outside of this, my major problem with submodules is they use separate
work trees and repositories, and thus require lots of extra
housekeeping to get anything done. I'd be much happier if submodules
would share the same objects/packs/.gitdir/refs/indexfile as the
superproject, and the *only* thing special about them would be that
the superproject's tree points at a commit object instead of a tree
object. In other words, I think the actual repo format is correct
as-is, but the tools surrounding it cause a lot of confusion.
Imagine if cloning a superproject also checked out the subproject
transparently, and committing dirty data inside the subproject's tree
created a new commit object for the subproject, then tacked that
commit object into the superproject's index for a later commit
(exactly as changing a subdir creates a new tree object that the
parent directory can refer to).
This doesn't solve some use cases, however, such as ones where people
really don't want to check out (or even fetch) the contents of some
submodules, even when they check out the superproject. The current
implementation *does* handle that situation. I'm not sure how many
people rely on that behaviour, though. (And maybe the correct
solution to *that* is proper support for sparse clone/checkout
regardless of submodules.)
Have fun,
Avery
Am 04.01.2010 18:51, schrieb Nguyen Thai Ngoc Duy:
Incidentally I was just drafting git-super.sh it see how far it goes.
The goal was to implement some cross-module operations over time. "git
super status", "git super commit" and others could be handy.
Hm, i'm not sure if this will really help us. I would rather see "git
status" and friends do the right thing for submodules too. Maybe this
has to be configurable but i think the separate commands that one has
to use for submodules now are part of the usability problems we are
seeing.
IMHO putting the functionality of "git submodule summary" into "git
diff" was a step in the right direction. This thread is about adding a
line to the diff output when diffing against the working directory and
a submodule has a dirty working directory too. Then you can ask "git
diff" and it tells you anything you need to know about the submodule
before committing or checking out in the supermodule (And IMO later on
"git status" should give us this information too).
For me one big problem comes down to producing accurate output for
'git log'. git submodules assume that the history inside the module
is entirely separate (you need to run multiple 'git log' instances to
see the full history); git-subtree assumes that it's entirely
integrated. In that sense, git-subtree is somewhat more in line with
the core principle of git (we track the history of "the content", not
any particular file or subdir). Unfortunately, it also exposes a
problem with that core principle: taken to its extreme, "the content"
includes all data in the universe. And while git could branch and
merge the universe very efficiently in about O(log n) time, 'git log'
output gets less useful about O(n) with the size of the tree.
Neither git-subtree nor git submodules seem to help with this "log
pollution" problem very much - but I don't know what to do that would
be better.
I think this depends extremely on the use case and may even differ
from submodule to submodule. It might be desirable to be able to
specify which submodule logs you want to see, because only the user
knows what is important for him. But you should be able to ask "git
log" directly without forking it in every submodule you care about,
no?
There has been a thread between Junio and Heiko about group mappings
for submodules. Maybe the configuration could be extended to contain
information about what submodule should add to the superprojects log?
http://thread.gmane.org/gmane.comp.version-control.git/130928/
Outside of this, my major problem with submodules is they use separate
work trees and repositories, and thus require lots of extra
housekeeping to get anything done. I'd be much happier if submodules
would share the same objects/packs/.gitdir/refs/indexfile as the
superproject, and the *only* thing special about them would be that
the superproject's tree points at a commit object instead of a tree
object. In other words, I think the actual repo format is correct
as-is, but the tools surrounding it cause a lot of confusion.
I don't care deeply where the objects live but agree about the repo
format and the confusion ;-)
Imagine if cloning a superproject also checked out the subproject
transparently,
That would be great (at least at checkout time, after clone you
might wanna decide which submodules to initialize first - unless
group mappings are working). Right now we use post-checkout hooks
to do that.
and committing dirty data inside the subproject's tree
created a new commit object for the subproject, then tacked that
commit object into the superproject's index for a later commit
(exactly as changing a subdir creates a new tree object that the
parent directory can refer to).
That would be a nice feature.
This doesn't solve some use cases, however, such as ones where people
really don't want to check out (or even fetch) the contents of some
submodules, even when they check out the superproject. The current
implementation *does* handle that situation. I'm not sure how many
people rely on that behaviour, though. (And maybe the correct
solution to *that* is proper support for sparse clone/checkout
regardless of submodules.)
We do rely on this behavior. But sparse clone or group mappings
could replace that need.
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:47:58
Hi,
On Mon, 4 Jan 2010, Jens Lehmann wrote:
Am 04.01.2010 10:44, schrieb Johannes Schindelin:
quoted
The real problem is that submodules in the current form are not very
well designed.
IMVHO using the tree sha1 for a submodule seems to be the 'natural' way
to include another git repo. And it gives the reproducibility i expect
from a scm. Or am i missing something?
You do remember the discussion at the Alles wird Git about the need for
Subversion external-like behavior, right?
It looks to me as most shortcomings come from the fact that most git
commands tend to ignore submodules (and if they don't, like git gui and
gitk do now, they e.g. only show certain aspects of their state).
It is not only ignoring. It is not being able to cope with the state only
submodules can be in (see below).
Submodules are in heavy use in our company since last year. Virtually
every patch i submitted for submodules came from that experience and
scratched an itch i or one of my colleagues had (and the situation did
already improve noticeably by the few things we changed). We are still
convinced that using submodules was the right decision. But some work
has still to be done to be able to use them easily and to get rid of
some pitfalls.
Submodules may be the best way you have in Git for your workflow ATM.
But that does not mean that the submodule design is in any way
thought-through.
Just a few shortcomings that do show up in my main project (and to a
small extent in msysGit, as you are probably aware):
- submodules were designed with a strong emphasis on not being forced to
check them out. But Git makes it very unconvenient to actually check
submodules out, let alone check them out at clone-time. And it is
outright impossible to _enforce_ a submodule to be checked out.
- among other use cases, submodules are recommended for sharing content
between two different repositories. But it is part of the design that it
is _very_ easy to forget to commit, or push the changes in the submodule
that are required for the integrity of the superproject.
- that use case -- sharing content between different repositories -- is
not really supported by submodules, but rather an afterthought. This is
all too obvious when you look at the restriction that the shared content
must be in a single subdirectory.
- submodules would be a perfect way to provide a fast-forward-only media
subdirectory that is written to by different people (artists) than to
the superproject (developers). But there is no mechanism to enforce
shallow fetches, which means that this use case cannot be handled
efficiently using Git.
- related are the use cases where it is desired not to have a fixed
submodule tip committed to the superproject, but always to update to the
current, say, master (like Subversion's externals). This use case has
been wished away by the people who implemented submodules in Git. But
reality has this nasty habit of ignoring your wishes, does it not?
- there have been patches supporting rebasing submodules, i.e.
submodules where a "git submodule update" rebases the current branch to
the revision committed to the superproject rather than detaching the
HEAD, which everybody who ever contributed to a project with submodules
should agree is a useful thing. But the patches only have been discussed
to death, to the point where the discussion's information content was
converging to zero, yet the patches did not make it into Git. (FWIW
this is one reason why I refuse to write patches to git-submodule.sh: I
refuse to let my time to be wasted like that.)
- working directories with GIT_DIRs are a very different beast from single
files. That alone leads to a _lot_ of problems. The original design of
Git had only a couple of states for named content (AKA files): clean,
added, removed, modified. The states that are possible with submodules
are for the most part not handled _at all_ by most Git commands (and it
is sometimes very hard to decide what would be the best way to handle
those states, either). Just think of a submodule at a different
revision than committed in the superproject, with uncommitted changes,
ignored and unignored files, a few custom hooks, a bit of additional
metadata in the .git/config, and just for fun, a few temporary files in
.git/ which are used by the hooks.
- while it might be called clever that the submodules' metadata are stored
in .gitmodules in the superproject (and are therefore naturally tracked
with Git), the synchronization with .git/config is performed exactly
once -- when you initialize the submodule. You are likely to miss out
on _every_ change you pulled into the superproject.
All in all, submodules are very clumsy to work with, and you are literally
forced to provide scripts in the superproject to actually work with the
submodules.
quoted
In ths short run, we can paper over the shortcomings of the submodules
by introducing a command line option "--include-submodules" to
update-refresh, diff-files and diff-index, though.
Maybe this is the way to go for now (and hopefully we can turn this
option on by default later because we did the right thing ;-).
I do not think that --include-submodules is a good default. It is just
too expensive in terms of I/O even to check the status in a superproject
with a lot of submodules.
Besides, as long as there is enough reason to have out-of-Git alternative
solutions such as repo, submodules deserve to be 2nd-class citizens.
Ciao,
Dscho
From: Shawn O. Pearce <hidden> Date: 2016-06-15 22:47:58
Johannes Schindelin [off-list ref] wrote:
Besides, as long as there is enough reason to have out-of-Git alternative
solutions such as repo, submodules deserve to be 2nd-class citizens.
If I didn't think I'd be shot by current submodule users, I'd offer
to write a full replacement based around the current in repository
format, but with sane features like we have in repo.
Actually, that's why repo happened. I felt like submodules was
already too frozen to accept a different approach. And another
guy here thought XML might be a solution to a problem... :-|
--
Shawn.
On Mon, Jan 4, 2010 at 5:27 PM, Shawn O. Pearce [off-list ref] wrote:
Johannes Schindelin [off-list ref] wrote:
quoted
Besides, as long as there is enough reason to have out-of-Git alternative
solutions such as repo, submodules deserve to be 2nd-class citizens.
If I didn't think I'd be shot by current submodule users, I'd offer
to write a full replacement based around the current in repository
format, but with sane features like we have in repo.
Perhaps write it and call it 'git sub' or something. Put them both
in, and let users decide which they want to use. Or, like git
subtree, maintain it separately.
Personally, I've avoided tools like repo because they seem to try to
kidnap my *entire* git experience, most of which is already fine.
It's just submodules that are crazy. I think it's probably similar
for other people.
Avery
On Mon, Jan 4, 2010 at 5:29 PM, Johannes Schindelin
[off-list ref] wrote:
On Mon, 4 Jan 2010, Jens Lehmann wrote:
quoted
IMVHO using the tree sha1 for a submodule seems to be the 'natural' way
to include another git repo. And it gives the reproducibility i expect
from a scm. Or am i missing something?
You do remember the discussion at the Alles wird Git about the need for
Subversion external-like behavior, right?
I'm not sure why this is such an issue. Basically, non-version-locked
submodules are about the easiest thing in the world; that's why CVS
and SVN supported them first. (SVN later added version-locking like
git has.)
All you need is a .gitignore entry and a trivial script that checks
out the external. If you want to be fancy, this operation could be
part of git, but it's such a totally different case (and an easy one,
no less) that I think it ought to be treated totally seperately.
- among other use cases, submodules are recommended for sharing content
between two different repositories. But it is part of the design that it
is _very_ easy to forget to commit, or push the changes in the submodule
that are required for the integrity of the superproject.
[...]
- working directories with GIT_DIRs are a very different beast from single
files. That alone leads to a _lot_ of problems. The original design of
Git had only a couple of states for named content (AKA files): clean,
added, removed, modified. The states that are possible with submodules
are for the most part not handled _at all_ by most Git commands (and it
is sometimes very hard to decide what would be the best way to handle
those states, either). Just think of a submodule at a different
revision than committed in the superproject, with uncommitted changes,
ignored and unignored files, a few custom hooks, a bit of additional
metadata in the .git/config, and just for fun, a few temporary files in
.git/ which are used by the hooks.
I think this is primarily because checked-out submodules currently
have their own .git directories (with their own config, index, etc).
If they were considered *part* of the subproject's repo checkout, and
updated upon switching branches, etc, this whole class of problems
would go away.
- that use case -- sharing content between different repositories -- is
not really supported by submodules, but rather an afterthought. This is
all too obvious when you look at the restriction that the shared content
must be in a single subdirectory.
I haven't found the subdir requirement to be much of an issue, at
least on Unix where I can simply work around it using symlinks from
the superproject into the subproject. It's obviously more gross on
Windows, but I've worked around it there too. This one isn't a daily
aggravation for me, though maybe it is for others. And any cure I can
think of sounds rather worse than the disease.
- submodules would be a perfect way to provide a fast-forward-only media
subdirectory that is written to by different people (artists) than to
the superproject (developers). But there is no mechanism to enforce
shallow fetches, which means that this use case cannot be handled
efficiently using Git.
I doubt you want to "enforce" shallow fetches. And if you just want
to "allow" shallow fetches, or default to shallow fetches, I'd think
it would be pretty easy to add. This hasn't been important to me
either. (It seems to be not too important to git users in general, or
git's support *in general* for shallow repositories would be more
featureful.)
- while it might be called clever that the submodules' metadata are stored
in .gitmodules in the superproject (and are therefore naturally tracked
with Git), the synchronization with .git/config is performed exactly
once -- when you initialize the submodule. You are likely to miss out
on _every_ change you pulled into the superproject.
This could be fixed too, though I gave up on git-submodule before I
bothered to fix it myself.
The correct solution here is simply to not ever copy the settings from
.gitmodules into .git/config. Instead, git-submodule should read
.gitmodules as defaults, and then override those defaults with
anything in .git/config. 99% of users will probably not need to ever
put any of their settings in .git/config, and so this problem
disappears.
All in all, submodules are very clumsy to work with, and you are literally
forced to provide scripts in the superproject to actually work with the
submodules.
Agreed; I do this in every project which uses git-submodule. (And
from doing so, I learned that the value-added of git-submodule is
nearly zero. My script does most of the work, and it could just as
easily check out the submodule as a git repo too. I could even choose
to version-lock or not version-lock the checked-out submodule: just
hardcode the commitid into my script!)
I do not think that --include-submodules is a good default. It is just
too expensive in terms of I/O even to check the status in a superproject
with a lot of submodules.
I've thought about this a lot, and I think having a special case for
submodules here is the wrong line of thinking. A big project
*without* submodules has this same problem. The "real" solution is to
just make status checks faster.
(This is actually possible to do: in the extreme case, you just have a
daemon running with inotify or the Windows equivalent. TortoiseSvn
reputedly does something like this. I've thought of writing such a
daemon myself to just twiddle --assume-{un,}changed flags at the right
times, particularly since status checks in Windows are so ridiculously
slow. But I got frustrated when it was *still* slow even after
setting --assume-unchanged on all the files in the index. git still
scans directories to detect *unknown* files, and there seems to be no
way to turn it off or, moreover, to provide the list of unknown files
from some other source.)
Have fun,
Avery
You do remember the discussion at the Alles wird Git about the need for
Subversion external-like behavior, right?
Yup. But never having used svn, let alone externals, i think i just
did not get it then ;-)
- submodules were designed with a strong emphasis on not being forced to
check them out. But Git makes it very unconvenient to actually check
submodules out, let alone check them out at clone-time. And it is
outright impossible to _enforce_ a submodule to be checked out.
Absolutely. But i think the group mappings discussed by Junio and Heiko
are a good starting point to solve that problem:
http://thread.gmane.org/gmane.comp.version-control.git/130928/
This should be solvable by putting the necessary information into
.gitmodules and have git clone use it.
- among other use cases, submodules are recommended for sharing content
between two different repositories. But it is part of the design that it
is _very_ easy to forget to commit, or push the changes in the submodule
that are required for the integrity of the superproject.
Definitely (and if i got that right, svn externals have the same problem).
What about checking for every submodule before a push in the superproject
that its HEAD is on a remote branch? I don't think we can provide full
safety here, but we could handle the 99% case of a forgotten push in the
submodule. This could even be done with a rather simple hook (if we had a
pre-push hook that is :-).
- that use case -- sharing content between different repositories -- is
not really supported by submodules, but rather an afterthought. This is
all too obvious when you look at the restriction that the shared content
must be in a single subdirectory.
I don't see that as a problem (and it's the same with svn externals, no?).
And having worked for a long time with a RCS variant which allowed
"projects" to contain an arbitrary list of files, i don't think this is
a problem (but forgetting to add new files to this list really is, so
putting everything in one directory is *much* safer IMHO).
And: almost all files were properly grouped in directories after a decade
of development even though that was not enforced by the scm at all.
- related are the use cases where it is desired not to have a fixed
submodule tip committed to the superproject, but always to update to the
current, say, master (like Subversion's externals). This use case has
been wished away by the people who implemented submodules in Git. But
reality has this nasty habit of ignoring your wishes, does it not?
Having read up about svn externals in the meantime, what about something
like this:
- Add a command like "git submodule forward" (as update is already in
use) that takes an optional -b <branchname>. It does a fetch in the
submodule, then tries to fast forward (or rebase) to master or the
branch given and stages this commit in the superproject. This should
be the equivalent to doing an "svn update" in a repo with externals.
Or am i missing something?
(And we could avoid the detached HEAD in the fast forward case by
really checking out the branch in the submodule)
- We could also add an option to "git submodule add" to specify the
default branch name for forward.
- while it might be called clever that the submodules' metadata are stored
in .gitmodules in the superproject (and are therefore naturally tracked
with Git), the synchronization with .git/config is performed exactly
once -- when you initialize the submodule. You are likely to miss out
on _every_ change you pulled into the superproject.
Yes. This synchronization could be either obsoleted by only using
.gitmodules or automated.
Besides, as long as there is enough reason to have out-of-Git alternative
solutions such as repo, submodules deserve to be 2nd-class citizens.
I think in the long run to make submodules first class citizens the
following submodule commands must be obsoleted by their regular git
parts: init (by git clone), status (by git status), update (by git
checkout), summary (already in git diff thanks to your patch) and sync
(maybe Avery's idea of only relying on .gitmodules and not copying data
int .git/config would solve this).
That would leave git submodule add, foreach and maybe a command to do
what svn update does for externals and another to manipulate things like
group membership etc..
Which reminds me of Sverre's quote from Alles Wird Git:
"Yes, it is possible. But it will be hard."
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:47:58
Hi,
On Tue, 5 Jan 2010, Jens Lehmann wrote:
Am 04.01.2010 23:29, schrieb Johannes Schindelin:
quoted
- submodules were designed with a strong emphasis on not being forced
to check them out. But Git makes it very unconvenient to actually
check submodules out, let alone check them out at clone-time. And
it is outright impossible to _enforce_ a submodule to be checked
out.
Absolutely. But i think the group mappings discussed by Junio and Heiko
are a good starting point to solve that problem:
http://thread.gmane.org/gmane.comp.version-control.git/130928/
This should be solvable by putting the necessary information into
.gitmodules and have git clone use it.
And of course, existing Git versions will not handle it correctly.
Judging from the rebasing-submodule patch, the next Git version will not
handle it either.
But you're correct, one has to start _somewhere_.
quoted
- among other use cases, submodules are recommended for sharing
content between two different repositories. But it is part of the
design that it is _very_ easy to forget to commit, or push the
changes in the submodule that are required for the integrity of the
superproject.
Definitely (and if i got that right, svn externals have the same problem).
Yes, svn externals have that problem. But we do not need to take the svn
externals example more seriously than it deserves: it illustrates a valid
use case that is not handled by submodules. But svn externals are not
what I would call "elegant design" either.
What about checking for every submodule before a push in the
superproject that its HEAD is on a remote branch? I don't think we can
provide full safety here, but we could handle the 99% case of a
forgotten push in the submodule. This could even be done with a rather
simple hook (if we had a pre-push hook that is :-).
The problem with hooks is that for security reasons, every user has to
install them in every repository herself (unless she is working on a
machine serviced by an overzealous administrator).
quoted
- that use case -- sharing content between different repositories --
is not really supported by submodules, but rather an afterthought.
This is all too obvious when you look at the restriction that the
shared content must be in a single subdirectory.
I don't see that as a problem (and it's the same with svn externals, no?).
And having worked for a long time with a RCS variant which allowed
"projects" to contain an arbitrary list of files, i don't think this is
a problem (but forgetting to add new files to this list really is, so
putting everything in one directory is *much* safer IMHO).
And: almost all files were properly grouped in directories after a decade
of development even though that was not enforced by the scm at all.
That happens to be the case here, I agree.
But I have a use case here where the shared content is _not_ a library
that can live in a subdirectory naturally.
quoted
- related are the use cases where it is desired not to have a fixed
submodule tip committed to the superproject, but always to update to
the current, say, master (like Subversion's externals). This use
case has been wished away by the people who implemented submodules
in Git. But reality has this nasty habit of ignoring your wishes,
does it not?
Having read up about svn externals in the meantime, what about something
like this:
- Add a command like "git submodule forward" (as update is already in
use) that takes an optional -b <branchname>. It does a fetch in the
submodule, then tries to fast forward (or rebase) to master or the
branch given and stages this commit in the superproject. This should
be the equivalent to doing an "svn update" in a repo with externals.
Or am i missing something?
Yes. It is not the decision of the fetcher, but of the guy who adds the
submodule to decide what it is.
- We could also add an option to "git submodule add" to specify the
default branch name for forward.
That's an obvious precondition for proper always-tip-submodules. But
Git's core data structure, the index, does not allow for it. _That_ is
the difficulty, not what the user interface would look like.
quoted
- while it might be called clever that the submodules' metadata are
stored in .gitmodules in the superproject (and are therefore
naturally tracked with Git), the synchronization with .git/config is
performed exactly once -- when you initialize the submodule. You
are likely to miss out on _every_ change you pulled into the
superproject.
Yes. This synchronization could be either obsoleted by only using
.gitmodules or automated.
I start to wonder whether the insistence that .gitmodules' settings must
be overrideable makes any sense in practice.
quoted
Besides, as long as there is enough reason to have out-of-Git
alternative solutions such as repo, submodules deserve to be 2nd-class
citizens.
I think in the long run to make submodules first class citizens the
following submodule commands must be obsoleted by their regular git
parts: init (by git clone), status (by git status), update (by git
checkout), summary (already in git diff thanks to your patch) and sync
(maybe Avery's idea of only relying on .gitmodules and not copying data
int .git/config would solve this).
Avery's idea was to make .gitmodules overrideable in .git/config, which
would share almost all the shortcomings I listed for the current solution.
That would leave git submodule add, foreach and maybe a command to do
what svn update does for externals and another to manipulate things like
group membership etc..
Which reminds me of Sverre's quote from Alles Wird Git: "Yes, it is
possible. But it will be hard."
Yeah, it will be hard. Especially since the fact that submodule is a
bloated shell script has outlived its usefulness by far. (It would be
different if it was a nice, small, elegant script, but you have looked at
it, so you know why I am disgusted.)
Ciao,
Dscho
But I have a use case here where the shared content is _not_ a library
that can live in a subdirectory naturally.
Yes, we had to reorganize a major part of one project too. Heiko could
tell more about that.
quoted
Having read up about svn externals in the meantime, what about something
like this:
- Add a command like "git submodule forward" (as update is already in
use) that takes an optional -b <branchname>. It does a fetch in the
submodule, then tries to fast forward (or rebase) to master or the
branch given and stages this commit in the superproject. This should
be the equivalent to doing an "svn update" in a repo with externals.
Or am i missing something?
Yes. It is not the decision of the fetcher, but of the guy who adds the
submodule to decide what it is.
quoted
- We could also add an option to "git submodule add" to specify the
default branch name for forward.
That's an obvious precondition for proper always-tip-submodules. But
Git's core data structure, the index, does not allow for it. _That_ is
the difficulty, not what the user interface would look like.
I have never experienced (and never had the need for) such an always-tip
scenario and therefore still seem to have difficulties to grok it. I
assume you always want to have the newest tip at /checkout/ time, not at
/commit/ time? Then my proposal would really not help you.
I start to wonder whether the insistence that .gitmodules' settings must
be overrideable makes any sense in practice.
I know of none, maybe someone else can speak up here?
(And even if it is overrideable, do the settings necessarily have to be
copied into .git/config when they aren't even overridden?)
On Tue, Jan 05, 2010 at 10:46:11AM +0100, Johannes Schindelin wrote:
On Tue, 5 Jan 2010, Jens Lehmann wrote:
quoted
Yes. This synchronization could be either obsoleted by only using
.gitmodules or automated.
I start to wonder whether the insistence that .gitmodules' settings must
be overrideable makes any sense in practice.
I just read this and felt the need to comment.
Yes, it definitely makes sense in practise to have it overrideable
otherwise we loose the distributed nature of git for submodules.
Imagine you fork a project and you want to work with others on a change
that involves chaning a subproject. If you can not override .gitmodules
you can only work on the central repository.
I am actually working like this in practise. I have a private clone of
all the subprojects msysgit has and commit/push locally first. Once I
sense the change is going to be useful for a wider audience I send it
upstream. This would be more uncomfortable if it is not overideable.
But I know what you mean by the general confusion about manual updates.
So how about an approach like this:
* clone will initialise all submodules in .git/config from .gitmodules
* if a change in .gitmodules happens git scans .git/config for that
entry and in case nothing is there it syncronises the new one and
notifies the user.
* if a change in .gitmodules happens and the entry before was the same
in .git/config we also automatically update that entry there.
* In every other case we just leave .git/config alone.
Did I miss anything? I think you should get the idea and that it could
get rid of the confusion caused by manual .gitmodule updates.
cheers Heiko
P.S.: Additionally (for my use case) we could add a "hint mechanism"
which allows git to "guess" a new submodules address. For example in
case I have all my local clones on "git@my.server.net:<modulename>.git".
Now when a new submodule gets seen in .gitmodules it will infer the
address from the hint configuration and not take the original one from
upstream.
From: Johan Herland <hidden> Date: 2016-06-15 22:47:58
On Tuesday 05 January 2010, Heiko Voigt wrote:
P.S.: Additionally (for my use case) we could add a "hint mechanism"
which allows git to "guess" a new submodules address. For example in
case I have all my local clones on
"git@my.server.net:<modulename>.git". Now when a new submodule gets
seen in .gitmodules it will infer the address from the hint
configuration and not take the original one from upstream.
This can be achieved today, if the upstream .gitmodules uses relative
submodule URLs. I normally place super-repo and submodules in a single
directory on the server, and use submodule URLs of the
form "../<modulename>.git". Now, downstream developers can "git
clone --mirror" the repos from my server, and - as long as they
preserve the directory layout - provide their own complete server
mirror, without editing .gitmodules. Granted, the existing submodule
tools don't make working with relative submodule URLs particularily
easy...
...Johan
--
Johan Herland, [off-list ref]
www.herland.net
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:47:58
Hi,
On Tue, 5 Jan 2010, Heiko Voigt wrote:
On Tue, Jan 05, 2010 at 10:46:11AM +0100, Johannes Schindelin wrote:
quoted
On Tue, 5 Jan 2010, Jens Lehmann wrote:
quoted
Yes. This synchronization could be either obsoleted by only using
.gitmodules or automated.
I start to wonder whether the insistence that .gitmodules' settings must
be overrideable makes any sense in practice.
I just read this and felt the need to comment.
Yes, it definitely makes sense in practise to have it overrideable
otherwise we loose the distributed nature of git for submodules.
AFAICT you can use url.<base>.insteadOf for that.
Or maybe even better use a different remote for that, as you are likely
wanting to stay up-to-date with the upstream projects even if you work on
the stuff locally.
But I know what you mean by the general confusion about manual updates.
So how about an approach like this:
* clone will initialise all submodules in .git/config from .gitmodules
* if a change in .gitmodules happens git scans .git/config for that
entry and in case nothing is there it syncronises the new one and
notifies the user.
* if a change in .gitmodules happens and the entry before was the same
in .git/config we also automatically update that entry there.
* In every other case we just leave .git/config alone.
I'm sorry, but this is the kind of stuff I am seeing in Git: a lot of
really complicated design with a lot of corner cases, put on top of a
really simple and elegant design.
So I'd like to see a solution that is obviously superior by being
plain simple.
Ciao,
Dscho
From: Pau Garcia i Quiles <hidden> Date: 2016-06-15 22:47:58
Hello,
Let me pop here to support Johannes: I agree with every single point
he enumerated. Every. Single. Point.
For instance, I'd like to have a 'cmake' repository where I store all
the FindBlah.cmake modules, so that I can share them from every
repository, and not worry about users changing and committing in the
main project instead of the submodule. I can't. Subversion externals
still rule in that regard.
On Mon, Jan 4, 2010 at 11:29 PM, Johannes Schindelin
[off-list ref] wrote:
Hi,
On Mon, 4 Jan 2010, Jens Lehmann wrote:
quoted
Am 04.01.2010 10:44, schrieb Johannes Schindelin:
quoted
The real problem is that submodules in the current form are not very
well designed.
IMVHO using the tree sha1 for a submodule seems to be the 'natural' way
to include another git repo. And it gives the reproducibility i expect
from a scm. Or am i missing something?
You do remember the discussion at the Alles wird Git about the need for
Subversion external-like behavior, right?
quoted
It looks to me as most shortcomings come from the fact that most git
commands tend to ignore submodules (and if they don't, like git gui and
gitk do now, they e.g. only show certain aspects of their state).
It is not only ignoring. It is not being able to cope with the state only
submodules can be in (see below).
quoted
Submodules are in heavy use in our company since last year. Virtually
every patch i submitted for submodules came from that experience and
scratched an itch i or one of my colleagues had (and the situation did
already improve noticeably by the few things we changed). We are still
convinced that using submodules was the right decision. But some work
has still to be done to be able to use them easily and to get rid of
some pitfalls.
Submodules may be the best way you have in Git for your workflow ATM.
But that does not mean that the submodule design is in any way
thought-through.
Just a few shortcomings that do show up in my main project (and to a
small extent in msysGit, as you are probably aware):
- submodules were designed with a strong emphasis on not being forced to
check them out. But Git makes it very unconvenient to actually check
submodules out, let alone check them out at clone-time. And it is
outright impossible to _enforce_ a submodule to be checked out.
- among other use cases, submodules are recommended for sharing content
between two different repositories. But it is part of the design that it
is _very_ easy to forget to commit, or push the changes in the submodule
that are required for the integrity of the superproject.
- that use case -- sharing content between different repositories -- is
not really supported by submodules, but rather an afterthought. This is
all too obvious when you look at the restriction that the shared content
must be in a single subdirectory.
- submodules would be a perfect way to provide a fast-forward-only media
subdirectory that is written to by different people (artists) than to
the superproject (developers). But there is no mechanism to enforce
shallow fetches, which means that this use case cannot be handled
efficiently using Git.
- related are the use cases where it is desired not to have a fixed
submodule tip committed to the superproject, but always to update to the
current, say, master (like Subversion's externals). This use case has
been wished away by the people who implemented submodules in Git. But
reality has this nasty habit of ignoring your wishes, does it not?
- there have been patches supporting rebasing submodules, i.e.
submodules where a "git submodule update" rebases the current branch to
the revision committed to the superproject rather than detaching the
HEAD, which everybody who ever contributed to a project with submodules
should agree is a useful thing. But the patches only have been discussed
to death, to the point where the discussion's information content was
converging to zero, yet the patches did not make it into Git. (FWIW
this is one reason why I refuse to write patches to git-submodule.sh: I
refuse to let my time to be wasted like that.)
- working directories with GIT_DIRs are a very different beast from single
files. That alone leads to a _lot_ of problems. The original design of
Git had only a couple of states for named content (AKA files): clean,
added, removed, modified. The states that are possible with submodules
are for the most part not handled _at all_ by most Git commands (and it
is sometimes very hard to decide what would be the best way to handle
those states, either). Just think of a submodule at a different
revision than committed in the superproject, with uncommitted changes,
ignored and unignored files, a few custom hooks, a bit of additional
metadata in the .git/config, and just for fun, a few temporary files in
.git/ which are used by the hooks.
- while it might be called clever that the submodules' metadata are stored
in .gitmodules in the superproject (and are therefore naturally tracked
with Git), the synchronization with .git/config is performed exactly
once -- when you initialize the submodule. You are likely to miss out
on _every_ change you pulled into the superproject.
All in all, submodules are very clumsy to work with, and you are literally
forced to provide scripts in the superproject to actually work with the
submodules.
quoted
quoted
In ths short run, we can paper over the shortcomings of the submodules
by introducing a command line option "--include-submodules" to
update-refresh, diff-files and diff-index, though.
Maybe this is the way to go for now (and hopefully we can turn this
option on by default later because we did the right thing ;-).
I do not think that --include-submodules is a good default. It is just
too expensive in terms of I/O even to check the status in a superproject
with a lot of submodules.
Besides, as long as there is enough reason to have out-of-Git alternative
solutions such as repo, submodules deserve to be 2nd-class citizens.
Ciao,
Dscho
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Pau Garcia i Quiles
http://www.elpauer.org
(Due to my workload, I may need 10 days to answer)
On Tue, Jan 05, 2010 at 10:46:11AM +0100, Johannes Schindelin wrote:
quoted
On Tue, 5 Jan 2010, Jens Lehmann wrote:
quoted
Yes. This synchronization could be either obsoleted by only using
.gitmodules or automated.
I start to wonder whether the insistence that .gitmodules' settings must
be overrideable makes any sense in practice.
I just read this and felt the need to comment.
Yes, it definitely makes sense in practise to have it overrideable
otherwise we loose the distributed nature of git for submodules.
Imagine you fork a project and you want to work with others on a change
that involves chaning a subproject. If you can not override .gitmodules
you can only work on the central repository.
I am actually working like this in practise. I have a private clone of
all the subprojects msysgit has and commit/push locally first. Once I
sense the change is going to be useful for a wider audience I send it
upstream. This would be more uncomfortable if it is not overideable.
But I know what you mean by the general confusion about manual updates.
So how about an approach like this:
* clone will initialise all submodules in .git/config from .gitmodules
* if a change in .gitmodules happens git scans .git/config for that
entry and in case nothing is there it syncronises the new one and
notifies the user.
* if a change in .gitmodules happens and the entry before was the same
in .git/config we also automatically update that entry there.
* In every other case we just leave .git/config alone.
Did I miss anything? I think you should get the idea and that it could
get rid of the confusion caused by manual .gitmodule updates.
cheers Heiko
P.S.: Additionally (for my use case) we could add a "hint mechanism"
which allows git to "guess" a new submodules address. For example in
case I have all my local clones on "git@my.server.net:<modulename>.git".
Now when a new submodule gets seen in .gitmodules it will infer the
address from the hint configuration and not take the original one from
upstream.
Thanks for sharing your thoughts. I find this discussion very interesting.
I found this other discussion in the design area enlightening.
http://thread.gmane.org/gmane.comp.version-control.git/47466/focus=47621
It was before I started using git heavily and I don't see many people who were in the discussion yet in the current thread, but I think it is worth reading.
P.S. A happy new year to everybody!
--
Nanako Shiraishi
http://ivory.ap.teacup.com/nanako3/
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:47:58
Hi,
On Tue, 5 Jan 2010, Pau Garcia i Quiles wrote:
For instance, I'd like to have a 'cmake' repository where I store all
the FindBlah.cmake modules, so that I can share them from every
repository, and not worry about users changing and committing in the
main project instead of the submodule.
... which reminds me... it was you who wanted to provide a working recipe
to compile and install CMake on msysGit, right?
What happened in the meantime?
Ciao,
Dscho
Could you be so kind and summarize the result of the thread in something
like 2000 characters?
I am sorry, but what with the recent trend of a precious few Git mailing
list members using up my weekly Git time budget in less than half a day,
just by me reading their mails, it would be nice if at least _some_
discussions on the list could be concise and to the point.
Thanks,
Dscho
From: Pau Garcia i Quiles <hidden> Date: 2016-06-15 22:47:58
On Wed, Jan 6, 2010 at 12:06 AM, Johannes Schindelin
[off-list ref] wrote:
Hi,
On Tue, 5 Jan 2010, Pau Garcia i Quiles wrote:
quoted
For instance, I'd like to have a 'cmake' repository where I store all
the FindBlah.cmake modules, so that I can share them from every
repository, and not worry about users changing and committing in the
main project instead of the submodule.
... which reminds me... it was you who wanted to provide a working recipe
to compile and install CMake on msysGit, right?
Right
What happened in the meantime?
What happened is I was very busy until November. Now I've got some free time.
At this moment, what stops me from beginning this project is a simple
question: is it worth my time? From the discussion a few months ago,
it looked like it would the a second-class citizen and never replace
the existing buildsystems, so I really wonder if I should spend me
time porting git to CMake, or I should focus on other projects which
would gladly receive my contributions. If you honestly think it's
worth it, just tell me and I'll start the port to CMake immediately.
--
Pau Garcia i Quiles
http://www.elpauer.org
(Due to my workload, I may need 10 days to answer)
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:47:58
Hi,
On Wed, 6 Jan 2010, Pau Garcia i Quiles wrote:
On Wed, Jan 6, 2010 at 12:06 AM, Johannes Schindelin
[off-list ref] wrote:
quoted
On Tue, 5 Jan 2010, Pau Garcia i Quiles wrote:
quoted
For instance, I'd like to have a 'cmake' repository where I store all
the FindBlah.cmake modules, so that I can share them from every
repository, and not worry about users changing and committing in the
main project instead of the submodule.
... which reminds me... it was you who wanted to provide a working
recipe to compile and install CMake on msysGit, right?
Right
quoted
What happened in the meantime?
What happened is I was very busy until November. Now I've got some free
time.
At this moment, what stops me from beginning this project is a simple
question: is it worth my time?
Well, I thought you wanted to show that CMake is superior to what we have
right now, and for me as msysGit maintainer, that implies that CMake
actually works within msysGit.
Now, I do not think that it is hard to get CMake to compile in msysGit,
but then, I just lost access to the last Windows computer, so I cannot do
that myself.
As Miles said, it is up to you to decide whether it is so complicated, or
whether CMake is likely not to convince, that the time balance turns out
positive or negative.
Ciao,
Dscho
Could you be so kind and summarize the result of the thread in something
like 2000 characters?
Sorry, but I only said "enlightening". There wasn't a conclusion that lets you stop thinking and just go ahead implementing the design specified in the thread, if that is what you are looking for.
Instead, let me tell you an example of what I found enlightening. It isn't a summary of the result. I don't think there was a *result*; otherwise somebody already would have implemented it.
I often wonder why 'git-submodule init' copies data to .git/config file. If .gitmodules file gives the default and I can use .git/config file to override it, it seems stupid to copy entries between these files. I can just keep using data from .gitmodules file until I need to override something.
Reading the thread made me realize how wrong I was. It became very clear why .gitmodules file shouldn't even be the default that is read when no entries is in .git/config file and why .git/config file should be the only thing that is used at runtime.
Unfortunately I can't summarize the reason in '2000 characters', so you need read the thread yourself if you are interested. The key concept that I was missing was that remote repositories can move or change over time, and you may want to check out and interact with a very old version of your supermodule. The .gitmodules file checked out in such a case still records old information. Treating .gitmodules file as a hint and always looking into .git/config file is a part of the fundamental solution to that problem, but I didn't even realize that such an issue existed when I read the current discussion until I found the old thread.
I think the 'git-submodule' script is mainly based on the 'three-level thing Steven Grimm suggested', but it doesn't seem to implement all the ideas in the thread yet. It gives no interactive prompt to suggest URL from 'git-submodule init' command. Neither it records which URLs have been seen with subproject.*.seen variable. But the issues that high level design must take into account looks very well thought out already.
--
Nanako Shiraishi
http://ivory.ap.teacup.com/nanako3/