From: Junio C Hamano <hidden> Date: 2016-08-11 20:12:34
Jakub Narebski [off-list ref] writes:
By the way, in todo branch, in Subpro.txt, there is talk about adding
link to submodule trees in _commit object_... well link to submodule tree
or commit, with the "mount point".
That was shot down by Linus and I agree with him. "bind" was a
bad idea because binding of a particular subproject commit into
a tree is a property of the tree, not one of the commits that
happen to have that tree.
From: Martin Waitz <hidden> Date: 2016-08-11 19:16:34
hoi :)
On Fri, Dec 01, 2006 at 03:09:40PM -0800, Linus Torvalds wrote:
On Fri, 1 Dec 2006, sf wrote:
quoted
If you do not want to fetch all of the supermodule then do not fetch the
supermodule.
So why do you want to limit it? There's absolutely no cost to saying "I
want to see all the common shared infrastructure, but I'm actually only
interested in this one submodule that I work with".
An interesting way to support this "only fetch some modules" use-case is
to use several supermodules.
So you could have one supermodule which is geared towards developers and
only contains the modules they use. Another supermodule contails all
the toolchain sources. And then there is the supermodule used for
releases which is just a merge of all the other supermodules.
The concept is so flexible that you don't have to introduce lots of
other things as module namespaces. Just use the tools you have in a
creative way ;-)
--
Martin Waitz
From: Josef Weidendorfer <hidden> Date: 2016-08-11 19:17:01
On Friday 01 December 2006 23:12, Martin Waitz wrote:
hoi :)
On Fri, Dec 01, 2006 at 11:06:40PM +0100, Josef Weidendorfer wrote:
quoted
quoted
Well, I would actually argue that you may often want to have a
supermodule and then at least have the _option_ to decide to not
fetch all the submodules.
If you want to allow this, you have to be able to cut off fetching the
objects of the supermodule at borders to given submodules, the ones you
do not want to track. With "border" I mean the submodule commit in some
tree of the supermodule.
I don't think this is something special to submodules. There has been
interest in checking out only a part of the tree even before talking
about submodules and I really think this feature should be independent
to submodules.
It's not about checking out part of the tree, it's about fetching only
part of the objects: If you have a slow modem and want to clone a
supermodule, you are not interested in fetching all the objects from
some submodules.
So it is more like a shallow clone. But even here, submodules are special
as you have defined borders between supermodule and submodules. This gives
you the freedom the introduce a submodule namespace, and allows you to
point to a submodule: "I do not want you!".
With shallow clone, there you do not have this option, so there, you need
to use something like grafting.
BTW: In your submodule implementation, is the user allowed to change the
relative path of the root of some submodule, e.g. with "git-mv" ?
Josef
On 12/15/06, Josef Weidendorfer [off-list ref] wrote:
That all sounds fine, but how do you create such symlinks in practice?
I'm very open to suggestions here, but the concept growing in my head
is based around Linus 'module'-file and keep things simple. A git
configuration file that specifies:
* link name for reference
* local path to link
* submodule source
* submodule path to tree/blob
* submodule commit / HEAD / branch
* options (depth-limit , ...)
I'm reconsidering having the path-name in the link, it should be
sufficient to have two SHA1's, one for the commit and one for the
tree/blob. Super-module should have the tree/blob in it's database so
that the link part only is there for version information and reference
(checking dirty state or history on the submodule). This way it easy
to clone the super-project and use it without having to map up all
sub-project sources. Sub-project sources is not important for version
information and could always be specified in the project in a
README-type of file.
Especially, what is the SCM user supposed to do to change the link
target, ie. from
<commit>/path/to/subtree
to
<commit>/path2/to2/subtree2
?
Should this do a re-checkout at the other point?
That would be a change in the modules file, maybe through a command
that also fixes the link. The link will have to be updated in the
index and commited as normal.
By linking a file from a submodule, such a link seems to force that
this file has to be at a fixed position in the submodule. Otherwise,
some magic has to happen when the file is moved in the submodule,
possibly leading to a dangling link, eg. if the whole subdirectory
specified in the link is removed.
Since we have the SHA1 (this is what we're using) and tree/blob
information in the super-modules database the change itself is not a
problem. The problem is to track renames/moves and your remove case in
the submodule. The tool that tracks the submodule should probably
warn/exit here and we would fix up the modules file manually.
IMHO this is getting way to complex.
One of complex situation here as I see it is the ability to handle to
track/checkout only a subset (tree/blob) of the submodule. This is
also quite an important feature - in my example it means the
difference of tracking one header file versus the whole source.
If you only want to check out part of a submodule, this should be
done with path-limiting checkouts, which should be a feature totally
independent from submodules.
If we can do path-limiting checkouts on a repo (module) we also can do
it on a sub-module since they are exactly the same. This is a very
powerful feature and it'd be a huge waste if it wasn't allowed for a
super-module to do on submodules.
And if you want to limit the number of objects transferred in cloning
of a subproject, it is better to further split this subproject into
multiple subprojects itself.
What if we have no control of the submodule? This can be tracked from
upstream, sourceforge, another company, etc. The submodule will often
live their own life and could be X, kernel, gcc, cairo, whatever, ...
The problem is not the representation in the git repository, but the
checked out module/submodule, where you need to use normal UNIX file semantics.
To move submodules around, the user should be able to just use
the normal UNIX "mv" commands, and git should be able to detect move
actions after the fact.
If we disregard the commit info, the link will act exactly as a normal
tree/blob. Git can know we're moving a subproject by watching the
module file. The main problem is to keep modules file up-to-date with
reality. We could enforce module file validity by disallowing such
operations and let the user do a "force" operation which also alters
the modules file.
This now becomes a problem if you use symlinks to "unify" multiple checkouts
of the same submodule at multiple places in the supermodule, and move
the symlink around, as it easily can get dangling this way. Thus, you would
not have a way to see what submodule this link was talking about.
The symlink only exists in the modules file. We only have the SHA1's
at the tree-level and there we have everything underneath the
tree/blob SHA1 in our database. We will only know if the modules
symlink file is dangling next time we fetch from the submodule - here
we would notify the user but our database is still consistent.
If you have a source commit chain A => B => C => D, you want
to make any build commits totally independent: you first only
are interested in a build commit for source versions A and D,
and later find out that a build commit for B and C would be nice,
too. If you force build commits into some history order, this
order now would be A => D => B => C, which makes no sense.
It makes no sense because the user seem to have act irrationally. The
commit-chain is completely valid as it has tracked the correct history
of the builds. I can't see any problems here, the build-project is
independent of the source-project with it's own history. We can hope
the user has given good explanations for his/her actions in the commit
messages though.
Think of it this way: one common use for submodules is really to just
(occasionally) track somebody elses code. The submodule should be a
totally pristine copy from somebody else (ie it might be the "intel driver
for X.org" submodule, maintained within intel), and the supermodule just
refers to it indirectly (ie the supermodule might be the "Fedora Core X
group" which contains all the different drivers from different people).
Could you please be a little bit more specific about how you would store the
"pristine copy".
Note that it's not necessarily "pristine", since the submodule clearly is
a local git repository in its own right. So like _any_ git repository, you
can (and may well end up) having your own local branches in the submodule,
with your own local modifications.
So I'm not claiming that a submodule must always match some external git
tree 100%, and that it must be read-only or anything like that. I'm just
saying that I suspect that quite often, one of the MOST IMPORTANT parts is
that the submodule is really something that somebody else technically
maintains, and that this is actually one of the _reasons_ why it is a
submodule in the first place.
For example, a lot of projects end up having some kind of "library
component" as a submodule. Take something like a video player project,
which would have something like ffmpeg as a submodule, not because you'd
maintain ffmpeg yourself, but simply because (let's say) the library
interface changes enough, or you need a specific version with some of your
own fixes that haven't been released widely yet, so you want to carry all
the libraries you need _with_ you, even though you don't really maintain
that submodule. You at most have some small extensions of your own.
Now, in this situation, it's relaly really _important_ that the submodule
really is totally independent of the supermodule, for several reasons.
For example, since you don't "really" own that project, carrying around
your own fixes is really really painful. We know it happens all the time,
and a lot of projects end up needing their own version, but the _last_
thing you want is to be in merge hell all the time. So as a supermodule
maintainer, the best possible thing for you is to be able to push back
those local changes to the original project maintainer, so that you
_don't_ have to maintain your own changes.
True. But if you need the changes to the submodule for your supermodule
to function, and upstream either does not want to merge your changes or
the merge will be available only after a long time, then what is the
alternative? You must be able to keep local changes, and you must be
able to keep pulling from upstream. Of course, what you describe is the
ideal case: You find a bug, push the fix upstream, and in no time at all
your fix is merged and you can just pull a new version into your
superproject, but that might be wishful thinking.
But you need to realize that the real maintainer of the submodule is
TOTALLY UNINTERESTED in your supermodule. He's not going to maintain it,
and in fact, if you have anything in the submodule that ends up talking
about your supermodule, that's just going to make it a lot less likely
that the upstream maintainer will ever pull your changes. He might take a
diff from you, but in a perfect world, you'd actually be able to tell him:
"Hey, I've got a git repository with a few fixes to your ffmpeg git tree,
please pull from git://myhost.com/submodule.git to get these fixes:
... explanation of fixes and commits that are relevant to
ffmpeg, and have nothing to do with the supermodule, except
that you need those bug-fixes because you _use_ ffmpeg ...
Thanks"
See?
No! All you need is a naming scheme to address the commit of the
subproject that should be pulled. The extreme case would be to just
address it with its id (well, currently you cannot do that with git
pull, but that is fixable). But I already proposed a syntax for naming
commits which are "hidden" in a superproject: Just name the path as
described in git-rev-parse and append double slashes (to indicate that
you mean the commit, not the tree it contains). So no manual work needs
be done by upstream.
[snipped: about independence of submodule branches]
quoted
There seems to be some agreement to store the commit id of
the submodule instead of a plain tree id in the supermodules tree object, and
that all objects that are reachable from this commit are made part of the
supermodule repository (either fetched or via alternates). Do you agree?
Well, I would actually argue that you may often want to have a supermodule
and then at least have the _option_ to decide to not fetch all the
submodules.
[transmeta example snipped]
So no, I don't think the supermodule should even _force_ people to always
get all the submodules. It migth be the default case, but at the same
time, it's just being polite to let users decide on their own whether they
really want _all_ of the build infrastructure sources.
If you want to track some chosen submodules there are two easy solutions:
1. If you want to track their state as it appears from the supermodule's
view, pull from master:<submodule>//
2. If you want to track their state from their own development branches,
pull from <submodule>/master
Can you see the difference?
quoted
If I understand you correctly you cannot make any changes to the submodules
code _in the supermodule's repository_, no bugfixes, no extensions, no
adaptions, nothing. Do you mean that?
Yes. I think you should make all changes _within_ the submodule, because
the submodule should still be an independent git tree in its own right.
Every commit is a git tree in its own right, is it not?
[description of independent submodule development snipped]
And the importance of keeping the submodule independent is partly just
stability and sanity, but partly also scalability. For example, the
"index" in a supermodule should NOT include the indexes of all the
submodules. That's really important, because the index doesn't really
scale. Things do slow down with large indexes.
For example, git can handle tens of thousands of files easily. I suspect
it scales well to hundreds of thousands of filenames. But with
supermodules, you really can end up in the situation where you have _tens_
of these submodules, maybe even hundreds. And if you try to maintain one
unified index for the _whole_ thing, I guarantee you that you'll start
feeling the pain. Indexing millions of files is just not going to be
pretty.
I am not sure I understand what you say.
1. If you are working on a submodule, then the supermodule never enters
the picture. You are working independently. So far, so good.
2. If you are working on the supermodule, git will not be able to
function? How would you work without submodules, in which case you would
have simply one large project?
So just from a git stability and scalability point, it's important to keep
subprojects _separate_. There is obviously integration stuff, but they
should still be seen as truly independent projects. Even the supermodule
should have clearly its own life even _regardless_ of submodules, because
(as I said) quite often you may want the supermodule, but you don't want
to have _all_ of the submodules.
But it's more than that stability and scalability thing too - keeping them
separate is what allows you to do pulls and pushes on an individual
subproject basis, and have people really work at that level. For example,
if you're the compiler guy at a company, you really do want to work with
other compiler people _outside_ the company, but you sure as hell may not
be able to give them access to your supermodule. But you may want to work
on _just_ the compiler parts (or at least share some branches in public),
which means that the subproject really has to be able to work
_independently_ of the supermodule.
I totally agree. When I try to explain why submodules work that only
exist as part of one or more supermodules, I do not mean to say that you
cannot or should not have independent branches or repositories for the
submodules' code.
So "independent" here is really key, for several reasons. And that all
means, for example, that here must NEVER be any "backpointers". A
subproject really can _never_ have backpointers to the superproject,
because that fundamentally means that the above kind of "compiler guy
works on the compiler subproject in public" cannot work, if your
supermodule isn't public.
I took that for granted: from a commit you only ever look backwards (in
time/history dimension) or downwards (in content dimension).
Regards
From: Martin Waitz <hidden> Date: 2016-08-11 19:19:45
hoi :)
On Fri, Dec 01, 2006 at 02:51:49PM +0100, Stephan Feder wrote:
If you work in the supermodule, even if it is in the code of the
submodule, you only commit to the supermodule. The submodule does not
"know" about these changes after step 1.
I think we are using totally different definitions of "submodule".
For me a submodule is responsible for everything in or below a certain
directory. So by definition when you change something in this
directory, you have to change it in the submodule.
You can't change the submodule contents in the supermodule without also
changing the submodule.
This is just like you can't commit a change to a file without also
changing the file.
Then the supermodule just records the current content of the entire
tree. The only new thing is that instead of simple files there are now
submodules and that are also recorded.
Why do you mix up supermodule and submodule? The way I see your proposal
you cannot change submodule and supermodule independently. That is a
huge drawback.
No, this is the benefit you get by introducing submodules.
Why would you want to introduce a submodule when it is not linked to the
supermodule?
--
Martin Waitz
From: Daniel Barkalow <hidden> Date: 2016-08-11 19:20:11
On Sat, 25 Nov 2006, Linus Torvalds wrote:
On Sun, 26 Nov 2006, Yann Dirson wrote:
quoted
Also, I'd rather expect "git-commit -a" outside of any submodule to
commit everything in the supermodule, triggering submodule commits as an
intermediate step when needed - just like "git-commit -a" does not
require to manually specify subdirectories to inclue in the commit. I'd
rather expect a special flag to exclude submodules from a commit.
So, how do you do commit messages? It generally doesn't make sense to
share the same commit message for submodules - the sub-commits generally
do different things.
The same way you do the first commit message. Ask independantly for each
commit message in sequence with enough context in the comment section that
you know what you're talking about.
I'd actually suggest that "git commit -a" with non-clean submodules error
out for that reason, with something like
submodule 'src/xyzzy' is not up-to-date, please commit changes to
that first.
exactly because you really generally should consider the submodule commits
to be a separate phase.
I think this is getting close to the classic usability blunder of having
the program tell you what you should have done instead of what you did,
and then making you do it yourself, rather than just doing it.
Just have it run "git commit -a" in each dirty submodule recursively as
part of preparing the index, since that's what the user wants to do
anyway, and nothing already done would be affected.
"git commit -a -m <message>" should probably fail, of course.
-Daniel
From: Martin Waitz <hidden> Date: 2016-08-11 19:20:16
hoi :)
On Fri, Dec 01, 2006 at 01:03:48PM +0100, sf wrote:
Andreas Ericsson wrote:
2. You can never prune the main (the supermodule's) repository, at least
not with what git provides today.
It even already works (well, not with what git provides today, but with
my implementation). git-prune simply walks all the submodules, too, when
doing it's reachability analysis.
What does not work is a prune inside the submodule, because it does not
know about all the commits used by the supermodule.
--
Martin Waitz
Unfortunately, during development, you've switched libsubmodule1 to
branch "development", but supermodule isn't tracking libsubmodule1/HEAD
it's tracking libsubmodule1/master. Your supermodule commit doesn't
capture a snapshot of the tree you're using.
Or maybe not a merge, but worse, you'd *replace* the
previously committed master with what's in your dev branch.
Right, you would be replacing the prior branch of that submodule with
the new submodule branch.
I think the safety valve you are looking for here is two things:
* don't automatically update the submodule's HEAD into the
supermodule's index.
* make sure the submodule's HEAD is a fast-forward of the
supermodule's index, with a --force option to force it
anyway.
Otherwise the developer just has to know what he/she is doing.
Today you can put stuff that isn't ready for prime-time into a
repository on the wrong branch just by applying the wrong patch,
or cherry-picking the wrong commit, etc... the user can (and
will) make mistakes. But they can also easily recover from them
by rewinding history and redoing it.
On a related note, it would be great from a usability point of view if
there were a way to say "I always want to be on the same branch in all
submodules and the supermodule."
That's not really an issue.
A branch doesn't exist just because you checked-out the branch, or
because you created it. A branch exists because there were two or
more commits (B and C) which use the same parent (A) and two or more
of those commits survive, e.g. they have refs which point to them
(directly or indirectly) or they were merged into another commit
which itself survives.
Therefore if the supermodule is on the "development branch" the
submodules are also immediately on the same branch, because their
HEADs are derived from whatever is stored in the supermodule's tree.
And that tree is derived from whatever "development branch" means.
Really what you want/need is a special head in the submodule
which acts as the "branch that corresponds to the supermodule".
This probably should just be a naked SHA1 stored in HEAD, which
is committable only because a supermodule exists in a higher level
directory.
The fact that the submodule project has branches *at all* is
totally irrelevant once you start to speak about that submodule
within the supermodule, as its the supermodule which determines
the branch of the submodule.
But I think the Perforce-style
"compose a single workspace out of different bits of a larger project"
model is hugely useful
That's a mess.
You start to get into weird cases where the directory structure
expected by the build process is no longer intact, because the user
has sliced it apart in weird ways. And there's no single version
which corresponds to that workspace as (if I recall correctly)
you can pick different tags or branches at will. I believe that
ClearCase has the same bug.
You also can't version that now spliced workspace, aside from taking
the configuration file and putting that under version control too.
However I think the proposal on the table will support that to some
degree, in that you can take any version of any repository and embed
it at any directory of any other repository. This means you can
for example embed the Linux kernel, glibc and gcc projects into
a larger "embedded device" repository, but you cannot alter the
structure of any of those three projects without making your own
locally developed branch of them. Which is actually the correct
thing to do as any subslicing of a repository is exactly that:
a locally developed branch of that repository.
--
So yeah, it's a bit hacky, but for the reasons I've tried to outline, I
actually think that users _want_ hacky. Exactly because "deep integration"
ends up having so many _bad_ features, so it's better to have a thin and
simple layer that you can actually see past if you want to.
Thin and simple sounds very good. Let's try it with an example. Lets
say we have one apllication App1 and three librarys (Lib1, Lib2, Lib3)
with the following dependency-graph:
App1
/\
/ \
Lib1 Lib2
\ /
\ /
Lib3 (don't really needed for this example but looks nice)
All components can be used individually and have their own upstream,
maintainer etc.
To compile App1 however, I need some files from both Lib1 and Lib2
specifying it's API. To satisfy these dependencies, It sounds
reasonable to link Lib2 and Lib3 submodules from App1. In your
concept, can I construct a modules file to fetch the API files and
Here's an real-world example that doesn't contradict:
And I'll add the note that people who do things like submodules aren't
generally even _used_ to them being "seamless", and most of the time
probably don't even want complete seamlessness.
As the example that Torgil points to shows, people are quite used to
actually even naming the submodules separately, and things like having the
"default" set of submodules not equal the "complete" set.
In other words, I don't think people expect or want something hugely more
complicated than the CVS/modules kind of file.
What people _do_ want (and that CVS in general is horribly bad at, and
this is not a module-specific issue) is to have the _versioning_ work
well. When you check out a specific version of a module, you want any
_linked_ modules to follow along too.
This is the same reason why CVS users use tags a lot: because even
_within_ a single project (no modules, no nothing), it's often hard to
re-create the exact state of a version any other way. So you tag every
single file and do insane things like that, because CVS just isn't very
good at guaranteeing consistency across the whole project.
The exact same thing is true about subprojects. I don't think that people
who have used CVS subprojects a lot really mind the CVS/modules file
itself (but hey, maybe I'm wrong - I've seen _other_ people maintain
modules in CVS, but I've never done it myself), but they do mind the fact
that it's hard as hell to do something as simple as "get all modules back
to version X" without lots and lots of careful crud (ie tagging every
singl emodule, things like that).
Now, I'm not exactly sure who wants to use git modules, so this is the
time to ask: did you hate the CVS/modules file? Or was it something you
set up once, and then basically forgot about? People clearly use the
ability to mark certain modules as depending on each other, and aliases to
say "if you ask for this module, you actually get a set of _these_
modules".
_I_ suspect that that isn't the problem people had, and isn't what they
have any problems with. What CVS didn't do very well (or at all, afaik) is
to say "I want supermodule version XYZ", and then got all the submodules
automatically to that (reliable) state. And THAT is something I think is
really important for submodules, and it's why I think the most important
part isn't actually all the veneer to make "git clone" and "git pull" work
(which is really about the CVS/modules kind of wrapper parsing), but
actually about the supermodule "tree" object pointing to a very specific
version, so that you get the exact same "atomic snapshotting" of multiple
trees that you get within a single git tree.
In other words, I _suspect_ that that is really what module users are all
about. They want the ability to specify an arbitrary collection of these
atomic snapshots (for releases etc), and just want a way to copy and move
those things around, and are less interested in making everything else
very seamless (because most people are happy to do the actual
_development_ entirely within the submodules, so the "development" part
is actually not that important for the supermodule, the supermodule is
mostly for aggregation and snapshots, and tying different versions of
different submodules together).
So that's where I come from. And maybe I'm totally wrong. I'd like to hear
what people who actually _use_ submodules think.
Put it another way, I see the proposal of allowing tree entries to be
commits in addition to trees and blobs, akin to recording the submodule
_history_ inside the _tree_, which I feel precisely violates the
distinction you want to keep between those 2 concepts.
No. Linus is right. Submodule commits belong in the tree.
We want to record a specific subtree within a larger tree. There are
three ways we can refer to a tree: by its tree SHA1, by a commit
which points at the tree SHA1, or by a tag which points at a
commit which points at the tree SHA1, or by a tag which points
at a tag which points at a commit which points at a tree SHA1.
Which is basically a tree-ish.
The advantage of linking to the commit-ish (commit or tag) and
not the tree-ish for a submodule is that it also provides you quick
access to answer the "how did this tree arive at this state" question
as the answer cannot come solely from the top level commit chain.
The reason... keep reading...
What about the case of a subproject that would evolve fast, and for
which we may not want intermediate versions to be part of the
supermodule ? (just exploring an idea without real connection to the
one discussed above)
Right. The submodule is free to be committed to an infinite number
of times for any given commit in the supermodule.
It is expected that users will commit to a submodule say hundreds of
times for every commit they make to the supermodule. Or thousands.
This is especially true if the submodule is some very large project,
e.g. the Linux kernel, and the supermodule "upgrades" the kernel it
is using after 3 months of staying on the same version. Suddenly the
supermodule has only 1 commit which covers maybe 10,000 commits in
the submodule.
Yet we still want to be able to efficiently perform operations like
"git bisect" within the scope of that submodule, to help narrow down
a particular bug that is within that submodule. To do that we need
the commit chain (all 10,000 of those commits) in the submodule.
To get those we really need a commit-ish and not a tree-ish, as
going from a tree-ish to a commit-ish is not only not unique but
is also pretty infeasible to do (you need to scan *every* commit).
--
From: Michael K. Edwards <hidden> Date: 2016-08-11 19:22:45
(I wrote most of this a couple of days ago, so it's not at the tip of
the conversational tree, so to speak. But it's effectively a response
to Linus's "what do you want to do with submodules" question, with
some thoughts on implementation. Sorry it's so long; like Blaise
Pascal, "I would have written a shorter letter, but I did not have the
time.")
The supermodule concept, implemented right, could really improve
cooperation among embedded platform integrators, boutique distro
publishers, and other editorial contributors to sprawling metaprojects
who don't want to run kernel.org-scale mirrors. To make this work,
you need sparse repositories (conserving resources when fetching, by
omitting the bulk of currently un-needed submodules that can reliably
be obtained later from elsewhere) and shallow cloning (conserving
resources when publishing, by referring cloners to a third-party
repository for universally available content).
For instance, it would be a wonderful thing if the pile-o-patches
nightmare that is PTXdist (and crosstool and buildtool and every other
approach I have seen for ongoing maintenance of embedded toolchains
and userlands) were obsoleted by a git supermodule. Its submodules
would mostly track external projects, but would also logically contain
the fix-up patches worked out during platform integration, checked in
to branches anchored at each upstream release point. The supermodule
would contain all of the build automation, log auditing, and remote
unit testing stuff, as well as the metadata for each submodule
involved in this platform build cycle.
At a content level, the sparsely populated / shallowly published
supermodule wouldn't be much different from today's PTXdist. But the
pay-off comes when you merge forward to a new release of some base
component (compiler, library, etc.) and discover that some of your
fix-ups have been adopted or obsoleted upstream, and new fix-ups are
needed for components that depend on the updated bit, and the set of
configurables has changed (for which you need to compensate in the
meta-configurator). Instead of piling up versioned patch directories,
you commit fix-ups to the sub-modules, which other integration
branches can ignore (if they aren't affected), merge, or cherry-pick.
As I understand it, in today's git, every content object is a patch to
the _data_ of one and only one git repository, containing the label of
the preceding _data_ state plus a diff of file contents and
attributes. Assuming this model is retained, any clean state of a
"leaf" module (one with no submodules) can be reached by replaying a
series of patches, starting from the repository's root node (an empty
directory with the hopefully unique label generated by init-db). The
label (SHA1) of the last patch is therefore a perfectly good label for
this _data_ state.
If all we were trying to do with supermodules was to capture and track
various states of the submodules' data, we could extend the format of
content objects to include "state X of submodule with init-db label
Y". That would have the effect of capturing submodule states as
_data_ in non-"leaf" modules. We would have to help cloners find a
place from which to pull these states, of course; and it's easy to get
sidetracked onto that part of the problem. But that's not where the
bang for the buck is in supermodules.
The whole model of distributed supermodules, with references to
slightly diverging submodules whose content should mostly be fetched
from external sources, smells to me just like LVM. The external
sources (like an LVM volume of which you have taken a "snapshot") make
up the bulk of the content pool. They also give you a window into
developments on the submodule's own branches (like being able to peek
forward and merge changes from the original volume). The supermodule
(the snapshot volume) provides most of the interesting refs (submodule
commits referenced by supermodule tags and branch heads), along with
enough "journaled" content to replay forward from some checkpoint
guaranteed to be available in each external source to any of these
refs.
The implication here is that submodule states are not just SHA1 labels
to be embedded within supermodule data diffs. One ought to be able to
clone a supermodule without immediately cloning full copies of any of
its submodules. This ought to populate the clone's content database
with all of the quanta of submodule content that aren't guaranteed to
be available from any not-too-stale submodule mirror. When cloning,
you don't want to have to inspect every supermodule state for
submodule states that are outside the global subset. So the
supermodule needs to maintain a set of supplemental refs from which
all referenced submodule states can be reached. This allows you to
traverse the portion of the pool of submodule content that can't be
reached from true submodule branch heads.
On 12/1/06, Linus Torvalds [off-list ref] wrote:
Yes, you do need to have a list of submodules somewhere, and you'd need to
maintain that separately. One of the results of having the submodules be
independent from the supermodule is that it's not all "automatically
integrated", and thus the supermodule does end up having to have things
like that maintained separately.
This is not a defect; it's a virtue. It's important for every commit
to the supermodule to contain the information of which submodule
branches you're currently on and how far along them you've crawled.
Any particular supermodule commit point is likely to reflect an
integration milestone visible only to the person working at the
supermodule level. No content object should ever cross a submodule
boundary, because then you wouldn't be able to apply it to the
submodule in isolation (or in another supermodule state) or identify
it when it is applied upstream and propagates back to you in a pull.
But the supermodule can also contain supplemental refs (heads and
tags) that don't exist in the submodule (and shouldn't necessarily be
pushed to it); the commits they refer to are localized to the
submodule but may not be reachable from any of the submodule's branch
heads.
And yes, if you screw that up, you wouldn't be able to fetch submodules
properly etc, even if you see the supermodule, and yes, this sounds more
like the CVS "Entries" kind of file that is more "tacked on" than really
deeply integrated. But I think the separation is _more_ than worth the
fact that you can see things being separate.
There is an opportunity for useful deep integration here. The same
algorithm that does reachability analysis for "git prune" can dig from
supermodule down to submodules, copying objects into the supermodule
database until it hits a commit that is advertised as "global" by the
submodule. "git clone" of the supermodule can then pull the bulk of
the submodules (a superset of the "global" subset) from (a mirror of)
the canonical place for each, and use the supermodule object database
as an alternate source for commits that don't exist in the "canonical"
submodule.
In fact, I'm very much arguing for keeping things as separate as possible,
while just integrating to the smallest possible degree (just _barely_
enough that you can do things like "git clone" and it will fetch multiple
repositories and put them all in the right places, and "git diff" and
friends will do reasonably sane things).
Keep it simple, stupid.
As simple as possible; but no simpler. The "alternates" / "git clone
--reference" model is already almost powerful enough for the
supermodule to contain a "journal" of submodule commits that haven't
yet been retired to the canonical subset (guaranteed present in each
mirror). The only difference is that the supermodule should be
considered a "weak alternates" source. Commit objects in the
supermodule's database should be visible to submodule-level operations
(so that commits which are accepted upstream get flowed in nicely
during "git pull").
But if a commit becomes reachable from a ref that is really in the
submodule (not just one of the supermodule's "supplemental refs",
which should _not_ be visible to submodule operations), then it should
be copied into the submodule's object database. (The refs internal to
the submodule should retain their integrity even if the supermodule is
inaccessible.) The existing "strong alternates" mechanism should be
reserved for repos which are at least as public and persistent as the
submodule, and supermodules don't qualify (e. g., Linus's transmeta
scenario).
On Sat, 2 Dec 2006, Josef Weidendorfer wrote:
quoted
The thing I wanted to discuss is whether such names would need to be globally
unique in the project containing submodles, or not.
My preference would be for it to be "local", just because (as I
mentioned), with mirroring etc, it might well be that you want to fetch
things from the _closest_ repository. That's really not a global decision,
it's a local one.
I think "global resource, local provider" is the way to go, with each
provider advertising what checkpoints of what resources it can supply.
When I clone or pull, I should be able to consult a local mapping of
submodule URIs to "mirrors" (which may well be local repositories
containing content and branches that aren't in the "official"
upstream). The only thing that may need "global" agreement is the
boundaries of the "global" subset for each submodule, i. e., the set
of commit objects that can reliably be obtained from any mirror of the
"official" upstream repository. That doesn't need to be terribly
clever; "at least three days old on a globally published branch" would
probably be a perfectly good heuristic.
quoted
If yes, it IMHO makes a lot of sense to introduce "submodule objects" which contain
these submodule names, and which are used as pointers to submodule commits in
supermodule trees.
You could do it that way, and then it would be global. It would work, and
in many ways it would probably be "simpler" on a supermodule level.
I think the implication of "submodule objects" is that supermodule
diffs would say "roll submodule X from commit-id A to commit-id B". I
don't think that would work very well for pulls/merges in the sparsely
populated scenario, because you want to be able to pull the
non-canonical subset of the individual diffs between states A and B
into the supermodule's object pool. When you decide later to flesh
out submodule X, you should only have to clone some canonical mirror
and then fast-forward to state B using objects you already have in the
supermodule pool.
The merge case is even clearer. Suppose I pull updates from two
remote branches of the supermodule onto my master branch. Each remote
branch has added the same submodule, cloned from third-party
repositories whose clone history goes back to the same origin. (The
example I have in mind is when some project switches to git from some
other SCM, and the maintainers of the remote branches port their
integration patches over from their git-svn tracker submodule to a
clone of upstream's new git repo.) I should be able to postpone the
merge effort, come back later and clone the upstream repo, then merge
the non-canonical commits that were pulled earlier.
I might want to decide at supermodule pull time to postpone pulling
the bodies of the submodule commits; but I want the full sequence of
submodule commit IDs in the supermodule commit object. So it's not so
much the supermodule _state_ that has a hierarchical structure; it's
the supermodule _diffs_ and _object_pool_ that become hierarchical.
The advantage of a global namespace is that you can much more easily
update it - "git fetch" will just fetch the new file(s) that describe the
subprojects very naturally if they are all global. Putting them in a local
.git/config file has it's advantages (see above), but it also makes it
very hard to version them, and to update the list - it would have to
become manual.
I think the only global-to-local-namespace mapping applies to the
different labels for the "empty repository" state generated at init-db
time. Given the init-db SHA1 of the linux kernel repository, I should
be able to choose any mirror or clone of that repository as a source
for objects in its "global set". I expect this provider not to
scribble on globally published branches, but that isn't even all that
critical; anything outside the canonical set is kept in the
supermodule's object pool, so I can always blow the submodule away and
regenerate it from a different mirror.
There are possibly combinations of the two approaches: have a "global
namespace" that describes the canonical place to get the subprojects, but
have some way to add local "translation" of the canonical names into
locally preferred versions (eg you could just have a way to say "this is
the local mirror for that global canonical place")
Maybe that would work?
Sure. But all you really need from the canonical place is its init-db
SHA1 (permanent) and its list of globally published branches
(monotonically expanding). A URL for it is a convenient shorthand but
doesn't have to be persistent.
Cheers,
What about my other argument for a submodule namespace:
You want to be able to move the relative root path of a submodule
inside of your supermodule, but yet want to have a unique name
for the submodule:
- to be able to just clone a submodule without having to know
the current position in HEAD
Umm? I don't get the issue. A submodule is a git repo in its own right,
and you clone it exactly like you'd clone any other repo. It _does_ have a
HEAD. It has it's own branches. It has everything.
So when you clone a submodule, you always get all those branches. The
supermodule will not _point_ to them all (the branches are local to the
submodule, and _will_ depend on things like "which upstreams module am I
tracking"), but they'll have to be there, exactly _because_ the submodule
has an existence and is tracked on its own.
In the trivial case where the submodule doesn't even _have_ any external
existence at all (ie it's always maintained as _just_ a submodule, it
would probably tend to have just one branch, and a clone would get
whatever that branch is), but that's just a degenerate special case of the
much richer "this submodule actually has a life of its own" case.
- more practically, e.g. to be able to name a submodule
independent from any current commit you are on in the supermodule,
e.g. to be able to store some meta information about a submodule:
The current commit within the supermodule would be _totally_ invisible to
the submodule.
Of course, if HEAD _differs_ from that commit within the supermodule, then
a "git diff" (when done from within the supermodule) should show that, but
again, that's actually only as seen from the _supermodule_.
- "Where is the official upstream of this submodule?"
That's entirely a question for the submodule. You cannot ask that question
within the confines of the supermodule, because it's not even a relevant
question in that context. Two different supermodule repositories may well
decide to get their submodules from difference places, just because they
got cloned from different places (or even just for practical reasons like
"that other site is closer to me").
So the official upstream of a submodule must NOT be encoded inside the
supermodule (or at least not within its _objects_). Exactly because the
upstream location is not a "global" thing - it's per-repository, and thus
must not be encoded in the global data (ie the objects).
It should be be encoded in some _ephemeral_ place, eg in the ".git/config"
file or in a ".git/remotes/origin"-like file (either in the supermodule or
the submodule, and I would seriously suggest you do it within in the
submodule itself, because you'll want it exactly when you decide to work
on the submodule and upgrade _that_).
- "Should git allow to commit rewind actions of this submodule
in the supermodule?" (which, AFAICS, exactly has the same
problems as publishing a rewound branch: you will get into
merge hell when you want to pull upstream changes into the
supermodule)
The only thing that a submodule must NOT be allowed to do on its own is
pruning (and it's distant cousin "git repack -d"). You must always prune
from the supermodule, because the submodule cannot really know on its own
what references point into it.
(There are alternatives. One alternative is to never allow rewinding - or
deletion - of branches in a submodule, and thus solve the problem that
way. That is the easier solution, because it also means that a "clone" of
a supermodule can just recursively clone the submodules independently
_without_ having to worry about reachability, but it's really _really_
draconian).
- "Should this submodule be checked out?"
This, I think, requires too much configuration to say separately for every
possible submodule, so I would suggest that the way to make that decision
is:
- "git clone" by default will fetch and check out all submodules (and
obviously they have to be described some way outside of the object
database, just so that you don't have to parse the _whole_ history of
the _whole_ supermodule just to find all possible submodules. So the
supermodule _will_ need some "list of submodules and where to get them"
in a config file or other).
- add a flag (possibly just re-use the current "-n" flag) that disables
that recursive fetching of submodules entirely.
- have a way to fetch individual submodules one-by-one (that capacity
obviously has to be there anyway, since the "recursive" git clone has
to be able to do it, so this is likely just "git clone" again, with
just logic added to say "when you clone something and are _already_
within a superproject, the clonee becomes a subproject automatically"
I dunno. And I'd also like to point out that things don't have to all work
fully before we can do at least some cases of this. For example, if the
initial version just always clones everything, big deal. I'm not saying
that we have to have support for things like this on "Day 1", I'm just
saying that I think people will want to be able to not fetch and check out
everything, so the design should _allow_ for it.
(But I also think that as long as submodules are independent enough, the
"design" part should fall out on its own, and it just becomes a "small
matter of programming" to actually get it to work).
From: Sven Verdoolaege <hidden> Date: 2016-08-11 19:24:39
FWIW, here's my view on this issue.
On Sun, Nov 26, 2006 at 12:49:08AM +0100, Yann Dirson wrote:
Also, I'd rather expect "git-commit -a" outside of any submodule to
commit everything in the supermodule, triggering submodule commits as an
intermediate step when needed - just like "git-commit -a" does not
require to manually specify subdirectories to inclue in the commit. I'd
rather expect a special flag to exclude submodules from a commit.
A commit should record the content changes that have been made, not change
any content itself. Some VCSs change the contents of a file when you
commit them (e.g., keyword substitution). Git, rightly, doesn't do that.
Likewise, when you commit in the superproject, it should simply record
the changes to the "content" of the subproject and not change it.
And the content of the subproject is a commit, so a commit in the
superproject should not change the content of the subproject by creating
another commit in the subproject.
From: Andreas Ericsson <hidden> Date: 2016-08-11 19:24:43
sf wrote:
That is one of the points Martin Waitz and I are discussing.
If I understand you correctly you cannot make any changes to the
submodules code _in the supermodule's repository_, no bugfixes, no
extensions, no adaptions, nothing. Do you mean that?
That would be a third alternative. In my opinion the usefulness of
submodules would be unnecessarily restricted if it comes to the choice
of either using the code from upstream as is or do not use submodules at
all. What is the point of the restriction?
That depends on your definition of submodule. In my eyes, a submodule is
a separate repo that can be committed to separately (and generally also
built separately), although it's usually built into something else. I'm
imagining most submodules will contain only library code and its testing
routines.
Insofar as I've envisioned submodules, it's a separate git repo where
you simply record a certain snapshot of the sub-repo with a commit in
the super-module, like so:
$ git commit ssl-functions/*.[ch] openssl -m "Upgraded openssl with
necessary changes to core code"
(yes, I know it's horrid to use -m to commit, and I daily advocate
against it where I work, but you get the idea, I'm sure)
Isn't this how it's supposed to work? Enlighten me, and please remember
that I'm drunk atm, so make it obvious ;-)
--
Andreas Ericsson andreas.ericsson@op5.se
OP5 AB www.op5.se
Worse, if you allow that to happen, the supermodule can commit a state that
cannot be retrieved from the submodule's repository. The ONLY thing a
supermodule can record about a submodule is a commit.
So what? You have a submodule commit that only exists in the
supermodule. I fail to see the problem. The changes you made to the
submodule _in the supermodule_ can later be pulled from wherever you want.
Regards
Stephan
Well, I just rather have than <sha1 of tree/blob> the definition
of sparse checkout (for example subdirectory name, or file name,
or glob pattern).
This is entirely an UI issue:
On 12/16/06, Torgil Svensson [off-list ref] wrote:
is based around Linus 'module'-file and keep things simple. A git
configuration file that specifies:
* link name for reference
* local path to link
* submodule source
* submodule path to tree/blob
* submodule commit / HEAD / branch
* options (depth-limit , ...)
On 12/16/06, Jakub Narebski [off-list ref] wrote:
And if you have that, you don't need <sha1 of tree/blob> in repository, in link object.
Correct. Since the commit contains all the version information, the
following combinations should give the same information iff we keep
the commit in the database:
1. <sha1 of commit> + <sha1 of tree/blob>
2. <sha1 of commit> + <symlink to tree/blob>
I used the sha1 because I wanted them to behave exactly like
trees/blobs in the database for operations that can disregard the
commit info. Now, if we keep the commit in the database as Linus
suggests we can reach the target from there with a symlink. This would
be more readable but also cost a few object lookups extra iterating
over the symlink.
With sparse (for example defined by 'src/*.h') or partial (for example
defined by 'Documentation/') checkout you should be able to merge
upstream... unless conflicts are in the not checked out part.
This would be a great feature! Will this conflict with path shortcuts?
If so, we might consider two types of objects: "link" which cannot
merge upstream and "module" which can merge upstream and contains a
.git object repository.
IMHO, "module" is a more intuitive name for specifying a
(functionality wise fully fledged) submodule with a repository inside.
"link" could be used for just mirroring a tree/blob. I'm not sure if a
separation is needed on a technical level.
From: Daniel Barkalow <hidden> Date: 2016-08-11 19:25:23
On Tue, 28 Nov 2006, Andreas Ericsson wrote:
Daniel Barkalow wrote:
quoted
On Sat, 25 Nov 2006, Linus Torvalds wrote:
quoted
I'd actually suggest that "git commit -a" with non-clean submodules error
out for that reason
Just have it run "git commit -a" in each dirty submodule recursively as part
of preparing the index, since that's what the user wants to do anyway, and
nothing already done would be affected.
Running "commit -a" is definitely the wrong thing to do, as it prevents one
from using the index at all. Erroring out if the submodules are dirty, or just
accepting the fact that they are and taking whatever commit HEAD points to is
*always* preferrable.
I don't think anyone would actually use the index in submodules but not in
the supermodule. If submodules are seen mostly as ordinary directories as
far as the supermodule's working directory is concerned, it wouldn't make
sense to not commit dirty state in a subdirectory with -a just because
it's a submodule.
It would be wrong to do "commit -a" in submodules if the supermodule
weren't being committed with -a, of course.
quoted
"git commit -a -m <message>" should probably fail, of course.
Why? There's no reason to rob this command of its power just because we're
using submodules.
It should fail if there are dirty submodules, because the user needs to
provide a commit message for each of them, and only one commit message can
be provided this way, and -m inhibits invoking an editor.
-Daniel
Thinking about this...
You have to make very sure to always update the caching layer containing
the backlinks on every addition of a further object. You can do this
because you always reached this new object by some other object, which
exactly is the backpointer.
You're missing the big issue.
The issue is that a cache like that would ABSOLUTELY SUCK.
You could speed up the non-common operations with it, but:
- any changes would become a LOT more expensive to do, because they all
need to update every single object they add (ie a "commit" would now
have to add backpointers TO EVERY SINGLE BLOB).
Imagine what this does to something like the kernel, where a commit
reaches 22,000 files!
You can do it at a finer granularity (ie do just the direct backlinks
and only do the "tree->blob" and "tree->tree" things rather than the
full commit reachability, but it's still going to be MUCH more painful
than what we do now.
- the cache would be a lot bigger than the current pack-files, and it
would be fragile as hell to boot. Because it needs to get rewritten for
every operation, it gets corrupted much more easily, and that's
ignoring things like race conditions, so it would now need a ton of
locking that git simply doesn't do at all.
- everything would basically slow down.
- you couldn't do shared object databases AT ALL, because backpointers
wouldn't work. The whole _reason_ you can share object databases is the
same reason we can't have backpointers: objects are immutable and never
change depending on circustances.
The _only_ downside of the current situation is literally the 24 or 28
bytes per object that we look at. For most operations, we don't even look
at that many objects, so it's really the worst-case things.
In fact, this "cache" can be created with a usual object traversal
(which has the original memory requirement), but as long as we do
not add objects to the database, further traversals would only need
a fraction of memory.
Right. If the project is totally read-only, the cache would work well.
For real development, it would SUCK. It would make things like "git reset"
very expensive indeed, for example (you'd have to unwind the whole cache:
either regenerating it - which would take minutes - or being very careful
indeed and being able to always remove objects properly and keeping track
of them 100%).
IOW, it's nasty nasty nasty. And it doesn't really even help anything but
a case that we actually already handle really well (I spent a lot of
effort on making the memory footprint minimal).
But it does mean that you do NOT want to traverse a hundred different
project "as if" they were one. That's really the only thing it means.
And since you can do submodules as independent projects, and you SHOULD do
them that way for tons of other reasons _anyway_, even that isn't a reason
to screw up all the _wonderful_ properties of the git object database.
So what I'm trying to say is that the immutable non-backpointer nature of
the git database is what makes it so WONDERFUL. It's efficient, it's
dense, it's stable, and it allows us all the clever things we do. But it
means that we do end up alway spending 28 bytes per object, and we can
never throw those 28 bytes away during a single "traversal" run.
From: Martin Waitz <hidden> Date: 2016-08-11 19:26:23
hoi :)
On Fri, Dec 01, 2006 at 08:49:20AM -0800, Linus Torvalds wrote:
Think of it this way: one common use for submodules is really to just
(occasionally) track somebody elses code. The submodule should be a
totally pristine copy from somebody else (ie it might be the "intel driver
for X.org" submodule, maintained within intel), and the supermodule just
refers to it indirectly (ie the supermodule might be the "Fedora Core X
group" which contains all the different drivers from different people).
Yes, but it is not only about tracking, also about distributing
submodules.
One Fedora X developer fixes a bug in the intel driver, commits that to
the submodule and then updates the supermodule to the new version (by
calling "git-update-index drivers/intel && git-commit" or something). Then
another Feora X developer updates his X repository. By pulling the
supermodule he also gets a new version of the submodule.
And this new version of the submodule is stored in a branch which can be
accessed by the submodule.
A supermodule can never "contain changes" to a submodule.
The supermodule always contains _the_entire_ submodule with its complete
history, so it also does contain changes. But it does not per-se
contain changes, only indirectly (i.e. the commits in the submodule are
not part of the supermodule commit chain).
A supermodule would always just point to the submodule, and not have
any changes what-so-ever of its own. The submodule is self-sufficient,
and always contains all its _own_ changes.
From: Steven Grimm <hidden> Date: 2016-08-11 19:26:47
Linus Torvalds wrote:
So I actually think that submodules should at least start out as something
rather independent, where a "commit -a" in the supermodule will _only_
commit the supermodule itself - and if you haven't committed the submodule
yet, you'll just get the current HEAD state of the submodule.
That would make it impossible to atomically commit a change that affects
two submodules, yes? I think cross-submodule commit is highly desirable
and will be a fairly common use case for submodules if it's supported.
For example, if you have "client" and "server" submodules and someone
makes a protocol change, you don't want some unwitting developer to pull
just half of the change and end up with incompatible code in the two
submodules.
I have no problem with making the "only commit the supermodule" behavior
the default and requiring a command-line option for the "commit
everything" case, but I think "commit everything" is useful. And
honestly IMO it should be the default since it'll behave in a less
surprising way; when I do a "commit -a" I expect all my changes to be
committed, whether they're in submodules or not.
From: Martin Waitz <hidden> Date: 2016-08-11 19:27:49
hoi :)
On Sat, Dec 02, 2006 at 12:44:20PM -0800, Linus Torvalds wrote:
And watch the memory usage.
hmm, really sad, it was such a nice concept until now...
You are right, I have to think more about scalability. O(N) anywhere is
really bad for submodules. They really should be able to bundle the kernel,
mozilla, qt and whatnot into one project and that will get huge.
--
Martin Waitz
From: Jakub Narebski <hidden> Date: 2016-08-11 19:28:44
Andreas Ericsson wrote:
Indeed. With the "tight" integration option we'd also have to have the
mechanism to rewrite the tree-entries with the location where the
submodule is located in the working tree. This might be needed anyways,
but it sure as hell seems a lot easier to just tack that part on when
doing a checkout and actually creating all the files.
Excellent idea! This way most of the concerns for "separate repositories for
submodules" layout about ability to rename directory the submodule resides
in, or move submodule are resolved. The other part would be to use
submodule-aware git-mv to move submodule(s).
Perhaps the following solution would work best:
* refs/submodules/<module> holds sha1 of top commit in submodule
* objects/info/submodules is a file which can be automatically generated
(or at least automatically updated) on checkout, with the following
contents:
<module> TAB or SPC <path to submodule, or GIT_DIR of submodule, or
GIT_OBJECT_DIRECTORY of submodule>
with the usual rule that # and ; means comment, \ at end of line is used
for continuations, empty lines doesn't matter etc.
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
On 12/16/06, Torgil Svensson [off-list ref] wrote:
I'm very open to suggestions here, but the concept growing in my head
is based around Linus 'module'-file and keep things simple. A git
configuration file that specifies:
* link name for reference
* local path to link
* submodule source
* submodule path to tree/blob
* submodule commit / HEAD / branch
* options (depth-limit , ...)
I'm reconsidering having the path-name in the link, it should be
sufficient to have two SHA1's, one for the commit and one for the
tree/blob. Super-module should have the tree/blob in it's database so
that the link part only is there for version information and reference
(checking dirty state or history on the submodule). This way it easy
to clone the super-project and use it without having to map up all
sub-project sources. Sub-project sources is not important for version
information and could always be specified in the project in a
README-type of file.
See it as the link only is there for the version handling between
different modules and it's the module file that give an UI to the the
link (which project, branch, ....). Many users will not care whats
behind those links, but if they want to edit the link they have to
create the modules file or fetch it somewhere - it may even be
provided and version controlled in the project itself.
example tree object:
100644 blob <sha1 of blob> README
100644 blob <sha1 of blob> REPORTING-BUGS
100644 link <sha1 of blob> <sha1 of commit>
040000 tree <sha1 of tree> arch
040000 tree <sha1 of tree> block
040000 link <sha1 of tree> <sha1 of commit>
Note that the links functions exactly as the blobs and trees in the
database. The difference is that they origin from _a_subproject_ (we
don't care which in this stage) with the specified commit SHA1. If the
link isn't represented in the modules file, it's no big deal, it can
be added later on if needed.
If the blame-game begins or if we want to check what we're using on a
submodule level we can always pinpoint the exact file/tree content and
From: Josef Weidendorfer <hidden> Date: 2016-08-11 19:30:09
On Saturday 02 December 2006 11:04, Andy Parkins wrote:
quoted
So what do you do with deleted submodules?
You wouldn't want them to still sit around in your working directory,
but you still have to preserve them.
Now that is a tricky one. Mind you, I think that problem exists for any
implementation. I haven't got a good answer for that.
That suggests that it is probably better to separate submodule repositories
from their checked out working trees. Why not put the GITDIRs of the submodules
in subdirectories of the supermodules GITDIR instead?
From: Martin Waitz <hidden> Date: 2016-08-11 19:31:27
hoi :)
On Sat, Dec 02, 2006 at 12:46:01PM -0800, Linus Torvalds wrote:
On Sat, 2 Dec 2006, Martin Waitz wrote:
quoted
What makes a submodule so special that now we suddenly have to store
those stuff in the object database?
I'm not sure it is. I suspect a pure commit link with just a CVS-style
"modules" file is sufficient. I'm just saying that I don't think it is
_wrong_ to possibly want to expand it.
If we later see that we really want to have it we can always introduce
it later. I don't think we should do it now if we don't see clear
benefits _now_.
So I was not against the link object itself (initially I wanted to do it
this way, too), only agains the information which was proposed to be
stored there. Up to now I haven't found anything which makes sense to
store next to the submodule commit to define the identity of the
submodule.
--
Martin Waitz
What makes a submodule so special that now we suddenly have to store
those stuff in the object database?
I'm not sure it is. I suspect a pure commit link with just a CVS-style
"modules" file is sufficient. I'm just saying that I don't think it is
_wrong_ to possibly want to expand it.
From: Daniel Barkalow <hidden> Date: 2016-08-11 19:32:50
On Tue, 28 Nov 2006, Sven Verdoolaege wrote:
On Tue, Nov 28, 2006 at 12:28:47PM -0500, Daniel Barkalow wrote:
quoted
It would be wrong to do "commit -a" in submodules if the supermodule
weren't being committed with -a, of course.
What if you say "git commit submodule" ?
Obviously no -a, as I said.
If you agree with the above, then why should "git commit -a"
do any different from "git commit submodule" if submodule was
the only thing that got changed ?
If submodule was the only thing that got changed, it's not dirty; if it
were dirty, some of its contents would also have gotten changed. Surely:
"git commit submodule/foo bar"
should do "git commit foo" in submodule, and then commit the supermodule
with the new commit for the submodule and the change to bar. And so
"submodule/foo" is something you could commit changes to, so it should get
picked up by -a.
Of course, if submodule *is* the *only* thing that changed (e.g., you did
a fast-forward merge in it, or you've previously committed it completely),
there won't be a "commit -a" in it, because that would just generate a
gratuitous commit.
-Daniel
From: Sam Vilain <hidden> Date: 2016-08-11 19:32:55
Michael K. Edwards wrote:
who don't want to run kernel.org-scale mirrors. To make this work,
you need sparse repositories (conserving resources when fetching, by
omitting the bulk of currently un-needed submodules that can reliably
be obtained later from elsewhere) and shallow cloning (conserving
resources when publishing, by referring cloners to a third-party
repository for universally available content).
Did you see GitTorrent? http://gittorrent.utsl.gen.nz/ A lot of
similar ideas to what you mention. Sorry, still no prototype :)
I'd see the submodules thing as a good way to glue together a whole
bunch of repositories, so that the core mirror servers only have to
mirror a small-ish number of repositories.
Sam.
On Sat, Nov 25, 2006 at 01:53:38AM -0500, Shawn Pearce wrote:
Yann Dirson [off-list ref] wrote:
quoted
We don't need to have commits in the tree for this. We'll just have
submodule commits which are not attached to a supermodule commit, and we
can access the whole submodule history through the submodule .git/HEAD,
just like we do for a standard git project.
No. You cannot do that.
How do we setup .git/HEAD when bisecting the supermodule?
Or merging it? Or doing anything else with it?
Would there be any problem assuming git-update-ref would take care of
updating it ?
Ideally the .git/HEAD of every submodule should seek to the commit
that points at the tree of the submodule which the supermodule
is referencing.
You mean, whenever we seek the HEAD of the supermodule, right ?
This lets you then perform a bisect within the
submodule when you identify the supermodule commit which caused
the breakage.
That is, first bisect the supermodule (which naturally bisects the
submodule with rough granularity, assuming there are many submodule
commits for at least some supermodule commits), then bisect the submodule
between the two commits identified at supermodule level, right ?
We need the submodule commits to do this. Doing it without is
too expensive.
Maybe I missed something again, but I'm still not convinced :)
--
From: R. Steve McKown <hidden> Date: 2016-08-11 19:33:52
On Saturday 02 December 2006 12:41 pm, Linus Torvalds wrote:
In other words, I _suspect_ that that is really what module users are all
about. They want the ability to specify an arbitrary collection of these
atomic snapshots (for releases etc), and just want a way to copy and move
those things around, and are less interested in making everything else
very seamless (because most people are happy to do the actual
_development_ entirely within the submodules, so the "development" part
is actually not that important for the supermodule, the supermodule is
mostly for aggregation and snapshots, and tying different versions of
different submodules together).
So that's where I come from. And maybe I'm totally wrong. I'd like to hear
what people who actually _use_ submodules think.
Here's some thoughts on subprojects from my company's perspective. I
apologize for the long message.
Abstract: We use submodules heavily in CVS and SVN. I like what I've read
from Linus about the "thin veneer" approach of integrating subprojects. It
seems conceptually to provide the support we desire. For us, it's important
that the mandated linkage between a master project and a subproject is
minimal to maximize our flexibility in building our processes.
We develop and maintain a lot of embedded applications. Both for higher level
systems (ex: 32MB RAM/32MB storage) running the Linux kernel and a customized
set of libs/app support code and more deeply embedded environments (ex: 8KB
of RAM and 32KB of storage). Even though these two cases are very different
in many repects, the version management issues are the same.
- We (mostly) track everything needed to build historical versions of code
with 100% fidelity. This includes all of the tools used to compile, build,
test, deploy, debug, etc. the actual build results themselves. I initially
looked at Vesta several years ago. I love their conceptual approach to this
problem (integrated build system that caches mid-level build results within
the repository itself), but it's too unwieldy, very hard to set up (lots of
up-front effort), and lacks many useful features.
- Most of our "applications" are a relatively small amount of app-specific
code with references to several/many shared modules. Shared modules can
contain support tools, like build/test/debug/deploy support for a given
embedded platform, in-house developed shared app code, or shared code
developed by third parties.
- We use CVS to manage our larger system development projects. The repo is
about 2GB and has several dozen application-code submodules. We use the
"third party sources" approach to tracking submodules as outlined in Ch.13 of
the CVS manual. Additionally, we manage our "buildox" (similar to buildroot
in concept) in another CVS repo. All prior interesting versions of the
buildroot can be built from source (toolchains, everything), if necessary.
Applications contain metadata (a file...) in the repo so the app-level build
system can ensure it is being ran under the correct version of buildbox;
clunky but serviceable. CVS is a nightmare because of its poor
branch/tagging facilities, and many of the things we *ought* to be doing with
revision control we don't because of the complexity.
- We use SVN to manage our deeply embedded system projects. The repo is about
250MB in size. Applications use the svn:externals property to reference
needed modules. We aren't using a buildbox in this environment yet (bad!).
SVN's simple branching and svn:externals are a giant leap forward in
comparison to CVS's capabilities.
Below are some common use case scenarios that are to varying degrees unweildy
in CVS and/or SVN. Many of these involving non-trivial branching and merging
operations are nearly impractical in CVS, and the lack of merge tracking (to
support repeated safe merging from one branch to another) makes some of these
a bit tricky in SVN too. Of course neither repo supports
disconnected/distributed operation, which would make a number of activities
that much simpler as well.
- Round trip module management. A specific app requires a change to a shared
module, so it makes a local branch to develop the change. The "diff" is
presented to the maintainer (who may be inhouse). The next interesting
maintainer version of the module gets imported into our repo (if in house,
it's already there), where the app can reference it. This merge process may
leave changes not yet implemented (or never to be implemented) by the module
maintainer in the local branch used by the apps. Other apps are unaffected,
as they are linking to a prior version in the local branch.
- Pragmatic development. It's typical that in developing an application, a
developer will need to simultaneously make changes to one or more submodules.
If more than trivial, he/she should branch the submodules and continually
tracking the HEAD of those branches in the relevant app. This is so complex
and fraught with problems in CVS that it doesn't get done, and developers
house too much change over time in their working directories. With SVN and
svn:externals, the process is workable. It is nice that an svn:external can
point to (the HEAD of) a branch when making changes.
- An application implements a new feature internally (say support for a new
digital chipset in the embedded world) which later needs to be "promoted" to
a subproject for use by others. Pretty easy in SVN. A challenge in CVS;
it's really not possible to "convert" app code into a "third party source"
and retain an historical link.
- Updating build tools. In concept no different than updating a shared code
module. In practice, due to the buildbox strategy, it's a bit convoluted. I
don't expect this to get much smoother. Getting Vesta-like features, where
integrated build suport can cache lower-level build results in a version-safe
manner (like the binary code built when the cross toolchain was built) would
be killer, but that's surely OT for the submodules discussion.
Thanks,
From: Sven Verdoolaege <hidden> Date: 2016-08-11 19:34:32
On Fri, Dec 08, 2006 at 12:29:14PM -0600, Jon Loeliger wrote:
Why not have a "glue" object at the "stub"-interface of
the supermodule tree that provides policy mappings to
the sub-modules. Perhaps indicating git URL location,
mappings of branch names between super- and sub- modules,
special commit SHA1s, user policy or config choices at
the boundary, and things like that.
Is that the sort of direction we are headed?
Not unless you have something useful in mind that could be put in
these glue objects. URLs and branch names, in particular, should
not be stored in the repository itself, but in configuration files,
since they will be different for different copies of the repo.
From: Martin Waitz <hidden> Date: 2016-08-11 19:35:27
hoi :)
On Fri, Dec 01, 2006 at 02:21:20PM +0100, sf wrote:
I just had a short (really short) look at your work. My impression is
that your repository setup is much too complicated.
Well, I'm not really satisfied with the UI part.
What exactly do you find complicated?
As I proposed elsewhere: For submodules to work you only need to allow
commits in tree objects (that is what your implementation requires as
well). Everything else is in the tools. Much simpler.
I do not quite get your point.
The core of my work allows to put commits into tree objects.
Then there is some more (but not quite finished) work to make the tools
work together with submodules. So no, not everything is there yet.
--
Martin Waitz
The only problem I'm seeing atm is that the supermodule somehow has to
mark whatever commits it's using from the submodule inside the submodule
repo so that they effectively become un-prunable, otherwise the
supermodule may some day find itself with a history that it can't restore.
One could circumvent that by creating a separate repo for the submodule
at checkout time and pull the needed objects in the supermodule's odb
when commiting the supermodule. This way prune in the submodule cannot
do any harm, because in it's odb are no objects that are important for
the supermodule.
Uwe
--
Uwe Kleine-Koenig
Well, I would actually argue that you may often want to have a supermodule
and then at least have the _option_ to decide to not fetch all the
submodules.
If you want to allow this, you have to be able to cut off fetching the
objects of the supermodule at borders to given submodules, the ones you
do not want to track. With "border" I mean the submodule commit in some
tree of the supermodule.
This looks a little bit like a shallow clone
No.
I would say that it looks more like a "partial checkout" than a shallow
clone.
A shallow clone limits the data in "time" - we have _some_ data, but we
don't have all of the history of that data.
In contrast, a submodule that we don't fetch is an all-or-nothing
situation: we simply don't have the data at all, and it's really a matter
of simply not recursing into that submodule at all - much more like not
checking out a particular part of the tree.
So if a shallow clone is a "limit in time", a lack of a module (or a lack
of a checkout for a subtree in general - you could certainly imagine doing
the same thing even _within_ a git repository, and indeed, we did discuss
exactly that at one point in time) is more of a "limit in space".
From: Josef Weidendorfer <hidden> Date: 2016-08-11 19:36:02
On Saturday 02 December 2006 21:58, Martin Waitz wrote:
So I was not against the link object itself (initially I wanted to do it
this way, too), only agains the information which was proposed to be
stored there. Up to now I haven't found anything which makes sense to
store next to the submodule commit to define the identity of the
submodule.
Isn't it enough reason that a porcelain probably wants to store meta
information for a given submodule, giving the need to put a name/identity
to it?
Josef
From: Sven Verdoolaege <hidden> Date: 2016-08-11 19:37:00
On Tue, Nov 28, 2006 at 01:37:54PM -0500, Daniel Barkalow wrote:
If submodule was the only thing that got changed, it's not dirty; if it
were dirty, some of its contents would also have gotten changed.
For me, the commit is the only "content" of the subproject that the
superproject should care about, so the submodule being dirty or not
is completely irrelevant (for committing), but it seems you see the
subproject more as a (working) tree than as a commit. Of course, as
Linus already mentioned, a "git commit" could still warn you if the
subproject was dirty.
Surely:
"git commit submodule/foo bar"
I wouldn't dream of doing such an operation, because it doesn't make
sense to me. (So as far as I'm concerned, you can make it do whatever
you'd like it to do.) You can only commit the subproject as a whole.
should do "git commit foo" in submodule, and then commit the supermodule
with the new commit for the submodule and the change to bar. And so
"submodule/foo" is something you could commit changes to, so it should get
picked up by -a.
From: Andy Parkins <hidden> Date: 2016-08-11 19:37:53
On Tuesday 2006 November 28 09:36, Andreas Ericsson wrote:
I'd actually prefer the second solution here and let git print a list of
submodules with dirty state and ask for some sort of user-response
before creating the actual commit. As non-interactive commits should
always be clean, requiring user intervention on non-clean state should
be a safe thing to do.
I'd agree. However, is there a need to require user intervention? Can we not
make the following analogies to normal git operation:
file in working directory -> submodule working directory
file in index -> submodule repository
It's perfectly possible to make a commit with different contents in the index
and the working directory - it shows up in the git-status output very nicely.
Why not deal with submodules in the same way?
Now imagine the following repository:
file1
file2
submodule1/file3
Make changes to file2 and file3, but don't update-index or commit. git-status
would show:
# Changed but not updated:
# (use git-update-index to mark for commit)
#
# modified: file2
# dirty: submodule1
#
nothing to commit
Now "git-update-index file2" and git-status
# Updated but not checked in:
# (will commit)
#
# modified: file2
#
# Changed but not updated:
# (use git-update-index to mark for commit)
#
# dirty: submodule1/
#
Now do a commit in submodule1/ and git-status in the supermodule.
# Updated but not checked in:
# (will commit)
#
# modified: file2
# submodule: submodule1/
#
Obviously the detail would be different, but you get the idea. There is
almost no difference between git-with-submodules and git-as-normal.
I suppose there would actually need to be an extra step were the submodule is
added to the supermodule index. So really there would be three states from
git-status:
# Updated but not checked in:
# (will commit)
#
# modified: file2
#
# Changed but not updated:
# (use git-update-index to mark for commit)
#
# modified: file1
# submodule: submodule1/
#
# Dirty submodules:
# (commit changes in the submodule to clean)
#
# dirty: submodule1/
#
Which means: since the last supermodule commit there has been
* a change to file2, which is in the index and would be committed.
* a change to file1, which is not in the index and won't be committed.
* a commit to submodule1, which won't be committed
* changes to the submodule working directory
This really reinforces Linus's interpretation that submodules are
directories - they would presumably just get a new object type and be
referenced in the tree object. git-update-index would be blind to dirty
submodules with no new commit, just as git-update-index on an unchanged file
has no effect.
Has this question been answered yet? How does the supermodule know which
branch to track in the submodule? Does it simply track HEAD or when the
submodule is added to the supermodule is it told which branch to track? I
suppose it's got to be HEAD really hasn't it?
Andy
--
Dr Andy Parkins, M Eng (hons), MIEE
From: Jakub Narebski <hidden> Date: 2016-08-11 19:38:27
Torgil Svensson wrote:
On 12/16/06, Torgil Svensson [off-list ref] wrote:
quoted
example tree object:
100644 blob <sha1 of blob> README
100644 blob <sha1 of blob> REPORTING-BUGS
100644 link <sha1 of blob> <sha1 of commit>
040000 tree <sha1 of tree> arch
040000 tree <sha1 of tree> block
040000 link <sha1 of tree> <sha1 of commit>
Sorry, I was sloppy and forgot the names:
100644 blob <sha1 of blob> README
100644 blob <sha1 of blob> REPORTING-BUGS
100644 link <sha1 of blob> <sha1 of commit> AUTHORS
040000 tree <sha1 of tree> arch
040000 tree <sha1 of tree> block
040000 link <sha1 of tree> <sha1 of commit> misc
Now it doesn't looks like trees/blobs anymore so maybe a link object
is handy:
100644 blob <sha1 of blob> README
100644 blob <sha1 of blob> REPORTING-BUGS
100644 link <sha1 of link> AUTHORS
040000 tree <sha1 of tree> arch
040000 tree <sha1 of tree> block
040000 link <sha1 of link> misc
link-object:
<sha1 of commit>
<sha1 of tree/blob>
What do you need <sha1 of tree/blob> for in link-object? Wouldn't you
use usually the sha1 of top tree of a commit, which is uniquely defined
by commit object, so you need only <ahs1 of commit>?
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
Side note - you can do submodules other ways, but if you do, you'll almost
certainly go crazy.
You could, for example, make submodules be some kind of "union
filesystem", where you allow overlapping trees. It's conceptually
possible. It's also horribly horribly wrong, if only because I guarantee
that you'll have so many problems with it that you will only end up with a
mess that is even worse than "branches" in CVS.
That was shot down by Linus and I agree with him. "bind" was a
bad idea because binding of a particular subproject commit into
a tree is a property of the tree, not one of the commits that
happen to have that tree.
Yes. I think it would be a _fine_ idea to have a new tree-entry type that
points to a sub-commit, but it really does need to be on a "tree level",
not a commit level.
If it's on a tree level, getting things like "git diff" etc to work is not
impossible, and it will also fit very well into the whole git
infrastructure.
So right now a tree entry can be another tree or a blob - and the only
extension would be to add a "commit" type (which would largely _act_ as a
tree entry, at least for sorting, ie it would use the same "sorts as if it
had a '/' at the end" logic).
Now, to get everything to work seamlessly within such a commit thing
might be a fair amount of work, but I'm not sure you even _need_ to. It
might be ok to just say "subproject 'xyzzy' differs" in the diff, for
example, and have some rudimentary support for "git status" etc talking
about subprojects that need to be committed.
So that's where I come from. And maybe I'm totally wrong. I'd like to hear
what people who actually _use_ submodules think.
I think you'd rather hear from people who _would_ use submodules; I've
worked on a number of projects that would have benefitted from that
general functionality, but nobody trusted the implementation enough to
actually use it.
At my work, we're doing a bunch of stuff with microcontrollers. We've got
about a dozen different boards with microcontrollers, and each of them has
different firmware. We also have a bunch of code that can go on any of the
boards.
The way things are organized currently is that each board has its own
project, and there's a "common-micro" project with the common code. This
sort of works, but it means that when you change things in common-micro,
you never know what effect this will have on boards other than the one
you're actually working on. What I'd like to have is that each project has
a "common-micro" subdirectory, and changes to each of these can be merged
into each other, but that doesn't happen automaticly, and each board's
revisions include the common-micro revision they were created with.
Our setup and requirements at work are exactly the same: We have a few
main projects that are developed independently and we have one "helper"
project for code that is general enough to be reused. So work on the
helper project is only done while working on one of the main projects.
When we switch to another main project we integrate the changes to the
"helper" project.
That's the theory, at least.
Regards
Stephan
Side note: that's not to say that I would really see why you'd want to
have both the tree and the commit SHA1's, and why you seemingly think that
the links don't need a filename. Hmm?
I really want that file-name back - we can call it a mind short-circuit.
If you require the tree objects to be in the database, you might as well
require that the commit object be there. But you could make rules that say
that subprojects don't need the whole commit history, for example (which
is just a shallow clone in the subproject).
You have a very good point here, this would give us the history of the
From: Josef Weidendorfer <hidden> Date: 2016-08-11 19:40:17
On Thursday 14 December 2006 22:27, Torgil Svensson wrote:
This example is somewhat complex since the build for lib1.so and the
header-file might not has gone through the same commit on the lib1
subproject. Consider this example:
If you want to track build results for some source,
why would you ever want these builds go out of sync with the source?
As the built files depend on the source (and other things), the
source should be a submodule of the build project.
Hmm... I think I see a problem / wish for submodules here.
With the current submodule proposal, we force submodules to be
subdirectories inside of a supermodule.
Your example has the folling submodule dependence
("X ==> Y" means Y being a submodule of X):
App ==> Lib
^ ^
| |
AppBuild ==> LibBuild
If we force submodules to be subdirectories of supermodules,
Lib needlessly will have to appear two times in a checkout of
AppBuild.
However, there is nothing wrong with it. Yet, you perhaps want
the 2 Lib submodules not to go out of sync. This easily
can be done with symlinking the Lib checkouts. As they are submodules,
everything should work fine.
Perhaps an option you want to have is to force a checkout
of AppBuild to make these symlinking itself when it detects
identical submodules links.
Hmmm... the only problem with a symlink is that it can go wrong
when moved. Unfortunately, I do not have a good solution for
this. We can not make UNIX symlinks smart in any way.
Hardlinking directories would be a solution, but that is not
possible.
Another thing:
With normal "$buildroot != $srcroot" environments, the source
can not be a subdirectory of the build directory.
Yet, we want to specify submodule/supermodule relation.
This is difficult to do with a submodule object, as it needs
to appear in trees in the supermodule.
Actually, the best workaround for this is to make Lib a direct
submodule of AppBuild, and specify the relationship of
LibBuild ==> Lib only in AppBuild.
BTW, build project commits probably should not depend on any
history of other build commits.
So you actually want all build commits to be root commits, and
have a tag name which could include the source commit id from
which the build was done. This gives some loose coupling.
Link: /headers/lib1.h -> <lib1-commit3>/src/lib1.h
Link: /bin/lib1.so -> <build1-commit>/i386/lib1/lib1.so
Link: /bin/app1 -> <build1-commit>/i386/app1/app1
<lib1-commit1>, <lib1-commit2> and <lib1-commit3> should be the same,
dictated by the app1 project.
I do not see any problem here. Symlinks are stored in the git repository.
As the AppBuild commit depends on App and LibBuild submodule commits, the
symlinks always should be correct.
Can we enforce this in the modules file
or should the different supermodules fix this somehow using
scripts/hooks?
I do not see any need for an hook. But of course, a checkout hook should
be able to generate files/links. However, IMHO this should be not
done with hooks but with Makefile targets.
How do the super-projects in this case get access to the blobs pointed
by the links - transparent or explicit in the build-process?
Submodules should automatically be checked out when checking out the
supermodule. So the blobs should already be there.
Or do I miss something?
From: Andreas Ericsson <hidden> Date: 2016-08-11 19:41:52
Jon Loeliger wrote:
On Fri, 2006-12-01 at 14:13, Linus Torvalds wrote:
quoted
So this is why it's really important that the submodule really is a git
repository in its own right, and why committing stuff in the supermodule
NEVER affect the submodule itself directly (it might _cause_ you to also
do a commit in the submodule indirectly, but the submodule commit MUST be
totally independent, and stand on its own).
An implication of this is that the entire administrative
responsibility for having some super-sub module interaction
lies entirely with the supermodule.
That's a good thing. I wouldn't want the openssl maintainers to have to
bother with every project that uses their code, and I'm fairly certain
they feel the same.
--
Andreas Ericsson andreas.ericsson@op5.se
OP5 AB www.op5.se
This lets you then perform a bisect within the
submodule when you identify the supermodule commit which caused
the breakage.
That is, first bisect the supermodule (which naturally bisects the
submodule with rough granularity, assuming there are many submodule
commits for at least some supermodule commits), then bisect the submodule
between the two commits identified at supermodule level, right ?
Right. That is how you _must_ do it.
The reason is:
- the supermodule will not track every release of the submodule. One of
the biggest reasons for using submodules in the first place is that the
submodules have their own development _independently_ of the
supermodule, and usually the supermodule will import new versions of
submodules only occasionally (eg the supermodule might choose to track
only major releases of the submodule, for example)
(And yes, I realize that this is not necessarily the only submodule
usage: sometimes the submodules are literally _only_ developed as
submodules, and you'd never develop them independently. It depends on
the situation)
- As a resule of the above, you MUST NOT do bisection at the submodule
level at first: it's entirely possible that the supermodule never ever
actually used the submodule state at a finer granularity, and
"bisecting" into such state would be idiotic (it's really no different
from "bisecting" a regular commit by splitting up a commit into patches
against individual files - sure, it's a smaller granularity, but it's a
granularity that never _existed_, and was never tested or intended to
work!)
So yes, you should expect that
(a) submodule changes "jump around" in the supermodule - even to the
point of going backwards in time as far as the submodule is concerned
(ie the supermodule might have tested a new release of a submodule,
committed that, found a problem, and decided to just go back to an
earlier version of the submodule again, and committed that again)
(b) This implies very much that there can be a n:m relationship between
submodule and supermodule commits. A supermodule commit does _not_
imply a commit in the submodule (it might commit changes to the
top-level makefile or to _another_ submodule), but equally, a
submodule commit does _not_ imply a commit in the supermodule
(because the submodule might be independently changed in some other
repository where it's the _primary_ development, not a submodule)
So you shouldn't expect submodules to be very "tightly" coupled, and I
don't think you even want the workflow to _be_ that tight. I think it's ok
if submodules show up as such, and that "git diff" etc don't try to make
it all "seamless".
It often _shouldn't_ be seamless: you should be able to commit to a
supermodule without committing the submodule state: it's really no
different from committing individual files (it migth be somethign that is
_discouraged_ as a workflow for some project, the same way you might
discourage using "git commit one/file" over "git commit -a", and for the
same reason: you're committing some state that doesn't match what your
tree actually looks like).
Similarly, doing a "git commit -a" within a submodule should really just
commit _that_ submodule, and not even _try_ to know about supermodules
etc, because the submodule really should be a totally independent git
repository.
[ Side note: you may well want to set up submodules so that they share the
object store with the supermodule: that may be the simplest way to make
operations that traverse things recursively work out, since it means
that you can do object lookups for everythign you traverse without
having to even think about it.
On the other hand, this could equally easily be done by just making
every submodule an "alternates" directory in the supermodule: that keeps
the object databases separate, but means that anybody in the supermodule
will always be able to look up all the objects in the submodules. So
even here, we certainly _can_ keep things separated, without even
introducing any new concepts. ]
So I actually think that submodules should at least start out as something
rather independent, where a "commit -a" in the supermodule will _only_
commit the supermodule itself - and if you haven't committed the submodule
yet, you'll just get the current HEAD state of the submodule.
Add some trivial help in "git status" to _warn_ about the fact that
submodules haven't been committed and are dirty, but I really think that
it should be a very explicit thing where you really do see things as
submodules, not as "one big module".
From: Martin Waitz <hidden> Date: 2016-08-11 19:42:35
hoi :)
On Fri, Dec 01, 2006 at 02:05:33PM +0100, sf wrote:
quoted
On Fri, Dec 01, 2006 at 01:09:49PM +0100, sf wrote:
quoted
Martin Waitz wrote:
quoted
So you not only store your submodule HEAD commit in the supermodule
when you do commit to the supermodule, it also means that your
submodule HEAD will be updated when you update your supermodule.
Why the magic? The typical workflow in git is
1. You work on a branch, i.e. edit and commit and so on.
2. At some point, you decide to share the work you did on that branch
(e-mail a patch, merge into another branch, push upstream or let it by
pulled by upstream)
3. Other people want to use your new work.
Sorry, if that was not obvious: You actually procceed with one of the
options I listed in Step 2. What I wanted to state is that with git you
do not mix up committing (which is local to your repository and your
branch) and publishing.
I guess you are refering to not mix up committing to the submodule and
updating the supermodule index.
These are really two separate steps, I just combined them above because I
wanted to put emphasis on the other part: it is not a one-way flow, it
is bidirectional, so your HEAD would have to changed if the supermodule
gets updated.
And I consider changing HEAD, without looking at the branch it points
to, to be a bad thing.
--
Martin Waitz
From: Stephan Feder <hidden> Date: 2016-08-11 19:43:13
Martin Waitz wrote:
hoi :)
On Fri, Dec 01, 2006 at 02:05:33PM +0100, sf wrote:
quoted
quoted
On Fri, Dec 01, 2006 at 01:09:49PM +0100, sf wrote:
quoted
Martin Waitz wrote:
quoted
So you not only store your submodule HEAD commit in the supermodule
when you do commit to the supermodule, it also means that your
submodule HEAD will be updated when you update your supermodule.
Why the magic? The typical workflow in git is
1. You work on a branch, i.e. edit and commit and so on.
2. At some point, you decide to share the work you did on that branch
(e-mail a patch, merge into another branch, push upstream or let it by
pulled by upstream)
3. Other people want to use your new work.
Sorry, if that was not obvious: You actually procceed with one of the
options I listed in Step 2. What I wanted to state is that with git you
do not mix up committing (which is local to your repository and your
branch) and publishing.
I guess you are refering to not mix up committing to the submodule and
updating the supermodule index.
The opposite: If you work in the supermodule, even if it is in the code
of the submodule, you only commit to the supermodule. The submodule does
not "know" about these changes after step 1.
These are really two separate steps, I just combined them above because I
wanted to put emphasis on the other part: it is not a one-way flow, it
is bidirectional, so your HEAD would have to changed if the supermodule
gets updated.
Why do you mix up supermodule and submodule? The way I see your proposal
you cannot change submodule and supermodule independently. That is a
huge drawback.
Regards
From: Andreas Ericsson <hidden> Date: 2016-08-11 19:44:18
Uwe Kleine-Koenig wrote:
Hello,
Andreas Ericsson wrote:
quoted
The only problem I'm seeing atm is that the supermodule somehow has to
mark whatever commits it's using from the submodule inside the submodule
repo so that they effectively become un-prunable, otherwise the
supermodule may some day find itself with a history that it can't restore.
One could circumvent that by creating a separate repo for the submodule
at checkout time and pull the needed objects in the supermodule's odb
when commiting the supermodule. This way prune in the submodule cannot
do any harm, because in it's odb are no objects that are important for
the supermodule.
Yes, but then you'd lose history connectivity (I'm assuming you'd only
pull in the tree and blob objects from the submodule, and prefix the
tree-entrys with whatever directory you're storing the submodul in).
--
Andreas Ericsson andreas.ericsson@op5.se
OP5 AB www.op5.se
So you not only store your submodule HEAD commit in the supermodule when you
do commit to the supermodule, it also means that your submodule HEAD
will be updated when you update your supermodule.
Why the magic? The typical workflow in git is
1. You work on a branch, i.e. edit and commit and so on.
2. At some point, you decide to share the work you did on that branch
(e-mail a patch, merge into another branch, push upstream or let it by
pulled by upstream)
I fail to understand why these two steps have to be mixed up. Someone
care to explain?
Regards
Stephan
From: Jakub Narebski <hidden> Date: 2016-08-11 19:44:28
[cut]
From this discussion I think it follows that supermodule should track HEAD
version of submodule. Perhaps the supermodule index should have sha1 of
submodule commit, so (as usual) you have to update-index in supermodule to
record changes in submodule; the difference being that you update to HEAD
version, not to working directory version. Or you can just git-commit -a
in supermodule which would take working directory version of files, and HEAD
version of submodules.
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
From: Andreas Ericsson <hidden> Date: 2016-08-11 19:45:04
Torgil Svensson wrote:
On 12/4/06, Linus Torvalds [off-list ref] wrote:
quoted
So yeah, it's a bit hacky, but for the reasons I've tried to outline, I
actually think that users _want_ hacky. Exactly because "deep
integration"
ends up having so many _bad_ features, so it's better to have a thin and
simple layer that you can actually see past if you want to.
Thin and simple sounds very good. Let's try it with an example. Lets
say we have one apllication App1 and three librarys (Lib1, Lib2, Lib3)
with the following dependency-graph:
App1
/\
/ \
Lib1 Lib2
\ /
\ /
Lib3 (don't really needed for this example but looks nice)
All components can be used individually and have their own upstream,
maintainer etc.
To compile App1 however, I need some files from both Lib1 and Lib2
specifying it's API. To satisfy these dependencies, It sounds
reasonable to link Lib2 and Lib3 submodules from App1. In your
concept, can I construct a modules file to fetch the API files and
their history without checking out the whole Lib1 and Lib2 source?
I think not. Then it wouldn't be a submodule anymore, but just some
random sources from an upstream project. Not that it's an uncommon
workflow or anything, but it's sort of akin to just importing the SHA1
implementation (a few source-files with no real interest in the history
of those source-files) from openssl into a different project rather than
actually using the entire openssl lib (which would be nice to have as a
submodule).
--
Andreas Ericsson andreas.ericsson@op5.se
OP5 AB www.op5.se
From: Martin Waitz <hidden> Date: 2016-08-11 19:45:16
hoi :)
On Fri, Dec 01, 2006 at 11:26:22PM +0100, Josef Weidendorfer wrote:
It's not about checking out part of the tree, it's about fetching only
part of the objects: If you have a slow modem and want to clone a
supermodule, you are not interested in fetching all the objects from
some submodules.
So when you want to suppress one submodule, how is this not about only
checking out part of the tree?
Ok, you also want to avoid downloading the submodule, but you first have
to solve the partial checkout.
BTW: In your submodule implementation, is the user allowed to change the
relative path of the root of some submodule, e.g. with "git-mv" ?
In principle: yes.
However there are some links between both repositories that have to be
updated manually (for the shared object repository and for ignoring
submodule files in the supermodule).
But I expect that much of this configuration stuff will vanish when
submodules are better integrated in git.
Rename detection for submodules would be another interesting thing to
have. It should be much easier as for files because we can simply check
for common ancestors and do not have to guess based on the diff.
--
Martin Waitz
From: Jon Loeliger <hidden> Date: 2016-08-11 19:45:25
On Fri, 2006-12-01 at 14:13, Linus Torvalds wrote:
So this is why it's really important that the submodule really is a git
repository in its own right, and why committing stuff in the supermodule
NEVER affect the submodule itself directly (it might _cause_ you to also
do a commit in the submodule indirectly, but the submodule commit MUST be
totally independent, and stand on its own).
An implication of this is that the entire administrative
responsibility for having some super-sub module interaction
lies entirely with the supermodule.
Why not have a "glue" object at the "stub"-interface of
the supermodule tree that provides policy mappings to
the sub-modules. Perhaps indicating git URL location,
mappings of branch names between super- and sub- modules,
special commit SHA1s, user policy or config choices at
the boundary, and things like that.
Is that the sort of direction we are headed?
jdl
From: Martin Waitz <hidden> Date: 2016-08-11 19:45:31
hoi :)
On Sat, Dec 02, 2006 at 12:34:31AM +0100, sf wrote:
quoted
Now your submodule is no longer seen as an independent git repository
and I think this would cause problems when you want to push/pull between
the submodule and its upstream repository.
You can always pick a single commit or several commits out of a larger
repository and have a complete git repository.
And I already explained how to push and pull even from within superprojects.
Sure it you are able to make it work, but it needs more work on the UI part.
How do you handle the index? How do you allow to clone only the
submodule?
I really thought about such a setup too, but then decided that it is
much easier to work with submodules when you can really see it as a
repository of its own.
quoted
But you could still call the "xdiff" part of the git repository a
submodule. And then changes to the xdiff directory result in a new
submodule commit, even when there is no direct reference to it.
So you'd still "commit to the xdiff submodule".
Let's make certain that we understand each other. I see a clear
distinction between the submodule code in a supermodule branch (commits
in the supermodule's tree and nothing else) and submodule branches which
are independent of the superproject. Supermodule branches and submodule
branches do not interact, only if I want them to.
Agreed.
I think the thing which caused some discussion is that I make the
current submodule commit which is used by the supermodule available in a
refs/head in the submodule.
So there is one "branch" in the submodule which corresponds to the
version used by the supermodule, but this is just for user interface.
It's most important purpose is to give this special commit a name, so
that it can be used in merges, etc.
By selecting another refs/heads "branch" in the submodule you can also
easily detach the submodule from the supermodule.
It is really important to understand that you can't branch the submodule
alone and still have it connected to the supermodule, because the
supermodule always tracks only one commit for each submodule.
So every branch that affects the project has to be done on project
(topmost supermodule) level.
But of course the submodule can have other branches which are not
tracked by the supermodule.
So by checking out refs/heads/master (as it is used in my
implementation) you can attach the submodule to the supermodule (attach
as in: bring the working directory in sync with the whole project), and
you can detach it by selecting another refs/heads (the submodule is
still part of the supermodule, but not in the state which is currently
visible in the working directory).
This may sound confusing, but it really is the only semantic for
submodule branches that makes sense.
There are fears that you may commit something that does not match your
current working directory. Sure, but you explicitly asked for it and I
think it won't be a problem if git-status tells about this fact.
The double slashes is the only way I can think of that clearly indicates
that I do not mean the contents named by the path, but the commit that
you find there. Once you have named a commit in that way, you can
continue to apply other revision naming suffixes, paths, and so on.
With the current semantics, you can already get to the submodule commit
(just leave out your double slashes), but what is missing is simply to
apply all the modifiers again on this submodule commit.
So I think we can do without the double slashes.
--
Martin Waitz
From: Josef Weidendorfer <hidden> Date: 2016-08-11 19:45:53
On Friday 01 December 2006 23:41, sf wrote:
Linus Torvalds wrote:
...
quoted
In contrast, a submodule that we don't fetch is an all-or-nothing
situation: we simply don't have the data at all, and it's really a matter
of simply not recursing into that submodule at all - much more like not
checking out a particular part of the tree.
If you do not want to fetch all of the supermodule then do not fetch the
supermodule. Instead fetch only the submodules you are interested in.
You do not have to fetch the whole repository.
But what, when I *want* to fetch the supermodule because of the source
tree which is only available in the supermodule?
Of course, you can argue that the only objects in trees of a supermodule
should be submodule commits, but this is quite restricting the usage of
supermodules.
See further arguments for a submodule namespace in my other mail.
You probably want to specify policies about submodule handling. This
information has to be indexed by some name independent from a
supermodule commit.
Josef
From: Martin Waitz <hidden> Date: 2016-08-11 19:47:13
On Fri, Dec 01, 2006 at 05:38:44PM +0100, Andreas Ericsson wrote:
quoted
But if the supermodule contains changes to the submodule, you still
have to change the submodule. And this implies changing the submodule
HEAD or some branch.
Not really. I fail to see why HEAD needs to be changed so long as the
commit is in the submodule's odb.
Because I want the submodule to act as a normal git repository.
Please note that I also voted against changing HEAD directly, but that
the new commit which came from the supermodule is just stored in one
branch of the submodule, as part of the supermodule checkout.
--
Martin Waitz
From: Martin Waitz <hidden> Date: 2016-08-11 19:47:53
hoi :)
On Fri, Dec 01, 2006 at 04:12:10PM -0800, Linus Torvalds wrote:
It only gets interesting for commands that fetch new objects, ie do a
"pull/fetch" op, and you'd need to know where/how to fetch new objects for
the xyzzy subproject, so that's a "naming" issue. You have a few choices:
- get all the objects directly from the subproject as if it was one big
project.
I actually think this sucks. Why? Because it puts an insane load on the
server side, which basically needs to traverse the object list of the
_sum_ of all projects. An initial clone (or a really big pull, which
comes to the same thing) would be absolutely horrendous
I don't buy your scalability argument.
By dividing the object traversal in separate steps you do not win
anything. The complexity of the operation still stays the same, as you
still have to traverse the exact same amount of objects.
By separating the repositories you just make reachability analyis be
totally awkward, without winning anything.
--
Martin Waitz
From: Jakub Narebski <hidden> Date: 2016-08-11 19:48:42
Torgil Svensson wrote:
On 12/16/06, Jakub Narebski [off-list ref] wrote:
quoted
All fine, but this does not and I think cannot protect us from the
fact that we can have <sha1 of tree/blob> which doesn't match
<sha1 of commit>.
True, that will be a real problem. Unless we have a bug in git, do you
see a scenario in which this is likely to happen?
Well, I just rather have than <sha1 of tree/blob> the definition
of sparse checkout (for example subdirectory name, or file name,
or glob pattern).
Besides you need the name of directory (for tree) or file (for blob),
otherwise you would have no way to update it when submodule advances
version, and you want to use new submodule version. And if you have
that, you don't need <sha1 of tree/blob> in repository, in link object.
You might want it in the index, for performance reasons, though.
quoted
I think it would be better to have sparse/partial checkout first.
But that is just my idea. Because with <sha1 of tree/blob> which
is not sha1 of commit tree you might loose (I think) the ability
to merge, for example your changes to submodule with upstream.
That's correct. I also want a sparse/partial checkout but I don't want
the full submodule path. I'm also perfectly fine (for my current
use-cases) with not being able to merge upstream unless we're tracking
the commit tree (here, we might not want to specify the tree SHA1).
With sparse (for example defined by 'src/*.h') or partial (for example
defined by 'Documentation/') checkout you should be able to merge
upstream... unless conflicts are in the not checked out part.
I'm not trying to impose a technically fragile solution here [I don't
believe it is, but I'm not the most competent to say that either], I'm
trying to find solutions for my use cases and I had problems adapting
them to the current suggestion.
From: Josef Weidendorfer <hidden> Date: 2016-08-11 19:49:02
On Saturday 02 December 2006 21:43, Martin Waitz wrote:
On Sat, Dec 02, 2006 at 02:50:45PM +0100, Josef Weidendorfer wrote:
quoted
On Saturday 02 December 2006 11:04, Andy Parkins wrote:
quoted
quoted
So what do you do with deleted submodules?
You wouldn't want them to still sit around in your working directory,
but you still have to preserve them.
Now that is a tricky one. Mind you, I think that problem exists for any
implementation. I haven't got a good answer for that.
That suggests that it is probably better to separate submodule repositories
from their checked out working trees. Why not put the GITDIRs of the submodules
in subdirectories of the supermodules GITDIR instead?
Why not simply use a shared object database instead?
Sure. I have no problem with this.
But can we go one step further?
AFAICS your submodules store the .git/ directories of submodules directly
at submodule position in the working tree - but you have a link .git/objects
into the object database of the supermodule.
When the user wants to delete the submodule, he would remove this .git/ directory,
too. So you loose the .git/refs of the submodule etc. I would suggest to put
the submodule .git dirs into the .git dir of the supermodule.
But if the supermodule contains changes to the submodule, you still
have to change the submodule. And this implies changing the submodule
HEAD or some branch.
Not really. I fail to see why HEAD needs to be changed so long as the commit
is in the submodule's odb.
Right. A commit in the supermodule should _not_ imply a commit in the
submodule.
Maybe I should take a look at the code, but it sounds like people are
still trying to "mix" submodules too much.
Think of it this way: one common use for submodules is really to just
(occasionally) track somebody elses code. The submodule should be a
totally pristine copy from somebody else (ie it might be the "intel driver
for X.org" submodule, maintained within intel), and the supermodule just
refers to it indirectly (ie the supermodule might be the "Fedora Core X
group" which contains all the different drivers from different people).
So anything that mixes super-modules and sub-modules too much will always
break this kind of model.
A supermodule can never "contain changes" to a submodule. A supermodule
would always just point to the submodule, and not have any changes
what-so-ever of its own. The submodule is self-sufficient, and always
contains all its _own_ changes.
From: Josef Weidendorfer <hidden> Date: 2016-08-11 19:49:53
On Friday 15 December 2006 18:43, Torgil Svensson wrote:
On 12/15/06, Josef Weidendorfer [off-list ref] wrote:
quoted
However, there is nothing wrong with it. Yet, you perhaps want
the 2 Lib submodules not to go out of sync. This easily
can be done with symlinking the Lib checkouts. As they are submodules,
everything should work fine.
This is interesting. In my notation:
/path/to/link/name -> <commit>/path/to/subtree
means that there is a link named "name" in the tree object for
"path/to/link". The link points to a "link object" specifying a
subtree or blob of the tree that is pointed to in a submodule commit.
Ah, now I understand. I somehow missed this notation.
This is not currently implemented but has at least the following
advantages:
1. You can access files in a submodule without fetching the whole
submodule (which may be very large). (App1 is only interested in
lib1.h, the rest is toally irrelevant)
2. Superproject can access referenced (linked) files in it's own
folder-structure without being forced a structure by the subproject.
That all sounds fine, but how do you create such symlinks in practice?
Do you want to introduce special porcelain commands to create them?
Especially, what is the SCM user supposed to do to change the link
target, ie. from
<commit>/path/to/subtree
to
<commit>/path2/to2/subtree2
?
Should this do a re-checkout at the other point?
By linking a file from a submodule, such a link seems to force that
this file has to be at a fixed position in the submodule. Otherwise,
some magic has to happen when the file is moved in the submodule,
possibly leading to a dangling link, eg. if the whole subdirectory
specified in the link is removed.
IMHO this is getting way to complex.
Much simpler is to include the full submodule at some path in
the supermodule, and create normal symlinks from the supermodule
into the submodule.
If you only want to check out part of a submodule, this should be
done with path-limiting checkouts, which should be a feature totally
independent from submodules.
And if you want to limit the number of objects transferred in cloning
of a subproject, it is better to further split this subproject into
multiple subprojects itself.
If you do a symlink instead, doesn't you loose versioning information?
Of course, you need the submodule fully checked out somewhere in the
supermodule, and the link goes into the submodule directory. The
versioning is given by the supermodule/submodule link.
What happens with the symlinks if someone clones the superproject?
As already said: the link has to go into a submodule directory, which
will be checked out automatically with the clone of the supermodule.
quoted
Perhaps an option you want to have is to force a checkout
of AppBuild to make these symlinking itself when it detects
identical submodules links.
Hmmm... the only problem with a symlink is that it can go wrong
when moved. Unfortunately, I do not have a good solution for
this. We can not make UNIX symlinks smart in any way.
Hardlinking directories would be a solution, but that is not
possible.
Wouldn't specifying the submodule path in the link object fit in well
here? Then each "link object" can represent a checked out tree from
the subproject in the superproject directory-structure.
The problem is not the representation in the git repository, but the
checked out module/submodule, where you need to use normal UNIX file semantics.
To move submodules around, the user should be able to just use
the normal UNIX "mv" commands, and git should be able to detect move
actions after the fact.
The simple thing here is that currently, git does not have this problem
as it tracks content, and does not even try to detect any moves at
commit time. This is different with submodules, as there, you want to
be able to track moves of any submodule root directories.
This now becomes a problem if you use symlinks to "unify" multiple checkouts
of the same submodule at multiple places in the supermodule, and move
the symlink around, as it easily can get dangling this way. Thus, you would
not have a way to see what submodule this link was talking about.
And for this thing, I do not see how your link object could help.
So it is better to use a simple submodule concept, and for this corner
cases, we perhaps could expect the user to fix e.g. a dangling symlink
to a previous submodule checkout himself, using a meaningful error message.
quoted
BTW, build project commits probably should not depend on any
history of other build commits.
Why? Can you give an example here.
If you have a source commit chain A => B => C => D, you want
to make any build commits totally independent: you first only
are interested in a build commit for source versions A and D,
and later find out that a build commit for B and C would be nice,
too. If you force build commits into some history order, this
order now would be A => D => B => C, which makes no sense.
Build commit independence can easily be achieved by making every commit
parentless, without further history. You still have the link
to the source version via the submodule link in the tree.
But to not loose any such build commits, they have to appear
as tags or refs (unless integrated in another superproject
build commit).
quoted
quoted
Link: /headers/lib1.h -> <lib1-commit3>/src/lib1.h
Link: /bin/lib1.so -> <build1-commit>/i386/lib1/lib1.so
Link: /bin/app1 -> <build1-commit>/i386/app1/app1
<lib1-commit1>, <lib1-commit2> and <lib1-commit3> should be the same,
dictated by the app1 project.
I do not see any problem here. Symlinks are stored in the git repository.
As the AppBuild commit depends on App and LibBuild submodule commits, the
symlinks always should be correct.
The main reason for these "links" are for versioning purposes: the
uniqe SHA1 of the "link" representing a tree/blob in a version of the
submodule should be "included" in the supermodules commit. Symlinks
won't give that at all.
The version coupling will be there if the whole submodule is available
at some path in the supermodule checkout, as said above.
I think I've talked myself into the position where it definitely has to be
HEAD being tracked in the submodules; anything else is a disaster waiting to
happen because commit doesn't check in your current tree.
Yes, but not only that, HEAD is the only thing that fits with the
rest of the git repository/index/working directory model.
Lets review...
What's HEAD? Its the commit which matches the index state as
closely as possible, with the only differences being the changes in
progress that are being prepared for the next commit (whose parent
will be HEAD). If the index and working directory are both clean
(no changes) then its also the current content of this directory,
right?
What's the index? Its what you are about to commit.
What's the working directory? Its the current content, which may
also be partially checked out or dirty.
So HEAD in a submodule is the current content of that submodule.
Therefore any update-index call on a submodule should load HEAD
(totally ignoring whatever branch it refers to) into the supermodule
index.
--
Think of it this way: one common use for submodules is really to just
(occasionally) track somebody elses code. The submodule should be a
totally pristine copy from somebody else (ie it might be the "intel driver
for X.org" submodule, maintained within intel), and the supermodule just
refers to it indirectly (ie the supermodule might be the "Fedora Core X
group" which contains all the different drivers from different people).
Could you please be a little bit more specific about how you would store the
"pristine copy".
Note that it's not necessarily "pristine", since the submodule clearly is
a local git repository in its own right. So like _any_ git repository, you
can (and may well end up) having your own local branches in the submodule,
with your own local modifications.
So I'm not claiming that a submodule must always match some external git
tree 100%, and that it must be read-only or anything like that. I'm just
saying that I suspect that quite often, one of the MOST IMPORTANT parts is
that the submodule is really something that somebody else technically
maintains, and that this is actually one of the _reasons_ why it is a
submodule in the first place.
For example, a lot of projects end up having some kind of "library
component" as a submodule. Take something like a video player project,
which would have something like ffmpeg as a submodule, not because you'd
maintain ffmpeg yourself, but simply because (let's say) the library
interface changes enough, or you need a specific version with some of your
own fixes that haven't been released widely yet, so you want to carry all
the libraries you need _with_ you, even though you don't really maintain
that submodule. You at most have some small extensions of your own.
Now, in this situation, it's relaly really _important_ that the submodule
really is totally independent of the supermodule, for several reasons.
For example, since you don't "really" own that project, carrying around
your own fixes is really really painful. We know it happens all the time,
and a lot of projects end up needing their own version, but the _last_
thing you want is to be in merge hell all the time. So as a supermodule
maintainer, the best possible thing for you is to be able to push back
those local changes to the original project maintainer, so that you
_don't_ have to maintain your own changes.
But you need to realize that the real maintainer of the submodule is
TOTALLY UNINTERESTED in your supermodule. He's not going to maintain it,
and in fact, if you have anything in the submodule that ends up talking
about your supermodule, that's just going to make it a lot less likely
that the upstream maintainer will ever pull your changes. He might take a
diff from you, but in a perfect world, you'd actually be able to tell him:
"Hey, I've got a git repository with a few fixes to your ffmpeg git tree,
please pull from git://myhost.com/submodule.git to get these fixes:
... explanation of fixes and commits that are relevant to
ffmpeg, and have nothing to do with the supermodule, except
that you need those bug-fixes because you _use_ ffmpeg ...
Thanks"
See?
So this is why it's really important that the submodule really is a git
repository in its own right, and why committing stuff in the supermodule
NEVER affect the submodule itself directly (it might _cause_ you to also
do a commit in the submodule indirectly, but the submodule commit MUST be
totally independent, and stand on its own).
Now, you don't _have_ to push things upstream, of course. You can always
just maintain your own submodule branch, and every once in a while, inside
the submodule, you do
# fetch the development in the origin/master branch
git fetch submodule-origin origin/master
# rebase our own special magic sauce on top of that
git rebase origin/master
to update your submodule, and _then_ you do a commit in the supermodule
(after testing that the update is all ok, of course) which will update the
"commit" pointer in the supermodule.
Notice? In this example, we really maintained the submodule AS a
submodule. It was independent, but tied into the supermodule, so that when
we clone the supermodule, or do things like bisection on a supermodule, we
always end up cloning the submodule too (and in the case of bisection, we
really only bisect the supermodule, but the submodule always gets
"tracked" in the sense that we would always check out the state of the
submodule that was appropriate for that particular commit in the
supermodule).
There seems to be some agreement to store the commit id of
the submodule instead of a plain tree id in the supermodules tree object, and
that all objects that are reachable from this commit are made part of the
supermodule repository (either fetched or via alternates). Do you agree?
Well, I would actually argue that you may often want to have a supermodule
and then at least have the _option_ to decide to not fetch all the
submodules.
For an example of this kind of usage, let me tell you how we operated at
Transmeta a few years ago, which I'm not saying is the _only_ way to
operate, but it's ONE way to do it, and I'll also explain _why_ we did it,
and why we had submodules.
In the case of transmeta, we had our own tools, our own programs, and we
"owned" all of those. We _also_ used a lot of external tools, like gcc
etc. However, different people worked on different parts, and if you
worked on the actual x86 JIT part, you probably didn't want to have all of
the gcc stuff in your tree _too_. That just took a lot of space, and you
really didn't want to compile the whole toolchain (which took hours),
since there were precompiled binaries readily available.
Still, from a _release_ standpoint, when we released a new binary, that
binary very much depended not just on the actual JIT sources, but on the
whole toolchain. So if you wanted to be able to re-create a release, you
really needed _everything_. You couldn't just take the "current version"
of the toolchain, you needed to have the toolchain that was used AT THE
TIME OF THE RELEASE.
And this is a _classic_ example of when you'd want to use submodules.
Notice how everybody wanted _some_ of the submodules, but really only the
release people wanted them _all_. The higher up the chain you were, the
less likely you were to really want to muck around with the compiler and
the linker, for example.
And nobody really owned all modules.
So what you really want is:
- a supermodule maintainer that is not really the maintainer of _any_ of
the submodules, but that does the main "build world" infrastructure
(and generally would tend to also maintain the source control
infrastructure itself)
- submodules that had their own maintainers, and where the maintainers
may or may not have wanted the supermodule, but even when they wanted
the supermodule, they might not want _all_ of the submodules, simply
because they just didn't care.
- some of the submodules then have _upstream_ sources that were totally
independent, and that you would want to track, but you had zero power
AT ALL over them, and yet you migt well want to push back at least some
of the fixes you did - at least the ones that made sense even outside
your own project - just to avoid having to maintain a _huge_ set of
internal patches.
So no, I don't think the supermodule should even _force_ people to always
get all the submodules. It migth be the default case, but at the same
time, it's just being polite to let users decide on their own whether they
really want _all_ of the build infrastructure sources.
If I understand you correctly you cannot make any changes to the submodules
code _in the supermodule's repository_, no bugfixes, no extensions, no
adaptions, nothing. Do you mean that?
Yes. I think you should make all changes _within_ the submodule, because
the submodule should still be an independent git tree in its own right.
But obviously, you'd often use a private _branch_ in the submodule beause
you end up having whatever private extensions. That's always true: we
always have the "master" branch that is kind of the default "private
branch" for any repository, but obviously that is often extended upon, and
you may have several private branches.
For example, after you've done a big update (from some external upstream
source) in the submodule that you are using, you migth decide that you do
all the work on that new big update in a _new_ private branch within the
submodule - and get the submodule changes all squared away on its own
_before_ you then decide to commit the end result (the tip of that new
private branch) within the supermodule.
Ie, you very much should be able to to do
git clone supermodule/that/one/submodule my-own-version-of-submodule
to clone a submodule _without_ getting anything else (but still get all
the work you did within he submodule - very much including your own
private branch work).
And the importance of keeping the submodule independent is partly just
stability and sanity, but partly also scalability. For example, the
"index" in a supermodule should NOT include the indexes of all the
submodules. That's really important, because the index doesn't really
scale. Things do slow down with large indexes.
For example, git can handle tens of thousands of files easily. I suspect
it scales well to hundreds of thousands of filenames. But with
supermodules, you really can end up in the situation where you have _tens_
of these submodules, maybe even hundreds. And if you try to maintain one
unified index for the _whole_ thing, I guarantee you that you'll start
feeling the pain. Indexing millions of files is just not going to be
pretty.
So just from a git stability and scalability point, it's important to keep
subprojects _separate_. There is obviously integration stuff, but they
should still be seen as truly independent projects. Even the supermodule
should have clearly its own life even _regardless_ of submodules, because
(as I said) quite often you may want the supermodule, but you don't want
to have _all_ of the submodules.
But it's more than that stability and scalability thing too - keeping them
separate is what allows you to do pulls and pushes on an individual
subproject basis, and have people really work at that level. For example,
if you're the compiler guy at a company, you really do want to work with
other compiler people _outside_ the company, but you sure as hell may not
be able to give them access to your supermodule. But you may want to work
on _just_ the compiler parts (or at least share some branches in public),
which means that the subproject really has to be able to work
_independently_ of the supermodule.
So "independent" here is really key, for several reasons. And that all
means, for example, that here must NEVER be any "backpointers". A
subproject really can _never_ have backpointers to the superproject,
because that fundamentally means that the above kind of "compiler guy
works on the compiler subproject in public" cannot work, if your
supermodule isn't public.
On 12/15/06, Josef Weidendorfer [off-list ref] wrote:
If you want to track build results for some source,
why would you ever want these builds go out of sync with the source?
I don't, bad wording by me. That was the problem I wanted to address.
Your example has the folling submodule dependence
("X ==> Y" means Y being a submodule of X):
App ==> Lib
^ ^
| |
AppBuild ==> LibBuild
In my example "AppBuild" and "LibBuild" were the same project but this
scenario is relevant as well.
If we force submodules to be subdirectories of supermodules,
Lib needlessly will have to appear two times in a checkout of
AppBuild.
However, there is nothing wrong with it. Yet, you perhaps want
the 2 Lib submodules not to go out of sync. This easily
can be done with symlinking the Lib checkouts. As they are submodules,
everything should work fine.
This is interesting. In my notation:
/path/to/link/name -> <commit>/path/to/subtree
means that there is a link named "name" in the tree object for
"path/to/link". The link points to a "link object" specifying a
subtree or blob of the tree that is pointed to in a submodule commit.
This is not currently implemented but has at least the following
advantages:
1. You can access files in a submodule without fetching the whole
submodule (which may be very large). (App1 is only interested in
lib1.h, the rest is toally irrelevant)
2. Superproject can access referenced (linked) files in it's own
folder-structure without being forced a structure by the subproject.
If you do a symlink instead, doesn't you loose versioning information?
What happens with the symlinks if someone clones the superproject?
Perhaps an option you want to have is to force a checkout
of AppBuild to make these symlinking itself when it detects
identical submodules links.
Hmmm... the only problem with a symlink is that it can go wrong
when moved. Unfortunately, I do not have a good solution for
this. We can not make UNIX symlinks smart in any way.
Hardlinking directories would be a solution, but that is not
possible.
Wouldn't specifying the submodule path in the link object fit in well
here? Then each "link object" can represent a checked out tree from
the subproject in the superproject directory-structure.
Another thing:
With normal "$buildroot != $srcroot" environments, the source
can not be a subdirectory of the build directory.
This is true for symlinks and would also be corrected if we have a
(sparse) submodule checkout there in it's place.
BTW, build project commits probably should not depend on any
history of other build commits.
Why? Can you give an example here.
quoted
Link: /headers/lib1.h -> <lib1-commit3>/src/lib1.h
Link: /bin/lib1.so -> <build1-commit>/i386/lib1/lib1.so
Link: /bin/app1 -> <build1-commit>/i386/app1/app1
<lib1-commit1>, <lib1-commit2> and <lib1-commit3> should be the same,
dictated by the app1 project.
I do not see any problem here. Symlinks are stored in the git repository.
As the AppBuild commit depends on App and LibBuild submodule commits, the
symlinks always should be correct.
The main reason for these "links" are for versioning purposes: the
uniqe SHA1 of the "link" representing a tree/blob in a version of the
submodule should be "included" in the supermodules commit. Symlinks
won't give that at all.
quoted
How do the super-projects in this case get access to the blobs pointed
by the links - transparent or explicit in the build-process?
Submodules should automatically be checked out when checking out the
supermodule. So the blobs should already be there.
Or do I miss something?
Probably not as that was a piece of the puzzle that I was missing.
From: Andreas Ericsson <hidden> Date: 2016-08-11 19:51:17
Martin Waitz wrote:
hoi :)
On Fri, Dec 01, 2006 at 02:05:33PM +0100, sf wrote:
quoted
quoted
On Fri, Dec 01, 2006 at 01:09:49PM +0100, sf wrote:
quoted
Martin Waitz wrote:
quoted
So you not only store your submodule HEAD commit in the supermodule
when you do commit to the supermodule, it also means that your
submodule HEAD will be updated when you update your supermodule.
Why the magic? The typical workflow in git is
1. You work on a branch, i.e. edit and commit and so on.
2. At some point, you decide to share the work you did on that branch
(e-mail a patch, merge into another branch, push upstream or let it by
pulled by upstream)
3. Other people want to use your new work.
Sorry, if that was not obvious: You actually procceed with one of the
options I listed in Step 2. What I wanted to state is that with git you
do not mix up committing (which is local to your repository and your
branch) and publishing.
I guess you are refering to not mix up committing to the submodule and
updating the supermodule index.
These are really two separate steps, I just combined them above because I
wanted to put emphasis on the other part: it is not a one-way flow, it
is bidirectional, so your HEAD would have to changed if the supermodule
gets updated.
And I consider changing HEAD, without looking at the branch it points
to, to be a bad thing.
So a commit in the supermodule turns into a commit in the submodule?
That's just plain wrong. If it doesn't, why would the submodule HEAD
have to change?
--
Andreas Ericsson andreas.ericsson@op5.se
OP5 AB www.op5.se
In contrast, a submodule that we don't fetch is an all-or-nothing
situation: we simply don't have the data at all, and it's really a matter
of simply not recursing into that submodule at all - much more like not
checking out a particular part of the tree.
If you do not want to fetch all of the supermodule then do not fetch the
supermodule.
So why do you want to limit it? There's absolutely no cost to saying "I
want to see all the common shared infrastructure, but I'm actually only
interested in this one submodule that I work with".
If you need a common infrastructure to be able to work with the
submodule, then the submodule is not independent of of the supermodule.
I see a contradiction in your requirements.
Also, anybody who works on just the build infrastructure simply may not
care about all the submodules. The submodules may add up to hundreds of
gigs of stuff. Not everybody wants them. But you may still want to get the
common build infrastructure.
See above.
In other words, your "all or nothing" approach is
(a) not friendly
and
(b) has no real advantages anyway, since modules have to be independent
enough that you _can_ split them off for other reasons anyway.
So forcing that "you have to take everything" mentality onyl has
negatives, and no positives. Why do it?
(There have been lots of use cases for shallow clones but for a long
time git did not support them).
If you can extend this partial fetch feature to the non-subproject case
I would agree with your reasoning. What makes the subprojects so special
in this regard. Do I have to turn a plain tree into a subproject to be
able to ignore it? Once you can restrict fetches to parts of the
contents you get the ability to restrict fetches to the "common
infrastructure" and selected submodules for free.
Regards
Stephan
From: Martin Waitz <hidden> Date: 2016-08-11 19:51:57
hoi :)
On Fri, Dec 01, 2006 at 11:55:03PM +0100, Josef Weidendorfer wrote:
What about my other argument for a submodule namespace:
You want to be able to move the relative root path of a submodule
inside of your supermodule, but yet want to have a unique name
for the submodule:
- to be able to just clone a submodule without having to know
the current position in HEAD
- more practically, e.g. to be able to name a submodule
independent from any current commit you are on in the supermodule,
e.g. to be able to store some meta information about a submodule:
- "Where is the official upstream of this submodule?"
you can always have a bare repository for all used modules lying around
in some defined location. There is no need for a unique submodule-name.
--
Martin Waitz
On Fri, Dec 01, 2006 at 06:33:12PM +0100, Stephan Feder wrote:
quoted
We are in agreement about two fundamental parts of the implementation
and their meaning:
1. A submodule is stored as a commit id in a tree object.
2. Every object that is reachable from the submodule's commit are
reachable from the supermodule's repository.
Correct.
Good. For me that is the main point. As I said before the user interface
is not so important because it can be changed anytime, but to change the
object database later is close to impossible.
...
Let's see if I understand you correctly:
You don't want to create an additional .git directory for the submodule
and just handle everything with one toplevel .git repository for the
whole project.
Yes.
Without the .git directory, you of course do not have refs/heads inside
the submodule.
Correct..
So this is a different user-interface approach to submodules when
compared to my approach. But the basis is the same and both could
inter-operate.
Big YES.
Now your submodule is no longer seen as an independent git repository
and I think this would cause problems when you want to push/pull between
the submodule and its upstream repository.
You can always pick a single commit or several commits out of a larger
repository and have a complete git repository.
And I already explained how to push and pull even from within superprojects.
No technical problems, but UI-problems because now your submodule is
handled completly different to a "normal" repository.
Yes and no. You can always have branches that are only concerned with
submodules' code, say, in refs/heads/submodules/<submodule>/.
"submodules" here is simply an example and has not deeper meaning. You
could call it foo or whatever you like. Or you could use
refs/heads/<submodule>/ if it suits you.
But if you mean the submodule as seen from the supermodule, then there
is a difference. Naturally, because the concept of submodules is new to git.
quoted
quoted
Yes, but if it would have been integrated as a submodule it obviously
would have been committed to the xdiff submodule inside the git
repository.
No. The xdiff submodule would only exist as part of the git repository.
But you could still call the "xdiff" part of the git repository a
submodule. And then changes to the xdiff directory result in a new
submodule commit, even when there is no direct reference to it.
So you'd still "commit to the xdiff submodule".
Let's make certain that we understand each other. I see a clear
distinction between the submodule code in a supermodule branch (commits
in the supermodule's tree and nothing else) and submodule branches which
are independent of the superproject. Supermodule branches and submodule
branches do not interact, only if I want them to.
quoted
You could, f.e., access the xdiff commit in git HEAD as HEAD:xdiff//
(again my proposed syntax). HEAD:xdiff//~2:xemit.c would give you the
grandparent of xemit.c in the xdiff submodule.
git-cat-file commit HEAD:xdiff already works out of the box (even
cat-file tree to get the submodule tree). But up to now revision
parsing follows the file name only once.
What about just separating things with "/"?
commit HEAD
tree HEAD/
blob HEAD/Makefile
commit HEAD/xdiff
tree HEAD/xdiff/
blob HEAD/xdiff~2/xemit.c
this may add some confusion when used with hierarchical branches, but
it's still unique:
refs/heads/master/xdiff/xemit.c
Just use as many path components until a matching reference is found,
then start peeling.
Or just use / between super and submodule:
refs/heads/master:xdiff/xemit.c
I think this is easier to read then
refs/heads/master:xdiff//:xemit.c
The double slashes is the only way I can think of that clearly indicates
that I do not mean the contents named by the path, but the commit that
you find there. Once you have named a commit in that way, you can
continue to apply other revision naming suffixes, paths, and so on.
Let's try. What does git cat-file -p
master:dir/sub//^^^:sub/dir/sub//^:dir/file mean?
Explanation: Take branch master and go to path dir/sub. There you will
find a commit. Take its grand-grandparent and go to path sub/dir/sub
(the first sub is a subproject as well but we do not care). There you
will, again, find a commit. Take its parent and go to path dir/file
which happens to be a blob the contents of which you want to cat.
In reality you will never see these kinds of complex paths. Have you
ever seen something like git cat-file -p
bd2c39f58f915af532b488c5bda753314f0db603~12^{commit}^2^5~8^2~308:README ?
quoted
If you mean the submodule repository created by init-module I
understand. But why create this "helper repository at all"?
From: Martin Waitz <hidden> Date: 2016-08-11 19:52:19
hoi :)
On Sat, Dec 02, 2006 at 12:17:44AM +0100, Josef Weidendorfer wrote:
After some thinking, a submodule namespace even is important for checking
out only parts of a supermodule, exactly because the root of a submodule
potentially can change at every commit.
have you ever thought about the idea that the location may be an
important thing to consider for your decision.
Perhaps the submodule is now used for something else (this is why it was
moved) and that now you'd like to keep it?
Anyway, you can just create several supermodules or implement generic
partial tree support for git. I do not see any reason to special case
submodules here.
--
Martin Waitz
From: Daniel Barkalow <hidden> Date: 2016-08-11 19:53:00
On Tue, 28 Nov 2006, Sven Verdoolaege wrote:
On Tue, Nov 28, 2006 at 01:37:54PM -0500, Daniel Barkalow wrote:
quoted
If submodule was the only thing that got changed, it's not dirty; if it
were dirty, some of its contents would also have gotten changed.
For me, the commit is the only "content" of the subproject that the
superproject should care about, so the submodule being dirty or not
is completely irrelevant (for committing), but it seems you see the
subproject more as a (working) tree than as a commit.
I think we agree on the tree/commit/object database model part.
I think we disagree on how the working *directories* relate. I see the
checked-out state of a submodule as being relevant to the checked-out
state of the supermodule, such that dirty state in the submodule directory
is dirty state in the supermodule directory.
quoted
Surely:
"git commit submodule/foo bar"
I wouldn't dream of doing such an operation, because it doesn't make
sense to me. (So as far as I'm concerned, you can make it do whatever
you'd like it to do.) You can only commit the subproject as a whole.
I'm thinking that users of subprojects will often want to work on the
subprojects rather than exclusively using commits prepared by other
people, and it's too much trouble to have to do the work in a repository
for just the subproject and pull it into the superproject's submodule to
test it. So the submodule working directory needs to function as a working
directory for the subproject. Then
"cd submodule; git commit foo"
does the obvious thing, but that should be the same as
"git commit submodule/foo" (since it normally is)
and then it makes sense to let you do multiple commits with a single
command when the paths end in different modules, since that's obviously
what you're requesting, and then -a must do all of them.
-Daniel
Linus Torvalds wrote:
...
> Think of it this way: one common use for submodules is really to just
> (occasionally) track somebody elses code. The submodule should be a
> totally pristine copy from somebody else (ie it might be the "intel
driver
> for X.org" submodule, maintained within intel), and the supermodule just
> refers to it indirectly (ie the supermodule might be the "Fedora Core X
> group" which contains all the different drivers from different people).
Could you please be a little bit more specific about how you would store
the "pristine copy". There seems to be some agreement to store the
commit id of the submodule instead of a plain tree id in the
supermodules tree object, and that all objects that are reachable from
this commit are made part of the supermodule repository (either fetched
or via alternates). Do you agree?
...
> A supermodule can never "contain changes" to a submodule. A supermodule
> would always just point to the submodule, and not have any changes
> what-so-ever of its own. The submodule is self-sufficient, and always
> contains all its _own_ changes.
That is one of the points Martin Waitz and I are discussing.
If I understand you correctly you cannot make any changes to the
submodules code _in the supermodule's repository_, no bugfixes, no
extensions, no adaptions, nothing. Do you mean that?
That would be a third alternative. In my opinion the usefulness of
submodules would be unnecessarily restricted if it comes to the choice
of either using the code from upstream as is or do not use submodules at
all. What is the point of the restriction?
Regards