From: Junio C Hamano <hidden> Date: 2016-06-15 22:42:16
Linus Torvalds [off-list ref] writes:
I'd suggest adding a new kind of object ("gitlink") which has some
well-specified format (20-byte SHA1 + ASCII C string "name" - the name
translation to external repository would be done in the .git/config file
of the "outer" project). Then a special file mode to indicate that in the
"struct tree", and support for "git-update-cache" to understand how such
an object is really tied into the "<pathname>/.git/HEAD" file rather than
the rest of the directory contents.
Then a "git fetch" would have to be taught to recursively fetch the other
subproject when the "gitlink" changes.
There are two positive properties about this setup, and one
negative:
+ The contained project is kept totally independent and does
not have to know it is contained.
+ The tree for the contained project can be rooted anywhere in
the containing project's tree.
- The contained project cannot be rooted at the same level or
higher than the containing project; the containing project
can only delegate a whole subdirectory to the contained
project.
The "embedded software" example Simon originally suggested can
be represented with the above. I'll think aloud for a while
here, because I am of a slow kind who needs a more-or-less
concrete illustration to understand what is being discussed
(that is primarily why I have not said anything on this topic so
far).
The "containing" project would have a handful "gitlink" objects
among other things. The toplevel tree object from a commit in
such a project might look like this (mode bits 0160000 is
S_IFDIR|S_IFLNK, which is what this thing is):
$ git ls-tree HEAD
0100644 blob 012345... Makefile
0100644 blob 123456... README
0160000 link 234567... gcc-4.0
0160000 link 345678... linux-2.6
0040000 tree 456789... src
$ git cat-file -t 345678
link
$ git cat-file link 345678
commit 87530db5ec7d519c7ba334e414307c5130ae2da8
url git://...torvalds/linux-2.6.git/
The upstream Linux 2.6 repository.
$ cd linux-2.6 && git-rev-parse --verify HEAD
87530db5ec7d519c7ba334e414307c5130ae2da8
URL will be used as a suggestion for people who cloned this tree
to set up their repository. The place and method you clone from
Linus tree might be different, so this has to stay suggestion
and should be overridable by the repository owner. And to help
people at an unusual location you could have textual comment at
the end, just like tags.
How would this get set up initially? Here is one way.
$ git init-db
$ edit Makefile README src/*
$ git clone git://...torvalds/linux-2.6.git/ linux-2.6
$ git clone git://.../gcc-4.0.git/ gcc-4.0
$ link=$(echo 'The upstream Linux 2.6 repository.' |
git-mklink linux-2.6)
$ git update-index --add --cacheinfo 0160000 $link linux-2.6
$ : ;# same for gcc-4.0
$ git add . ;# add the rest as usual
$ git commit
I presume that the index file have the "gitlink" object just
like in a tree object. The usual merge rules would apply to
those index entries; we should be able to treat gitlinks just
like we handle symlinks.
Interesting would be "git checkout-index linux-2.6" (or what
"git read-tree -u" does in this "containing" project for
linux-2.6 subdirectory). After descending into linux-2.6, it
should not just do "git reset --hard $commit" for the commit
recorded in the gitlink (the user may have local modifications
in the subtree). Doing "git update-ref HEAD $commit" there is
not quite right either because the index there would then need
to be adjusted as well. Perhaps the real core level commands
such as "checkout-index" and "read-tree -u" should fail when the
subproject tree is dirty, just like "read-tree -m old new" does
not always have to succeed.
What does "git-diff-index/git-diff-tree/git-diff-files" would do
with them?
$ git-diff-files linux-2.6
would compare the commit recorded in the link and what is
checked out in the linux-2.6/.git/HEAD and report that
difference. So do other git-diff-* siblings. At the core level
we do not have to recurse and look at linux-2.6/.git/index (we
may end up doing so at the end, I dunno; initially we said at
the core level we do not have to generate patches but we ended
up having -p option go all of git-diff-* siblings).
Fetching/cloning at the core level is easy. "git-fetch-pack"
would just need to do one level, but Porcelains need to address
how to actually arrange the subprojects cloning to happen, which
is harder.
"git clone" would say: "Ah, now I see these gitlinks; we need to
clone them. linux-2.6 directory needs to be populated with
commit 87530d from git://...torvalds/linux-2.6.git/ repository.
Would this work for you, or would you use different mirror?"
and then it clones the repository and sets linux-2.6/.git/HEAD
to the named commit and does a checkout. The URL used for this
actual subcloning would need to be stored somewhere in $GIT_DIR/,
perhaps in config as you suggested. I do not think we need a
separate name for it -- we can probably say "linux-2.6" for this
(i.e. use the pathname itself as the key).
What happens if the containing project wants to move these
gitlinks (or remove them)? When checking out such a commit with
"git-read-tree -u", would the subproject directory be wiped out
(again, such a "read-tree" would be prevented if it would result
in information loss)?
All of this sounds quite a lot of change with brittleness.
Now I'll think aloud about a completely different design.
We could simply overlay the projects. I think this is what
Johannes suggested earlier.
You keep one branch for each "subproject", and make commits into
each branch (i.e. if you modified files for the upstream kernel,
the change is committed to the branch for linux-2.6 subproject),
but when checking things out, you do an equivalent of octopus
merge across subprojects.
One downside of this approach is we cannot re-root the
subprojects until we update read-tree and write-tree, but I
suspect that would be a lot smaller change. Once that is done,
we could:
$ git init-db
$ mkdir linux-2.6
$ H=$(git-fetch-pack -k git://...torvalds/linux-2.6.git/ master)
$ echo $H >.git/refs/heads/kernel
$ : ;# same for gcc-4.0
$ cat .git/config <<EOF
[core]
branchroot = linux-2.6 for kernel
branchroot = gcc-4.0 for gcc
EOF
$ git add . ;# add src and stuff
$ git commit ;# commits only the scaffolding into "master"
So far, we fetched the kernel and gcc HEAD with needed objects
and stored them into separate branches. Then:
$ git setup-overlay embed master kernel gcc ;# works like an octopus
The setup-overlay command would create a new branch "embed" to
hold an octopus merge across named branches "master", "kernel",
and "gcc", and mark that the repository is in a funny "overlay"
mode, in which various commands work differently from usual:
$ edit linux-2.6/CREDITS gcc-4.0/COPYING Makefile
$ git commit -a
The "commit" needs to be taught to look at what setup-overlay
left for us, pick out paths that belong to each constituent
branch and do a re-rooting write-tree, for each branch.
This would keep changes to subprojects independent painlessly,
but we would also need a way to tie the versions of subprojects
together (i.e. "this version of src was done with this
particular version of linux-2.6"). This can be done by
committing the octopus to "embed" branch. Probably easiest
would be to make one commit each to modified constituent branch,
and after that make another commit to "embed" to commit the
octopus to keep track of the aggregation --- the commit would
have the parents set to the previous embed and top commit of
each constituent branch.
If we do not need re-rooting (e.g. redo your slurping gitk into
git.git), I think all of the above can be done without any core
changes. It would be a lot of Porcelainish work, but I suspect
the core impact would be smaller.
+ The contained project is kept totally independent and does
not have to know it is contained.
+ The tree for the contained project can be rooted anywhere in
the containing project's tree.
Right.
- The contained project cannot be rooted at the same level or
higher than the containing project; the containing project
can only delegate a whole subdirectory to the contained
project.
Yes.
However, I think this is actually a _huge_ advantage.
The thing is, if you do the contained projects as "union projects" as you
suggest, I will bet that it will really really suck, because it ends up
losing the two positives above.
In particular, any real independent project will have it's own "Makefile"
or "configure-in", and often its own "src" subdirectory or other
pseudo-standard names.
And the "contained project as a link" approach has zero problems with that
at all, exactly because it keeps the projects clearly separate - just
linked (one way).
What does "git-diff-index/git-diff-tree/git-diff-files" would do
with them?
I would actually argue that git itself wouldn't do a whole lot with them.
There are real advantages to seeing only the diffs wrt _one_ of the
projects, and I'd argue that
git-diff-*
would actually act like they now act for directories that they don't
recurse into, ie you'd see something like
:160000 160000 5eb57670... 3f1a42aa... M sub-project
and it would be up to higher-level porcelain to recurse.
Why? Partly because that's actually likely enough for a lot of users: you
_can_ use just the raw git programs by just doing
cd sub-project
git diff
..
git commit
and so technically you aren't really missing a lot. The capabilities are
there, you just have to do some more by hand (but in many ways that is
_good_: it makes it obvious that you're really committing a _different_
subproject).
The other reason? A lot of the git infrastructure really does only work on
the "one project" level. The programs work with _one_ index, not two.
Reading two trees is perfectly possible, but unless you keep them in
separate stages, you can't separate them afterwards. IOW, trying to be
recursive really does end up being a big change, for very little gain (and
for a lot of potential bugs and instability).
In contrast, doing it at a higher level means that you have a simple and
reliable lower level that you can trust. Layering is good.
Fetching/cloning at the core level is easy. "git-fetch-pack"
would just need to do one level, but Porcelains need to address
how to actually arrange the subprojects cloning to happen, which
is harder.
"git clone" would say: "Ah, now I see these gitlinks; we need to
clone them.
Actually, I would say no - that's actually not a "clone" operation so much
as a "checkout" operation. There are strong arguments that you should
_not_ clone sub-projects when you clone the top-level project: there's no
reason to. Anybody else who clones it will have all the information you
have, so cloning th esub-project is just extra work.
So only if you actually check it out (which is often in practice the
second stage of the cloning, of course) do you want to fetch the
subproject too. But even then you might want to ask the user (he may have
a local repository for that sub-project somewhere else, so going to the
"canonical name" might be the wrong thing to do - and he might not even
care, because he might want to work _just_ on the top-level project).
Now I'll think aloud about a completely different design.
We could simply overlay the projects. I think this is what
Johannes suggested earlier.
You keep one branch for each "subproject", and make commits into
each branch (i.e. if you modified files for the upstream kernel,
the change is committed to the branch for linux-2.6 subproject),
but when checking things out, you do an equivalent of octopus
merge across subprojects.
I think this one has serious disadvantages:
- it's much less obvious when there are common names and especially
common subdirectories.
- in _practice_, almost all sub-projects are kept in sub-directories. Are
you doing to change the sub-project git tree? How are you going to
merge back to the original sub-project?
- iow, I think this only works for sub-projects that are totally
controlled by the top-level project - in which case they might as well
just be totally merged into the top level (the way we did with the
"tools" project, and largely with "gitk").
in the "gitk" case, we could actually continue to keep gitk a separate
project, but that was really fortunate: it's purely because gitk ends up
being a single file, with no Makefile at all to build it independently
etc. The moment we integrated the "tools" sub-project into git, we lost
the ability to do that, exactly because they now needed to share Makefiles
etc, making all further development very inter-twined.
Put another way: the moment you have linkages going both ways between the
subproject and the top-level project, it's no longer two separate
projects. At that point, it in practice becomes one, since the sub-project
can no longer do independent development without merging becoming a big
issue.
The advantage of having a "git link" is exactly the fact that the
dependency goes only one way. The subproject remains truly independent.
Linus
From: A Large Angry SCM <hidden> Date: 2016-06-15 22:42:16
So far I've not seen any convincing arguments why the sub-projects can
not be managed by the Makefile, or equivalent, of the super-project.
Particularly when the sub-projects have a life of their own.
So far I've not seen any convincing arguments why the sub-projects can not be
managed by the Makefile, or equivalent, of the super-project. Particularly
when the sub-projects have a life of their own.
Now, from a developer standpoint I actually agree with you. I find
sub-projects totally useless - I'm much happier just having separate
trees.
The advantage (as far as I can tell) of sub-projects is not that they are
easier to develop in, but that it's a total nightmare for the technical
_user_ to download ten different projects from ten different sites, and
configure them properly and install them in the right order, and keep them
up-to-date.
There are projects that I simply gave up even trying to track: I wasn't
interested in being a developer per se, but I _was_ interested in trying
to test and give feedback to the current development tree - but it was
just too damn confusing to get it working.
If I could have just done a "git clone <top-level>" to get it all, I'd
have been a much more productive user.
This is why I think sub-projects are more about "git checkout" and an
automated "git fetch" than anything else. Doing actual development etc you
can easily do one project at a time. "git diff" and "git commit" wouldn't
need any real ability to recurse into subprojects and try to make it
seamless. And if you do a "git pull" that needs to do anything but
fast-forward, you might as well resolve the sub-projects one by one.
Linus
From: A Large Angry SCM <hidden> Date: 2016-06-15 22:42:16
Linus Torvalds wrote:
On Sat, 14 Jan 2006, A Large Angry SCM wrote:
quoted
So far I've not seen any convincing arguments why the sub-projects can not be
managed by the Makefile, or equivalent, of the super-project. Particularly
when the sub-projects have a life of their own.
Now, from a developer standpoint I actually agree with you. I find
sub-projects totally useless - I'm much happier just having separate
trees.
The advantage (as far as I can tell) of sub-projects is not that they are
easier to develop in, but that it's a total nightmare for the technical
_user_ to download ten different projects from ten different sites, and
configure them properly and install them in the right order, and keep them
up-to-date.
There are projects that I simply gave up even trying to track: I wasn't
interested in being a developer per se, but I _was_ interested in trying
to test and give feedback to the current development tree - but it was
just too damn confusing to get it working.
If I could have just done a "git clone <top-level>" to get it all, I'd
have been a much more productive user.
$ make get_sub_components
This can work with most any SCM (depending on your environment), is
amazingly flexible, and does not require special support in the SCM.
The "get" rule for each sub-project could be something like:
git_sub-project:
mkdir sub-project
cd sub-project
git-init-db
git-fetch <fetch-options> <repository> <refspec>
git-checkout <branch>
$(MAKE) get_sub_components
This is why I think sub-projects are more about "git checkout" and an
automated "git fetch" than anything else. Doing actual development etc you
can easily do one project at a time. "git diff" and "git commit" wouldn't
need any real ability to recurse into subprojects and try to make it
seamless. And if you do a "git pull" that needs to do anything but
fast-forward, you might as well resolve the sub-projects one by one.
And all of this can be done today, without changing git, with more
flexibility, with Make rules.
From: Alexander Litvinov <hidden> Date: 2016-06-15 22:42:16
On Saturday 14 January 2006 14:59, Junio C Hamano wrote:
Now I'll think aloud about a completely different design.
We could simply overlay the projects. I think this is what
Johannes suggested earlier.
You keep one branch for each "subproject", and make commits into
each branch (i.e. if you modified files for the upstream kernel,
the change is committed to the branch for linux-2.6 subproject),
but when checking things out, you do an equivalent of octopus
merge across subprojects.
If I cleary understand this idea it is NOT that I dreaming about. Almost all
our sub-projects are used in more than one project (imaging network layer
library). So variant with gitlink is that I willing.
From: Alex Riesen <hidden> Date: 2016-06-15 22:42:16
On 1/14/06, Linus Torvalds [off-list ref] wrote:
quoted
So far I've not seen any convincing arguments why the sub-projects can not be
managed by the Makefile, or equivalent, of the super-project. Particularly
when the sub-projects have a life of their own.
Now, from a developer standpoint I actually agree with you. I find
sub-projects totally useless - I'm much happier just having separate
trees.
The advantage (as far as I can tell) of sub-projects is not that they are
easier to develop in, but that it's a total nightmare for the technical
_user_ to download ten different projects from ten different sites, and
configure them properly and install them in the right order, and keep them
up-to-date.
There are projects that I simply gave up even trying to track: I wasn't
interested in being a developer per se, but I _was_ interested in trying
to test and give feedback to the current development tree - but it was
just too damn confusing to get it working.
If I could have just done a "git clone <top-level>" to get it all, I'd
have been a much more productive user.
This is why I think sub-projects are more about "git checkout" and an
automated "git fetch" than anything else. Doing actual development etc you
can easily do one project at a time. "git diff" and "git commit" wouldn't
need any real ability to recurse into subprojects and try to make it
seamless. And if you do a "git pull" that needs to do anything but
fast-forward, you might as well resolve the sub-projects one by one.
That is exactly how subprojects are used in Perforce- and ClearCase-like SCM:
the working tree is "configured" to contain the super-project (build
configuration)
and the actual work happens in the subproject and _only_ there. The mentioned
systems even have heavily used permission system just to prevent either
checkout or commit anywhere outside the area of responsibility of a developer.
(The "permissions" are somehow pointless in git context, just mentioned them
to underline the main point).
From: Andreas Ericsson <hidden> Date: 2016-06-15 22:42:16
Alexander Litvinov wrote:
On Saturday 14 January 2006 14:59, Junio C Hamano wrote:
quoted
Now I'll think aloud about a completely different design.
We could simply overlay the projects. I think this is what
Johannes suggested earlier.
You keep one branch for each "subproject", and make commits into
each branch (i.e. if you modified files for the upstream kernel,
the change is committed to the branch for linux-2.6 subproject),
but when checking things out, you do an equivalent of octopus
merge across subprojects.
If I cleary understand this idea it is NOT that I dreaming about. Almost all
our sub-projects are used in more than one project (imaging network layer
library). So variant with gitlink is that I willing.
Then it isn't so much a subproject as a separate project of its own.
Otherwise glibc would be a subproject of pretty much everything and
that's hardly a sane setup.
--
Andreas Ericsson andreas.ericsson@op5.se
OP5 AB www.op5.se
Tel: +46 8-230225 Fax: +46 8-230231
The "containing" project would have a handful "gitlink" objects
among other things. The toplevel tree object from a commit in
such a project might look like this (mode bits 0160000 is
S_IFDIR|S_IFLNK, which is what this thing is):
$ git ls-tree HEAD
0100644 blob 012345... Makefile
0100644 blob 123456... README
0160000 link 234567... gcc-4.0
0160000 link 345678... linux-2.6
0040000 tree 456789... src
$ git cat-file -t 345678
link
$ git cat-file link 345678
commit 87530db5ec7d519c7ba334e414307c5130ae2da8
url git://...torvalds/linux-2.6.git/
The upstream Linux 2.6 repository.
$ cd linux-2.6 && git-rev-parse --verify HEAD
87530db5ec7d519c7ba334e414307c5130ae2da8
URL will be used as a suggestion for people who cloned this tree
to set up their repository.
I'd prefer to have the objects needed to get the linux-2.6 tree in the
object db of the containing project. Then "url" is not needed, and you
could directly use the commit as value for the link. i.e.
$ git ls-tree HEAD
0100644 blob 012345... Makefile
0100644 blob 123456... README
0160000 link 435363... gcc-4.0
0160000 link 87530d... linux-2.6
0040000 tree 456789... src
(You could now rename "link" to "commit", but it would break the
layout.)
Moreover I prefer the the link approach over the bind method. The
reason is, that binds use information from the commit object to build
the wc other than the tree. Moreover the condition that the
"containing" tree must not have an entry named linux-2.6 is handled
implicitly with links.
Please correct me if I'm wrong somewhere. It's some time ago I read the
patches and this thread. This mail is the result of some thoughts in my
vacation.
Best regards
Uwe
--
Uwe Zeisberger
http://www.google.com/search?q=1+year+divided+by+3+in+seconds