From: Junio C Hamano <hidden> Date: 2016-06-15 22:43:11
Jan Hudec [off-list ref] writes:
IMHO it makes more sense to fetch during fetch of superproject:
- If you don't fetch the superproject, it won't start refering to
unavailable commit of subproject. So should only need to fetch subproject
after fetching superproject.
Eh, I was suggesting that the subproject fetch would come after
checkout in "fetch and then checkout" sequence of the
superproject, and if you are arguing against it, you should
justify why it should not happen before checkout, as we both
agree it should come after fetch of superproject. Your argument
is like saying you have to git-init before doing anything so
you should fetch when you git-init. That's not a justification.
- If you fetch from more than one location, you want to fetch subproject
from location corresponding to where you fetch superproject from.
Not at all. There is no reason to believe that the case that
superproject and subproject come from related URLs is more
common. One of the reasons to do a separated project
organization is to allow looser bindings of the project from
project administrative viewpoint. The integrator may not
necessarily have any control over what the subproject guys do,
and more importantly, the subproject people do not even care nor
be aware of the fact that their project is sometimes bound
inside other peoples' superprojects. Think of the embedded
appliance vendor binding the kernel, libc and busybox in their
superproject that holds them together with the build
infrastructure. The kernel folks certainly do not particularly
care about the vendor.
From: Sven Verdoolaege <hidden> Date: 2016-06-15 22:43:11
On Sun, May 20, 2007 at 11:33:17AM -0700, Junio C Hamano wrote:
Jan Hudec [off-list ref] writes:
quoted
- If you fetch from more than one location, you want to fetch subproject
from location corresponding to where you fetch superproject from.
Not at all. There is no reason to believe that the case that
superproject and subproject come from related URLs is more
common. One of the reasons to do a separated project
organization is to allow looser bindings of the project from
project administrative viewpoint. The integrator may not
necessarily have any control over what the subproject guys do,
and more importantly, the subproject people do not even care nor
be aware of the fact that their project is sometimes bound
inside other peoples' superprojects. Think of the embedded
appliance vendor binding the kernel, libc and busybox in their
superproject that holds them together with the build
infrastructure. The kernel folks certainly do not particularly
care about the vendor.
I think what Jan means is that if there are two (or more) copies
of the superproject then it is more likely that the subproject
commit can be found in the subproject repo "pointed to" (e.g.,
through my submodule.*.url) by the superproject repo you fetched from.
skimo
From: Jan Hudec <hidden> Date: 2016-06-15 22:43:11
Hello,
I gave the problem some more thought, and though I follow up on my previous
comment below, I can now see this:
So far it was discussed what should happen in fetch (+ checkout). But I think
the following are the interesting cases. Please read ALL before responding,
they are in somewhat random order:
For following, assume there is a repo of project super has two branches,
master and next. The next branch adds subproject sublib. In that state, I fetch
refs/heads/*:refs/remotes/origin/* and check out master. So I have
a repository, that contains revision with submodule, but I did not check it
out yet.
- Some time later, *without* fetching again, I simply
git checkout --submodules -b feature1 remotes/origin/next
Obviously it needs to give me the module.
- Should that checkout work without network access?
- Ok, now I start hacking on feature1 and find a bug in sublib, that I need
to fix for it to work. Therefore I change something within sublib.
However few days later I am asked to fix a bug in stable release of super.
Therefore I: git checkout master Now, where does sublib go? It contains
precious data!
For the worst case situation assume, that the master branch also has
directory sublib, so it can't stay where it was as unversioned.
- The fix in master is done, back to our feature1, right? git checkout
--submodules feature1 Obviously re-fetching from upstram won't work. The
head feature1 now refers to a commit that I made and only exists localy.
- Now the maintainer of super wants to test the feature1. However sublib
upstream did not accept the bugfix yet (and is perhaps waiting for
confirmation, that the fixed version really works well for super, so we
have to test).
Therefore I push feature1 to my public repo, set up a public repo with my
fixes to sublib and configure my public super repo to know about it.
The maintainer already has a repo of super including sublib submodule. But
when he pulls from me, he does not have the repo with my fixes.
- The maintainer reviewed my feature1 and now needs to work on feature2.
That however requires new upstream version of super. Therefore he needs to
pull alternatively from both upstream and my repo with super, depending on
what he works on.
For the most complex case, assume here that I add more fixes to sublib
while author of feature2 uses more and more bleeding edge stuff, so the
maintainer really needs futher changes in sublib from both repos.
- Also git has to fail safe if I forget to push the sublib, so when the
maintainer tries to pull super, the refered revision of sublib simply
won't be found.
I am not sure how to handle these cases. But they are cases that can happen
in real life and should be handled somehow. Even if some of them just require
some manual configuration.
Here is one possible idea:
We could store the GIT_DIR of submodule within the GIT_DIR of the
superproject instead of the submodule directory itself. So instead of:
/
/.git
/subdir
/subdir/.git
There would be:
/
/.git
/subdir
/.git/submodules/submodule-name.git
This would require changes to the logic how git finds GIT_DIR (which would be
really deep change), but it would provide place to store the submodule data
while the submodule is not being checked out.
This does not address the last two cases above with mutliple sources, each
containing some revisions. There I see two options:
- The submodules are fetched during superproject fetch (based on them being
configured, even if they are not checked out) and the URL might depend on
url configured for superproject. That is:
git fetch --submodules foobar
would do roughty:
for GIT_DIR in $GIT_DIR/submodules/*.git; do
git fetch foobar || git fetch
done
So if you configured source of the same name for the subproject, it would
be pulled, otherwise the default one would.
Checkout would then be local-only operation, because subprojects are
up-to-date.
- The superproject checkout would try fetching all sources of the
subproject, until the requested revision is found.
This could be extended to normal checkout doing it as well --
"git checkout sha1" would try fetching all configured sources if the
revision was not found.
Perhaps we could actually do both. That is, "git fetch --subprojects" to
also fetch all of "$GIT_DIR/submodules/*.git" and checkout to try fetching if
it can't find the desired revision.
On Sun, May 20, 2007 at 11:33:17 -0700, Junio C Hamano wrote:
Jan Hudec [off-list ref] writes:
quoted
IMHO it makes more sense to fetch during fetch of superproject:
- If you don't fetch the superproject, it won't start refering to
unavailable commit of subproject. So should only need to fetch subproject
after fetching superproject.
Eh, I was suggesting that the subproject fetch would come after
checkout in "fetch and then checkout" sequence of the
superproject, and if you are arguing against it, you should
justify why it should not happen before checkout, as we both
agree it should come after fetch of superproject. Your argument
is like saying you have to git-init before doing anything so
you should fetch when you git-init. That's not a justification.
It definitely has to come after fetch on superproject. My original thought
was, that it would be weird if it was part of the checkout itself, meaning
even checkout that does not follow a fetch. However I thought about it some
more and that might conflict with other requirements.
quoted
- If you fetch from more than one location, you want to fetch subproject
from location corresponding to where you fetch superproject from.
Not at all. There is no reason to believe that the case that
superproject and subproject come from related URLs is more
common. One of the reasons to do a separated project
I definitely don't think it's more common. But it's the harder case and it
might happen. Generally it will happen if some people work on both the
superproject and the subproject. Of course the argument is that than it
should not be separate projects, but maybe the teams just partly overlap.
Example of this situation is given above. IMHO it needs to be handled
somehow (probably git would have to check all potential sources whether they
have the revision in question).
--
Jan 'Bulb' Hudec [off-list ref]
From: Sven Verdoolaege <hidden> Date: 2016-06-15 22:43:11
On Mon, May 21, 2007 at 06:59:38PM +0200, Jan Hudec wrote:
We could store the GIT_DIR of submodule within the GIT_DIR of the
superproject instead of the submodule directory itself. So instead of:
/
/.git
/subdir
/subdir/.git
There would be:
/
/.git
/subdir
/.git/submodules/submodule-name.git
I have this already, except that I use /.git/submodules/submodule-name/.git
because I was too lazy to figure out how to get clone to use the above
without using --bare, because --bare disables separate-remotes.
Is there any good reason, btw for --bare not to do separate-remotes ?
We could throw out a lot of speical cases, especially the --bare http
fetch if we would simply always do a separate-remotes.
This would require changes to the logic how git finds GIT_DIR (which would be
really deep change),
From: Jan Hudec <hidden> Date: 2016-06-15 22:43:12
On Mon, May 21, 2007 at 20:05:06 +0200, Sven Verdoolaege wrote:
On Mon, May 21, 2007 at 06:59:38PM +0200, Jan Hudec wrote:
quoted
We could store the GIT_DIR of submodule within the GIT_DIR of the
superproject instead of the submodule directory itself. So instead of:
/
/.git
/subdir
/subdir/.git
There would be:
/
/.git
/subdir
/.git/submodules/submodule-name.git
I have this already, except that I use /.git/submodules/submodule-name/.git
because I was too lazy to figure out how to get clone to use the above
without using --bare, because --bare disables separate-remotes.
Is there any good reason, btw for --bare not to do separate-remotes ?
We could throw out a lot of speical cases, especially the --bare http
fetch if we would simply always do a separate-remotes.
Glad to hear it. I didn't really have time to read through all the patches.
quoted
This would require changes to the logic how git finds GIT_DIR (which would be
really deep change),
Euhm.... I just add a symlink...
Yes. Except I am not sure mingw supports that.
--
Jan 'Bulb' Hudec [off-list ref]
From: Martin Waitz <hidden> Date: 2016-06-15 22:43:12
hoi :)
On Mon, May 21, 2007 at 06:59:38PM +0200, Jan Hudec wrote:
Here is one possible idea:
We could store the GIT_DIR of submodule within the GIT_DIR of the
superproject instead of the submodule directory itself. So instead of:
/
/.git
/subdir
/subdir/.git
There would be:
/
/.git
/subdir
/.git/submodules/submodule-name.git
This would require changes to the logic how git finds GIT_DIR (which would be
really deep change), but it would provide place to store the submodule data
while the submodule is not being checked out.
I agree that we need something like that.
We don't have to move the entire subproject.git into the superproject,
but we need to have all _referenced_ objects in the .git dir of the
superproject.
There are several possibilities to do so:
* move the entire .git dir
* move .git/objects
* explicitly copy all referenced objects
I have some experimental code to configure a per-subproject directory
in the superproject/.git as alternate object store for the submodule
to make the last two solutions possible. Perhaps I should dig it out again
and adapt it to current git.
If there is a 1:1 relationship between subproject and object store then
even efficient fsck and repack/prune are possible for the submodule without
loosing objects.
But such a 1:1 relationship is bad when you move subprojects to another
location (or include the same subproject several times in different
locations of the tree).
Perhaps the user should be able to choose which one he wants.
quoted
Not at all. There is no reason to believe that the case that
superproject and subproject come from related URLs is more
common. One of the reasons to do a separated project
I definitely don't think it's more common. But it's the harder case and it
might happen. Generally it will happen if some people work on both the
superproject and the subproject. Of course the argument is that than it
should not be separate projects, but maybe the teams just partly overlap.
I think it will be _very_ common to store super and subprojects in
related locations. First to be independent from third-party servers
while working on the superproject.
Second (and I think more important) because many times there will
be superproject related adaptations in the subproject. Yes they
are independent, and exactly for that reason the subproject upstream
maintainers may not take every change which is needed to satisfy the
superproject. We _now_ see that in all Linux distributions already.
So when you use superprojects to integrate several independent projects,
then the superproject maintainer/administrator should really keep a
clone of all subprojects handy on his site.
--
Martin Waitz
From: Jan Hudec <hidden> Date: 2016-06-15 22:43:12
On Mon, May 21, 2007 at 23:11:34 +0200, Martin Waitz wrote:
On Mon, May 21, 2007 at 06:59:38PM +0200, Jan Hudec wrote:
quoted
There would be:
/
/.git
/subdir
/.git/submodules/submodule-name.git
This would require changes to the logic how git finds GIT_DIR (which would be
really deep change), but it would provide place to store the submodule data
while the submodule is not being checked out.
I agree that we need something like that.
We don't have to move the entire subproject.git into the superproject,
but we need to have all _referenced_ objects in the .git dir of the
superproject.
There are several possibilities to do so:
* move the entire .git dir
* move .git/objects
* explicitly copy all referenced objects
I believe we really need entire .git dir. When the superporject checks out
revision which does not reference that subproject, we still need to preserve
not only the objects of subproject, but also the refs and config.
I have some experimental code to configure a per-subproject directory
in the superproject/.git as alternate object store for the submodule
to make the last two solutions possible. Perhaps I should dig it out again
and adapt it to current git.
If there is a 1:1 relationship between subproject and object store then
even efficient fsck and repack/prune are possible for the submodule without
loosing objects.
But such a 1:1 relationship is bad when you move subprojects to another
location (or include the same subproject several times in different
locations of the tree).
Perhaps the user should be able to choose which one he wants.
That's why there should be the extra level of indirection using .gitmodules.
It should map the directory name to the object store name, so you can
relocate the subproject.
Including the same project several times is indeed interesting. Maybe the
subprojects should be "light checkouts" (I believe something like this was
already discussed on the list sometime). Those would be .git dirs, that would
only have HEAD and pointer to another .git dir with everything else.
quoted
quoted
Not at all. There is no reason to believe that the case that
superproject and subproject come from related URLs is more
common. One of the reasons to do a separated project
I definitely don't think it's more common. But it's the harder case and it
might happen. Generally it will happen if some people work on both the
superproject and the subproject. Of course the argument is that than it
should not be separate projects, but maybe the teams just partly overlap.
I think it will be _very_ common to store super and subprojects in
related locations. First to be independent from third-party servers
while working on the superproject.
Second (and I think more important) because many times there will
be superproject related adaptations in the subproject. Yes they
are independent, and exactly for that reason the subproject upstream
maintainers may not take every change which is needed to satisfy the
superproject. We _now_ see that in all Linux distributions already.
So when you use superprojects to integrate several independent projects,
then the superproject maintainer/administrator should really keep a
clone of all subprojects handy on his site.
Yes, repositories with distribution-specific patches will add a large class
of cases requiring multiple sources support.
--
Jan 'Bulb' Hudec [off-list ref]
From: Martin Waitz <hidden> Date: 2016-06-15 22:43:12
On Tue, May 22, 2007 at 09:37:06PM +0200, Jan Hudec wrote:
quoted
We don't have to move the entire subproject.git into the superproject,
but we need to have all _referenced_ objects in the .git dir of the
superproject.
There are several possibilities to do so:
* move the entire .git dir
* move .git/objects
* explicitly copy all referenced objects
I believe we really need entire .git dir. When the superporject checks out
revision which does not reference that subproject, we still need to preserve
not only the objects of subproject, but also the refs and config.
but all the other refs do not belong to the superproject.
For those who are working on the subproject there are of course a lot
of refs which they have to work with, but that can be dealt with
outside of the superproject scope. The subproject is still a normal
Git repository, after all.
That is, you can have remote entries, branches and what not.
But all that is not interesting in the superproject scope.
So I thing moving the entire subproject.git into the superproject.git is too
much. The superproject is only interested in the objects and in one
ref -- the one stored inside its tree.
quoted
I have some experimental code to configure a per-subproject directory
in the superproject/.git as alternate object store for the submodule
to make the last two solutions possible. Perhaps I should dig it out again
and adapt it to current git.
If there is a 1:1 relationship between subproject and object store then
even efficient fsck and repack/prune are possible for the submodule without
loosing objects.
But such a 1:1 relationship is bad when you move subprojects to another
location (or include the same subproject several times in different
locations of the tree).
Perhaps the user should be able to choose which one he wants.
That's why there should be the extra level of indirection using .gitmodules.
It should map the directory name to the object store name, so you can
relocate the subproject.
Including the same project several times is indeed interesting. Maybe the
subprojects should be "light checkouts" (I believe something like this was
already discussed on the list sometime). Those would be .git dirs, that would
only have HEAD and pointer to another .git dir with everything else.
Well, even if they might share a lot of objects they might be included
for completely different reasons and so might need to work with
different communities (remote entries, branches, etc.).
So sharing objects makes sense, sharing the rest of .git is not
neccessary.
quoted
I think it will be _very_ common to store super and subprojects in
related locations. First to be independent from third-party servers
while working on the superproject.
Second (and I think more important) because many times there will
be superproject related adaptations in the subproject. Yes they
are independent, and exactly for that reason the subproject upstream
maintainers may not take every change which is needed to satisfy the
superproject. We _now_ see that in all Linux distributions already.
So when you use superprojects to integrate several independent projects,
then the superproject maintainer/administrator should really keep a
clone of all subprojects handy on his site.
Yes, repositories with distribution-specific patches will add a large class
of cases requiring multiple sources support.
You don't really need multiple sources for it.
The subproject contains both upstream and local changes, but I think
it makes sense to keep the entire object store local (the same way
to keep all the entire history local even if you only want to add to it
in a normal repository). Those people who work on the subproject and
communicate with its upstream developers of course need remote entries
and have to synchronize the subproject with upstream. But that is
not related to the superproject at all.
So yes, you have different sources but you don't need extra support
in the subproject implementation for it.
--
Martin Waitz
From: Jakub Narebski <hidden> Date: 2016-06-15 22:43:12
Martin Waitz wrote:
On Tue, May 22, 2007 at 09:37:06PM +0200, Jan Hudec wrote:
quoted
Including the same project several times is indeed interesting. Maybe the
subprojects should be "light checkouts" (I believe something like this was
already discussed on the list sometime). Those would be .git dirs, that would
only have HEAD and pointer to another .git dir with everything else.
Well, even if they might share a lot of objects they might be included
for completely different reasons and so might need to work with
different communities (remote entries, branches, etc.).
So sharing objects makes sense, sharing the rest of .git is not
neccessary.
One of the final ideas for "lightweight checkout" was having in
.git/config the location of "true" $GIT_DIR (or parts of it:
GIT_INDEX_FILE, GIT_OBJECT_DIRECTORY), and "shadowing" the rest
of "true $GIT_DIR" with what is present in .git. It means that
you can have .git/index and .git/HEAD, and if you don't find
appropriate .git/refs/heads/master file you look to "true $GIT_DIR".
--
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
From: Jan Hudec <hidden> Date: 2016-06-15 22:43:12
On Thu, May 24, 2007 at 17:48:33 +0200, Martin Waitz wrote:
On Tue, May 22, 2007 at 09:37:06PM +0200, Jan Hudec wrote:
quoted
quoted
We don't have to move the entire subproject.git into the superproject,
but we need to have all _referenced_ objects in the .git dir of the
superproject.
There are several possibilities to do so:
* move the entire .git dir
* move .git/objects
* explicitly copy all referenced objects
I believe we really need entire .git dir. When the superporject checks out
revision which does not reference that subproject, we still need to preserve
not only the objects of subproject, but also the refs and config.
but all the other refs do not belong to the superproject.
For those who are working on the subproject there are of course a lot
of refs which they have to work with, but that can be dealt with
outside of the superproject scope. The subproject is still a normal
Git repository, after all.
That is, you can have remote entries, branches and what not.
But all that is not interesting in the superproject scope.
So I thing moving the entire subproject.git into the superproject.git is too
much. The superproject is only interested in the objects and in one
ref -- the one stored inside its tree.
No, the way I mean it the subproject and superproject don't share a single
bit. The subproject.git dir is subdirectory of superproject.git, but has no
thing in common with it.
quoted
quoted
I have some experimental code to configure a per-subproject directory
in the superproject/.git as alternate object store for the submodule
to make the last two solutions possible. Perhaps I should dig it out again
and adapt it to current git.
Ah. My bad. Didn't notice this. I do NOT want to share any objects between
subproject and superproject. At least not unless the user explicitely asks
for that, which might make sense if the subproject was carved out of the
superproject.
quoted
quoted
If there is a 1:1 relationship between subproject and object store then
even efficient fsck and repack/prune are possible for the submodule without
loosing objects.
But such a 1:1 relationship is bad when you move subprojects to another
location (or include the same subproject several times in different
locations of the tree).
Perhaps the user should be able to choose which one he wants.
That's why there should be the extra level of indirection using .gitmodules.
It should map the directory name to the object store name, so you can
relocate the subproject.
Including the same project several times is indeed interesting. Maybe the
subprojects should be "light checkouts" (I believe something like this was
already discussed on the list sometime). Those would be .git dirs, that would
only have HEAD and pointer to another .git dir with everything else.
Well, even if they might share a lot of objects they might be included
for completely different reasons and so might need to work with
different communities (remote entries, branches, etc.).
So sharing objects makes sense, sharing the rest of .git is not
neccessary.
No, I didn't mean the subproject and superproject would share anything.
The case I talk about is if project foo has subdirs A and B and they both
contain (different revisions of) the same subproject. The .gitmodules
definition is:
[submodule "A"]
name=bar
[submodule "B"]
name=bar
In such case A/.git and B/.git can't both be symlinks to
.git/subprojects/bar.git, because they have different HEAD, but everything
else should be defined by .git/subprojects/bar.git
quoted
quoted
I think it will be _very_ common to store super and subprojects in
related locations. First to be independent from third-party servers
while working on the superproject.
Second (and I think more important) because many times there will
be superproject related adaptations in the subproject. Yes they
are independent, and exactly for that reason the subproject upstream
maintainers may not take every change which is needed to satisfy the
superproject. We _now_ see that in all Linux distributions already.
So when you use superprojects to integrate several independent projects,
then the superproject maintainer/administrator should really keep a
clone of all subprojects handy on his site.
Yes, repositories with distribution-specific patches will add a large class
of cases requiring multiple sources support.
You don't really need multiple sources for it.
The subproject contains both upstream and local changes, but I think
Upstream + local is not the interesting case. Multiple upstreams is.
it makes sense to keep the entire object store local (the same way
to keep all the entire history local even if you only want to add to it
in a normal repository). Those people who work on the subproject and
communicate with its upstream developers of course need remote entries
and have to synchronize the subproject with upstream. But that is
not related to the superproject at all.
So yes, you have different sources but you don't need extra support
in the subproject implementation for it.
Well, if the subproject is not auto-fetched, there's no need for extra
support. But if there is auto-fetch, it should be aware of possibility to
have multiple upstreams.
--
Jan 'Bulb' Hudec [off-list ref]