[RFD/PATCH] submodule doc: describe where we can configure them

This is similar to the gitignore document, but doesn't mirror
the current situation. It is rather meant to start a discussion for
the right approach for mirroring repositories with submodules.

Signed-off-by: Stefan Beller <redacted>
---

 Jonathan, is this something you had in mind?

 Documentation/git-submodule.txt | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
index 13adebf..b5559e5 100644
--- a/Documentation/git-submodule.txt
+++ b/Documentation/git-submodule.txt

@@ -59,6 +59,22 @@ instead of treating the other project as a submodule. Directories
 that come from both projects can be cloned and checked out as a whole
 if you choose to go that route.
 
+Submodule operations can be configured using the following mechanisms
+(from highest to lowest precedence):
+
+ * the command line for those commands that support taking submodule specs.
+
+ * the configuration file `$GIT_DIR/config`.
+
+ * the configuration file `config` found in the `refs/submodule/config` branch.
+   This can be used to overwrite the upstream configuration in the `.gitmodules`
+   file without changing the history of the project.
+   Useful options here are overwriting the base, where relative URLs apply to,
+   when mirroring only parts of the larger collection of submodules.
+
+ * the `.gitmodules` file inside the repository. A project usually includes this
+   file to suggest defaults for the upstream collection of repositories.
+
 COMMANDS
 --------
 add::

-- 
2.8.0.rc4.10.geb92688.dirty

Stefan Beller wrote:

This is similar to the gitignore document, but doesn't mirror
the current situation. It is rather meant to start a discussion for
the right approach for mirroring repositories with submodules.

Ooh.

[...]

(This documentation probably belongs in gitmodules(5) --- then,
git-submodule(1) could focus on command-line usage and point there for
configuration information.)

There are two aspects of this to be separated: what governs the behavior
of commands running locally, and where we get information about
submodules from a remote repository.

Local commands
--------------
The original submodule design was that local commands rely on
information from .git/config, and that information gets copied there
from .gitmodules when a submodule is initialized.  That way, a local
user can specify their preferred mirror or other options using some
straightforward 'git config' commands.

As a side effect, the settings in .git/config tell git which submodules
to pay attention to (which submodules were initialized).

When .gitmodules changes, the settings in .git/config are left alone,
since the end user *might* have manually set something up and we don't
want to trample on it.

This design is somewhat problematic for a few reasons:

- When I want to stop paying attention to a particular submodule and
  start paying attention to it again later, all my local settings are
  gone.

- When upstream adds a new submodule, I have to do the same manual
  work to change the options for that new submodule.

- When upstream changes submodule options (perhaps to fix a URL
  typo), I don't get those updates.

A fix is to use settings from .git/config when present and fall back
to .gitmodules when not.  I believe the submodule code has been slowly
moving in that direction for new features.  Perhaps we can do so for
existing features (like submodule.*.url) too.

An alternative would have been to introduce a .git/info/submodules
file that overrides settings from .gitmodules, analagous to
.git/info/excludes overriding .gitignore and .git/info/attributes
overriding .gitattributes.  We are already using .git/config for
this so that doesn't seem necessary.

Remote repositories
-------------------
The .gitmodules file has some odd properties as a place to put
configuration:

- it is versioned.  There is no way to change URLs in an old version
  of .gitmodules retroactively when a URL has changed.

- it is controlled by whoever writes history.  There is no way for me
  to change the URLs in my mirror of https://gerrit.googlesource.com/gerrit
  to match my mirror's different filesystem layout without producing
  my own history that diverges from the commits I am mirroring.

When the URLs in .gitmodules are relative URLs, this means that if
I mirror a superproject, I have to mirror all its submodules, too,
with the same layout.  It's not so easy for me to publish my copy
of the parent project and the one subproject I made changes in --- I
have to mirror everything.  In particular, this means I can't mirror
https://gerrit.googlesource.com/gerrit to github.

When the URLs in .gitmodules are absolute URLs, this means that if
I mirror a superproject, I cannot ask people consuming my mirror to
use my mirrors of child projects, too.  I cannot publish my copy of
the parent project and the one subproject I made changes in and
expect people to be able to "git clone --recurse-submodules" the
result successfully.

It is as though refs were stored in a .gitrefs file, with all the
attendant disadvantages, instead of being a separate component of
the repository that a particular repository owner can manipulate
without changing history.

To fix this, we could allow additional .gitmodules settings to be put
in another ref (perhaps something like "refs/repository/config" to allow
sharing additional repository-specific configuration in other files
within the same tree --- e.g., branch descriptions).  The semantics:

* If there is a gitmodules file in refs/repository/config in the
  repository I clone, then the submodule settings from it are stored
  locally somewhere that overrides .gitmodules.  Perhaps
  .git/info/<remotename>/gitmodules?

* Later fetches from the remote would also update this gitmodules
  file.

* Settings from this gitmodules file can be overridden locally
  using 'git config' until an explicit "git submodule sync" to
  override the local configuration.

What do you think?

If two different remotes provide conflicting values for a setting
in their gitmodules files, git would error out and ask the user
to intervene with a tie-breaking "git config" setting.

Thanks,
Jonathan

On Tue, May 3, 2016 at 4:56 PM, Jonathan Nieder [off-list ref] wrote:

Stefan Beller wrote:

quoted

This is similar to the gitignore document, but doesn't mirror
the current situation. It is rather meant to start a discussion for
the right approach for mirroring repositories with submodules.

Ooh.

Thanks for writing such a detailed answer. :)

[...]

quoted

--- a/Documentation/git-submodule.txt
+++ b/Documentation/git-submodule.txt

@@ -59,6 +59,22 @@ instead of treating the other project as a submodule. Directories
 that come from both projects can be cloned and checked out as a whole
 if you choose to go that route.

+Submodule operations can be configured using the following mechanisms
+(from highest to lowest precedence):
+
+ * the command line for those commands that support taking submodule specs.
+
+ * the configuration file `$GIT_DIR/config`.
+
+ * the configuration file `config` found in the `refs/submodule/config` branch.
+   This can be used to overwrite the upstream configuration in the `.gitmodules`
+   file without changing the history of the project.
+   Useful options here are overwriting the base, where relative URLs apply to,
+   when mirroring only parts of the larger collection of submodules.
+
+ * the `.gitmodules` file inside the repository. A project usually includes this
+   file to suggest defaults for the upstream collection of repositories.

(This documentation probably belongs in gitmodules(5) --- then,
git-submodule(1) could focus on command-line usage and point there for
configuration information.)

That makes sense!

There are two aspects of this to be separated: what governs the behavior
of commands running locally, and where we get information about
submodules from a remote repository.

After reading the first time, this seems to also contain "historical context".

Local commands
--------------
The original submodule design was that local commands rely on
information from .git/config, and that information gets copied there
from .gitmodules when a submodule is initialized.  That way, a local
user can specify their preferred mirror or other options using some
straightforward 'git config' commands.

As a side effect, the settings in .git/config tell git which submodules
to pay attention to (which submodules were initialized).

When .gitmodules changes, the settings in .git/config are left alone,
since the end user *might* have manually set something up and we don't
want to trample on it.

This design is somewhat problematic for a few reasons:

- When I want to stop paying attention to a particular submodule and
  start paying attention to it again later, all my local settings are
  gone.

- When upstream adds a new submodule, I have to do the same manual
  work to change the options for that new submodule.

- When upstream changes submodule options (perhaps to fix a URL
  typo), I don't get those updates.

A fix is to use settings from .git/config when present and fall back
to .gitmodules when not.  I believe the submodule code has been slowly
moving in that direction for new features.  Perhaps we can do so for
existing features (like submodule.*.url) too.

An alternative would have been to introduce a .git/info/submodules
file that overrides settings from .gitmodules, analagous to
.git/info/excludes overriding .gitignore and .git/info/attributes
overriding .gitattributes.  We are already using .git/config for
this so that doesn't seem necessary.

I don't know if it is a worthwhile goal nevertheless to move
the information about submodules to .git/info/submodules eventually
as that brings consistency across different features of Git?

Remote repositories
-------------------
The .gitmodules file has some odd properties as a place to put
configuration:

- it is versioned.  There is no way to change URLs in an old version
  of .gitmodules retroactively when a URL has changed.

I would not call it odd for having one versioned place. Consider your
build process is updated and the new build process produces new
intermediate files. You would add these files to the .gitignore file
eventually, but when building old revisions with the new build chain
you'd be surprised by all those untracked files being displayed.
Or another example: Recently in git.git some test helper files were moved.
By checking out an older version of git you see a lot of test-* files
in your worktree although they were ignored at another revision.

That paragraph got longer than expected, but I just wanted to say that
being versioned can be either good or bad.

- it is controlled by whoever writes history.  There is no way for me
  to change the URLs in my mirror of https://gerrit.googlesource.com/gerrit
  to match my mirror's different filesystem layout without producing
  my own history that diverges from the commits I am mirroring.

To come up with an analogy to ignored files:
If I use a project and use a different build system, I may see untracked
files as they are not ignored by the .gitignore file.

Then I have a way of ignoring them nevertheless in .git/info/excludes.
Sharing this information beyond this repository is hard though, but
that wasn't seen as a feature yet?

When the URLs in .gitmodules are relative URLs, this means that if
I mirror a superproject, I have to mirror all its submodules, too,
with the same layout.  It's not so easy for me to publish my copy
of the parent project and the one subproject I made changes in --- I
have to mirror everything.  In particular, this means I can't mirror
https://gerrit.googlesource.com/gerrit to github.

because the way repository URLs work are different for these 2 hosts.
googlesource.com allows to have URLs that are nested in another level
e.g. Gerrit references "../plugins/download-commands", such that
remote URL becomes https://gerrit.googlesource.com/plugins/download-commands

At Github we cannot create another level of nesting as their naming follows the
owner/name scheme.

When the URLs in .gitmodules are absolute URLs, this means that if
I mirror a superproject, I cannot ask people consuming my mirror to
use my mirrors of child projects, too.  I cannot publish my copy of
the parent project and the one subproject I made changes in and
expect people to be able to "git clone --recurse-submodules" the
result successfully.

It is as though refs were stored in a .gitrefs file, with all the
attendant disadvantages, instead of being a separate component of
the repository that a particular repository owner can manipulate
without changing history.

To fix this, we could allow additional .gitmodules settings to be put
in another ref (perhaps something like "refs/repository/config" to allow
sharing additional repository-specific configuration in other files
within the same tree --- e.g., branch descriptions).  The semantics:

* If there is a gitmodules file in refs/repository/config in the
  repository I clone, then the submodule settings from it are stored
  locally somewhere that overrides .gitmodules.  Perhaps
  .git/info/<remotename>/gitmodules?

* Later fetches from the remote would also update this gitmodules
  file.

* Settings from this gitmodules file can be overridden locally
  using 'git config' until an explicit "git submodule sync" to
  override the local configuration.

What do you think?

If two different remotes provide conflicting values for a setting
in their gitmodules files, git would error out and ask the user
to intervene with a tie-breaking "git config" setting.

Let's look at an example with C mirroring from B, who mirrors from A.

The user who clones the superproject from C may want to obtain submodules
from either C or B or A. All this can be configured in
the refs/repository/config value of C, but in case it is not configured in C,
it may fall back to the same branch from B. When and how would B get
that branch?

Thanks for writing out this detailed brain dump :)
Stefan

Thanks,
Jonathan

On Tue, May 03, 2016 at 05:59:58PM -0700, Stefan Beller wrote:

On Tue, May 3, 2016 at 4:56 PM, Jonathan Nieder [off-list ref] wrote:

quoted

Stefan Beller wrote:

quoted

This is similar to the gitignore document, but doesn't mirror
the current situation. It is rather meant to start a discussion for
the right approach for mirroring repositories with submodules.

Ooh.

Thanks for writing such a detailed answer. :)

BTW, here is a pointer to the discussion (and what I wrote down) about
this from back in 2014:

https://github.com/jlehmann/git-submod-enhancements/wiki/Ideas#special-ref-overriding-gitmodules-values

quoted

To fix this, we could allow additional .gitmodules settings to be put
in another ref (perhaps something like "refs/repository/config" to allow
sharing additional repository-specific configuration in other files
within the same tree --- e.g., branch descriptions).  The semantics:

* If there is a gitmodules file in refs/repository/config in the
  repository I clone, then the submodule settings from it are stored
  locally somewhere that overrides .gitmodules.  Perhaps
  .git/info/<remotename>/gitmodules?

* Later fetches from the remote would also update this gitmodules
  file.

* Settings from this gitmodules file can be overridden locally
  using 'git config' until an explicit "git submodule sync" to
  override the local configuration.

What do you think?

If two different remotes provide conflicting values for a setting
in their gitmodules files, git would error out and ask the user
to intervene with a tie-breaking "git config" setting.

Let's look at an example with C mirroring from B, who mirrors from A.

The user who clones the superproject from C may want to obtain submodules
from either C or B or A. All this can be configured in
the refs/repository/config value of C, but in case it is not configured in C,
it may fall back to the same branch from B. When and how would B get
that branch?

I think B has to setup that branch on its own when it starts to mirror
A and uses different submodule urls or other configs.

Jonathan you suggested to copy the content from a remote to
.git/info/<remotename>/gitmodules locally. How would one get it to the
remote side? It seems to me as if we would need to implement additional
infrastructure to do this. Would it not be simpler if we just kept it on
a ref on the local side as well? We already have the infrastructure to
read those values from a ref. We only would need to add something to
write them. Then a simple push, which could be aliased in form of a
git-submodule subcommand, suffices to get the values to the remote.

That also solves issues when people clone from their working copy.

I would like to think a little bit further about the conflict situation
when two remotes are providing values. Configuring this looks to me like
a nightmare for users. Maybe there is some sort of elegant solution?
E.g. like we use the values from remote A during a fetch from A, the
ones from B during a fetch from B and no values from a special ref in
case there is no remote operation involved. Since the main goal is to
support forking of submodules isn't there always a remote operation
involved?

My suggested scheme above does not solve the currently quite typical use
case where you might 'git fetch' without submodules first and then do
the submodule fetches during a 'git submodule update'. On the other hand
in the 'ideal future world' where submodules behave like "normal files" the
fetch will be done during the superproject fetch so in that case we
could solve such conflicts.

The main thing which we could keep in mind is that we only allow certain
values in such special refs. E.g. only the ones needed to support the
fork workflow. BTW, do we actually need to change other values than the
URL? Addtionally we ignore other values that are more related to the
overall project structure. E.g. like submodule.<name>.ignore.

Ok after writing this it really feels like special casing a lot. I would
not really call it elegant. At the same time limiting these special refs
to one special use case (forking) might help us to keep the user
interface[1] simpler and conflict free in the long run. Not sure. What
do you think?

Cheers Heiko

[1] Which is not the simplest already.

On Wed, May 4, 2016 at 8:01 AM, Heiko Voigt [off-list ref] wrote:

On Tue, May 03, 2016 at 05:59:58PM -0700, Stefan Beller wrote:

quoted

On Tue, May 3, 2016 at 4:56 PM, Jonathan Nieder [off-list ref] wrote:

quoted

Stefan Beller wrote:

quoted

This is similar to the gitignore document, but doesn't mirror
the current situation. It is rather meant to start a discussion for
the right approach for mirroring repositories with submodules.

Ooh.

Thanks for writing such a detailed answer. :)

BTW, here is a pointer to the discussion (and what I wrote down) about
this from back in 2014:

https://github.com/jlehmann/git-submod-enhancements/wiki/Ideas#special-ref-overriding-gitmodules-values

Thanks for pointing at the prior discussion!
Although not much happened since then (code wise)?

quoted

quoted

To fix this, we could allow additional .gitmodules settings to be put
in another ref (perhaps something like "refs/repository/config" to allow
sharing additional repository-specific configuration in other files
within the same tree --- e.g., branch descriptions).  The semantics:

* If there is a gitmodules file in refs/repository/config in the
  repository I clone, then the submodule settings from it are stored
  locally somewhere that overrides .gitmodules.  Perhaps
  .git/info/<remotename>/gitmodules?

* Later fetches from the remote would also update this gitmodules
  file.

* Settings from this gitmodules file can be overridden locally
  using 'git config' until an explicit "git submodule sync" to
  override the local configuration.

What do you think?

If two different remotes provide conflicting values for a setting
in their gitmodules files, git would error out and ask the user
to intervene with a tie-breaking "git config" setting.

Let's look at an example with C mirroring from B, who mirrors from A.

The user who clones the superproject from C may want to obtain submodules
from either C or B or A. All this can be configured in
the refs/repository/config value of C, but in case it is not configured in C,
it may fall back to the same branch from B. When and how would B get
that branch?

I think B has to setup that branch on its own when it starts to mirror
A and uses different submodule urls or other configs.

Jonathan you suggested to copy the content from a remote to
.git/info/<remotename>/gitmodules locally. How would one get it to the
remote side? It seems to me as if we would need to implement additional
infrastructure to do this. Would it not be simpler if we just kept it on
a ref on the local side as well? We already have the infrastructure to
read those values from a ref. We only would need to add something to
write them. Then a simple push, which could be aliased in form of a
git-submodule subcommand, suffices to get the values to the remote.

That is good idea!

That also solves issues when people clone from their working copy.

I would like to think a little bit further about the conflict situation
when two remotes are providing values. Configuring this looks to me like
a nightmare for users. Maybe there is some sort of elegant solution?
E.g. like we use the values from remote A during a fetch from A, the
ones from B during a fetch from B and no values from a special ref in
case there is no remote operation involved. Since the main goal is to
support forking of submodules isn't there always a remote operation
involved?

Here is what I imagine
When B mirrors from A, B sets up this special ref for its repository,
e.g. refs/meta/submodule-B and have a symbolic ref pointing at that.
(e.g. SUBMODULE_CONFIG pointing at refs/meta/submodule-B,
which has a worktree which contains a .gitmodules files which
sets up
  "submodule.baz.url = http://B/baz"
  "submodule.relativeBase = http://A"

That way anyone cloning from B would get
the superproject and the submodule baz from B while the
rest of the submodules are found at A.

When C mirrors from A, they add another branch  refs/meta/submodule-C,
which can either be a fork of refs/meta/submodule-B with some changes on
top of it or it can add a reference to refs/meta/submodule-B, i.e. the
configuration
would be:

  "submodule.baseConfig = refs/meta/submodule-B"
  "submodule.foo.url = ssh://C/foo"

and SUBMODULE_CONFIG would point at refs/meta/submodule-C.

When cloning from C, the user would get

 * the superproject from C
 * submodule foo from C
 * submodule baz from B
 * all other submodules from A

By the inheriting property of the branch of B there are no conflicting values.
C could just overwrite submodule.baseConfig for example.

My suggested scheme above does not solve the currently quite typical use
case where you might 'git fetch' without submodules first and then do
the submodule fetches during a 'git submodule update'. On the other hand
in the 'ideal future world' where submodules behave like "normal files" the
fetch will be done during the superproject fetch so in that case we
could solve such conflicts.

The main thing which we could keep in mind is that we only allow certain
values in such special refs. E.g. only the ones needed to support the
fork workflow. BTW, do we actually need to change other values than the
URL? Addtionally we ignore other values that are more related to the
overall project structure. E.g. like submodule.<name>.ignore.

Maybe we want to have a dedicated protocol field, eventually.
A,B,C may have different standards on what they use by default.
e.g. Use ssh at kernel.org, but http in a corporate mirror, because http is
the only protocol not blocked by firewall. So I could imagine that a
complete mirror of submodules with relative URLs wants to only replace
ssh by http.

Ok after writing this it really feels like special casing a lot. I would
not really call it elegant. At the same time limiting these special refs
to one special use case (forking) might help us to keep the user
interface[1] simpler and conflict free in the long run. Not sure. What
do you think?

Cheers Heiko

[1] Which is not the simplest already.

Hi,

On Wed, May 04, 2016 at 01:50:24PM -0700, Stefan Beller wrote:

On Wed, May 4, 2016 at 8:01 AM, Heiko Voigt [off-list ref] wrote:

quoted

On Tue, May 03, 2016 at 05:59:58PM -0700, Stefan Beller wrote:

quoted

On Tue, May 3, 2016 at 4:56 PM, Jonathan Nieder [off-list ref] wrote:

quoted

Stefan Beller wrote:

quoted

This is similar to the gitignore document, but doesn't mirror
the current situation. It is rather meant to start a discussion for
the right approach for mirroring repositories with submodules.

Ooh.

Thanks for writing such a detailed answer. :)

BTW, here is a pointer to the discussion (and what I wrote down) about
this from back in 2014:

https://github.com/jlehmann/git-submod-enhancements/wiki/Ideas#special-ref-overriding-gitmodules-values

Thanks for pointing at the prior discussion!
Although not much happened since then (code wise)?

Yes, IIRC nothing happened code wise. It went so far that a rough
consensus was made but nobody actually stepped in to scratch that itch.

quoted

Jonathan you suggested to copy the content from a remote to
.git/info/<remotename>/gitmodules locally. How would one get it to the
remote side? It seems to me as if we would need to implement additional
infrastructure to do this. Would it not be simpler if we just kept it on
a ref on the local side as well? We already have the infrastructure to
read those values from a ref. We only would need to add something to
write them. Then a simple push, which could be aliased in form of a
git-submodule subcommand, suffices to get the values to the remote.

That is good idea!

Thanks.

quoted

That also solves issues when people clone from their working copy.

I would like to think a little bit further about the conflict situation
when two remotes are providing values. Configuring this looks to me like
a nightmare for users. Maybe there is some sort of elegant solution?
E.g. like we use the values from remote A during a fetch from A, the
ones from B during a fetch from B and no values from a special ref in
case there is no remote operation involved. Since the main goal is to
support forking of submodules isn't there always a remote operation
involved?

Here is what I imagine
When B mirrors from A, B sets up this special ref for its repository,
e.g. refs/meta/submodule-B and have a symbolic ref pointing at that.
(e.g. SUBMODULE_CONFIG pointing at refs/meta/submodule-B,
which has a worktree which contains a .gitmodules files which
sets up
  "submodule.baz.url = http://B/baz"
  "submodule.relativeBase = http://A"

That way anyone cloning from B would get
the superproject and the submodule baz from B while the
rest of the submodules are found at A.

This sounds sensible. But my imagination of a conflict was in a
different way. E.g. project A has a submodule B. And now A has a remote
1 where you publish and maybe another remote 2 where someone else (a
colleague?) publishes. Which configuration do you use? Here the two
remotes are independent instead of subsequent forks. In this case my
solution would be to use the configuration branch from 1 for B when
interacting with 1. I do not have completely checked whether we always
have a remote at hand for such a resolution.

When C mirrors from A, they add another branch  refs/meta/submodule-C,
which can either be a fork of refs/meta/submodule-B with some changes on
top of it or it can add a reference to refs/meta/submodule-B, i.e. the
configuration
would be:

  "submodule.baseConfig = refs/meta/submodule-B"
  "submodule.foo.url = ssh://C/foo"

and SUBMODULE_CONFIG would point at refs/meta/submodule-C.

When cloning from C, the user would get

 * the superproject from C
 * submodule foo from C
 * submodule baz from B
 * all other submodules from A

By the inheriting property of the branch of B there are no conflicting values.
C could just overwrite submodule.baseConfig for example.

So that means in the default case we create a chain of all previous
forks embedded in repository database. I am not saying that this is
necessarily a bad thing but I feel that it is a new property which we
should think about. It helps because users will get updated values from
sources that are in the chain. On the other hand it adds a lot of
dependencies which are point of failures in case a remote disappears. I
am undecided on this. I would prefer if we could let people play with it
a little (maybe on pu?) and then decide if there are practical pitfalls
with this.

quoted

My suggested scheme above does not solve the currently quite typical use
case where you might 'git fetch' without submodules first and then do
the submodule fetches during a 'git submodule update'. On the other hand
in the 'ideal future world' where submodules behave like "normal files" the
fetch will be done during the superproject fetch so in that case we
could solve such conflicts.

The main thing which we could keep in mind is that we only allow certain
values in such special refs. E.g. only the ones needed to support the
fork workflow. BTW, do we actually need to change other values than the
URL? Addtionally we ignore other values that are more related to the
overall project structure. E.g. like submodule.<name>.ignore.

Maybe we want to have a dedicated protocol field, eventually.
A,B,C may have different standards on what they use by default.
e.g. Use ssh at kernel.org, but http in a corporate mirror, because http is
the only protocol not blocked by firewall. So I could imagine that a
complete mirror of submodules with relative URLs wants to only replace
ssh by http.

By this you mean 'submodule.relativeBase' that you introduced above
right? Or something similar. These values I would still consider them
URL'ish. But my question was more geared towards this direction: Are
there other values than the ones used to assemble the URL that make
sense to share?

E.g.: Someone might want to fork a repository and might want to change
the default set of submodules that are populated with 'git submodule
update --init'. Is this something we should allow via these special refs
or is this actually changing the project structure and should also be
reflected in project history? IMO the latter is the case.

Only things like the technical organisation (like the place where a
repository can be found) justify to be outside of the repository IMO.

A repository without submodules does have one collection of remote
repository urls. To me adding proper fork support seems be the switch
from one collection for one repository to many collections for many
repositories. Since this one collection is already outside of the
superproject it makes sense to do the same for the submodules. So my
question reformulated could be: Are there more values we currently keep
inside the repository for submodules that actually belong outside? A
good indication could be that they are already outside in the
superproject.

I did not find any flaw in these statements yet, but maybe I am
oversimplifying?

Cheers Heiko

quoted

Here is what I imagine
When B mirrors from A, B sets up this special ref for its repository,
e.g. refs/meta/submodule-B and have a symbolic ref pointing at that.
(e.g. SUBMODULE_CONFIG pointing at refs/meta/submodule-B,
which has a worktree which contains a .gitmodules files which
sets up
  "submodule.baz.url = http://B/baz"
  "submodule.relativeBase = http://A"

That way anyone cloning from B would get
the superproject and the submodule baz from B while the
rest of the submodules are found at A.

This sounds sensible. But my imagination of a conflict was in a
different way. E.g. project A has a submodule B. And now A has a remote
1 where you publish and maybe another remote 2 where someone else (a
colleague?) publishes. Which configuration do you use? Here the two
remotes are independent instead of subsequent forks. In this case my
solution would be to use the configuration branch from 1 for B when
interacting with 1. I do not have completely checked whether we always
have a remote at hand for such a resolution.

I think it is the responsibility of the pusher to make sure the
configuration is sane.
So if I were to push to remote 2 and you push to remote 1, we'd both configure
the special branch of our superprojects for these remotes for that submodule.

If the superproject has relative urls for the submodule, all we had to do was
unset (or overwrite) the submodule.baseConfig.

quoted

When C mirrors from A, they add another branch  refs/meta/submodule-C,
which can either be a fork of refs/meta/submodule-B with some changes on
top of it or it can add a reference to refs/meta/submodule-B, i.e. the
configuration
would be:

  "submodule.baseConfig = refs/meta/submodule-B"
  "submodule.foo.url = ssh://C/foo"

and SUBMODULE_CONFIG would point at refs/meta/submodule-C.

When cloning from C, the user would get

 * the superproject from C
 * submodule foo from C
 * submodule baz from B
 * all other submodules from A

By the inheriting property of the branch of B there are no conflicting values.
C could just overwrite submodule.baseConfig for example.

So that means in the default case we create a chain of all previous
forks embedded in repository database.

Not necessarily. I was just pointing out that this was possible. The
intermediate
party could decide that their upstream is too unreliable and not point
to their upstream.

This would incur the cost of having to clone all submodules and
overwriting the absolute
urls. For the relative URLs this would just work as of now.

All I wanted with that example is to offer the flexibility to not have
to clone all the
submodule, but I can fork a mega-project with 100s of submodules and maybe
just fiddle with one of them and then publish that.

I am not saying that this is
necessarily a bad thing but I feel that it is a new property which we
should think about. It helps because users will get updated values from
sources that are in the chain. On the other hand it adds a lot of
dependencies which are point of failures in case a remote disappears. I
am undecided on this. I would prefer if we could let people play with it
a little (maybe on pu?) and then decide if there are practical pitfalls
with this.

quoted

quoted

My suggested scheme above does not solve the currently quite typical use
case where you might 'git fetch' without submodules first and then do
the submodule fetches during a 'git submodule update'. On the other hand
in the 'ideal future world' where submodules behave like "normal files" the
fetch will be done during the superproject fetch so in that case we
could solve such conflicts.

The main thing which we could keep in mind is that we only allow certain
values in such special refs. E.g. only the ones needed to support the
fork workflow. BTW, do we actually need to change other values than the
URL? Addtionally we ignore other values that are more related to the
overall project structure. E.g. like submodule.<name>.ignore.

Maybe we want to have a dedicated protocol field, eventually.
A,B,C may have different standards on what they use by default.
e.g. Use ssh at kernel.org, but http in a corporate mirror, because http is
the only protocol not blocked by firewall. So I could imagine that a
complete mirror of submodules with relative URLs wants to only replace
ssh by http.

By this you mean 'submodule.relativeBase' that you introduced above
right? Or something similar. These values I would still consider them
URL'ish. But my question was more geared towards this direction: Are
there other values than the ones used to assemble the URL that make
sense to share?

E.g.: Someone might want to fork a repository and might want to change
the default set of submodules that are populated with 'git submodule
update --init'. Is this something we should allow via these special refs
or is this actually changing the project structure and should also be
reflected in project history? IMO the latter is the case.

That sounds reasonable.

Only things like the technical organisation (like the place where a
repository can be found) justify to be outside of the repository IMO.

A repository without submodules does have one collection of remote
repository urls. To me adding proper fork support seems be the switch
from one collection for one repository to many collections for many
repositories. Since this one collection is already outside of the
superproject it makes sense to do the same for the submodules. So my
question reformulated could be: Are there more values we currently keep
inside the repository for submodules that actually belong outside? A
good indication could be that they are already outside in the
superproject.

I did not find any flaw in these statements yet, but maybe I am
oversimplifying?

They sound right to me.

Cheers Heiko

Thanks for the discussion :)
Stefan

On Mon, May 09, 2016 at 10:32:50AM -0700, Stefan Beller wrote:

quoted

quoted

Here is what I imagine
When B mirrors from A, B sets up this special ref for its repository,
e.g. refs/meta/submodule-B and have a symbolic ref pointing at that.
(e.g. SUBMODULE_CONFIG pointing at refs/meta/submodule-B,
which has a worktree which contains a .gitmodules files which
sets up
  "submodule.baz.url = http://B/baz"
  "submodule.relativeBase = http://A"

That way anyone cloning from B would get
the superproject and the submodule baz from B while the
rest of the submodules are found at A.

This sounds sensible. But my imagination of a conflict was in a
different way. E.g. project A has a submodule B. And now A has a remote
1 where you publish and maybe another remote 2 where someone else (a
colleague?) publishes. Which configuration do you use? Here the two
remotes are independent instead of subsequent forks. In this case my
solution would be to use the configuration branch from 1 for B when
interacting with 1. I do not have completely checked whether we always
have a remote at hand for such a resolution.

I think it is the responsibility of the pusher to make sure the
configuration is sane.
So if I were to push to remote 2 and you push to remote 1, we'd both configure
the special branch of our superprojects for these remotes for that submodule.

If the superproject has relative urls for the submodule, all we had to do was
unset (or overwrite) the submodule.baseConfig.

What if (because we work together) you and me have both remotes in our
local repository. We only push to our private remotes but fetch from
both. Since we work together we also forked the same submodule B and
have different URL configurations for it. I push to B1 and you to B2.
Now we both have two special branches (one from B1 and one from B2) in
our local repositories, since on either of our private remotes there is
one special branch.

Which values are valid now? I see you are advocating for a symbolic ref
SUBMODULE_CONFIG that points to a single special branch in charge, but
maybe we can avoid that. In this case there actually is no real
conflict, since we can just add both remotes B1, B2 to the submodule B.
Which one is used is a choice of the user during push.

For submodule.relativeBase we could try a similar solution and just add
all remotes that can be constructed with the different configurations.
Probably under the same name as in the superproject.

So if we limit ourselves to only allow URL'ish (actually remote'ish is
probably a better term) we can actually avoid conflict resolution and
just add/use them all. If we limit ourselves to the fork use case and my
hypothesis that we only need to allow remote'ish values in these special
branches for it is true, we can actually keep it quite simple and have
no conflict resolution at all I think (and realize now).

What do you think?

quoted

quoted

When C mirrors from A, they add another branch  refs/meta/submodule-C,
which can either be a fork of refs/meta/submodule-B with some changes on
top of it or it can add a reference to refs/meta/submodule-B, i.e. the
configuration
would be:

  "submodule.baseConfig = refs/meta/submodule-B"
  "submodule.foo.url = ssh://C/foo"

and SUBMODULE_CONFIG would point at refs/meta/submodule-C.

When cloning from C, the user would get

 * the superproject from C
 * submodule foo from C
 * submodule baz from B
 * all other submodules from A

By the inheriting property of the branch of B there are no conflicting values.
C could just overwrite submodule.baseConfig for example.

So that means in the default case we create a chain of all previous
forks embedded in repository database.

Not necessarily. I was just pointing out that this was possible. The
intermediate
party could decide that their upstream is too unreliable and not point
to their upstream.

This would incur the cost of having to clone all submodules and
overwriting the absolute
urls. For the relative URLs this would just work as of now.

All I wanted with that example is to offer the flexibility to not have
to clone all the
submodule, but I can fork a mega-project with 100s of submodules and maybe
just fiddle with one of them and then publish that.

Do you mean 'not having to fork all the submodules' here? Since 'without
cloning' is already possible, no?

I am assuming you meant fork. So submodule.relativeBase is meant to
solve that right? You set it and all relative submodule URLs that are
not configured otherwise relate to it.

My point was about the chaining with submodule.baseConfig. That is not
necessary to support partial forks of just a few submodules.

Actually while thinking about submodule.relativeBase now, I found it
might be nice to extend it a little. Imagine someone wants fork a set of
submodules and specify a relativeBase for them and then someone else
forking again wants to do that with another set of submodules. I imagine
subsequents forks are quite usual in git (like in the kernels workflow).
Maybe we can extend this scheme a little bit and allow to set
submodule.relativeBase for groups of submodules somehow?

quoted

I am not saying that this is
necessarily a bad thing but I feel that it is a new property which we
should think about. It helps because users will get updated values from
sources that are in the chain. On the other hand it adds a lot of
dependencies which are point of failures in case a remote disappears. I
am undecided on this. I would prefer if we could let people play with it
a little (maybe on pu?) and then decide if there are practical pitfalls
with this.

[...]

quoted

Only things like the technical organisation (like the place where a
repository can be found) justify to be outside of the repository IMO.

A repository without submodules does have one collection of remote
repository urls. To me adding proper fork support seems be the switch
from one collection for one repository to many collections for many
repositories. Since this one collection is already outside of the
superproject it makes sense to do the same for the submodules. So my
question reformulated could be: Are there more values we currently keep
inside the repository for submodules that actually belong outside? A
good indication could be that they are already outside in the
superproject.

I did not find any flaw in these statements yet, but maybe I am
oversimplifying?

They sound right to me.

Great. Then my simplification suggestion above should work as well.

Thanks for the discussion :)

Thank you for caring about this topic! I think this is quite some
important work to get submodule forking almost as simple as forking
their superprojects. I am happy to continue this discussion and
bounce ideas off of each other :)

Cheers Heiko