url.<base>.insteadOf vs. submodules

Hi all,

I am trying to make url.<base>.insteadOf work on the URLs inside 
.gitmodules, but it won't work (applying it to the repo itself works 
fine, to the config setting seems to be fine).

I do not want to modify .gitmodules: It is maintained upstream.

I cannot simply reconfigure submodule.<module>.url: the Configure script 
(regularly called during each compile) does
   git submodule sync
   git submodule update --init
I could tell upstream to change these commands if I can make a good 
argument; for them, it is relevant that they can change the submodule 
URL inside .gitmodule and have it "just work" for everybody downstream.

My own use case is that I want to be able to work with various 
experimental local clones even if I do not have Internet access.
I'm all ears if there's a way to do this without using insteadOf.


Here are the relevant two lines from the output of "git config -l" 
(after "git submodule init"):

url./home/jo/Projekte/perl6/bare-repos.insteadof=https://github.com
submodule.3rdparty/dynasm.url=https://github.com/MoarVM/dynasm.git


Here is what "git submodule update" does:

Cloning into '3rdparty/dyncall'...
fatal: unable to access 'https://github.com/MoarVM/dyncall.git/': Could 
not resolve host: github.com
fatal: clone of 'https://github.com/MoarVM/dyncall.git' into submodule 
path '3rdparty/dyncall' failed


Any help appreciated!

Regards,
Jo

On Sun, Feb 19, 2017 at 10:12:28PM +0100, Toolforger wrote:

I am trying to make url.<base>.insteadOf work on the URLs inside
.gitmodules, but it won't work (applying it to the repo itself works fine,
to the config setting seems to be fine).

The submodule operations happen in their own processes, and do not look
at the config of the parent repo. Are you setting the config in
.git/config of the super-project?

I don't know if there plans to make that work, but one workaround is to
set the config in ~/.gitconfig.

-Peff

On 20.02.2017 10:01, Jeff King wrote:

On Sun, Feb 19, 2017 at 10:12:28PM +0100, Toolforger wrote:

quoted

I am trying to make url.<base>.insteadOf work on the URLs inside
.gitmodules, but it won't work (applying it to the repo itself works fine,
to the config setting seems to be fine).

The submodule operations happen in their own processes, and do not look
at the config of the parent repo.

Ah, then we have a docbug.
git help config has this to say:

url.<base>.insteadOf
     Any URL that starts with this value will be rewritten to start,
     instead, with <base>.

The "Any" here is wrong, it would be "any except submodule" (possibly 
other exceptions).

 > Are you setting the config in

.git/config of the super-project?

Exactly.
My thinking was that since the submodule URLs are specified in the super 
project's .gitmodules, that setting should apply.

I don't know if there plans to make that work,

It would certainly help me out, though I guess it's going to be too late 
for my current project :-)

 > but one workaround is to set the config in ~/.gitconfig.

No can do - that's under version control.
My personal setup does not belong there I think ;-)

I am currently trying to write a shell script that
- does git submodule init
- pulls submodule configuration out of git config -l
- configures each submodule with insteadOf
It fits with my workflow because setting up the repositories is going to 
be done via script anyway.
I'm neither a shell nor a git expert, so any advice still appreciated.

Regards,
Jo

On Mon, Feb 20, 2017 at 09:31:40PM +0100, Toolforger wrote:

quoted

The submodule operations happen in their own processes, and do not look
at the config of the parent repo.

Ah, then we have a docbug.
git help config has this to say:

url.<base>.insteadOf
    Any URL that starts with this value will be rewritten to start,
    instead, with <base>.

The "Any" here is wrong, it would be "any except submodule" (possibly other
exceptions).

I'm not sure that "any" is wrong here. Repository-specific config does
not cross repository boundaries. That applies to this config value, and
to all the others, too (e.g., if you set "diff.renames" in the
super-project, it would not have an effect in the submodule).

I think if there is a doc bug, it is that the repo boundary between the
submodule and the super-project is not made more clear.

That said, I do think it would be a useful feature for the super-project
to rewrite URLs before handing them off to the submodule. But I do not
really work on submodules nor use them myself, so there may be
complications.

I suppose you could argue that failing to rewrite violates the "any" in
the quoted text. It doesn't say when the rewriting occurs, but it is
essentially "when the URL is accessed". So the super-project feeds the
raw URL to the submodule `git clone`, which then applies any URL
rewriting.

quoted

but one workaround is to set the config in ~/.gitconfig.

No can do - that's under version control.
My personal setup does not belong there I think ;-)

I'm not sure I understand. You have a project policy to use certain
URLs. But you, the user, want to override that. Why isn't the
user-specific config file the right place to put that?

(I think there _is_ a mismatch, in that the change is specific not just
to your user, but to the repo. So you would not want to rewrite other
references to the same URL in other repos. But that does not seem to be
your objection).

-Peff

On 20.02.2017 21:52, Jeff King wrote:
 > I think if there is a doc bug, it is that the repo boundary between the
 > submodule and the super-project is not made more clear.

It's not mentioned anywhere I'm aware of, particularly not on the 
insteadOf docs.

 > That said, I do think it would be a useful feature for the super-project
 > to rewrite URLs before handing them off to the submodule. But I do not
 > really work on submodules nor use them myself, so there may be
 > complications.

Agreed.

 > I suppose you could argue that failing to rewrite violates the "any" in
 > the quoted text. It doesn't say when the rewriting occurs, but it is
 > essentially "when the URL is accessed". So the super-project feeds the
 > raw URL to the submodule `git clone`, which then applies any URL
 > rewriting.



 >>> but one workaround is to set the config in ~/.gitconfig.
 >>
 >> No can do - that's under version control.
 >> My personal setup does not belong there I think ;-)
 >
 > I'm not sure I understand. You have a project policy to use certain
 > URLs. But you, the user, want to override that. Why isn't the
 > user-specific config file the right place to put that?

Ah right, I mistook ~/ for "project root" instead of "home dir".
Sorry for the confusion.

 > (I think there _is_ a mismatch, in that the change is specific not just
 > to your user, but to the repo. So you would not want to rewrite other
 > references to the same URL in other repos.

Indeed, and that's actually a problem.

The setup I'm aiming for is
   github -> local bare repo -> local clones with worktrees

If I place insteadOf rules in ~/.gitconfig, I will be unable to pull 
from github to my local bare repos.
Mmm... I could try to undo the insteadOf configuration from ~/.gitconfig 
in the local bare repos. Not sure whether I have to redirect from the 
github URL to itself.

Downside is that I'll have to remember to modify ~/.gitconfig whenever 
the upstream project changes its dependencies. Or whenever I want to 
reorganize my local project directory structure.
It's not totally out of the window, but right now it does not seem very 
attractive to me, and it's certainly not a good solution for everyone.

Regards,
Jo

On Tue, Feb 21, 2017 at 06:11:51AM +0100, Toolforger wrote:

quoted

I'm not sure I understand. You have a project policy to use certain
URLs. But you, the user, want to override that. Why isn't the
user-specific config file the right place to put that?

Ah right, I mistook ~/ for "project root" instead of "home dir".
Sorry for the confusion.

Ah, OK, that makes more sense.

quoted

(I think there _is_ a mismatch, in that the change is specific not just
to your user, but to the repo. So you would not want to rewrite other
references to the same URL in other repos.

Indeed, and that's actually a problem.

The setup I'm aiming for is
  github -> local bare repo -> local clones with worktrees

If I place insteadOf rules in ~/.gitconfig, I will be unable to pull from
github to my local bare repos.
Mmm... I could try to undo the insteadOf configuration from ~/.gitconfig in
the local bare repos. Not sure whether I have to redirect from the github
URL to itself.

Yeah, I think you would probably have to do a redirect-to-self to
override the global one.

At one point we discussed having conditional-config that would kick in
based on path-matching. I think it would be another way to do what you
want, but there's nothing merged.

I think anything involving ~/.gitconfig is basically a hack, though.
What you really want is for submodules to better support your
URL-rewriting case, and that's not an unreasonable thing to want.

We'll see if the submodule folks have any ideas on how to implement
that.

-Peff

On Mon, Feb 20, 2017 at 11:06 PM, Jeff King [off-list ref] wrote:

We'll see if the submodule folks have any ideas on how to implement
that.

So from reading your discussion, the user expectation is to have
`git submodule {init, update --init, sync}`
to pay attention to url.<base>.insteadOf when setting up the
submodule.<name>.URL, such that the modified URL is used for the
initial clone of the submodule (and hence any subsequent usage within
the submodule).

That sounds like a good idea to me.

Two caveates:

* After running `git submodule init`, you change url.<base>.insteadOf
  in the superproject. How do we need to word the documentation to
  have users expecting this change doesn't affect submodules?
  (See above Any vs. "Any except (initialized) submodules")

* So with the point above the insteadOf config only applies to the
  init/sync process, (i.e. once in time, ideally).
  Is that confusing or actually simplifying the submodule workflow?

Thanks,
Stefan

On Tue, Feb 21, 2017 at 10:19:38AM -0800, Stefan Beller wrote:

On Mon, Feb 20, 2017 at 11:06 PM, Jeff King [off-list ref] wrote:

quoted

We'll see if the submodule folks have any ideas on how to implement
that.

So from reading your discussion, the user expectation is to have
`git submodule {init, update --init, sync}`
to pay attention to url.<base>.insteadOf when setting up the
submodule.<name>.URL, such that the modified URL is used for the
initial clone of the submodule (and hence any subsequent usage within
the submodule).

Yeah, that was what I was envisioning.

Two caveates:

* After running `git submodule init`, you change url.<base>.insteadOf
  in the superproject. How do we need to word the documentation to
  have users expecting this change doesn't affect submodules?
  (See above Any vs. "Any except (initialized) submodules")

Good question.

I guess one answer is that this is the wrong approach entirely, and the
right one is something like: submodules should understand that they are
part of a superproject, and respect some whitelisted set of config from
the superproject .git/config file.

The second half is pretty easy to do (use git_config_from_file on the
super-project's $GIT_DIR/config, and pass a callback which filters the
keys before passing them along to the real callback).

I'm not sure about the first half (submodules know about their
superproject), though.

* So with the point above the insteadOf config only applies to the
  init/sync process, (i.e. once in time, ideally).
  Is that confusing or actually simplifying the submodule workflow?

Not sure. That's why I asked you. :)

One other caveat: I'm not sure if we do insteadOf recursively, but it
may be surprising to the child "git clone" that we've already applied
the insteadOf rewriting (especially if the rules are coming from
~/.gitconfig and may be applied twice).

-Peff

On Tue, Feb 21, 2017 at 3:00 PM, Jeff King [off-list ref] wrote:

On Tue, Feb 21, 2017 at 10:19:38AM -0800, Stefan Beller wrote:

quoted

On Mon, Feb 20, 2017 at 11:06 PM, Jeff King [off-list ref] wrote:

quoted

We'll see if the submodule folks have any ideas on how to implement
that.

So from reading your discussion, the user expectation is to have
`git submodule {init, update --init, sync}`
to pay attention to url.<base>.insteadOf when setting up the
submodule.<name>.URL, such that the modified URL is used for the
initial clone of the submodule (and hence any subsequent usage within
the submodule).

Yeah, that was what I was envisioning.

quoted

Two caveates:

* After running `git submodule init`, you change url.<base>.insteadOf
  in the superproject. How do we need to word the documentation to
  have users expecting this change doesn't affect submodules?
  (See above Any vs. "Any except (initialized) submodules")

Good question.

I guess one answer is that this is the wrong approach entirely, and the
right one is something like: submodules should understand that they are
part of a superproject, and respect some whitelisted set of config from
the superproject .git/config file.

This would break one of the core assumptions that submodules
are "independent" repos.

The way of action is a one way street:
* The superproject is aware of the submodule and when you invoke a
command on the superproject, you may mess around with the submodule,
e.g. update/remove it; absorb its git directory.
* The submodule is "just" a repository with weird .git link file and a
  respective core.worktree setup. Currently it doesn't know if it is
  guided by a superproject.


Though I do not know if this is actually a good assumption.
e.g. "[PATCH v2] git-prompt.sh: add submodule indicator"
https://public-inbox.org/git/1486075892-20676-2-git-send-email-email@benjaminfuchs.de/
really had trouble in the first version to nail down how to tell you are in
a submodule, but people want to know that.

The second half is pretty easy to do (use git_config_from_file on the
super-project's $GIT_DIR

There goes the "pretty easy"; currently there is no concept to find out
the existence of a super-project.

/config, and pass a callback which filters the
keys before passing them along to the real callback).

I'm not sure about the first half (submodules know about their
superproject), though.

Maybe we need to change that fundamental assumption.
So a more sophisticated way (thinking long term here) would be
to include the superprojects config file (with exceptions), and that
config file has more priority than e.g. the ~/.gitconfig file, but less
than the submodules own $GIT_DIR/config file.
Then a setting like the url rewriting would be "inherited" by the
submodule, with the option to overwrite the default as given by the
superproject.

quoted

* So with the point above the insteadOf config only applies to the
  init/sync process, (i.e. once in time, ideally).
  Is that confusing or actually simplifying the submodule workflow?

Not sure. That's why I asked you. :)

I think that would be ok. With the idea of inheriting the superprojects
config, we allow for not storing the rewritten url, so the submodule
handling is less of a corner case here, and as another advantage the
rewriting rule is applied in real time, e.g. you can change the superprojects
rule after the fact and the submodule would automagically make use of it.

One other caveat: I'm not sure if we do insteadOf recursively, but it
may be surprising to the child "git clone" that we've already applied
the insteadOf rewriting (especially if the rules are coming from
~/.gitconfig and may be applied twice).

When a rule is having effect twice the rule sounds broken. (the outcome
ought to be sufficiently different from the original?)

-Peff

Thanks,
Stefan

On Tue, Feb 21, 2017 at 03:16:27PM -0800, Stefan Beller wrote:

quoted

I guess one answer is that this is the wrong approach entirely, and the
right one is something like: submodules should understand that they are
part of a superproject, and respect some whitelisted set of config from
the superproject .git/config file.

This would break one of the core assumptions that submodules
are "independent" repos.

Yeah, that was the "first half" that I said was hard. :)

You could rationalize it under the fact that they _are_ independent
repos; we're just adding a new config source.  Arguably it could be a
feature for any repository embedded inside the working tree of another,
submodule or not, to consider the outer repository as a (limited) source
of config.

But there are probably a lot of irritating corner cases with the whole
concept unless we apply a strict whitelist of keys (e.g., you probably
don't want remote.* to be propagated). And as the recent
GIT_CONFIG_PARAMETERS whitelist showed, that approach ended up confusing
and annoying.

So maybe the whole thing is insane, and the right answer is that config
values should go into ~/.gitconfig. And we may need better tools there
for limiting that global config to certain parts of the tree (like Duy's
conditional include thing).

Though I do not know if this is actually a good assumption.
e.g. "[PATCH v2] git-prompt.sh: add submodule indicator"
https://public-inbox.org/git/1486075892-20676-2-git-send-email-email@benjaminfuchs.de/
really had trouble in the first version to nail down how to tell you are in
a submodule, but people want to know that.

Right, I think it's an interesting thing to know, but I agree there are
probably a lot of corner cases.

Maybe we need to change that fundamental assumption.
So a more sophisticated way (thinking long term here) would be
to include the superprojects config file (with exceptions), and that
config file has more priority than e.g. the ~/.gitconfig file, but less
than the submodules own $GIT_DIR/config file.

Yeah, that priority matches what I had been thinking.

quoted

One other caveat: I'm not sure if we do insteadOf recursively, but it
may be surprising to the child "git clone" that we've already applied
the insteadOf rewriting (especially if the rules are coming from
~/.gitconfig and may be applied twice).

When a rule is having effect twice the rule sounds broken. (the outcome
ought to be sufficiently different from the original?)

If you have:

  url.bar.insteadOf=foo
  url.baz.insteadOf=bar

do we convert "foo" to "baz"? If so, then I think applying the rules
again shouldn't matter. But if we don't, and only do a single level,
then having the caller rewrite the URL before it hands it to "git clone"
means we may end up unexpectedly doing two levels of rewriting.

-Peff

On Tue, Feb 21, 2017 at 3:40 PM, Jeff King [off-list ref] wrote:

quoted

quoted

One other caveat: I'm not sure if we do insteadOf recursively, but it
may be surprising to the child "git clone" that we've already applied
the insteadOf rewriting (especially if the rules are coming from
~/.gitconfig and may be applied twice).

When a rule is having effect twice the rule sounds broken. (the outcome
ought to be sufficiently different from the original?)

If you have:

  url.bar.insteadOf=foo
  url.baz.insteadOf=bar

do we convert "foo" to "baz"? If so, then I think applying the rules
again shouldn't matter. But if we don't, and only do a single level,
then having the caller rewrite the URL before it hands it to "git clone"
means we may end up unexpectedly doing two levels of rewriting.

I see. Thanks for the example. So really what we want is to record the
unencumbered URL (with no rewriting) and then at run time lookup various
places of url.*.insteadOf (which might change with the git version
that you use)

Thanks,
Stefan