Hi!
I have the feeling that “git submodule update --depth 1” is less clever
than it could be. Here is one example I observed with git 2.0.0:
git init foo
cd foo
git clone --single-branch \
-b v0.99 https://github.com/git/git.git git-scm
git submodule add https://github.com/git/git.git git-scm
git commit -m Submod
git clone --dissociate . ../bar
cd ../bar
git submodule update --init --depth 1 git-scm
This will download quite a bit of history, then result in an error message:
error: no such remote ref a3eb250f996bf5e12376ec88622c4ccaabf20ea8
Fetched in submodule path 'git-scm', but it did not contain
a3eb250f996bf5e12376ec88622c4ccaabf20ea8. Direct fetching of that
commit failed.
That seems so avoidable, since the commit in question is a tag, so it
would be perfectly possible to fetch that specific commit from the
server directly. Something like the following commands would do the trick:
git fetch $url $(git ls-remote $url | \
awk /$sha1/'{print $2}' | sed 's/\^{}//')
If the commit in question is NOT a ref, then whether asking for it by
unlisted SHA1 is supported will probably depend on the server's
uploadpack.allowReachableSHA1InWant setting. I guess this is a reason
why fb43e31 made the fetch for a specific SHA1 a fallback after the
fetch for the default branch. Nevertheless, in case of “--depth 1” I
think it would make sense to abort early: if none of the listed refs
matches the requested one, and asking by SHA1 isn't supported by the
server, then there is no point in fetching anything, since we won't be
able to satisfy the submodule requirement either way.
For the case of “--depth n” with n > 1, I was wondering whether it would
make sense to prefer the branch listed in submodule.‹name›.branch over
the default branch.
I think shallow submodules would be very useful to embed libraries into
projects, without too much care for history (and without the download
times getting it entails), but with efficient updates to affected files
only in case of a change in library version. But not being able to get a
specific tag as a shallow submodule is a major showstopper here, I think.
Greetings,
Martin von Gagern
On Mon, Jun 27, 2016 at 10:39 PM, Martin von Gagern
[off-list ref] wrote:
Hi!
I have the feeling that “git submodule update --depth 1” is less clever
than it could be. Here is one example I observed with git 2.0.0:
2.9.0 (as "Direct fetching of " is not part of 2.0.0 IIRC) ?
git init foo
cd foo
git clone --single-branch \
-b v0.99 https://github.com/git/git.git git-scm
git submodule add https://github.com/git/git.git git-scm
git commit -m Submod
git clone --dissociate . ../bar
cd ../bar
git submodule update --init --depth 1 git-scm
This will download quite a bit of history, then result in an error message:
error: no such remote ref a3eb250f996bf5e12376ec88622c4ccaabf20ea8
Fetched in submodule path 'git-scm', but it did not contain
a3eb250f996bf5e12376ec88622c4ccaabf20ea8. Direct fetching of that
commit failed.
Yeah there are a few things going on, which try to cover up an error
in design IMO.
* The depth is measured from the tip of a branch in the submodule,
not from the sha1 that the superproject points to.
* Shallowness is treated separately in the superproject and submodules as they
have a strong notion of being independent. It would be cool to have a thing
`git clone --recurse-submodules --depth=15
--submodule-depth-as-reachable-from-superproject`
which would obtain the submodules as shallow as possible, but it
includes all versions that
the 15 commits in the superproject points to. (may be 1 up to 15
different non-sequential versions)
That seems so avoidable, since the commit in question is a tag, so it
would be perfectly possible to fetch that specific commit from the
server directly. Something like the following commands would do the trick:
git fetch $url $(git ls-remote $url | \
awk /$sha1/'{print $2}' | sed 's/\^{}//')
* `git submodule update --init --depth 1` is using clone instead of fetch
currently when the submodule doesn't exist yet. The clone is buried in
the `submodule--helper update-clone` that is a mixture of listing
the submodules
and cloning multiple submodules in parallel if possible. So I would
assume it is
easier to teach git clone to behave correctly and then stop retrying
in git-submodule.sh
if `just_cloned` is set in the `cmd_update()`.
If the commit in question is NOT a ref, then whether asking for it by
unlisted SHA1 is supported will probably depend on the server's
uploadpack.allowReachableSHA1InWant setting. I guess this is a reason
why fb43e31 made the fetch for a specific SHA1 a fallback after the
fetch for the default branch. Nevertheless, in case of “--depth 1” I
think it would make sense to abort early: if none of the listed refs
matches the requested one, and asking by SHA1 isn't supported by the
server, then there is no point in fetching anything, since we won't be
able to satisfy the submodule requirement either way.
Makes sense! I think the easiest way forward to implement this will be:
* `git clone` learns a (maybe undocumented internal) option `--get-sha1`
`--branch` looks similar to what we want, but doesn't quite fit as we do not
know, whether we're on a tag or not. The submodule tells us just the
recorded sha1, not the branch/tag. So maybe we'd end up calling it
`--detach-at=<sha1>`, that will
-> inspect the ls-remote for the sha1 being there
-> if the sha1 is there (at least once) clone as if --branch <tag> was given
-> if not found and the server advertised allowReachableSHA1InWant,
try again inside the clone
* `submodule--helper update-clone` passes the `--get-sha1` to the
clones of the submodules
* cmd_update() in git-submodule.sh will only checkout submodules and
not try again
to fetch them if `just_cloned` is set as the cloning did the best it could.
For the case of “--depth n” with n > 1, I was wondering whether it would
make sense to prefer the branch listed in submodule.‹name›.branch over
the default branch.
Makes sense to me.
I think shallow submodules would be very useful to embed libraries into
projects, without too much care for history (and without the download
times getting it entails), but with efficient updates to affected files
only in case of a change in library version. But not being able to get a
specific tag as a shallow submodule is a major showstopper here, I think.
Thanks for taking your time to point this out and start this discussion!
Thanks,
Stefan
Greetings,
Martin von Gagern
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Stefan,
On 28.06.2016 19:20, Stefan Beller wrote:
quoted
I have the feeling that “git submodule update --depth 1” is less clever
than it could be. Here is one example I observed with git 2.0.0:
2.9.0 (as "Direct fetching of " is not part of 2.0.0 IIRC) ?
Yes, sorry. I had this tested with 2.8.3 at first, then waited for my
update to 2.9.0 to reproduce, and garbled the text while adjusting it.
Makes sense! I think the easiest way forward to implement this will be:
* `git clone` learns a (maybe undocumented internal) option `--get-sha1`
`--branch` looks similar to what we want, but doesn't quite fit as we do not
know, whether we're on a tag or not. The submodule tells us just the
recorded sha1, not the branch/tag. So maybe we'd end up calling it
`--detach-at=<sha1>`,
That name makes a lot of sense to me.
that will
-> inspect the ls-remote for the sha1 being there
-> if the sha1 is there (at least once) clone as if --branch <tag> was given
Clone but detach, to be consistent. Yes.
-> if not found and the server advertised allowReachableSHA1InWant,
try again inside the clone
All of this has to pass through transport and get-pack, right?
* `submodule--helper update-clone` passes the `--get-sha1` to the
clones of the submodules
* cmd_update() in git-submodule.sh will only checkout submodules and
not try again
to fetch them if `just_cloned` is set as the cloning did the best it could.
Sounds like a very reasonable roadmap to me.
Do you think there will be someone volunteering to tackle this?
Greetings,
Martin
On Tue, Jun 28, 2016 at 12:08 PM, Martin von Gagern
[off-list ref] wrote:
Hi Stefan,
On 28.06.2016 19:20, Stefan Beller wrote:
quoted
quoted
I have the feeling that “git submodule update --depth 1” is less clever
than it could be. Here is one example I observed with git 2.0.0:
2.9.0 (as "Direct fetching of " is not part of 2.0.0 IIRC) ?
Yes, sorry. I had this tested with 2.8.3 at first, then waited for my
update to 2.9.0 to reproduce, and garbled the text while adjusting it.
quoted
Makes sense! I think the easiest way forward to implement this will be:
* `git clone` learns a (maybe undocumented internal) option `--get-sha1`
`--branch` looks similar to what we want, but doesn't quite fit as we do not
know, whether we're on a tag or not. The submodule tells us just the
recorded sha1, not the branch/tag. So maybe we'd end up calling it
`--detach-at=<sha1>`,
That name makes a lot of sense to me.
quoted
that will
-> inspect the ls-remote for the sha1 being there
-> if the sha1 is there (at least once) clone as if --branch <tag> was given
Clone but detach, to be consistent. Yes.
Oh, right.
quoted
-> if not found and the server advertised allowReachableSHA1InWant,
try again inside the clone
All of this has to pass through transport and get-pack, right?
Yeah we have to go through the transport layer, i.e. from builtin/clone.c we
manipulate the transport object as defined in transport.h and code in
transport.c
What do you mean by get-pack, though?
quoted
* `submodule--helper update-clone` passes the `--get-sha1` to the
clones of the submodules
* cmd_update() in git-submodule.sh will only checkout submodules and
not try again
to fetch them if `just_cloned` is set as the cloning did the best it could.
Sounds like a very reasonable roadmap to me.
Do you think there will be someone volunteering to tackle this?
I have it on my (low priority) TODO list, so if you want it sooner
than later, go for
it yourself. Otherwise just wait. Maybe Duy has some interest as well.
Thanks,
Stefan
Greetings,
Martin