From: Junio C Hamano <hidden> Date: 2017-08-19 06:51:30
Stefan Beller [off-list ref] writes:
Jonathan brought up the following very long term vision:
Eventually the everyday git commands do not treat submodules
any special than trees, even the submodules git directory
may be non existent (everything is absorbed into the superproject);
so it really feels like a monorepo.
That may be one valid option to have but I do not see a reason why
it needs to be the only valid option. So I do not see why you are
bringing it up in this thread.
But that is a good starting point to discuss one possible future.
Let's think aloud how that world would look like.
* When you "git clone" a superproject (perhaps implicitly with the
"--recurse-submodules" option), the top-level project and all of
its submodules will be checked out on the same branch (presumably
the 'master' branch, which is the default).
* Your attempt to "git commit", "git branch", "git checkout -b",
etc. inside a submodule will either fail, or will implicitly
chdir up to the top-level superproject and turn into the
corresponding command with "--recurse-submodules".
* "git commit --recurse-submodules -a" from the top-level will grab
all the local changes, depth-first and recursively, in
submodules, makes a commit, binds the new commit to the index of
the superproject that immediately contains the submodule and
makes a commit in it, until it percolates all the way up to the
superproject. When working in this mode, branches in submodules
do not really matter; the gitlink in the superproject is the only
thing that matters.
* It naturally follows that between two adjacent commits C and C~1
in the superproject's history, the commit in a submodule bound to
C and the commit in a submodule bound to C~1 are either the same
(i.e. superproject made a commit but there was no change in the
submodule), or they are in direct parent-child relationship
(i.e. the local changes made to the submodule was recorded as a
single commit when the superproject made the commit).
* "git push --recurse-submodules" from the top-level will push the
history of the superproject out, together with the history of the
submodules.
I think it is doable, but a mechanism to enumerate all the commits
bound from submodules to a range of superproject's commits needs to
be invented to drive the pack-objects for efficient object transfer.
Having it would also help in fsck and gc---as branches are immaterial
in the submodule repositories, commits in superprojects that are
reachable from refs will have to serve as the connectivity anchors
for commit DAG in the submodule histories.
As long as we are talking about idealized future world (well, at
least an idea of somebody's "ideal", not necessarily shared by
everybody), I wonder if there is even any need to have commits in
submodules in such a world. To realize such a "monorepo" world, you
might be better off allowing a gitlink in the superproject to
directly point at a tree object in a submodule repository (making
them physically a single repository is an optional implementation
detail I choose to ignore in this discussion).
On Fri, Aug 18, 2017 at 11:51:13PM -0700, Junio C Hamano wrote:
As long as we are talking about idealized future world (well, at
least an idea of somebody's "ideal", not necessarily shared by
everybody), I wonder if there is even any need to have commits in
submodules in such a world. To realize such a "monorepo" world, you
might be better off allowing a gitlink in the superproject to
directly point at a tree object in a submodule repository (making
them physically a single repository is an optional implementation
detail I choose to ignore in this discussion).
IMO this is one step to far. One main use of submodules are shared
repositories that are used by many superprojects. The reason you want to
have commits in the submodule are so that you can push them
independently and all other users can pick up the changes. You could get
by by Using the superproject commits for the submodule once you push or
something but those do not necessarily make sense in the context of the
submodule.
So I think it is important that there are commits in the submodule so
its history makes sense independently for others.
Or how would you push out the history in the submodule in your idea?
Maybe I am missing something? What would be your use case with gitlinks
pointing to trees?
Cheers Heiko
From: Stefan Beller <hidden> Date: 2017-08-21 16:43:00
On Mon, Aug 21, 2017 at 9:05 AM, Heiko Voigt [off-list ref] wrote:
On Fri, Aug 18, 2017 at 11:51:13PM -0700, Junio C Hamano wrote:
quoted
As long as we are talking about idealized future world (well, at
least an idea of somebody's "ideal", not necessarily shared by
everybody), I wonder if there is even any need to have commits in
submodules in such a world. To realize such a "monorepo" world, you
might be better off allowing a gitlink in the superproject to
directly point at a tree object in a submodule repository (making
them physically a single repository is an optional implementation
detail I choose to ignore in this discussion).
IMO this is one step to far. One main use of submodules are shared
repositories that are used by many superprojects. The reason you want to
have commits in the submodule are so that you can push them
independently and all other users can pick up the changes. You could get
by by Using the superproject commits for the submodule once you push or
something but those do not necessarily make sense in the context of the
submodule.
So I think it is important that there are commits in the submodule so
its history makes sense independently for others.
Or how would you push out the history in the submodule in your idea?
Maybe I am missing something? What would be your use case with gitlinks
pointing to trees?
Well there are still commits, but in the superproject the UX feels more
as if the submodules were special trees. So if you want to interact with
the submodule specifically, you could do things like
git add /path/inside/sub
# works seamlessly from the superproject tree
git commit --submodule-commit-only
# When the flag is not give, you may get an editor
# asking for two commit messages, (sub+super)
git fetch --submodule
# When the flag is not given, we'd fetch superproject and
# on-demand
# You feel the superproject is in the way?
git worktree add --for-submodule <path/to/sub> ...
# The new submodule worktree puts the
# submodule only UX first. so it feels like its own
# repository, no need for specific flags.
On Mon, Aug 21, 2017 at 09:42:54AM -0700, Stefan Beller wrote:
On Mon, Aug 21, 2017 at 9:05 AM, Heiko Voigt [off-list ref] wrote:
quoted
On Fri, Aug 18, 2017 at 11:51:13PM -0700, Junio C Hamano wrote:
quoted
As long as we are talking about idealized future world (well, at
least an idea of somebody's "ideal", not necessarily shared by
everybody), I wonder if there is even any need to have commits in
submodules in such a world. To realize such a "monorepo" world, you
might be better off allowing a gitlink in the superproject to
directly point at a tree object in a submodule repository (making
them physically a single repository is an optional implementation
detail I choose to ignore in this discussion).
IMO this is one step to far. One main use of submodules are shared
repositories that are used by many superprojects. The reason you want to
have commits in the submodule are so that you can push them
independently and all other users can pick up the changes. You could get
by by Using the superproject commits for the submodule once you push or
something but those do not necessarily make sense in the context of the
submodule.
So I think it is important that there are commits in the submodule so
its history makes sense independently for others.
Or how would you push out the history in the submodule in your idea?
Maybe I am missing something? What would be your use case with gitlinks
pointing to trees?
Well there are still commits, but in the superproject the UX feels more
as if the submodules were special trees.
Ah ok then I misunderstood. So they only feel like trees.
So if you want to interact with
the submodule specifically, you could do things like
git add /path/inside/sub
# works seamlessly from the superproject tree
Would that mean that we need to loosen/keep the requirement loose for a
name from .gitmodules? I am asking because of my series for on-demand
fetch of renamed submodules. For the full functionality I would require
a name.
Would that be in a scenario where the user would then e.g. push the
submodule into the superproject?
Ah wait I misunderstood again. You mean a file in an existing
submodule right? Not adding submodule from a repository a user moved
there?
git commit --submodule-commit-only
# When the flag is not give, you may get an editor
# asking for two commit messages, (sub+super)
Or maybe something like
git commit --submodule path/to/submodule
so the user can specify which submodule she wants. I first wrote it
without the switch but but that collides with listing files which should
be added. IMO this shorter option is also more intuitive to understand
what it does (for this case).
git fetch --submodule
# When the flag is not given, we'd fetch superproject and
# on-demand
Yes like above we should add the path to the submodule right?
# You feel the superproject is in the way?
git worktree add --for-submodule <path/to/sub> ...
# The new submodule worktree puts the
# submodule only UX first. so it feels like its own
# repository, no need for specific flags.
I am not sure I understand this one. What would that do? Put a worktree
for submodule path/to/sub to ...?
Overall I like the direction of these ideas.
Cheers Heiko
From: Stefan Beller <hidden> Date: 2017-08-22 18:11:00
On Tue, Aug 22, 2017 at 8:33 AM, Heiko Voigt [off-list ref] wrote:
On Mon, Aug 21, 2017 at 09:42:54AM -0700, Stefan Beller wrote:
quoted
On Mon, Aug 21, 2017 at 9:05 AM, Heiko Voigt [off-list ref] wrote:
quoted
On Fri, Aug 18, 2017 at 11:51:13PM -0700, Junio C Hamano wrote:
quoted
As long as we are talking about idealized future world (well, at
least an idea of somebody's "ideal", not necessarily shared by
everybody), I wonder if there is even any need to have commits in
submodules in such a world. To realize such a "monorepo" world, you
might be better off allowing a gitlink in the superproject to
directly point at a tree object in a submodule repository (making
them physically a single repository is an optional implementation
detail I choose to ignore in this discussion).
IMO this is one step to far. One main use of submodules are shared
repositories that are used by many superprojects. The reason you want to
have commits in the submodule are so that you can push them
independently and all other users can pick up the changes. You could get
by by Using the superproject commits for the submodule once you push or
something but those do not necessarily make sense in the context of the
submodule.
So I think it is important that there are commits in the submodule so
its history makes sense independently for others.
Or how would you push out the history in the submodule in your idea?
Maybe I am missing something? What would be your use case with gitlinks
pointing to trees?
Well there are still commits, but in the superproject the UX feels more
as if the submodules were special trees.
Ah ok then I misunderstood. So they only feel like trees.
quoted
So if you want to interact with
the submodule specifically, you could do things like
git add /path/inside/sub
# works seamlessly from the superproject tree
Would that mean that we need to loosen/keep the requirement loose for a
name from .gitmodules? I am asking because of my series for on-demand
fetch of renamed submodules. For the full functionality I would require
a name.
Would that be in a scenario where the user would then e.g. push the
submodule into the superproject?
Ah wait I misunderstood again. You mean a file in an existing
submodule right? Not adding submodule from a repository a user moved
there?
Assuming the submodule is at /path in this example, the effect of
that command could be achieved today via
git -C /path add inside/sub
(i.e. for git-add we "just" detect that there is a submodule boundary
and run the git-add inside the submodule)
quoted
git commit --submodule-commit-only
# When the flag is not give, you may get an editor
# asking for two commit messages, (sub+super)
Or maybe something like
git commit --submodule path/to/submodule
Yes. In todays UX, you do
git -C path/to/submodule commit
for the command that you proposed. For a plain
git-commit in the superproject, we could envision
this sequence of todays commands:
git -C submodule commit
git add submodule
git commit
so the user can specify which submodule she wants. I first wrote it
without the switch but but that collides with listing files which should
be added. IMO this shorter option is also more intuitive to understand
what it does (for this case).
quoted
git fetch --submodule
# When the flag is not given, we'd fetch superproject and
# on-demand
Yes like above we should add the path to the submodule right?
yes.
quoted
# You feel the superproject is in the way?
git worktree add --for-submodule <path/to/sub> ...
# The new submodule worktree puts the
# submodule only UX first. so it feels like its own
# repository, no need for specific flags.
I am not sure I understand this one. What would that do? Put a worktree
for submodule path/to/sub to ...?
Yes, and at "..." you would have the UX of the submodule being
its own repository, no interaction with the superproject.
Overall I like the direction of these ideas.
Cheers Heiko
On Tue, Aug 22, 2017 at 11:10:52AM -0700, Stefan Beller wrote:
On Tue, Aug 22, 2017 at 8:33 AM, Heiko Voigt [off-list ref] wrote:
quoted
On Mon, Aug 21, 2017 at 09:42:54AM -0700, Stefan Beller wrote:
quoted
On Mon, Aug 21, 2017 at 9:05 AM, Heiko Voigt [off-list ref] wrote:
quoted
On Fri, Aug 18, 2017 at 11:51:13PM -0700, Junio C Hamano wrote:
# You feel the superproject is in the way?
git worktree add --for-submodule <path/to/sub> ...
# The new submodule worktree puts the
# submodule only UX first. so it feels like its own
# repository, no need for specific flags.
I am not sure I understand this one. What would that do? Put a worktree
for submodule path/to/sub to ...?
Yes, and at "..." you would have the UX of the submodule being
its own repository, no interaction with the superproject.
Are you sure that is what Junio meant? IMO that would be 'git worktree
clone' or 'git worktree checkout', no? In todays git terminology an
'add' is something that puts data into the repository / the index. That
is why I was/am confused.
Cheers Heiko
From: Stefan Beller <hidden> Date: 2017-08-25 16:38:18
On Fri, Aug 25, 2017 at 2:10 AM, Heiko Voigt [off-list ref] wrote:
On Tue, Aug 22, 2017 at 11:10:52AM -0700, Stefan Beller wrote:
quoted
On Tue, Aug 22, 2017 at 8:33 AM, Heiko Voigt [off-list ref] wrote:
quoted
On Mon, Aug 21, 2017 at 09:42:54AM -0700, Stefan Beller wrote:
quoted
On Mon, Aug 21, 2017 at 9:05 AM, Heiko Voigt [off-list ref] wrote:
quoted
On Fri, Aug 18, 2017 at 11:51:13PM -0700, Junio C Hamano wrote:
# You feel the superproject is in the way?
git worktree add --for-submodule <path/to/sub> ...
# The new submodule worktree puts the
# submodule only UX first. so it feels like its own
# repository, no need for specific flags.
I am not sure I understand this one. What would that do? Put a worktree
for submodule path/to/sub to ...?
Yes, and at "..." you would have the UX of the submodule being
its own repository, no interaction with the superproject.
Are you sure that is what Junio meant?
I am not sure at all, what Junio thought about this.
IMO that would be 'git worktree
clone' or 'git worktree checkout', no? In todays git terminology an
'add' is something that puts data into the repository / the index. That
is why I was/am confused.
I went with current gits:
$ git worktree
usage: git worktree add [<options>] <path> [<branch>]
or: git worktree list [<options>]
or: git worktree lock [<options>] <path>
or: git worktree prune [<options>]
or: git worktree unlock <path>
where the 'add' adds a new worktree (my mental emphasis was on the
option that says: "Make the new worktree for the submodule, such
that in that worktree the submodule feels like the toplevel
repository". If 'add' is a bad naming choice, we may want to
discuss&deprecate it in a new thread as that is related to the
worktree area?)
Thanks,
Stefan
From: Stefan Beller <hidden> Date: 2017-08-21 16:46:29
On Fri, Aug 18, 2017 at 11:51 PM, Junio C Hamano [off-list ref] wrote:
As long as we are talking about idealized future world (well, at
least an idea of somebody's "ideal", not necessarily shared by
everybody), I wonder if there is even any need to have commits in
submodules in such a world. To realize such a "monorepo" world, you
might be better off allowing a gitlink in the superproject to
directly point at a tree object in a submodule repository (making
them physically a single repository is an optional implementation
detail I choose to ignore in this discussion).
Then the sharing between superprojects (e.g. send an Android's linux
patch upstream or to another distro that also uses a superproject),
becomes cumbersome as the commit messages are missing and
potentially not specific to that subtree.