"Lauri Alanko" [off-list ref] writes:
I intend to work on a "subrepository" tool for git, but before I
embark on the actual programming, I thought to first invite comments
on the general design.
Some background first. I know that there are several existing
approaches already for managing nested repositories, but none of them
quite seems to fit my purposes. My primary goal is to use git for home
directory backup and mirroring, while the home directory itself may of
course contain repositories.
...
Submodules are a bit closer to what I want, but they have clearly been
designed for a different purpose: a repository with submodules is only
supposed to collate existing repositories, not act as a source for
them.
I have a repository that covers my home directory and some of its
subdirectories have their own repositories.
I had my home directory and its subdirectories before Git ever
existed, and I made my home directory and these subdirectories into
separate, nested Git repositories fairly early after I started
managing them with Git---way before submodules were invented. Now
the subdirectory repositories are bound as submodules of the top
level directory just fine.
I push these out for safekeeping purposes, all of my machines get
their copies from here, and some submodules are not cloned to work
machines (they house data of private nature). They are used just
like you are expected to use submodules. In fact, this is pretty
much vanilla use case of submodules, I think.
They _all_ originate from under my home directory, not "collating
existing repositories" at all.
Have you considered how you can _extend_ submodules support to
support your use case better? I think that would be a much more
useful approach, as you are likely to get help from other people who
do use submodules.
Quoting "Junio C Hamano" [off-list ref]:
Now
the subdirectory repositories are bound as submodules of the top
level directory just fine.
This is indeed possible, but with some serious caveats.
Firstly, if you simply do "git submodule add ./foo" (the obligatory
"./" being quite an unobvious pitfall), you get something quite
fragile, since now we have submodule.foo.url = ./foo. If the
submodules ever get reorganized and foo is moved to ./bar, then it is
impossible to check out older versions or alternate branches, since
the submodule is no longer where it is expected to be at the origin.
A more robust solution is to use submodule.foo.url =
./.git/modules/foo, since logical name of a module doesn't change.
This seems quite kludgy, though, and this cannot be how git-submodule
is supposed to be used.
But still, "git submodule update" only looks at the modules in the
currently checked-out tree. If we have other branches or old tags that
refer to other submodules, there's no simple way to fetch those, too.
And there is not even such a concept as a bare repository with modules.
So git-submodule is fundamentally a tool to attach repositories into a
tree, not to attach repositories into a repository. That's why it's
not really fit for my purposes.
The core problem is that to clone an entire repository and all its
submodules, there needs to be a way to list them all remotely. But the
git protocol doesn't just allow us to list the subdirectories under
.git/modules. Still, there are several ways to do this:
* Just read .gitmodules in every ref and find by brute force every
submodule referred to even by a single ref. This doesn't really scale.
* Maintain a list of all the submodules in a repository. This would
have to be in a separate metadata branch, and would get rather hairy
when we need to merge from a remote that has added other submodules.
* Represent the submodules as refs instead of independent
repositories. This is my proposal for subrepositories.
However, I feel that all of these are too drastic changes to make in
git-submodule, given that it is already well-established.
The minor problems, like lack of active branch tracking and multiple
mount points of a module, could in principle be fixed in
git-submodule. But again, I have no fondness for complex shell
programming. Perhaps it was justified when the only interface to git's
functionality were the command-line tools, but nowadays there are
various ways to manipulate git repositories from real programming
languages through real libraries (libgit2, dulwich, etc), and I prefer
to use those, so I don't really have any motivation to touch
git-submodule.
Lauri