Composing git repositories

(Changed subject)
(+CC: Peff; I suspect he'll be interested in the repository
composition discussion)

Jens Lehmann wrote:

Am 25.03.2013 20:57, schrieb Ramkumar Ramachandra:

quoted

Okay, I'll do it step-by-step now, with a live example:
[...]
 What is going on, seriously?

Pilot error, mostly omitting the --recursive option and some
- fixable - usability issues. Patches welcome.

As a user inexperienced with recursive submodules (I've only used them
in this repository), I found it highly confusing.  Thanks for clearing
them up.

It was highly exaggerated and dramatized to make the following points clear:

1. With the current limitation of cd-to-toplevel, it's extremely
unpleasant to work with many levels of nesting.

2. I thought no operation could be performed without cd-to-toplevel,
as even 'git submodule foo' complains about this (instead of saying
"command not found").  Having to go to each submodule subdirectory to
perform operations is just horrible.

3. To change a submodule, I'll have to propagate the change upwards,
creating new commits in each of the outer submodules, and the
superproject.

Apart from the implementation glitches, I don't like the design;
submodules don't compose well:

1. There's an inherent asymmetry between the superproject and each of
the subprojects, because the superproject owns all the object stores.
Why is it absolutely necessary to relocate the object stores?

2. The metadata is tracked as a file (.gitmodules) in the git
repository.  While it makes it possible for different branches to have
different submodules, it's impossible to say 'git submodule sync
a/b/c/d' (see: 'repo sync'), where b, c haven't been initialized.

3. The current implementation only allows me to compose with commit
objects, but what if I want to compose with refs?  ie. What if I want
to track the tip of the 'master' of a submodule in a superproject?  I
don't want to commit-cascade everytime I make a minor change in a
deeply nested submodule.

Other solutions such as repo (see: depot_tools) solve different
problems but are huge failures when it comes to composability; repo
uses a central manifest repository, for example.  Is it impossible to
build a tool that can truly compose git repositories?  I think
submodules gets a lot of things Right, and we'd have to start from
there.

quoted

This is just two levels of nesting: with more levels of nesting,
things only get worse.

Yes, you cannot have the cake and eat it. Either you incorporate
everything into a single repo (e.g. using subtree) and loose the
strong distinction which content belongs to which upstream repo
(which AFAIK is a valid choice unless you want to contribute back
to the submodule's upstream) or you'll have to cope with the
submodule borders showing up from time to time, reminding you
which part of the work tree has another upstream.

I don't think it's impossible.  We just haven't thought hard enough
about composition.

In message [off-list ref], Ramkumar Ramachandra writes:

    As a user inexperienced with recursive submodules (I've only used them
    in this repository), I found it highly confusing.  Thanks for clearing
    them up.

You may want to investigate third party alternatives like gitslave
http://gitslave.sf.net Depending on what problem you are trying to
solve, it can be better (or worse) to use compared to submodules.

It provides a usually conceptually easier method to group subprojects
together into a superproject.  You can replace practically any git
command you want with "gits" and it will usually work as you might
more or less expect.  Conceptually it has a list of git repositories
and will execute the listed git command on each in turn.

It seems to resolve most of the issues that you raise, but of course
it has some warts of its own.  Some could be resolved with sufficient
effort, others are fundamental.  (An example of the latter, you cannot
trivially tell what commit in other repositories a particular user was
at when he made a commit in a specific repository (absent a gits tag
being created).  An example of the former, if you have git output
paging turned on and many subprojects to check out, `gits clone` pages
the output and more to the point, blocks the clones until you page
through the output which you must typically do many times).

					-Seth Robertson

Seth Robertson wrote:

In message [off-list ref], Ramkumar Ramachandra writes:

    As a user inexperienced with recursive submodules (I've only used them
    in this repository), I found it highly confusing.  Thanks for clearing
    them up.

You may want to investigate third party alternatives like gitslave
http://gitslave.sf.net Depending on what problem you are trying to
solve, it can be better (or worse) to use compared to submodules.

I'm not looking for yet another specialized tool for my special
combined-repository workflow.  I'm looking to build one generalized
tool that will be able to do various kinds of compositions equally
well.

[...]

Thanks.  I learnt one more workflow that people want to use.

It's one large Perl script.  Quite an involved hack.