Re: [RFC] Submodules in GIT

Andy Parkins wrote:

On Thursday 2006 November 30 11:57, sf wrote:

quoted

quoted

Worse, if you allow that to happen, the supermodule can commit a state
that cannot be retrieved from the submodule's repository.  The ONLY thing
a supermodule can record about a submodule is a commit.

So what? You have a submodule commit that only exists in the
supermodule. I fail to see the problem. The changes you made to the
submodule _in the supermodule_ can later be pulled from wherever you want.

Eh?  The files aren't stored in the supermodule, they're stored in the 
submodule.  The ONLY thing in the supermodule is the commit hash.  The 
objects for the submodule are still /in/ the submodule.

But you have got the submodule on your local disk anyway. So just setup 
alternates and the supermodule contains all of the submodule.

It sounds like you're suggesting that the supermodule commit includes files 
from the submodule?  How can that work?   The two aren't separate entities 
then, it's just one big repository. 

It works as it always works in git: The supermodule commit contains the 
submodule commit, the submodule commit contains the submodule files, so 
the supermodule contains the submodule (at least the part of the 
submodule that is visible). It _must_ be one repository but it need not 
be big (once more, use alternates).

I mean, what would this supermodule commit look like?  Would it include a 
commit message?  Which module should that commit message be about?  Should 
the commit's parents be stored?  Which parents, the submodule HEAD or the 
supermodule HEAD?  Which tree object should it link to?  The one in the 
submodule doesn't exist, so it'll have to be a freshly made up one for the 
supermodule - except now you've put submodule paths in the supermodule.  
Nope.  That's never going to work.

Again I do not see the problem. Probably I have a much simpler picture 
of submodules: They are just commits in the supermodule's tree. 
Everything else follows naturally from how git currently behaves.

Of course it works. It is simple, it is the git way.

Am I missing the point?

Regards

Stephan

-- 
b.i.t.
beratungsgesellschaft für informations-technologie mbh
Stephan Feder
elisabethenstr. 62   fon: +49(0)6151/827575
64283 darmstadt      fax: +49(0)6151/827576
mailto:sf@b-i-t.de   www: http://www.b-i-t.de

On Fri, Dec 01, 2006 at 10:29:26AM +0000, Andy Parkins wrote:

I notice though that you avoided my question: what does YOUR submodule object 
contain?

He showed it to you in the example.  The "submodule object" is the COMMIT
of the submodule itself.

On Friday 2006 December 01 10:42, Sven Verdoolaege wrote:

He showed it to you in the example.  The "submodule object" is the COMMIT
of the submodule itself.

That's no different from mine.  I need more detail than that.

Is that commit in the submodule or the supermodule?  If it's in the submodule 
then we're talking about the same thing, as that's all I want.  If it's in 
the supermodule then I want to know what the tree object that that commit 
points to contains.  I also want to know how we tell the difference between a 
commit-in-supermodule and a 
commit-in-supermodule-which-is-actually-in-submodule.


Andy
-- 
Dr Andy Parkins, M Eng (hons), MIEE

Andy Parkins wrote:

On Thursday 2006, November 30 16:33, Sven Verdoolaege wrote:

quoted

quoted

Well, I know what the commit is /that/ was all that was stored.  So I

Then I have no idea what you are talking about.
A commit _contains_ all the history that lead up to that commit,
so if you have the commit, then you also have the history.

It's not so much an actual commit, as a reference to a commit in another 
repository.

Hmmm... I thought the idea was that submodule commit is available in the
object repository, be it via alternates mechanism pointing to the submodule
repository for alternate storage, or submodule being in "unrelated"
branch/tracking branch.
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

hoi :)

On Fri, Dec 01, 2006 at 12:16:00PM +0000, Andy Parkins wrote:

That's fine, I was only using the proxy object to allow additional
information into the submodule object.  Actually, I think it would
always be better to use a proxy object otherwise you have an error in
the tree object, because it will refer to an object that does not
exist.  The proxy object is allowed to refer to objects that don't
exist because it's not a tree object.

It is exactly the aim of my implementation to not have any reference to
something that is not accessible in the supermodule repository.

quoted

quoted

Is that commit in the submodule or the supermodule?

Well, logically that commit belongs to the submodule and is referenced
by the tree in the supermodule.
Phyisically it is stored in the projects object database which is
shared between the supermodule and all submodules (at least in my
implementation).

Hmmm, "shared"?  It must still be in the submodule physically though,
and presumably the supermodule uses alternatives to get access to it?
Otherwise the submodule will be impossible to separate from the
supermodule.

Yes, you can't separate it my just moving it out of the supermodule,
but you can always clone the submodule alone.

Okay.  I think I'm still a bit lost then.  I suppose I'll wait for your
patches to understand.

have a look at http://git.admingilde.org/tali/git.git/module2.
If you want to try it out, have a look at t/t7500-submodule.sh on how to
create submodules.

-- 
Martin Waitz

On Friday 2006 December 01 09:57, Martin Waitz wrote:

So why do you need the url hint committed to the supermodule?
We don't store remote information in the object database, too.

That's why it was a hint, probably configured when you first create the 
submodule connection.

Remember: this is still a distributed project, there is no one URL to
any submodule.

That point applies equally to your "tracking a submodule branch" point, except 
mine is only a URL hint, to help when first cloning that supermodule.  In 
truth, the clone will be perfectly able to get the submodule objects from the 
upstream supermodule, maintaining the distributed nature easily.

quoted

I say:
 submodulecommithash points at a commit /in the submodule/

But unluckily, this does not work.

Eh?  "Not work", we're talking about code that doesn't even exist, of course 
it doesn't "work".   Do you mean "doesn't work if we're using my 
implementation of submodules"?  Well that hardly seems like a fair attack.

You really have to be able to traverse the entire commit chain
from the supermodule into all submodules.

You can: when you hit a submodule tree object you set GIT_DIR to that 
submodule and continue.  If you don't do it like that then you have stored 
submodule trees in the supermodule and it's no longer a separate repository.  
Why you'd want to - I have no idea.  What purpose would you have for 
traversing the commit chain into the submodules?  The commit in the submodule 
is just a note of where that submodule was during the supermodule commit in 
question.

I notice though that you avoided my question: what does YOUR submodule object 
contain?  I really do want to know, as there is obviously a fundamental 
difference in what I think a submodule does and what you (and maybe everybody 
else) thinks a submodule does.  I'm perfectly willing to accept I'm wrong, 
but not without understanding how your method is going to work.


Andy
-- 
Dr Andy Parkins, M Eng (hons), MIEE

On Thursday 2006 November 30 15:50, Andreas Ericsson wrote:

quoted

Obviously, this is grossly simplified.  It also requires that HEAD be
allowed to be an arbitrary commit rather than a branch, but that's
already been generally agreed upon as a good thing.

It has? We're not talking supermodule specific things anymore, are we?

Not entirely, although I think it's going to be handy for submodules.  It was 
in a thread about remotes branches.  By allowing checkout of any commit 
rather than only those that have a ref/heads/ entry, you effectively have a 
read-only checkout.  You obviously couldn't commit to a repository like this, 
because HEAD wouldn't point at anything that is changeable.  It would be very 
easy to just git-branch from there and start work though.

I think it's going to be necessary for the submodule work, because without it 
the supermodule will have to create it's own temporary branches in the 
submodule in order to checkout an arbitrary commit.


Andy

-- 
Dr Andy Parkins, M Eng (hons), MIEE

On Friday 2006 December 01 12:34, Martin Waitz wrote:

It is exactly the aim of my implementation to not have any reference to
something that is not accessible in the supermodule repository.

Okay - I think you've put me right in another reply on this point - the 
submodule commit is in the supermodule; that was the part I hadn't got.

Yes, you can't separate it my just moving it out of the supermodule,
but you can always clone the submodule alone.

Ah - now that clarifies things a lot.  The fact that you can't separate it by 
moving it implies lots of things that take away many of my earlier worries.

have a look at http://git.admingilde.org/tali/git.git/module2.
If you want to try it out, have a look at t/t7500-submodule.sh on how to
create submodules.

Thanks.  I will look hard at this :-)  My apologies for bothering you so much 
with all these questions.  I just got a bit interested in it all :-)


Andy

-- 
Dr Andy Parkins, M Eng (hons), MIEE

On Fri, Dec 01, 2006 at 11:02:15AM +0000, Andy Parkins wrote:

On Friday 2006 December 01 10:42, Sven Verdoolaege wrote:

quoted

He showed it to you in the example.  The "submodule object" is the COMMIT
of the submodule itself.

That's no different from mine.  I need more detail than that.

You were proposing to create an extra object containing some random value
that is disconnected from the repo.

Is that commit in the submodule or the supermodule?

It's in BOTH.  That's why it's a *sub*module.

Someone else can try to expain it you.

hoi :)

On Fri, Dec 01, 2006 at 12:12:34PM +0000, Andy Parkins wrote:

On Friday 2006 December 01 11:10, Sven Verdoolaege wrote:

quoted

You were proposing to create an extra object containing some random value
that is disconnected from the repo.

Right, I think I've finally understood what Martin (and you) are
proposing.  You want every commit in the submodule to be propagated up
to the supermodule as well.  Okay.

I don't think it's right, but at least I understand.

Please note that the submodule commits are not part of the supermodule
commit chain, they are part of the supermodule _tree_.

It seems wrong because it's making commits in the supermodule that aren't 
commits to do with that project.

Of course they are part of your project, just like all the tree and blob
objects, too.

In my libxcb example; why should every project use libxcb in have to
store the entire history of libxcb?

Because you want to be able to use the submodule as a repository of its
own, too.  Be able to look at its history if you want to.
Be able to merge with new versions of the submodule.
This is what distiguishes a submodule from a pure file-based import of
another project.

When examining the supermodule history, I won't care about how libxcb
got to the state its in, and it's just noise in the supermodule
history.  What if I use 10 submodules, the supermodule history won't
show you anything useful - it's just unrelated submodule commits.

Again: the submodules are part of your supermodule _tree_, not it's
commit chain.  So you won't see the submodule commits when you invoke
git-log in the supermodule.

It gets worse, this is why I was asking for more detail: this commit
that you're storing in the supermodule.  It's the same commit as is in
the submodule?

It is _the_ commit from the submodule, yes.

What would the parent commit of that commit be?  It has to be the same
in both, because the commit-hash forces it to be.

It is the commit of the submodule, so its parents point to the submodule
history.

quoted

quoted

Is that commit in the submodule or the supermodule?

It's in BOTH.  That's why it's a *sub*module.

If it's in BOTH then the supermodule is a normal git repository.  You aren't 
tracking the submodule, you're just including it en masse.

The submodule is part of the entire project, so yes, it is included.
And the supermodule tracks submodule development by storing references
to the submodule history that was used at that time.


Lets try to paint a little diagram:


belongint to:
/--------- supermodule -------\    /---- submodule -------\

commit -> tree +-> blob
  |            +-> tree -> ...
  |            +-----------------> commit -> tree -> ...
  v                                  |
commit -> tree +-> ...               v
  |            +-----------------> commit -> ...
  |                                  |
  |                                  v
  |                                commit -> ...
  v                                  |
commit -> tree +-> ...               v
               +-----------------> commit


Both have their independent history, but they are linked as some
submodule versions are part of the supermodule tree.

-- 
Martin Waitz

On Thursday 2006 November 30 16:05, sf wrote:

Step 2: You commit to myproject. myproject now contains a new commit
object in path libxcb. (How to do that is up to the UI but at the
repository level the outcome should be obvious). This commit is local to
your repository.

Let's imagine a supermodule repository, and guess at it in more detail (I'll 
abbreviate some of the less interesting output):

$ git-cat-file -p HEAD
tree fb02e78085ecf2f29045603df858b5362e5bf8a4
parent 4f2dba685507e4a8e07dac298c4024feaec6bd7d
author Andy Parkins
committer Andy Parkins 
$ git-cat-file -p fb02e78085ecf2f29045603df858b5362e5bf8a4
100644 blob 46bd4e284a57e2faa539e7b72d62a38867075af5    Makefile
040000 tree 49ea01373a986a3db44d66702714aa75059ffa2c    doc
040000 subm d0a877464dc0198667a3e27ed3af8448ddacf947    libxcb

The "subm" type is our new ODB object that's going to store whatever we will 
need to access the submodule.  "libxcb" has already told us where this 
submodule is in the supermodule tree.

$ git-cat-file -p d0a877464dc0198667a3e27ed3af8448ddacf947
submodulecommithash ccddf1d4b0cf7fd3a699d8b33cf5bc4c5c4435b7
submoduleurlhint git://anongit.freedesktop.org/git/xcb/libxcb

Here "submodulecommithash" is telling us what commit in the submodule is 
stored in this supermodule tree.  The "submoduleurlhint" is to help when 
git-clone is used to clone this supermodule.

They key thing I wanted to point out here is the line:
  submodulecommithash ccddf1d4b0cf7fd3a699d8b33cf5bc4c5c4435b7
This is the ONLY link you have to the submodule.  I think this line represents 
the fundamental difference between our thinking on submodules.

I say:
 submodulecommithash points at a commit /in the submodule/
You say:
 "This commit is local to your repository".  i.e. it points at a commit in
 the supermodule, which in turn implies that the local commit object points
 at a local tree and local parents.

My question is therefore: tell me what that local commit's tree and parent's 
are?  At the moment I am having difficulty understanding what meaningful 
things you could have in those fields.



Andy
-- 
Dr Andy Parkins, M Eng (hons), MIEE

hoi :)

On Fri, Dec 01, 2006 at 10:29:26AM +0000, Andy Parkins wrote:

On Friday 2006 December 01 09:57, Martin Waitz wrote:

quoted

So why do you need the url hint committed to the supermodule?
We don't store remote information in the object database, too.

That's why it was a hint, probably configured when you first create the 
submodule connection.

In truth, the clone will be perfectly able to get the submodule
objects from the upstream supermodule, maintaining the distributed
nature easily.

that's exactly the reason why the hint is not needed.
Althogh you need to have one common project object database, storing the
objects of all modules.

quoted

quoted

I say:
 submodulecommithash points at a commit /in the submodule/

But unluckily, this does not work.

Eh?  "Not work", we're talking about code that doesn't even exist, of
course it doesn't "work".   Do you mean "doesn't work if we're using
my implementation of submodules"?  Well that hardly seems like a fair
attack.

Well, at first I started exactly as you described: only store the
submodule commit sha1 in the parent somewhere, but don't traverse it.
So this is a fair attack: your implementation already exists in
http://git.admingilde.org/tali/git.git/module ;-)
(ok, yes, it really is different to what you described as I stored the
sha1 differently, but I really learned that it is important to be able
to traverse the entire commit chain, from the root of the project to the
deepest submodule.)

quoted

You really have to be able to traverse the entire commit chain
from the supermodule into all submodules.

You can: when you hit a submodule tree object you set GIT_DIR to that
submodule and continue.  If you don't do it like that then you have
stored submodule trees in the supermodule and it's no longer a
separate repository.

Well, a submodule repository _is_ special in some ways:
fsck and prune have to take the references from the supermodule into
account.  In this sense it is _not_ separate from the supermodule.

I think that is important for the submodule repository to be independent
in other ways than its object database:  you should be able to exchange
commits with other repositories (be they stand-alone or a submodule in
another supermodule).  You should be able to use log/diff/blame/whatever
inside the submodule.

All this does not need an object database of its own.
So I chose to do it the easy way and use one object database for the
entire project - and disallow git-prune in a submodule.
There may be other/better ways to do this, but you have to be able
to access all objects which belong the project inside the toplevel
project repository.

Why you'd want to - I have no idea.  What
purpose would you have for traversing the commit chain into the
submodules?  The commit in the submodule is just a note of where that
submodule was during the supermodule commit in question.

Things get much simpler if you have one big graph of objects.

clone and especially fetch/pull naturally work at once.
You can ask for all objects inside the whole project which are needed to
be transferred between project version A and B, including all submodules.

You can even have one bare repository for the whole project.

I notice though that you avoided my question: what does YOUR submodule
object contain?  I really do want to know, as there is obviously a
fundamental difference in what I think a submodule does and what you
(and maybe everybody else) thinks a submodule does.

It really only stores the commit of the submodule directly.
So there is no new submodule object type.  The parent has a direct link
to the submodule commit in his tree object and in its index.  In order
to separate them from normal files or normal subdirectories, they get a
special mode: they are represented as socket.

-- 
Martin Waitz

Martin Waitz wrote:

hoi :)

On Fri, Dec 01, 2006 at 12:20:42PM +0000, Andy Parkins wrote:

quoted

Is there a public repository I can look at to see what you've done?
I'm interested in the sort of plumbing changes needed to make
something like this work.

link is in the mail that started this thread ;-).

And on GitWiki as well:
  http://git.or.cz/gitwiki/SubprojectSupport

-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

hoi :)

On Fri, Dec 01, 2006 at 01:59:58PM +0000, Andy Parkins wrote:

My apologies for bothering you so much with all these questions.

You were not bothering me.
Those were really interesting and valid questions.
In fact, it was a long way for me to come to the implementation I have
now.  And I really did ask many of those questions to me, too.

I should really write a nice paper about all of that, I think.

I just got a bit interested in it all :-)

Good :-)

-- 
Martin Waitz

On Thu, Nov 30, 2006 at 02:49:53PM +0000, Andy Parkins wrote:

How are these commits any different from just having one big repository?  If 

You can work on the submodule independently.

some of the development of the submodule is contained in the supermodule then 
it's not a submodule anymore.

On the contrary, that's exactly what a submodule is supposed to be.

Why bother with all the effort to make a separation between submodule and 
supermodule and then store the submodule commits in the supermodule.  That's 
not supermodule/submodule git - that's just normal git.

[..]

myproject (git root)
 |----- src
 |----- doc
 `----- libxcb (git root)

[..]

Submodules will solve this problem.  In the future I'll be able to check out 
any commit of myproject and it will automatically checkout the right commit 
from the libxcb repository.

How are you going to checkout the right commit of the lixcb repo if
you didn't store it in the supermodule ?

hoi :)

On Fri, Dec 01, 2006 at 02:11:19PM +0000, Andy Parkins wrote:

If I've understood; while the objects themselves are stored in the
supermodule ODB, they are still independent.  In fact, they're only in
the supermodule tree because it's most convenient to keep them there;
it sounds like it's very easy to strip them out again.

Yes.

Again, if I'm understanding, it's a bit like when you have an
additional root in a normal git repository, for example:

 * -- * -- * -- * (project1)
       \
        * -- * -- * (project1/stable)

   * -- * -- * -- * (project2)

Then to make project2 a submodule of project1, one of the project1
trees simply refers to a commit in project2.

Exactly.


-- 
Martin Waitz

On Friday 2006 December 01 11:10, Sven Verdoolaege wrote:

You were proposing to create an extra object containing some random value
that is disconnected from the repo.

Right, I think I've finally understood what Martin (and you) are proposing.  
You want every commit in the submodule to be propagated up to the supermodule 
as well.  Okay.

I don't think it's right, but at least I understand.

It seems wrong because it's making commits in the supermodule that aren't 
commits to do with that project.  In my libxcb example; why should every 
project use libxcb in have to store the entire history of libxcb?  When 
examining the supermodule history, I won't care about how libxcb got to the 
state its in, and it's just noise in the supermodule history.  What if I use 
10 submodules, the supermodule history won't show you anything useful - it's 
just unrelated submodule commits.

It gets worse, this is why I was asking for more detail: this commit that 
you're storing in the supermodule.  It's the same commit as is in the 
submodule?  What would the parent commit of that commit be?  It has to be the 
same in both, because the commit-hash forces it to be.

The only possibility would be that it's NOT the same hash in both, because the 
parents in the supermodule are inapplicable to the submodule, and the parent 
in the submodule is independent from the supermodule.  That means you have to 
store two commits: one for the submodule commit and one for the supermodule 
commit.  So what are you going to write in the supermodule commit?  Answer: a 
submodule commit hash - exactly as I said.

quoted

Is that commit in the submodule or the supermodule?

It's in BOTH.  That's why it's a *sub*module.

If it's in BOTH then the supermodule is a normal git repository.  You aren't 
tracking the submodule, you're just including it en masse.  Using semantics 
to justify a position isn't a very strong argument, calling it a "sub" module 
is just an easy bit of naming for us to hang the discussion on, it isn't 
necessarily a mathematical subset and superset.


Andy

-- 
Dr Andy Parkins, M Eng (hons), MIEE

On Fri, Dec 01, 2006 at 12:01:54AM +0000, Andy Parkins wrote:

On Thursday 2006, November 30 16:33, Sven Verdoolaege wrote:

quoted

quoted

Well, I know what the commit is /that/ was all that was stored.  So I

Then I have no idea what you are talking about.
A commit _contains_ all the history that lead up to that commit,
so if you have the commit, then you also have the history.

It's not so much an actual commit, as a reference to a commit in another 
repository.

This is heresy.  Any object referenced in a tree should be in the repo
(possibly via alternates).

Andy Parkins wrote:

On Thursday 2006 November 30 14:00, Stephan Feder wrote:

quoted

Again I do not see the problem. Probably I have a much simpler picture
of submodules: They are just commits in the supermodule's tree.
Everything else follows naturally from how git currently behaves.

How are these commits any different from just having one big repository?  If 
some of the development of the submodule is contained in the supermodule then 
it's not a submodule anymore.

Right now you only have commits of the top directory aka the super 
project. Every subdirectory is just that: a directory (which git stores 
as trees).

Now, if you have a subdirectory that git stores as a commit, not a tree, 
you have a subproject. It is a directory with history, and because the 
commit is part of your superprject, you have access to this history.

Why bother with all the effort to make a separation between submodule and 
supermodule and then store the submodule commits in the supermodule.  That's 
not supermodule/submodule git - that's just normal git.

No, it is not. Currently, there is no way to store a commit within the 
contents of another commit. You can only store trees and blobs.

Surely the whole point of having submodule's is so that you can take the 
submodule away.  Let me give you an example.  Let's say I have a project that 
uses the libxcb library (some random project out in the world that uses git).  
I've arranged it something like this:

myproject (git root)
 |----- src
 |----- doc
 `----- libxcb (git root)

This works fine; with one problem.  When I make a commit in myproject, there 
is no link into the particular snapshot of the libxcb that I used at that 
moment.  If libxcb moves on, and makes incompatible changes, then when I 
checkout an old version of myproject, it won't compile any more because I'll 
need to find out which commit of libxcb I used at the time.

OK.

Submodules will solve this problem.  In the future I'll be able to check out 
any commit of myproject and it will automatically checkout the right commit 
from the libxcb repository.

OK, I am still with you so far.

Now let's say I'm working away and find a bug in 
libxcb; I fix it, commit it.  That change had better be stored in the libxcb 
repository, and had better make no reference to the myproject repository.  If 
it doesn't, I'm going to have to pollute the libxcb upstream repository with 
myproject if I want to share those fixes.

Here comes the part where we did not meet before.

Of course you do not make any reference from your subproject to your 
superproject. You do exactly what you do in git today when you work with 
different branches:

Step 1: You fix a bug in myproject's subdirectory libxcb.

Step 2: You commit to myproject. myproject now contains a new commit 
object in path libxcb. (How to do that is up to the UI but at the 
repository level the outcome should be obvious). This commit is local to 
your repository.

Step 3: You propose your changes to the libxcb upstream (it might not be 
a repository you have write access to). I use the following made up 
syntax (see man git-rev-parse):

A suffix : followed by a path, _followed by a suffix //::_ names the 
_revision_ at the given path in the tree-ish object named by the part 
before the colon.

Step 3a: Generate a patch

git diff libxcb//^..libxcb//

Step 3b: Push your changes

git push <libxcb-repository> HEAD:libxcb//:<branch in libxcb-repository>

Step 3c: Let your changes be pulled

"Hello, please pull <myproject-repository> HEAD:libxcb//:<branch in 
libxcb-repository>"

Step 4: Pull upstream version (hopefully with your changes, otherwise 
you have to merge)

git pull <libxcb-repository> <branch in libxcb-repository>::HEAD:libxcb//

See, it works.

 From what I understand you want to do the commit and push steps in one 
go. How do you want to record local (to your superproject) changes to 
the subproject?

Regards

sf wrote:
...

A suffix : followed by a path, _followed by a suffix //::_ names the 
_revision_ at the given path in the tree-ish object named by the part 
before the colon.

Sorry, that was supposed to read: followed by a suffix //

hoi :)

On Thu, Nov 30, 2006 at 03:30:49PM +0000, Andy Parkins wrote:

Well, I know what the commit is /that/ was all that was stored.  So I 
(actually supermodule-git does):

cd $DIRECTORY_ASSOCIATED_WITH_SUBMODULE
git checkout -f $COMMIT_FROM_SUPERMODULE

Obviously, this is grossly simplified.  It also requires that HEAD be allowed 
to be an arbitrary commit rather than a branch, but that's already been 
generally agreed upon as a good thing.

It's not that easy.

You also have to make sure that all your submodule commits that _ever_
have been part of your submodule have be stay in your repository
forever.
Consider that your submodule switches to an other branch and some
old commits are not referenced by the current version any more.
These old commits still have to survive a git-prune, if they have been
part of some old supermodule version.
So you really have to connect both object databases and it's not enough
to just store the commit sha1 without actually parsing it by the GIT
core.

-- 
Martin Waitz

hoi :)

On Fri, Dec 01, 2006 at 09:19:04AM +0000, Andy Parkins wrote:

Let's imagine a supermodule repository, and guess at it in more detail (I'll 
abbreviate some of the less interesting output):

$ git-cat-file -p HEAD
tree fb02e78085ecf2f29045603df858b5362e5bf8a4
parent 4f2dba685507e4a8e07dac298c4024feaec6bd7d
author Andy Parkins
committer Andy Parkins 
$ git-cat-file -p fb02e78085ecf2f29045603df858b5362e5bf8a4
100644 blob 46bd4e284a57e2faa539e7b72d62a38867075af5    Makefile
040000 tree 49ea01373a986a3db44d66702714aa75059ffa2c    doc
040000 subm d0a877464dc0198667a3e27ed3af8448ddacf947    libxcb

at the moment, it is:
  140000 commit ccddf1d4b0cf7fd3a699d8b33cf5bc4c5c4435b7  libxcb

The "subm" type is our new ODB object that's going to store whatever we will 
need to access the submodule.  "libxcb" has already told us where this 
submodule is in the supermodule tree.

$ git-cat-file -p d0a877464dc0198667a3e27ed3af8448ddacf947
submodulecommithash ccddf1d4b0cf7fd3a699d8b33cf5bc4c5c4435b7
submoduleurlhint git://anongit.freedesktop.org/git/xcb/libxcb

So why do you need the url hint committed to the supermodule?
We don't store remote information in the object database, too.
Remember: this is still a distributed project, there is no one URL to
any submodule.

I say:
 submodulecommithash points at a commit /in the submodule/

But unluckily, this does not work.
You really have to be able to traverse the entire commit chain
from the supermodule into all submodules.

-- 
Martin Waitz

Andy Parkins wrote:

On Thursday 2006 November 30 15:20, Sven Verdoolaege wrote:

quoted

You can work on the submodule independently.

It's not independent if any part of it is in the supermodule.

quoted

quoted

some of the development of the submodule is contained in the supermodule
then it's not a submodule anymore.

On the contrary, that's exactly what a submodule is supposed to be.

I don't think so.  I think it's just made some complicated normal repository.

I believe that Andy meant "development history" in his above scentence. 
Naturally, using the code from the submodule while being capable of 
developing the submodule separately from the supermodule is what 
submodules are all about.

quoted

How are you going to checkout the right commit of the lixcb repo if
you didn't store it in the supermodule ?

Well, I know what the commit is /that/ was all that was stored.  So I 
(actually supermodule-git does):

cd $DIRECTORY_ASSOCIATED_WITH_SUBMODULE
git checkout -f $COMMIT_FROM_SUPERMODULE

Obviously, this is grossly simplified.  It also requires that HEAD be allowed 
to be an arbitrary commit rather than a branch, but that's already been 
generally agreed upon as a good thing.

It has? We're not talking supermodule specific things anymore, are we?

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se

On Thursday 2006 November 30 15:20, Sven Verdoolaege wrote:

You can work on the submodule independently.

It's not independent if any part of it is in the supermodule.

quoted

some of the development of the submodule is contained in the supermodule
then it's not a submodule anymore.

On the contrary, that's exactly what a submodule is supposed to be.

I don't think so.  I think it's just made some complicated normal repository.

How are you going to checkout the right commit of the lixcb repo if
you didn't store it in the supermodule ?

Well, I know what the commit is /that/ was all that was stored.  So I 
(actually supermodule-git does):

cd $DIRECTORY_ASSOCIATED_WITH_SUBMODULE
git checkout -f $COMMIT_FROM_SUPERMODULE

Obviously, this is grossly simplified.  It also requires that HEAD be allowed 
to be an arbitrary commit rather than a branch, but that's already been 
generally agreed upon as a good thing.

Andy
-- 
Dr Andy Parkins, M Eng (hons), MIEE

hoi :)

On Fri, Dec 01, 2006 at 12:20:42PM +0000, Andy Parkins wrote:

Is there a public repository I can look at to see what you've done?
I'm interested in the sort of plumbing changes needed to make
something like this work.

link is in the mail that started this thread ;-).

-- 
Martin Waitz

On Thursday 2006, November 30 16:33, Sven Verdoolaege wrote:

quoted

Well, I know what the commit is /that/ was all that was stored.  So I

Then I have no idea what you are talking about.
A commit _contains_ all the history that lead up to that commit,
so if you have the commit, then you also have the history.

It's not so much an actual commit, as a reference to a commit in another 
repository.

Andy

-- 
Dr Andrew Parkins, M Eng (Hons), AMIEE

On Friday 2006 December 01 09:32, Sven Verdoolaege wrote:

This is heresy.  Any object referenced in a tree should be in the repo
(possibly via alternates).

The "submodule" object would be in the local repository.  That would refer to 
another object, and is merely part of the submodule object.  Just as  
the "Author" and "Commiter" fields are part of the commit object but aren't 
actual objects in the tree.

Andy

-- 
Dr Andy Parkins, M Eng (hons), MIEE

On Thursday 2006 November 30 14:00, Stephan Feder wrote:

Again I do not see the problem. Probably I have a much simpler picture
of submodules: They are just commits in the supermodule's tree.
Everything else follows naturally from how git currently behaves.

How are these commits any different from just having one big repository?  If 
some of the development of the submodule is contained in the supermodule then 
it's not a submodule anymore.

Why bother with all the effort to make a separation between submodule and 
supermodule and then store the submodule commits in the supermodule.  That's 
not supermodule/submodule git - that's just normal git.

Surely the whole point of having submodule's is so that you can take the 
submodule away.  Let me give you an example.  Let's say I have a project that 
uses the libxcb library (some random project out in the world that uses git).  
I've arranged it something like this:

myproject (git root)
 |----- src
 |----- doc
 `----- libxcb (git root)

This works fine; with one problem.  When I make a commit in myproject, there 
is no link into the particular snapshot of the libxcb that I used at that 
moment.  If libxcb moves on, and makes incompatible changes, then when I 
checkout an old version of myproject, it won't compile any more because I'll 
need to find out which commit of libxcb I used at the time.

Submodules will solve this problem.  In the future I'll be able to check out 
any commit of myproject and it will automatically checkout the right commit 
from the libxcb repository.  Now let's say I'm working away and find a bug in 
libxcb; I fix it, commit it.  That change had better be stored in the libxcb 
repository, and had better make no reference to the myproject repository.  If 
it doesn't, I'm going to have to pollute the libxcb upstream repository with 
myproject if I want to share those fixes.



Andy
-- 
Dr Andy Parkins, M Eng (hons), MIEE

On Thu, Nov 30, 2006 at 03:30:49PM +0000, Andy Parkins wrote:

On Thursday 2006 November 30 15:20, Sven Verdoolaege wrote:

quoted

How are you going to checkout the right commit of the lixcb repo if
you didn't store it in the supermodule ?

Well, I know what the commit is /that/ was all that was stored.  So I 

Then I have no idea what you are talking about.
A commit _contains_ all the history that lead up to that commit,
so if you have the commit, then you also have the history.

hoi :)

On Fri, Dec 01, 2006 at 11:02:15AM +0000, Andy Parkins wrote:

On Friday 2006 December 01 10:42, Sven Verdoolaege wrote:

quoted

He showed it to you in the example.  The "submodule object" is the COMMIT
of the submodule itself.

That's no different from mine.

Well, there simply is no proxy object inbetween.

Is that commit in the submodule or the supermodule?

Well, logically that commit belongs to the submodule and is referenced
by the tree in the supermodule.
Phyisically it is stored in the projects object database which is
shared between the supermodule and all submodules (at least in my
implementation).

I also want to know how we tell the difference between a
commit-in-supermodule and a
commit-in-supermodule-which-is-actually-in-submodule.

There is no difference.

-- 
Martin Waitz

On Friday 2006 December 01 11:31, Martin Waitz wrote:

It really only stores the commit of the submodule directly.
So there is no new submodule object type.  The parent has a direct link
to the submodule commit in his tree object and in its index.  In order
to separate them from normal files or normal subdirectories, they get a
special mode: they are represented as socket.

Okay.  I think I've got it now.  I'm not convinced that the way you've chosen 
is the correct way, primarily because the separation between supermodule and 
submodule is not strong.  Regardless, as you're doing it, you get to pick :-) 
Is there a public repository I can look at to see what you've done?  I'm 
interested in the sort of plumbing changes needed to make something like this 
work.


Andy

-- 
Dr Andy Parkins, M Eng (hons), MIEE

On Friday 2006 December 01 11:46, Martin Waitz wrote:

quoted

That's no different from mine.

Well, there simply is no proxy object inbetween.

That's fine, I was only using the proxy object to allow additional information 
into the submodule object.  Actually, I think it would always be better to 
use a proxy object otherwise you have an error in the tree object, because it 
will refer to an object that does not exist.  The proxy object is allowed to 
refer to objects that don't exist because it's not a tree object.

quoted

Is that commit in the submodule or the supermodule?

Well, logically that commit belongs to the submodule and is referenced
by the tree in the supermodule.
Phyisically it is stored in the projects object database which is
shared between the supermodule and all submodules (at least in my
implementation).

Hmmm, "shared"?  It must still be in the submodule physically though, and 
presumably the supermodule uses alternatives to get access to it?  Otherwise 
the submodule will be impossible to separate from the supermodule.

quoted

I also want to know how we tell the difference between a
commit-in-supermodule and a
commit-in-supermodule-which-is-actually-in-submodule.

There is no difference.

Okay.  I think I'm still a bit lost then.  I suppose I'll wait for your 
patches to understand.


Andy
-- 
Dr Andy Parkins, M Eng (hons), MIEE

On Friday 2006 December 01 12:28, Martin Waitz wrote:

quoted

It seems wrong because it's making commits in the supermodule that aren't
commits to do with that project.

Of course they are part of your project, just like all the tree and blob
objects, too.

I wouldn't go as far as that; just because I use libxcb doesn't mean I want 
it's history merged with mine.  However, I think my worries are unfounded, 
your comment about being able to independently clone the libxcb tree helped 
me there.  If I've understood; while the objects themselves are stored in the 
supermodule ODB, they are still independent.  In fact, they're only in the 
supermodule tree because it's most convenient to keep them there; it sounds 
like it's very easy to strip them out again.

It is the commit of the submodule, so its parents point to the submodule
history.

Again, if I'm understanding, it's a bit like when you have an additional root 
in a normal git repository, for example:
 
 * -- * -- * -- * (project1)
       \
        * -- * -- * (project1/stable)
             
   * -- * -- * -- * (project2)

Then to make project2 a submodule of project1, one of the project1 trees 
simply refers to a commit in project2.

I think my original idea for how this works was correct with one minor flaw, 
and from that flaw all the other concerns flowed.  I imagined that there were 
two object databases - one for the supermodule and one for the submodule.  
The fault was that there aren't two ODBs there are two roots.  Which of 
course is a far easier way to blend to repositories.  Apart from that, I 
think I'm entirely in sync, and it was merely my wanting to put each of these 
roots in their own repository that caused all the confusion.




Andy
-- 
Dr Andy Parkins, M Eng (hons), MIEE

Sven Verdoolaege wrote:

On Fri, Dec 01, 2006 at 11:02:15AM +0000, Andy Parkins wrote:

quoted

On Friday 2006 December 01 10:42, Sven Verdoolaege wrote:

quoted

He showed it to you in the example.  The "submodule object" is the COMMIT
of the submodule itself.

That's no different from mine.  I need more detail than that.

You were proposing to create an extra object containing some random value
that is disconnected from the repo.

quoted

Is that commit in the submodule or the supermodule?

It's in BOTH.  That's why it's a *sub*module.

I would say it is only in the supermodule because that is the branch you 
are working on. If you are working on the submodule in an independent 
branch then you can pull from the submodule commit. But you do not want 
to pull the supermodule commit itself but only the commit in path libxcb 
(see my proposed syntax).

Regards

Stephan