[rfc] git submodules howto

hi,

i saw several "is there any step by step howto on how to use git
submodules?" question on irc, and as far as i think there is none
available at the moment

here is how i use it at the moment:

$ mkdir lib
$ cd lib
$ git init
Initialized empty Git repository in .git/
$ echo "libmakefile" > Makefile
$ dg add Makefile
$ git commit -m "libmakefile"
Created initial commit 57c1dce: libmakefile
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 Makefile
$ cd ..
$ mkdir main
$ cd main
$ git init
Initialized empty Git repository in .git/
$ echo "main makefile" > Makefile
$ git add Makefile
$ git commit -m "main makefile"
Created initial commit 8935291: main makefile
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 Makefile
$ git submodule add ../lib lib
Initialized empty Git repository in /home/vmiklos/scm/git/sub/main/lib/.git/
0 blocks
$ git commit -m "added lib submodule"
Created commit 9dbfedf: added lib submodule
 2 files changed, 4 insertions(+), 0 deletions(-)
 create mode 100644 .gitmodules
 create mode 160000 lib
$ cd ..
$ git clone main cloned
Initialized empty Git repository in /home/vmiklos/scm/git/sub/cloned/.git/
0 blocks
$ cd cloned
$ git submodule init
Submodule 'lib' (/home/vmiklos/scm/git/sub/lib/.git) registered for path 'lib'
$ git submodule update
Initialized empty Git repository in /home/vmiklos/scm/git/sub/cloned/lib/.git/
0 blocks
Submodule path 'lib': checked out '57c1dce0e083e9ee50d06111d6aa1523116c2e15'
$ cat Makefile
main makefile
$ cat lib/Makefile
libmakefile

my questions:

1) is this correct? :) i use it and it seem to do what i except, but
maybe it's not correct

2) does this worth adding to the documentation? maybe to a .txt under
Documentation/howto? or to git-submodule.txt?

thanks,
- VMiklos

Hi,

On Tue, 18 Sep 2007, Miklos Vajna wrote:

i saw several "is there any step by step howto on how to use git 
submodules?" question on irc, and as far as i think there is none 
available at the moment

AFAICT there is one, on the git Wiki, since yesterday.

Ciao,
Dscho

On Tue, 18 Sep 2007, Miklos Vajna wrote:

1) is this correct? :) i use it and it seem to do what i except, but
maybe it's not correct

Hi Miklos,

I had the same question, but you and I are both using it in a similar way 
so that's encouraging.

I put something in the Wiki the other day.

http://git.or.cz/gitwiki/GitSubmoduleTutorial

Mike

On Tue, Sep 18, 2007 at 12:55:38PM +0200, Miklos Vajna wrote:

1) is this correct? :) i use it and it seem to do what i except, but
maybe it's not correct

I'm not able to answer that question.

2) does this worth adding to the documentation? maybe to a .txt under
Documentation/howto? or to git-submodule.txt?

Could you add it as a new chapter to user-manual.txt (probably just
after the "git concepts" chapter), and then add links to that chapter
from git-submodule(1) and gitmodules(5)?

--b.

On Tue, Sep 18, 2007 at 09:29:40AM -0400, "J. Bruce Fields" [off-list ref] wrote:

quoted

2) does this worth adding to the documentation? maybe to a .txt under
Documentation/howto? or to git-submodule.txt?

Could you add it as a new chapter to user-manual.txt (probably just
after the "git concepts" chapter), and then add links to that chapter
from git-submodule(1) and gitmodules(5)?

hm, i did not know about the wiki page Michael created yesterday. so i
don't know what's the rule in case: if something is already in the wiki
then should or should not it be added to the 'official docs'?

- VMiklos

On Tue, Sep 18, 2007 at 05:47:34PM +0200, Miklos Vajna wrote:

On Tue, Sep 18, 2007 at 09:29:40AM -0400, "J. Bruce Fields" [off-list ref] wrote:

quoted

quoted

2) does this worth adding to the documentation? maybe to a .txt under
Documentation/howto? or to git-submodule.txt?

Could you add it as a new chapter to user-manual.txt (probably just
after the "git concepts" chapter), and then add links to that chapter
from git-submodule(1) and gitmodules(5)?

hm, i did not know about the wiki page Michael created yesterday. so i
don't know what's the rule in case: if something is already in the wiki
then should or should not it be added to the 'official docs'?

It should.  We also need submodules documentation for the "official"
documentation.

If you want to base that work off of that wiki page instead of your
original email, that's fine.  Just make sure you get Michael's
permission first.

--b.

[ adding Michael to CC. ]

On Tue, Sep 18, 2007 at 11:55:29AM -0400, "J. Bruce Fields" [off-list ref] wrote:

quoted

hm, i did not know about the wiki page Michael created yesterday. so i
don't know what's the rule in case: if something is already in the wiki
then should or should not it be added to the 'official docs'?

It should.  We also need submodules documentation for the "official"
documentation.

okay.

If you want to base that work off of that wiki page instead of your
original email, that's fine.  Just make sure you get Michael's
permission first.

Michael, i think the wiki version is better as my example does not
contain any extra to the wiki version. is it ok if i would send a patch
to include your work in the official docs?

i'm not sure how should i mention you, maybe in the commit message?

- VMiklos

On Tue, 18 Sep 2007, Miklos Vajna wrote:

Michael, i think the wiki version is better as my example does not
contain any extra to the wiki version. is it ok if i would send a patch
to include your work in the official docs?

Thanks, that would be great.

i'm not sure how should i mention you, maybe in the commit message?

Maybe signed-off-by is appropriate? Otherwise, don't worry about it.

Mike

Signed-off-by: Michael Smith <redacted>
Signed-off-by: Miklos Vajna <redacted>
---

On Tue, Sep 18, 2007 at 02:12:17PM -0400, Michael Smith [off-list ref] wrote:

On Tue, 18 Sep 2007, Miklos Vajna wrote:

quoted

Michael, i think the wiki version is better as my example does not
contain any extra to the wiki version. is it ok if i would send a patch
to include your work in the official docs?

Thanks, that would be great.

here it is. this version is a bit shorter than the wiki one, but i think it
does not contain less useful info

 Documentation/user-manual.txt |  175 +++++++++++++++++++++++++++++++++++++++++
 1 files changed, 175 insertions(+), 0 deletions(-)

diff --git a/Documentation/user-manual.txt b/Documentation/user-manual.txt
index ecb2bf9..ce0cf38 100644
--- a/Documentation/user-manual.txt
+++ b/Documentation/user-manual.txt

@@ -3155,6 +3155,181 @@ a tree which you are in the process of working on.
 If you blow the index away entirely, you generally haven't lost any
 information as long as you have the name of the tree that it described.
 
+[[submodules]]
+Submodules
+==========
+
+This tutorial explains how to create and publish a repository with submodules
+using the gitlink:git-submodule[1] command.
+
+Submodules maintain their own identity; the submodule support just stores the
+submodule repository location and commit ID, so other developers who clone the
+superproject can easily clone all the submodules at the same revision.
+
+To see how submodule support works, create (for example) four example
+repository that can be used later as a submodule:
+
+-------------------------------------------------
+$ mkdir ~/git
+$ cd ~/git
+$ for i in a b c d
+do
+	mkdir $i
+	cd $i
+	git init
+	echo "module $i" > $i.txt
+	git add $i.txt
+	git commit -m "Initial commit, submodule $mod"
+	cd ..
+done
+-------------------------------------------------
+
+Now create the superproject and add all the submodules:
+
+-------------------------------------------------
+$ mkdir super
+$ cd super
+$ git init
+$ echo hi > super.txt
+$ git add super.txt
+$ git commit -m "Initial commit of empty superproject"
+$ for i in a b c d
+do
+	git submodule add ~/git/$i
+done
+-------------------------------------------------
+
+See what files `git submodule` created:
+
+-------------------------------------------------
+$ ls -a
+.  ..  .git  .gitmodules  a  b  c  d  super.txt
+-------------------------------------------------
+
+The `git submodule add` command does a couple of things:
+
+- It clones the submodule under the current directory and by default checks out
+  the master branch.
+- It adds the submodule's clone path to the `.gitmodules` file and adds this
+  file to the index, ready to be committed.
+- It adds the submodule's current commit ID to the index, ready to be
+  committed.
+
+Commit the superproject:
+
+-------------------------------------------------
+$ git commit -m "Add submodules a, b, c, d."
+-------------------------------------------------
+
+Now clone the superproject:
+
+-------------------------------------------------
+$ cd ..
+$ git clone super cloned
+$ cd cloned
+-------------------------------------------------
+
+The submodule directories are there, but they're empty:
+
+-------------------------------------------------
+$ ls -a a
+.  ..
+$ git submodule status
+-d266b9873ad50488163457f025db7cdd9683d88b a
+-e81d457da15309b4fef4249aba9b50187999670d b
+-c1536a972b9affea0f16e0680ba87332dc059146 c
+-d96249ff5d57de5de093e6baff9e0aafa5276a74 d
+-------------------------------------------------
+
+Pulling down the submodules is a two-step process. First run `git submodule
+init` to add the submodule repository URLs to `.git/config`:
+
+-------------------------------------------------
+$ git submodule init
+-------------------------------------------------
+
+Now use `git submodule update` to clone the repositories and check out the
+commits specified in the superproject:
+
+-------------------------------------------------
+$ git submodule update
+$ cd a
+$ ls -a
+.  ..  .git  a.txt
+-------------------------------------------------
+
+One major difference between `git submodule update` and `git submodule add` is
+that `git submodule update` checks out a specific commit, rather than the tip
+of a branch. It's like checking out a tag: the head is detached, so you're not
+working on a branch.
+
+-------------------------------------------------
+$ git branch
+* (no branch)
+  master
+-------------------------------------------------
+
+If you want to make a change within a submodule, you should first check out a
+branch, make your changes, publish the change within the submodule, and then
+update the superproject to reference the new commit:
+
+-------------------------------------------------
+$ git branch
+* (no branch)
+  master
+$ git checkout master
+$ echo "adding a line again" >> a.txt
+$ git commit -a -m "Updated the submodule from within the superproject."
+$ git push
+$ cd ..
+$ git add a
+$ git commit -m "Updated submodule a."
+$ git push
+-------------------------------------------------
+
+NOTE: This means that you have to run `git submodule update` after `git pull`
+if you want to update the subprojects, too.
+
+Problems with submodules
+------------------------
+
+Always publish the submodule change before publishing the change to the
+superproject that references it. If you forget to publish the submodule change,
+others won't be able to clone the repository:
+
+-------------------------------------------------
+$ echo i added another line to this file >> a.txt
+$ git commit -a -m "doing it wrong this time"
+$ cd ..
+$ git add a
+$ git commit -m "Updated submodule a again."
+$ git push
+$ cd ~/git/cloned
+$ git pull
+$ git submodule update
+error: pathspec '261dfac35cb99d380eb966e102c1197139f7fa24' did not match any file(s) known to git.
+Did you forget to 'git add'?
+Unable to checkout '261dfac35cb99d380eb966e102c1197139f7fa24' in submodule path 'a'
+-------------------------------------------------
+
+It's not safe to run `git submodule update` if you've made changes within a
+submodule. They will be silently overwritten:
+
+-------------------------------------------------
+$ cat a.txt
+module a
+$ echo line added from private2 >> a.txt
+$ git commit -a -m "line added inside private2"
+$ cd ..
+$ git submodule update
+Submodule path 'a': checked out 'd266b9873ad50488163457f025db7cdd9683d88b'
+$ cd a
+$ cat a.txt
+module a
+-------------------------------------------------
+
+NOTE: The changes are still visible in the submodule's reflog.
+
 [[low-level-operations]]
 Low-level git operations
 ========================

-- 
1.5.3.1.1.g1e61-dirty

On Wed, Sep 19, 2007 at 07:42:50PM +0200, Miklos Vajna wrote:

+Submodules maintain their own identity; the submodule support just stores the
+submodule repository location and commit ID, so other developers who clone the
+superproject can easily clone all the submodules at the same revision.

[..]

+-------------------------------------------------
+$ mkdir super
+$ cd super
+$ git init
+$ echo hi > super.txt
+$ git add super.txt
+$ git commit -m "Initial commit of empty superproject"
+$ for i in a b c d
+do
+	git submodule add ~/git/$i
+done
+-------------------------------------------------

You may want to warn the reader not to use local URLs here if they
plan to publish their superproject.

+It's not safe to run `git submodule update` if you've made changes within a
+submodule. They will be silently overwritten:

This is only true if they didn't follow your advise of checking out
a branch first.

skimo

Hello,

Miklos Vajna wrote:

$ mkdir lib
$ cd lib
$ git init
Initialized empty Git repository in .git/
$ echo "libmakefile" > Makefile
$ dg add Makefile

[...]

1) is this correct? :) i use it and it seem to do what i except, but
maybe it's not correct

I get 

	bash: dg: command not found

:-)  Probably s/dg/git/

Best regards
Uwe

-- 
Uwe Kleine-König

http://www.google.com/search?q=half+a+cup+in+teaspoons

On Mon, Sep 24, 2007 at 09:11:35AM +0200, Uwe Kleine-König [off-list ref] wrote:

I get 

	bash: dg: command not found

:-)  Probably s/dg/git/

yes, thanks for the correction

- VMiklos