Bug in Submodule add

I believe I have found an issue with the way `submodule add` detects a submodule that already exists in the repository. 

To reproduce

1) add a git submodule in a specific location (we'll say it's at `./submodule/location`)
2) go through the normal steps of removing a submodule, as listed here - https://git.wiki.kernel.org/index.php/GitSubmoduleTutorial
3) Now the submodule is completely removed and there is no reference to it in .gitmodules or .git/config
4) Re-add a different repository at the same location (`./submodule/location`)

Expected - The new submodule repository will be set up at ./submodule/location and have the new repository as its origin

What Actually Happens - The new submodule uses the existing `$gitdir` (old repository) as the actual backing repository to the submodule, but the new repository is reflected in .gitmodules and .git/config.

So to recap, the result is that `git remote show origin`  in the submodule shows a different origin than is in .gitmodules and .git/config

One simple step to remedy this would be to add the deletion of the backing repository from the .git/modules directory, but again, I think an actual command to take care of all of these steps is in order anyways.  Not sure you want to encourage people poking around in the .git directory.

If this is already resolved in the newest versions, please disregard

Thanks!

Jonathan Johnson

http://jondavidjohn.com | me@jondavidjohn.com

Am 26.09.2012 06:18, schrieb Jonathan Johnson:

I believe I have found an issue with the way `submodule add` detects a submodule that already exists in the repository. 

Yes, this is an issue and thanks for the detailed report.

To reproduce

1) add a git submodule in a specific location (we'll say it's at `./submodule/location`)
2) go through the normal steps of removing a submodule, as listed here - https://git.wiki.kernel.org/index.php/GitSubmoduleTutorial
3) Now the submodule is completely removed and there is no reference to it in .gitmodules or .git/config
4) Re-add a different repository at the same location (`./submodule/location`)

Expected - The new submodule repository will be set up at ./submodule/location and have the new repository as its origin

What Actually Happens - The new submodule uses the existing `$gitdir` (old repository) as the actual backing repository to the submodule, but the new repository is reflected in .gitmodules and .git/config.

So to recap, the result is that `git remote show origin`  in the submodule shows a different origin than is in .gitmodules and .git/config

One simple step to remedy this would be to add the deletion of the backing repository from the .git/modules directory, but again, I think an actual command to take care of all of these steps is in order anyways.  Not sure you want to encourage people poking around in the .git directory.

Unfortunately just throwing away the old repository under .git/modules,
whether manually or by a git command, is no real solution here: it would
make it impossible to go back to a commit which records the old submodule
and check that out again.

The reason for this issue is that the submodule path is used as its name
by "git submodule add". While we could check this type of conflict locally,
we can't really avoid it due to the distributed nature of git (somebody
else could add a different repo under the same path - and thus the same
name - in another clone of the repo).

The only long term solution I can think of is to use some kind of UUID for
the name, so that the names of newly added submodules won't have a chance
to clash anymore. For the short term aborting "git submodule add" when a
submodule of that name already exists in .git/modules of the superproject
together with the ability to provide a custom name might at least solve
the local clashes.

Am 26.09.2012 22:56, schrieb Jens Lehmann:

Am 26.09.2012 06:18, schrieb Jonathan Johnson:

quoted

To reproduce

1) add a git submodule in a specific location (we'll say it's at `./submodule/location`)
2) go through the normal steps of removing a submodule, as listed here - https://git.wiki.kernel.org/index.php/GitSubmoduleTutorial
3) Now the submodule is completely removed and there is no reference to it in .gitmodules or .git/config
4) Re-add a different repository at the same location (`./submodule/location`)

Expected - The new submodule repository will be set up at ./submodule/location and have the new repository as its origin

What Actually Happens - The new submodule uses the existing `$gitdir` (old repository) as the actual backing repository to the submodule, but the new repository is reflected in .gitmodules and .git/config.

So to recap, the result is that `git remote show origin`  in the submodule shows a different origin than is in .gitmodules and .git/config

One simple step to remedy this would be to add the deletion of the backing repository from the .git/modules directory, but again, I think an actual command to take care of all of these steps is in order anyways.  Not sure you want to encourage people poking around in the .git directory.

Unfortunately just throwing away the old repository under .git/modules,
whether manually or by a git command, is no real solution here: it would
make it impossible to go back to a commit which records the old submodule
and check that out again.

The reason for this issue is that the submodule path is used as its name
by "git submodule add". While we could check this type of conflict locally,
we can't really avoid it due to the distributed nature of git (somebody
else could add a different repo under the same path - and thus the same
name - in another clone of the repo).

The only long term solution I can think of is to use some kind of UUID for
the name, so that the names of newly added submodules won't have a chance
to clash anymore. For the short term aborting "git submodule add" when a
submodule of that name already exists in .git/modules of the superproject
together with the ability to provide a custom name might at least solve
the local clashes.

This two patch series implements the short term solution described above.

Using some kind of UUID can easily be added in a subsequent patch, we just
have to replace 'sm_name="$sm_path"' with 'sm_name=$(<generate uuid>)' in
line 348 of git-submodule.sh. I think it'll be the best solution to just
use a random UUID for that, as doing anything clever (like using the SHA1
of the url to avoid copies of the same remote repo) might lead to subtle
breakages (e.g. because it assumes the url stays unique forever, which it
sometimes won't). But maybe the short term solution is sufficient as most
of the time people won't produce submodule name conflicts (and names
derived from paths are much more readable that UUIDs). Thoughts?


Jens Lehmann (2):
  Teach "git submodule add" the --name option
  submodule add: Fail when .git/modules/<name> already exists

 Documentation/git-submodule.txt |  7 ++++-
 Documentation/gitmodules.txt    |  4 ++-
 git-submodule.sh                | 35 ++++++++++++++++-------
 t/t7400-submodule-basic.sh      | 63 +++++++++++++++++++++++++++++++++++++++++
 t/t7406-submodule-update.sh     |  2 +-
 5 files changed, 97 insertions(+), 14 deletions(-)

-- 
1.7.12.1.430.g4fd6dc4

"git submodule add" initializes the name of a submodule to its path. This
was ok as long as the .git directory lived inside the submodule's work
tree, but since 1.7.8 it is stored in the .git/modules/<name> directory of
the superproject, making the submodule name survive the removal of the
submodule's work tree. This leads to problems when the user tries to add a
different submodule at the same path - and thus the same name - later, as
that will happily try to restore the submodule from the old repository
instead of the one the user specified and will lead to a checkout of the
wrong repository.

Add the new "--name" option to let the user provide a name for the
submodule. This enables the user to solve this conflict without having to
remove .git/modules/<name> by hand (which is no viable solution as it
makes it impossible to checkout a commit that records the old submodule
and populate it, as that will still check out the new submodule for the
same reason).

To achieve that the submodule's name is added to the parameter list of
the module_clone() helper function. This makes it possible to remove the
call of module_name() there because both callers of module_clone() already
know the name and can provide it as argument number two.

Reported-by: Jonathan Johnson <redacted>
Signed-off-by: Jens Lehmann <redacted>
---
 Documentation/git-submodule.txt |  7 ++++++-
 Documentation/gitmodules.txt    |  4 +++-
 git-submodule.sh                | 32 ++++++++++++++++++++---------
 t/t7400-submodule-basic.sh      | 45 +++++++++++++++++++++++++++++++++++++++++
 4 files changed, 76 insertions(+), 12 deletions(-)

diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
index 2de7bf0..22efca0 100644
--- a/Documentation/git-submodule.txt
+++ b/Documentation/git-submodule.txt

@@ -9,7 +9,7 @@ git-submodule - Initialize, update or inspect submodules
 SYNOPSIS
 --------
 [verse]
-'git submodule' [--quiet] add [-b branch] [-f|--force]
+'git submodule' [--quiet] add [-b branch] [-f|--force] [--name <name>]
 	      [--reference <repository>] [--] <repository> [<path>]
 'git submodule' [--quiet] status [--cached] [--recursive] [--] [<path>...]
 'git submodule' [--quiet] init [--] [<path>...]

@@ -266,6 +266,11 @@ OPTIONS
 	Initialize all submodules for which "git submodule init" has not been
 	called so far before updating.

+--name::
+	This option is only valid for the add command. It sets the submodule's
+	name to the given string instead of defaulting to its path. The name
+	must be valid as a directory name and may not end with a '/'.
+
 --reference <repository>::
 	This option is only valid for add and update commands.  These
 	commands sometimes need to clone a remote repository. In this case,

diff --git a/Documentation/gitmodules.txt b/Documentation/gitmodules.txt
index 4effd78..ab3e91c 100644
--- a/Documentation/gitmodules.txt
+++ b/Documentation/gitmodules.txt

@@ -18,7 +18,9 @@ working tree, is a text file with a syntax matching the requirements
 of linkgit:git-config[1].

 The file contains one subsection per submodule, and the subsection value
-is the name of the submodule. Each submodule section also contains the
+is the name of the submodule. The name is set to the path where the
+submodule has been added unless it was customized with the '--name'
+option of 'git submodule add'. Each submodule section also contains the
 following required keys:

 submodule.<name>.path::

diff --git a/git-submodule.sh b/git-submodule.sh
index 3e2045e..22febb1 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh

@@ -5,7 +5,7 @@
 # Copyright (c) 2007 Lars Hjemli

 dashless=$(basename "$0" | sed -e 's/-/ /')
-USAGE="[--quiet] add [-b branch] [-f|--force] [--reference <repository>] [--] <repository> [<path>]
+USAGE="[--quiet] add [-b branch] [-f|--force] [--name <name>] [--reference <repository>] [--] <repository> [<path>]
    or: $dashless [--quiet] status [--cached] [--recursive] [--] [<path>...]
    or: $dashless [--quiet] init [--] [<path>...]
    or: $dashless [--quiet] update [--init] [-N|--no-fetch] [-f|--force] [--rebase] [--reference <repository>] [--merge] [--recursive] [--] [<path>...]

@@ -29,6 +29,7 @@ files=
 nofetch=
 update=
 prefix=
+custom_name=

 # The function takes at most 2 arguments. The first argument is the
 # URL that navigates to the submodule origin repo. When relative, this URL

@@ -179,8 +180,9 @@ module_name()
 module_clone()
 {
 	sm_path=$1
-	url=$2
-	reference="$3"
+	name=$2
+	url=$3
+	reference="$4"
 	quiet=
 	if test -n "$GIT_QUIET"
 	then

@@ -189,8 +191,6 @@ module_clone()

 	gitdir=
 	gitdir_base=
-	name=$(module_name "$sm_path" 2>/dev/null)
-	test -n "$name" || name="$sm_path"
 	base_name=$(dirname "$name")

 	gitdir=$(git rev-parse --git-dir)

@@ -272,6 +272,11 @@ cmd_add()
 			reference="$1"
 			shift
 			;;
+		--name)
+			case "$2" in '') usage ;; esac
+			custom_name=$2
+			shift
+			;;
 		--)
 			shift
 			break

@@ -336,6 +341,13 @@ Use -f if you really want to add it." >&2
 		exit 1
 	fi

+	if test -n "$custom_name"
+	then
+		sm_name="$custom_name"
+	else
+		sm_name="$sm_path"
+	fi
+
 	# perhaps the path exists and is already a git repo, else clone it
 	if test -e "$sm_path"
 	then

@@ -348,7 +360,7 @@ Use -f if you really want to add it." >&2

 	else

-		module_clone "$sm_path" "$realrepo" "$reference" || exit
+		module_clone "$sm_path" "$sm_name" "$realrepo" "$reference" || exit
 		(
 			clear_local_git_env
 			cd "$sm_path" &&

@@ -359,13 +371,13 @@ Use -f if you really want to add it." >&2
 			esac
 		) || die "$(eval_gettext "Unable to checkout submodule '\$sm_path'")"
 	fi
-	git config submodule."$sm_path".url "$realrepo"
+	git config submodule."$sm_name".url "$realrepo"

 	git add $force "$sm_path" ||
 	die "$(eval_gettext "Failed to add submodule '\$sm_path'")"

-	git config -f .gitmodules submodule."$sm_path".path "$sm_path" &&
-	git config -f .gitmodules submodule."$sm_path".url "$repo" &&
+	git config -f .gitmodules submodule."$sm_name".path "$sm_path" &&
+	git config -f .gitmodules submodule."$sm_name".url "$repo" &&
 	git add --force .gitmodules ||
 	die "$(eval_gettext "Failed to register submodule '\$sm_path'")"
 }

@@ -594,7 +606,7 @@ Maybe you want to use 'update --init'?")"

 		if ! test -d "$sm_path"/.git -o -f "$sm_path"/.git
 		then
-			module_clone "$sm_path" "$url" "$reference"|| exit
+			module_clone "$sm_path" "$name" "$url" "$reference" || exit
 			cloned_modules="$cloned_modules;$name"
 			subsha1=
 		else

diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index 56a81cd..78bf739 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh

@@ -681,4 +681,49 @@ test_expect_success 'moving the superproject does not break submodules' '
 	)
 '

+test_expect_success 'submodule add --name allows to replace a submodule with another at the same path' '
+	(
+		cd addtest2 &&
+		(
+			cd repo &&
+			echo "$submodurl/repo" >expect &&
+			git config remote.origin.url >actual &&
+			test_cmp expect actual &&
+			echo "gitdir: ../.git/modules/repo" >expect &&
+			test_cmp expect .git
+		) &&
+		rm -rf repo &&
+		git rm repo &&
+		git submodule add -q --name repo_new "$submodurl/bare.git" repo >actual &&
+		test ! -s actual &&
+		echo "gitdir: ../.git/modules/submod" >expect &&
+		test_cmp expect submod/.git &&
+		(
+			cd repo &&
+			echo "$submodurl/bare.git" >expect &&
+			git config remote.origin.url >actual &&
+			test_cmp expect actual &&
+			echo "gitdir: ../.git/modules/repo_new" >expect &&
+			test_cmp expect .git
+		) &&
+		echo "repo" >expect &&
+		git config -f .gitmodules submodule.repo.path >actual &&
+		test_cmp expect actual &&
+		git config -f .gitmodules submodule.repo_new.path >actual &&
+		test_cmp expect actual&&
+		echo "$submodurl/repo" >expect &&
+		git config -f .gitmodules submodule.repo.url >actual &&
+		test_cmp expect actual &&
+		echo "$submodurl/bare.git" >expect &&
+		git config -f .gitmodules submodule.repo_new.url >actual &&
+		test_cmp expect actual &&
+		echo "$submodurl/repo" >expect &&
+		git config submodule.repo.url >actual &&
+		test_cmp expect actual &&
+		echo "$submodurl/bare.git" >expect &&
+		git config submodule.repo_new.url >actual &&
+		test_cmp expect actual
+	)
+'
+
 test_done

-- 
1.7.12.1.430.g4fd6dc4

When adding a new submodule it can happen that .git/modules/<name> already
contains a submodule repo, e.g. when a submodule is removed from the work
tree and another submodule is added at the same path. But then the work
tree of the submodule will be populated using the existing repository and
not the one the user provided.

Error out in that case and tell the user she should use a different name
for the submodule with the "--name" option to avoid this problem. In one
test in t7406 the --name option had to be added to "git submodule add", as
that test re-adds a formerly removed submodule.

Reported-by: Jonathan Johnson <redacted>
Signed-off-by: Jens Lehmann <redacted>
---
 git-submodule.sh            |  3 ++-
 t/t7400-submodule-basic.sh  | 18 ++++++++++++++++++
 t/t7406-submodule-update.sh |  2 +-
 3 files changed, 21 insertions(+), 2 deletions(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 22febb1..58cd053 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh

@@ -359,7 +359,8 @@ Use -f if you really want to add it." >&2
 		fi

 	else
-
+		test ! -d ".git/modules/$sm_name" ||
+		die "$(eval_gettext "Submodule name '\$sm_name' is already used. Please choose another name with the '--name' option.")"
 		module_clone "$sm_path" "$sm_name" "$realrepo" "$reference" || exit
 		(
 			clear_local_git_env

diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index 78bf739..a031a27 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh

@@ -726,4 +726,22 @@ test_expect_success 'submodule add --name allows to replace a submodule with ano
 	)
 '

+test_expect_success 'submodule add with an existing name fails' '
+	(
+		cd addtest2 &&
+		rm -rf repo &&
+		test_must_fail git submodule add -q --name repo_new "$submodurl/bare.git" repo &&
+		test ! -d repo &&
+		echo "repo" >expect &&
+		git config -f .gitmodules submodule.repo_new.path >actual &&
+		test_cmp expect actual&&
+		echo "$submodurl/bare.git" >expect &&
+		git config -f .gitmodules submodule.repo_new.url >actual &&
+		test_cmp expect actual &&
+		echo "$submodurl/bare.git" >expect &&
+		git config submodule.repo_new.url >actual &&
+		test_cmp expect actual
+	)
+'
+
 test_done

diff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh
index 1542653..2d44c51 100755
--- a/t/t7406-submodule-update.sh
+++ b/t/t7406-submodule-update.sh

@@ -627,7 +627,7 @@ test_expect_success 'submodule add properly re-creates deeper level submodules'
 	(cd super &&
 	 git reset --hard master &&
 	 rm -rf deeper/ &&
-	 git submodule add ../submodule deeper/submodule
+	 git submodule add --name deeper/submodule2 ../submodule deeper/submodule
 	)
 '

-- 
1.7.12.1.430.g4fd6dc4