[RFC/PATCH 0/4] Add submodule test harness

STALE3752d

Revision rfc of 2 in this series.

9 messages, 2 authors, 2016-06-15 · open the first message on its own page

[RFC/PATCH 0/4] Add submodule test harness

From: Jens Lehmann <hidden>
Date: 2016-06-15 23:00:31 

So here is my proposal for a single test harness for submodule updates. It is
intended to be easily applicable to all work tree manipulating commands. The
current version tests the status quo (and it documents two bugs in current
Git). This framework will be extended to cover recursive submodule update too
in a later series.

The first patch adds a simple helper function to the test lib which makes
it easier to test for an empty submodule directory.

The second patch contains the heavy lifting, it adds the test framework for
switching submodules. Currently only transitions without merge conflicts are
tested for, I intend to add others producing merge conflicts in a follow-up
series.

The third and forth patch show how to apply this framework to a simple
command (checkout) and a more complicated case where a helper function is
used to execute a preparing command (diff) which produces the input for the
then to-be-tested command (apply).

I'm currently working on adding this harness to more commands. read-tree and
reset are finished, merge and pull still show some oddities when used with
the --no-ff option, also cherry-pick and checkout-index are not working as
expected yet. And then there are am, apply, bisect, rebase, revert & stash
apply which still need to be covered.

Jens Lehmann (4):
  test-lib: add test_dir_is_empty()
  Submodules: Add the lib-submodule-update.sh test library
  checkout: call the new submodule update test framework
  apply: add t4137 for submodule updates

 t/lib-submodule-update.sh     | 627 ++++++++++++++++++++++++++++++++++++++++++
 t/t2013-checkout-submodule.sh |   5 +
 t/t4137-apply-submodule.sh    |  20 ++
 t/test-lib-functions.sh       |  11 +
 4 files changed, 663 insertions(+)
 create mode 100755 t/lib-submodule-update.sh
 create mode 100755 t/t4137-apply-submodule.sh

-- 
1.9.1.327.g3d8d896

[RFC/PATCH 1/4] test-lib: add test_dir_is_empty()

From: Jens Lehmann <hidden>
Date: 2016-06-15 23:00:31 

For the upcoming submodule test framework we often need to assert that an
empty directory exists in the work tree. Add the test_dir_is_empty()
function which asserts that the given argument is an empty directory.

Signed-off-by: Jens Lehmann <redacted>
---

I believe this one is pretty straightforward (unless I missed that this
functionality already exists someplace I forgot to look ;-).

 t/test-lib-functions.sh | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh
index 158e10a..93d10cd 100644
--- a/t/test-lib-functions.sh
+++ b/t/test-lib-functions.sh
@@ -489,6 +489,17 @@ test_path_is_dir () {
 	fi
 }

+# Check if the directory exists and is empty as expected, barf otherwise.
+test_dir_is_empty () {
+	test_path_is_dir "$1" &&
+	if test $(ls -a1 "$1" | wc -l) != 2
+	then
+		echo "Directory '$1' is not empty, it contains:"
+		ls -la "$1"
+		return 1
+	fi
+}
+
 test_path_is_missing () {
 	if [ -e "$1" ]
 	then
-- 
1.9.1.327.g3d8d896

[RFC/PATCH 2/4] Submodules: Add the lib-submodule-update.sh test library

From: Jens Lehmann <hidden>
Date: 2016-06-15 23:00:31 

Add this test library to simplify covering all combinations of submodule
update scenarios without having to add those to a test of each work tree
manipulating command over and over again.

The functions test_submodule_switch() and test_submodule_forced_switch()
are intended to be called from a test script with a single argument. This
argument is either a work tree manipulating command (including any command
line options) or a function (when more than a single git command is needed
to switch work trees from the current HEAD to another commit). This
command (or function) is passed a target branch as argument. The two new
functions check that each submodule transition is handled as expected,
which currently means that submodule work trees are not affected until
"git submodule update" is called. The "forced" variant is for commands
using their '-f' or '--hard' option and expects them to overwrite local
modifications as a result. Each of these two functions contains 14
tests_expect_* calls.

Calling one of these test functions the first time creates a repository
named "submodule_update_repo". At first it contains two files, then a
single submodule is added in another commit followed by commits covering
all relevant submodule modifications. This repository is newly cloned into
the "submodule_update" for each test_expect_* to avoid interference
between different parts of the test functions (some to-be-tested commands
also manipulate refs along with the work tree, e.g. "git reset").

Follow-up commits will then call these two test functions for all work
tree manipulating commands (with a combination of all their options
relevant to what they do with the work tree) making sure they work as
expected. Later this test library will be extended to cover merges
resulting in conflicts too. Also it is intended to be easily extendable
for the recursive update functionality, where even more combinations of
submodule modifications have to be tested for.

This version documents two bugs in current Git with expected failures:

*) When a submodule is replaced with a tracked file of the same name the
   submodule work tree including any local modifications (and even the
   whole history if it uses a .git directory instead of a gitfile!) is
   simply removed.

*) Forced work tree updates happily manipulate files in the directory of a
   submodule that has just been removed in the superproject (but is of
   course still present in the work tree due to the way submodules are
   currently handled). This becomes dangerous when files in the submodule
   directory are overwritten by files from the new superproject commit, as
   any modifications to the submodule files will be lost) and is expected
   to also destroy history in the - admittedly unlikely case - the new
   commit adds a file named ".git" to the submodule directory.

Signed-off-by: Jens Lehmann <redacted>
---

I think the first bug really needs to be fixed, as that behavior is extremely
nasty. We should always protect work tree modifications (unless forced) and
*never* remove a .git directory (even when forced).

I'm not so sure about the second one. Even though I believe the current
behavior is not correct (switching commits should never mess around in a
submodule directory) it may be that people who committed a directory =>
submodule transition (or vice versa) are depending on the current
behavior. Fixing the bug would forbid to repeat the submodule => directory
transition and only allow that after the user removed the submodule directory
manually. And as the potential fallout of this bug is not as disastrous as
for the first bug, I might be convinced we should not fix it.

 t/lib-submodule-update.sh | 627 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 627 insertions(+)
 create mode 100755 t/lib-submodule-update.sh

diff --git a/t/lib-submodule-update.sh b/t/lib-submodule-update.sh
new file mode 100755
index 0000000..de5d54e
--- /dev/null
+++ b/t/lib-submodule-update.sh
@@ -0,0 +1,627 @@
+# Create a submodule layout used for all tests below.
+#
+# The following use cases are covered:
+# - New submodule (no_submodule => add_sub1)
+# - Removed submodule (add_sub1 => remove_sub1)
+# - Updated submodule (add_sub1 => modify_sub1)
+# - Submodule updated to invalid commit (add_sub1 => invalid_sub1)
+# - Submodule updated from invalid commit (invalid_sub1 => valid_sub1)
+# - Submodule replaced by tracked files in directory (add_sub1 =>
+#   replace_sub1_with_directory)
+# - Directory containing tracked files replaced by submodule
+#   (replace_sub1_with_directory => replace_directory_with_sub1)
+# - Submodule replaced by tracked file with the same name (add_sub1 =>
+#   replace_sub1_with_file)
+# - Tracked file replaced by submodule (replace_sub1_with_file =>
+#   replace_file_with_sub1)
+#
+#                   --O-----O
+#                  /  ^     replace_directory_with_sub1
+#                 /   replace_sub1_with_directory
+#                /----O
+#               /     ^
+#              /      modify_sub1
+#      O------O-------O
+#      ^      ^\      ^
+#      |      | \     remove_sub1
+#      |      |  -----O-----O
+#      |      |   \   ^     replace_file_with_sub1
+#      |      |    \  replace_sub1_with_file
+#      |   add_sub1 --O-----O
+# no_submodule        ^     valid_sub1
+#                     invalid_sub1
+#
+create_lib_submodule_repo () {
+	git init submodule_update_repo &&
+	(
+		cd submodule_update_repo &&
+		echo "expect" >>.gitignore &&
+		echo "actual" >>.gitignore &&
+		echo "x" >file1 &&
+		echo "y" >file2 &&
+		git add .gitignore file1 file2 &&
+		git commit -m "Base" &&
+		git branch "no_submodule" &&
+
+		git checkout -b "add_sub1" &&
+		git submodule add ./. sub1 &&
+		git commit -m "Add sub1" &&
+		git checkout -b remove_sub1 &&
+		git revert HEAD &&
+
+		git checkout -b "modify_sub1" "add_sub1" &&
+		git submodule update &&
+		(
+			cd sub1 &&
+			git fetch &&
+			git checkout -b "modifications" &&
+			echo "z" >file2 &&
+			echo "x" >file3 &&
+			git add file2 file3 &&
+			git commit -m "modified file2 and added file3" &&
+			git push origin modifications
+		) &&
+		git add sub1 &&
+		git commit -m "Modify sub1" &&
+
+		git checkout -b "replace_sub1_with_directory" "add_sub1" &&
+		git submodule update &&
+		(
+			cd sub1 &&
+			git checkout modifications
+		) &&
+		git rm --cached sub1 &&
+		rm sub1/.git* &&
+		git config -f .gitmodules --remove-section "submodule.sub1" &&
+		git add .gitmodules sub1/* &&
+		git commit -m "Replace sub1 with directory" &&
+		git checkout -b replace_directory_with_sub1 &&
+		git revert HEAD &&
+
+		git checkout -b "replace_sub1_with_file" "add_sub1" &&
+		git rm sub1 &&
+		echo "content" >sub1 &&
+		git add sub1 &&
+		git commit -m "Replace sub1 with file" &&
+		git checkout -b replace_file_with_sub1 &&
+		git revert HEAD &&
+
+		git checkout -b "invalid_sub1" "add_sub1" &&
+		git update-index --cacheinfo 160000 0123456789012345678901234567890123456789 sub1 &&
+		git commit -m "Invalid sub1 commit" &&
+		git checkout -b valid_sub1 &&
+		git revert HEAD &&
+		git checkout master
+	)
+}
+
+# Helper function to replace gitfile with .git directory
+replace_gitfile_with_git_dir () {
+	(
+		cd "$1" &&
+		git_dir="$(git rev-parse --git-dir)" &&
+		rm -f .git &&
+		cp -a "$git_dir" .git &&
+		GIT_WORK_TREE=. git config --unset core.worktree
+	)
+}
+
+# Test that the .git directory in the submodule is unchanged (except for the
+# core.worktree setting)
+test_git_directory_is_unchanged () {
+	(
+		cd "$1" &&
+		git config core.worktree "../../../$1"
+	) &&
+	git diff -r ".git/modules/$1" "$1/.git" &&
+	(
+		cd "$1" &&
+		GIT_WORK_TREE=. git config --unset core.worktree
+	)
+}
+
+# Helper function to be executed at the start of every test below, it sets up
+# the submodule repo if it doesn't exist and configures the most problematic
+# settings for diff.ignoreSubmodules.
+prolog () {
+	(test -d submodule_update_repo || create_lib_submodule_repo) &&
+	test_config_global diff.ignoreSubmodules all &&
+	test_config diff.ignoreSubmodules all
+}
+
+# Helper function to bring work tree back into the state given by the
+# commit. This includes trying to populate sub1 accordingly if it exists and
+# should be updated to an existing commit.
+reset_work_tree_to () {
+	rm -rf submodule_update &&
+	git clone submodule_update_repo submodule_update &&
+	(
+		cd submodule_update &&
+		rm -rf sub1 &&
+		git checkout -f "$1" &&
+		git status -u -s >actual &&
+		test_must_be_empty actual &&
+		sha1=$(git ls-tree HEAD "sub1" 2>/dev/null | grep 160000 | tr '\t' ' ' | cut -d ' ' -f3) &&
+		if test -n "$sha1" &&
+		   test $(cd "sub1" && git rev-parse --verify "$sha1^{commit}")
+		then
+			git submodule update --init --recursive "sub1"
+		fi
+	)
+}
+
+# Test that the superproject contains the content according to commit "$1"
+# (the work tree must match the index for everything but submodules but the
+# index must exactly match the given commit including any submodule SHA-1s).
+test_superproject_content () {
+	git diff-index --cached "$1" >actual &&
+	test_must_be_empty actual &&
+	git diff-files --ignore-submodules >actual &&
+	test_must_be_empty actual
+}
+
+# Test that the given submodule at path "$1" contains the content according
+# to the submodule commit recorded in the superproject's commit "$2"
+test_submodule_content () {
+	if test $# != 2
+	then
+		echo "test_submodule_content needs two arguments"
+		return 1
+	fi &&
+	submodule="$1" &&
+	commit="$2" &&
+	test -d "$submodule"/ &&
+	if ! test -f "$submodule"/.git && ! test -d "$submodule"/.git
+	then
+		echo "Submodule $submodule is not populated"
+		return 1
+	fi &&
+	sha1=$(git ls-tree "$commit" "$submodule" 2>/dev/null | tr '\t' ' ' | cut -d ' ' -f3) &&
+	if test -z "$sha1"
+	then
+		echo "Couldn't retrieve SHA-1 of $submodule for $commit"
+		return 1
+	fi &&
+	(
+		cd "$submodule" &&
+		git status -u -s >actual &&
+		test_must_be_empty actual &&
+		git diff "$sha1" >actual &&
+		test_must_be_empty actual
+	)
+}
+
+# Test that the following transitions are correctly handled:
+# - Updated submodule
+# - New submodule
+# - Removed submodule
+# - Directory containing tracked files replaced by submodule
+# - Submodule replaced by tracked files in directory
+# - Submodule replaced by tracked file with the same name
+# - tracked file replaced by submodule
+#
+# The default is that submodule contents aren't changed until "git submodule
+# update" is run. And even then that command doesn't delete the work tree of
+# a removed submodule.
+#
+# Removing a submodule containing a .git directory must fail even when forced
+# to protect the history!
+#
+
+# Test that submodule contents are currently not updated when switching
+# between commits that change a submodule.
+test_submodule_switch () {
+	command="$1"
+	######################### Appearing submodule #########################
+	# Switching to a commit letting a submodule appear creates empty dir ...
+	test_expect_success "$command: added submodule creates empty directory" '
+		prolog &&
+		reset_work_tree_to no_submodule &&
+		(
+			cd submodule_update &&
+			git branch -t add_sub1 origin/add_sub1 &&
+			$command add_sub1 &&
+			test_superproject_content origin/add_sub1 &&
+			test_dir_is_empty sub1 &&
+			git submodule update --init --recursive &&
+			test_submodule_content sub1 origin/add_sub1
+		)
+	'
+	# ... and doesn't care if it already exists ...
+	test_expect_success "$command: added submodule leaves existing empty directory alone" '
+		prolog &&
+		reset_work_tree_to no_submodule &&
+		(
+			cd submodule_update &&
+			mkdir sub1 &&
+			git branch -t add_sub1 origin/add_sub1 &&
+			$command add_sub1 &&
+			test_superproject_content origin/add_sub1 &&
+			test_dir_is_empty sub1 &&
+			git submodule update --init --recursive &&
+			test_submodule_content sub1 origin/add_sub1
+		)
+	'
+	# ... unless there is an untracked file in its place.
+	test_expect_success "$command: added submodule doesn't remove untracked unignored file with same name" '
+		prolog &&
+		reset_work_tree_to no_submodule &&
+		(
+			cd submodule_update &&
+			git branch -t add_sub1 origin/add_sub1 &&
+			echo -n >sub1 &&
+			test_must_fail $command add_sub1 &&
+			test_superproject_content origin/no_submodule &&
+			test_must_be_empty sub1
+		)
+	'
+	# Replacing a tracked file with a submodule produces an empty
+	# directory ...
+	test_expect_success "$command: replace tracked file with submodule creates empty directory" '
+		prolog &&
+		reset_work_tree_to replace_sub1_with_file &&
+		(
+			cd submodule_update &&
+			git branch -t replace_file_with_sub1 origin/replace_file_with_sub1 &&
+			$command replace_file_with_sub1 &&
+			test_superproject_content origin/replace_file_with_sub1 &&
+			test_dir_is_empty sub1 &&
+			git submodule update --init --recursive &&
+			test_submodule_content sub1 origin/replace_file_with_sub1
+		)
+	'
+	# ... as does removing a directory with tracked files with a
+	# submodule.
+	test_expect_success "$command: replace directory with submodule" '
+		prolog &&
+		reset_work_tree_to replace_sub1_with_directory &&
+		(
+			cd submodule_update &&
+			git branch -t replace_directory_with_sub1 origin/replace_directory_with_sub1 &&
+			$command replace_directory_with_sub1 &&
+			test_superproject_content origin/replace_directory_with_sub1 &&
+			test_dir_is_empty sub1 &&
+			git submodule update --init --recursive &&
+			test_submodule_content sub1 origin/replace_directory_with_sub1
+		)
+	'
+
+	######################## Disappearing submodule #######################
+	# Removing a submodule doesn't remove its work tree ...
+	test_expect_success "$command: removed submodule leaves submodule directory and its contents in place" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t remove_sub1 origin/remove_sub1 &&
+			$command remove_sub1 &&
+			test_superproject_content origin/remove_sub1 &&
+			test_submodule_content sub1 origin/add_sub1
+		)
+	'
+	# ... especially when it contains a .git directory.
+	test_expect_success "$command: removed submodule leaves submodule containing a .git directory alone" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t remove_sub1 origin/remove_sub1 &&
+			replace_gitfile_with_git_dir sub1 &&
+			$command remove_sub1 &&
+			test_superproject_content origin/remove_sub1 &&
+			test_submodule_content sub1 origin/add_sub1 &&
+			test_git_directory_is_unchanged sub1
+		)
+	'
+	# Replacing a submodule with files in a directory must fail as the
+	# submodule work tree isn't removed ...
+	test_expect_success "$command: replace submodule with a directory fails" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t replace_sub1_with_directory origin/replace_sub1_with_directory &&
+			test_must_fail $command replace_sub1_with_directory &&
+			test_superproject_content origin/add_sub1 &&
+			test_submodule_content sub1 origin/add_sub1
+		)
+	'
+	# ... especially when it contains a .git directory.
+	test_expect_success "$command: replace submodule containing a .git directory with a directory fails" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t replace_sub1_with_directory origin/replace_sub1_with_directory &&
+			replace_gitfile_with_git_dir sub1 &&
+			test_must_fail $command replace_sub1_with_directory &&
+			test_superproject_content origin/add_sub1 &&
+			test_submodule_content sub1 origin/add_sub1 &&
+			test_git_directory_is_unchanged sub1
+		)
+	'
+	# Replacing it with a file must fail as it could throw away any local
+	# work tree changes ...
+	test_expect_failure "$command: replace submodule directory with a file must fail" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t replace_sub1_with_file origin/replace_sub1_with_file &&
+			test_must_fail $command replace_sub1_with_file &&
+			test_superproject_content origin/add_sub1 &&
+			test_submodule_content sub1 origin/add_sub1
+		)
+	'
+	# ... or even destroy unpushed parts of submodule history if that
+	# still uses a .git directory.
+	test_expect_failure "$command: replace submodule directory with a file must fail" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t replace_sub1_with_file origin/replace_sub1_with_file &&
+			replace_gitfile_with_git_dir sub1 &&
+			test_must_fail $command replace_sub1_with_file &&
+			test_superproject_content origin/add_sub1 &&
+			test_submodule_content sub1 origin/add_sub1 &&
+			test_git_directory_is_unchanged sub1
+		)
+	'
+
+	########################## Modified submodule #########################
+	# Updating a submodule sha1 doesn't update the submodule's work tree
+	test_expect_success "$command: modified submodule does not update submodule work tree" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t modify_sub1 origin/modify_sub1 &&
+			$command modify_sub1 &&
+			test_superproject_content origin/modify_sub1 &&
+			test_submodule_content sub1 origin/add_sub1 &&
+			git submodule update &&
+			test_submodule_content sub1 origin/modify_sub1
+		)
+	'
+
+	# Updating a submodule to an invalid sha1 doesn't update the
+	# submodule's work tree, subsequent update will fail
+	test_expect_success "$command: modified submodule does not update submodule work tree to invalid commit" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t invalid_sub1 origin/invalid_sub1 &&
+			$command invalid_sub1 &&
+			test_superproject_content origin/invalid_sub1 &&
+			test_submodule_content sub1 origin/add_sub1 &&
+			test_must_fail git submodule update &&
+			test_submodule_content sub1 origin/add_sub1
+		)
+	'
+	# Updating a submodule from an invalid sha1 doesn't update the
+	# submodule's work tree, subsequent update will succeed
+	test_expect_success "$command: modified submodule does not update submodule work tree from invalid commit" '
+		prolog &&
+		reset_work_tree_to invalid_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t valid_sub1 origin/valid_sub1 &&
+			$command valid_sub1 &&
+			test_superproject_content origin/valid_sub1 &&
+			test_dir_is_empty sub1 &&
+			git submodule update --init --recursive &&
+			test_submodule_content sub1 origin/valid_sub1
+		)
+	'
+}
+
+# Test that submodule contents are currently not updated when switching
+# between commits that change a submodule, but throwing away local changes in
+# the superproject is allowed.
+test_submodule_forced_switch () {
+	command="$1"
+	######################### Appearing submodule #########################
+	# Switching to a commit letting a submodule appear creates empty dir ...
+	test_expect_success "$command: added submodule creates empty directory" '
+		prolog &&
+		reset_work_tree_to no_submodule &&
+		(
+			cd submodule_update &&
+			git branch -t add_sub1 origin/add_sub1 &&
+			$command add_sub1 &&
+			test_superproject_content origin/add_sub1 &&
+			test_dir_is_empty sub1 &&
+			git submodule update --init --recursive &&
+			test_submodule_content sub1 origin/add_sub1
+		)
+	'
+	# ... and doesn't care if it already exists ...
+	test_expect_success "$command: added submodule leaves existing empty directory alone" '
+		prolog &&
+		reset_work_tree_to no_submodule &&
+		(
+			cd submodule_update &&
+			git branch -t add_sub1 origin/add_sub1 &&
+			mkdir sub1 &&
+			$command add_sub1 &&
+			test_superproject_content origin/add_sub1 &&
+			test_dir_is_empty sub1 &&
+			git submodule update --init --recursive &&
+			test_submodule_content sub1 origin/add_sub1
+		)
+	'
+	# ... unless there is an untracked file in its place.
+	test_expect_success "$command: added submodule does remove untracked unignored file with same name when forced" '
+		prolog &&
+		reset_work_tree_to no_submodule &&
+		(
+			cd submodule_update &&
+			git branch -t add_sub1 origin/add_sub1 &&
+			echo -n >sub1 &&
+			$command add_sub1 &&
+			test_superproject_content origin/add_sub1 &&
+			test_dir_is_empty sub1
+		)
+	'
+	# Replacing a tracked file with a submodule produces an empty
+	# directory ...
+	test_expect_success "$command: replace tracked file with submodule creates empty directory" '
+		prolog &&
+		reset_work_tree_to replace_sub1_with_file &&
+		(
+			cd submodule_update &&
+			git branch -t replace_file_with_sub1 origin/replace_file_with_sub1 &&
+			$command replace_file_with_sub1 &&
+			test_superproject_content origin/replace_file_with_sub1 &&
+			test_dir_is_empty sub1 &&
+			git submodule update --init --recursive &&
+			test_submodule_content sub1 origin/replace_file_with_sub1
+		)
+	'
+	# ... as does removing a directory with tracked files with a
+	# submodule.
+	test_expect_success "$command: replace directory with submodule" '
+		prolog &&
+		reset_work_tree_to replace_sub1_with_directory &&
+		(
+			cd submodule_update &&
+			git branch -t replace_directory_with_sub1 origin/replace_directory_with_sub1 &&
+			$command replace_directory_with_sub1 &&
+			test_superproject_content origin/replace_directory_with_sub1 &&
+			test_dir_is_empty sub1 &&
+			git submodule update --init --recursive &&
+			test_submodule_content sub1 origin/replace_directory_with_sub1
+		)
+	'
+
+	######################## Disappearing submodule #######################
+	# Removing a submodule doesn't remove its work tree ...
+	test_expect_success "$command: removed submodule leaves submodule directory and its contents in place" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t remove_sub1 origin/remove_sub1 &&
+			$command remove_sub1 &&
+			test_superproject_content origin/remove_sub1 &&
+			test_submodule_content sub1 origin/add_sub1
+		)
+	'
+	# ... especially when it contains a .git directory.
+	test_expect_success "$command: removed submodule leaves submodule containing a .git directory alone" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t remove_sub1 origin/remove_sub1 &&
+			replace_gitfile_with_git_dir sub1 &&
+			$command remove_sub1 &&
+			test_superproject_content origin/remove_sub1 &&
+			test_submodule_content sub1 origin/add_sub1 &&
+			test_git_directory_is_unchanged sub1
+		)
+	'
+	# Replacing a submodule with files in a directory must fail as the
+	# submodule work tree isn't removed ...
+	test_expect_failure "$command: replace submodule with a directory fails" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t replace_sub1_with_directory origin/replace_sub1_with_directory &&
+			test_must_fail $command replace_sub1_with_directory &&
+			test_superproject_content origin/add_sub1 &&
+			test_submodule_content sub1 origin/add_sub1
+		)
+	'
+	# ... especially when it contains a .git directory.
+	test_expect_failure "$command: replace submodule containing a .git directory with a directory fails" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t replace_sub1_with_directory origin/replace_sub1_with_directory &&
+			replace_gitfile_with_git_dir sub1 &&
+			test_must_fail $command replace_sub1_with_directory &&
+			test_superproject_content origin/add_sub1 &&
+			test_submodule_content sub1 origin/add_sub1 &&
+			test_git_directory_is_unchanged sub1
+		)
+	'
+	# Replacing it with a file must fail as it could throw away any local
+	# work tree changes ...
+	test_expect_failure "$command: replace submodule directory with a file must fail" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t replace_sub1_with_file origin/replace_sub1_with_file &&
+			test_must_fail $command replace_sub1_with_file &&
+			test_superproject_content origin/add_sub1 &&
+			test_submodule_content sub1 origin/add_sub1
+		)
+	'
+	# ... or even destroy unpushed parts of submodule history if that
+	# still uses a .git directory.
+	test_expect_failure "$command: replace submodule directory with a file must fail" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t replace_sub1_with_file origin/replace_sub1_with_file &&
+			replace_gitfile_with_git_dir sub1 &&
+			test_must_fail $command replace_sub1_with_file &&
+			test_superproject_content origin/add_sub1 &&
+			test_submodule_content sub1 origin/add_sub1 &&
+			test_git_directory_is_unchanged sub1
+		)
+	'
+
+	########################## Modified submodule #########################
+	# Updating a submodule sha1 doesn't update the submodule's work tree
+	test_expect_success "$command: modified submodule does not update submodule work tree" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t modify_sub1 origin/modify_sub1 &&
+			$command modify_sub1 &&
+			test_superproject_content origin/modify_sub1 &&
+			test_submodule_content sub1 origin/add_sub1 &&
+			git submodule update &&
+			test_submodule_content sub1 origin/modify_sub1
+		)
+	'
+	# Updating a submodule to an invalid sha1 doesn't update the
+	# submodule's work tree, subsequent update will fail
+	test_expect_success "$command: modified submodule does not update submodule work tree to invalid commit" '
+		prolog &&
+		reset_work_tree_to add_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t invalid_sub1 origin/invalid_sub1 &&
+			$command invalid_sub1 &&
+			test_superproject_content origin/invalid_sub1 &&
+			test_submodule_content sub1 origin/add_sub1 &&
+			test_must_fail git submodule update &&
+			test_submodule_content sub1 origin/add_sub1
+		)
+	'
+	# Updating a submodule from an invalid sha1 doesn't update the
+	# submodule's work tree, subsequent update will succeed
+	test_expect_success "$command: modified submodule does not update submodule work tree from invalid commit" '
+		prolog &&
+		reset_work_tree_to invalid_sub1 &&
+		(
+			cd submodule_update &&
+			git branch -t valid_sub1 origin/valid_sub1 &&
+			$command valid_sub1 &&
+			test_superproject_content origin/valid_sub1 &&
+			test_dir_is_empty sub1 &&
+			git submodule update --init --recursive &&
+			test_submodule_content sub1 origin/valid_sub1
+		)
+	'
+}
-- 
1.9.1.327.g3d8d896

[RFC/PATCH 3/4] checkout: call the new submodule update test framework

From: Jens Lehmann <hidden>
Date: 2016-06-15 23:00:31 

Test that the checkout command updates the work tree as expected with or
without the '-f' flag.

Signed-off-by: Jens Lehmann <redacted>
---

I think this should explain how to use the framework with a single command
and some options.

 t/t2013-checkout-submodule.sh | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/t/t2013-checkout-submodule.sh b/t/t2013-checkout-submodule.sh
index 06b18f8..6847f75 100755
--- a/t/t2013-checkout-submodule.sh
+++ b/t/t2013-checkout-submodule.sh
@@ -3,6 +3,7 @@
 test_description='checkout can handle submodules'

 . ./test-lib.sh
+. "$TEST_DIRECTORY"/lib-submodule-update.sh

 test_expect_success 'setup' '
 	mkdir submodule &&
@@ -62,4 +63,8 @@ test_expect_success '"checkout <submodule>" honors submodule.*.ignore from .git/
 	! test -s actual
 '

+test_submodule_switch "git checkout"
+
+test_submodule_forced_switch "git checkout -f"
+
 test_done
-- 
1.9.1.327.g3d8d896

[RFC/PATCH 4/4] apply: add t4137 for submodule updates

From: Jens Lehmann <hidden>
Date: 2016-06-15 23:00:31 

Test that the apply command updates the work tree as expected for the
'--index' and the '--3way' options (for submodule changes which don't
result in conflicts).

Signed-off-by: Jens Lehmann <redacted>
---

And this shows how to use the new framework when more than a single command
is needed to switch to a new work tree.

 t/t4137-apply-submodule.sh | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
 create mode 100755 t/t4137-apply-submodule.sh

diff --git a/t/t4137-apply-submodule.sh b/t/t4137-apply-submodule.sh
new file mode 100755
index 0000000..f0a0500
--- /dev/null
+++ b/t/t4137-apply-submodule.sh
@@ -0,0 +1,20 @@
+#!/bin/sh
+
+test_description='git am handling submodules'
+
+. ./test-lib.sh
+. "$TEST_DIRECTORY"/lib-submodule-update.sh
+
+apply_index () {
+    git diff --ignore-submodules=dirty "..$1" | git apply --index -
+}
+
+test_submodule_switch "apply_index"
+
+apply_3way () {
+    git diff --ignore-submodules=dirty "..$1" | git apply --3way -
+}
+
+test_submodule_switch "apply_3way"
+
+test_done
-- 
1.9.1.327.g3d8d896

Re: [RFC/PATCH 2/4] Submodules: Add the lib-submodule-update.sh test library

From: W. Trevor King <hidden>
Date: 2016-06-15 23:00:43 

On Tue, Mar 25, 2014 at 06:05:05PM +0100, Jens Lehmann wrote:
*) When a submodule is replaced with a tracked file of the same name
   the submodule work tree including any local modifications (and
   even the whole history if it uses a .git directory instead of a
   gitfile!) is simply removed.
…
I think the first bug really needs to be fixed, as that behavior is
extremely nasty. We should always protect work tree modifications
(unless forced) and *never* remove a .git directory (even when
forced).
I think this should be covered by the usual “don't allow checkouts
from dirty workdirs unless the dirty-ing changes are easily applied to
the target tree”.

Are we waiting to land this series (or a successor) before starting on
a fix for this issue?  There have been a number of submodule series in
flight recently, and I'm having trouble keeping track of them all ;).

*) Forced work tree updates happily manipulate files in the
   directory of a submodule that has just been removed in the
   superproject (but is of course still present in the work tree due
   to the way submodules are currently handled). This becomes
   dangerous when files in the submodule directory are overwritten
   by files from the new superproject commit, as any modifications
   to the submodule files will be lost) and is expected to also
   destroy history in the - admittedly unlikely case - the new
   commit adds a file named ".git" to the submodule directory.
…
I'm not so sure about the second one. Even though I believe the
current behavior is not correct (switching commits should never mess
around in a submodule directory)
This should also be covered by the usual “don't allow checkouts from
dirty workdirs unless the dirty-ing changes are easily applied to the
target tree”.  We don't implement this yet, but I'd like to force
users to move any about-to-be-clobbered state from their submodule
into .git/modules/<name>/ (via commits or stashes) before allowing
them to begin the checkout.  Once we've ensured that the state is
preserved out-of-tree, then clobber away ;).

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

Re: [RFC/PATCH 2/4] Submodules: Add the lib-submodule-update.sh test library

From: Jens Lehmann <hidden>
Date: 2016-06-15 23:00:43 

Am 17.04.2014 18:41, schrieb W. Trevor King:
On Tue, Mar 25, 2014 at 06:05:05PM +0100, Jens Lehmann wrote:
quoted
*) When a submodule is replaced with a tracked file of the same name
   the submodule work tree including any local modifications (and
   even the whole history if it uses a .git directory instead of a
   gitfile!) is simply removed.
…
I think the first bug really needs to be fixed, as that behavior is
extremely nasty. We should always protect work tree modifications
(unless forced) and *never* remove a .git directory (even when
forced).
I think this should be covered by the usual “don't allow checkouts
from dirty workdirs unless the dirty-ing changes are easily applied to
the target tree”.
Nope, the target tree will be removed completely and everything in
it is silently nuked. It should be allowed with '-f', but only if
the submodule contains a gitfile, and never if it contains a .git
directory (which is just what we do for rm too).

Are we waiting to land this series (or a successor) before starting on
a fix for this issue?
I think so, as this bug is there for a long time (so I see no urge
to fix it very soon) and my test harness is intended to document
this current bug (and then soon its fix).
quoted
*) Forced work tree updates happily manipulate files in the
   directory of a submodule that has just been removed in the
   superproject (but is of course still present in the work tree due
   to the way submodules are currently handled). This becomes
   dangerous when files in the submodule directory are overwritten
   by files from the new superproject commit, as any modifications
   to the submodule files will be lost) and is expected to also
   destroy history in the - admittedly unlikely case - the new
   commit adds a file named ".git" to the submodule directory.
…
I'm not so sure about the second one. Even though I believe the
current behavior is not correct (switching commits should never mess
around in a submodule directory)
This should also be covered by the usual “don't allow checkouts from
dirty workdirs unless the dirty-ing changes are easily applied to the
target tree”.  We don't implement this yet, but I'd like to force
users to move any about-to-be-clobbered state from their submodule
into .git/modules/<name>/ (via commits or stashes) before allowing
them to begin the checkout.  Once we've ensured that the state is
preserved out-of-tree, then clobber away ;).
I'm intending to fix this in the recursive checkout series, as I'm
a) not sure if any users currently depend on that for a submodule
to directory transition and b) recursive checkout is the place to
consistently care about submodule modifications (the submodule
script doesn't do that and it is impossible to change that without
causing trouble to a lot of users.

Re: [RFC/PATCH 2/4] Submodules: Add the lib-submodule-update.sh test library

From: W. Trevor King <hidden>
Date: 2016-06-15 23:00:44 

On Thu, Apr 17, 2014 at 11:08:06PM +0200, Jens Lehmann wrote:
Am 17.04.2014 18:41, schrieb W. Trevor King:
quoted
On Tue, Mar 25, 2014 at 06:05:05PM +0100, Jens Lehmann wrote:
quoted
*) When a submodule is replaced with a tracked file of the same name
   the submodule work tree including any local modifications (and
   even the whole history if it uses a .git directory instead of a
   gitfile!) is simply removed.
…
I think the first bug really needs to be fixed, as that behavior is
extremely nasty. We should always protect work tree modifications
(unless forced) and *never* remove a .git directory (even when
forced).
I think this should be covered by the usual “don't allow checkouts
from dirty workdirs unless the dirty-ing changes are easily applied to
the target tree”.
Nope, the target tree will be removed completely and everything in
it is silently nuked. It should be allowed with '-f', but only if
the submodule contains a gitfile, and never if it contains a .git
directory (which is just what we do for rm too).
I think it's not covered *now* because of a flaw in our “are dirty-ing
changes easily applied to the target tree” detection logic, and the
solution should involve updating that logic to hit on this case.

b) recursive checkout is the place to consistently care about
submodule modifications (the submodule script doesn't do that and it
is impossible to change that without causing trouble to a lot of
users.
I agree that the submodule script is not the place for this, and the
core checkout code is.  I'd like checkouts to always be recursive, and
see --[no-]recurse-submodules as a finger-breaking stop-gap until we
can complete that transition for checkout, bisect, merge, reset, and
other work-tree altering commands.  I think this is one reason why
some folks prefer the stiffer joints you get from a subtree approach.

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

Re: [RFC/PATCH 2/4] Submodules: Add the lib-submodule-update.sh test library

From: Jens Lehmann <hidden>
Date: 2016-06-15 23:00:44 

Am 17.04.2014 23:55, schrieb W. Trevor King:
On Thu, Apr 17, 2014 at 11:08:06PM +0200, Jens Lehmann wrote:
quoted
Am 17.04.2014 18:41, schrieb W. Trevor King:
quoted
On Tue, Mar 25, 2014 at 06:05:05PM +0100, Jens Lehmann wrote:
quoted
*) When a submodule is replaced with a tracked file of the same name
   the submodule work tree including any local modifications (and
   even the whole history if it uses a .git directory instead of a
   gitfile!) is simply removed.
…
I think the first bug really needs to be fixed, as that behavior is
extremely nasty. We should always protect work tree modifications
(unless forced) and *never* remove a .git directory (even when
forced).
I think this should be covered by the usual “don't allow checkouts
from dirty workdirs unless the dirty-ing changes are easily applied to
the target tree”.
Nope, the target tree will be removed completely and everything in
it is silently nuked. It should be allowed with '-f', but only if
the submodule contains a gitfile, and never if it contains a .git
directory (which is just what we do for rm too).
I think it's not covered *now* because of a flaw in our “are dirty-ing
changes easily applied to the target tree” detection logic, and the
solution should involve updating that logic to hit on this case.
Yup.
quoted
b) recursive checkout is the place to consistently care about
submodule modifications (the submodule script doesn't do that and it
is impossible to change that without causing trouble to a lot of
users.
I agree that the submodule script is not the place for this, and the
core checkout code is.  I'd like checkouts to always be recursive, and
see --[no-]recurse-submodules as a finger-breaking stop-gap until we
can complete that transition for checkout, bisect, merge, reset, and
other work-tree altering commands.  I think this is one reason why
some folks prefer the stiffer joints you get from a subtree approach.
We are definitely in the same boat here :-)
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help