Old submodules broken in 2.19rc1 and 2.19rc2

10 messages, 4 authors, 2018-09-08 · open the first message on its own page

Old submodules broken in 2.19rc1 and 2.19rc2

From: Allan Sandfeld Jensen <hidden>
Date: 2018-09-07 09:53:48

Submodules checked out with older versions of git not longer works in the 
latest 2.19 releases. A "git submodule update --recursive" command wil fail 
for each submodule with a line saying "fatal: could not open 
'<submodule>/.git' for writing> Is a directory.

I checked the release notes so far, and they do not say the old submodules 
from git 2.16- were no longer supported, so I assume it is a bug?

'Allan

Re: Old submodules broken in 2.19rc1 and 2.19rc2

From: Jeff King <hidden>
Date: 2018-09-07 15:03:30

On Fri, Sep 07, 2018 at 11:52:58AM +0200, Allan Sandfeld Jensen wrote:
Submodules checked out with older versions of git not longer works in the 
latest 2.19 releases. A "git submodule update --recursive" command wil fail 
for each submodule with a line saying "fatal: could not open 
'<submodule>/.git' for writing> Is a directory.
I couldn't reproduce after cloning with v1.6.6.3 (which creates ".git"
as a directory in the tree). Is it possible for you to bisect (or
perhaps share with us the broken example)?
I checked the release notes so far, and they do not say the old submodules 
from git 2.16- were no longer supported, so I assume it is a bug?
I don't think it was intentional. +cc Stefan for submodule expertise.

-Peff

Re: Old submodules broken in 2.19rc1 and 2.19rc2

From: Allan Sandfeld Jensen <hidden>
Date: 2018-09-07 15:19:01

On Freitag, 7. September 2018 17:03:27 CEST Jeff King wrote:
On Fri, Sep 07, 2018 at 11:52:58AM +0200, Allan Sandfeld Jensen wrote:
quoted
Submodules checked out with older versions of git not longer works in the
latest 2.19 releases. A "git submodule update --recursive" command wil
fail
for each submodule with a line saying "fatal: could not open
'<submodule>/.git' for writing> Is a directory.
I couldn't reproduce after cloning with v1.6.6.3 (which creates ".git"
as a directory in the tree). Is it possible for you to bisect (or
perhaps share with us the broken example)?
I discovered it by using Debian testing, and it is shipping the 2.17rcs for 
some reason. The example is just an old checkout of qt5.git with submodules, 
it is rather large.

I could try bisecting, but I am not sure I have the time anytime soon, I just 
wanted to report it to you first incase you knew of a change that suddenly 
assumed the new structure.

'Allan


Re: Old submodules broken in 2.19rc1 and 2.19rc2

From: Stefan Beller <hidden>
Date: 2018-09-07 17:08:57

On Fri, Sep 7, 2018 at 2:53 AM Allan Sandfeld Jensen [off-list ref] wrote:
Submodules checked out with older versions of git not longer works in the
latest 2.19 releases. A "git submodule update --recursive" command wil fail
for each submodule with a line saying "fatal: could not open
'<submodule>/.git' for writing> Is a directory.
Can you run the update again with

    GIT_TRACE=1 git submodule update ....

and post the output?

I have the suspicion that e98317508c0 (submodule:
ensure core.worktree is set after update, 2018-06-18)
might be the offender.

Could you try reverting that commit and check as well?

    git clone https://github.com/git/git && cd git
    git revert e98317508c0
    make install # installs to you home dir at ~/bin

and then try again, as well?
(though bisection may be more fruitful if this doesn't pan out)

Re: Old submodules broken in 2.19rc1 and 2.19rc2

From: Jonathan Nieder <hidden>
Date: 2018-09-07 20:14:44

Hi,

Allan Sandfeld Jensen wrote:
I discovered it by using Debian testing, and it is shipping the 2.17rcs for
some reason.
I believe you mean Debian unstable.  Debian testing has 2.18.0.
             The example is just an old checkout of qt5.git with submodules, 
it is rather large.
Do you have a reproduction recipe I can use (starting with a "git clone"
command I would run using an old version of Git, etc)?
I could try bisecting, but I am not sure I have the time anytime soon, I just 
wanted to report it to you first incase you knew of a change that suddenly 
assumed the new structure.
This is definitely not an intentional change, so more details would be
very welcome.

Thanks,
Jonathan

Re: Old submodules broken in 2.19rc1 and 2.19rc2

From: Jonathan Nieder <hidden>
Date: 2018-09-07 20:20:12

Stefan Beller wrote:
On Fri, Sep 7, 2018 at 2:53 AM Allan Sandfeld Jensen [off-list ref] wrote:
quoted
                      A "git submodule update --recursive" command wil fail
for each submodule with a line saying "fatal: could not open
'<submodule>/.git' for writing> Is a directory.
[...]
I have the suspicion that e98317508c0 (submodule:
ensure core.worktree is set after update, 2018-06-18)
might be the offender.
Oh!  That seems likely.

Allan, output from "strace -f git submodule update --init" would also
be interesting.

Jonathan

Re: Old submodules broken in 2.19rc1 and 2.19rc2

From: Jonathan Nieder <hidden>
Date: 2018-09-07 22:35:20

Stefan Beller wrote:
On Fri, Sep 7, 2018 at 2:53 AM Allan Sandfeld Jensen [off-list ref] wrote:
quoted
Submodules checked out with older versions of git not longer works in the
latest 2.19 releases. A "git submodule update --recursive" command wil fail
for each submodule with a line saying "fatal: could not open
'<submodule>/.git' for writing> Is a directory.
[...]
I have the suspicion that e98317508c0 (submodule:
ensure core.worktree is set after update, 2018-06-18)
might be the offender.
I still was not able to reproduce it, but after a bit of staring at
the code, I'm pretty sure I just did something wrong in the
reproduction process.  That commit is indeed the offender.

It introduces the following code (rewrapped for clarity) in
git-submodule.sh:

	if ! $(
		git config -f \
			"$(git rev-parse --git-common-dir)/modules/$name/config" \
			core.worktree
	) 2>/dev/null
	then
		git submodule--helper connect-gitdir-workingtree "$name" "$sm_path"
	fi

Staring at it for a while, you can see one problem: the 'if ! $(git
config)' should be simply 'if ! git config'.  This ends up trying to
run the core.worktree value as a command, which would usually fail.

That brings us into connect_work_tree_and_git_dir, which does

	/* Prepare .git file */
	strbuf_addf(&gitfile_sb, "%s/.git", work_tree_);
	if (safe_create_leading_directories_const(gitfile_sb.buf))
		die(_("could not create directories for %s"), gitfile_sb.buf);

	/* Prepare config file */
	strbuf_addf(&cfg_sb, "%s/config", git_dir_);
	if (safe_create_leading_directories_const(cfg_sb.buf))
		die(_("could not create directories for %s"), cfg_sb.buf);

	git_dir = real_pathdup(git_dir_, 1);
	work_tree = real_pathdup(work_tree_, 1);

	/* Write .git file */
	write_file(gitfile_sb.buf, "gitdir: %s",
		   relative_path(git_dir, work_tree, &rel_path));

The write_file runs into .git already existing as a directory, failing
with the message Allan saw.

This would happen in at least two cases:

- if the submodule exists both in .git/modules/ *and* in the worktree
  (due to flipping between Git versions and branches with and without
  the submodule), the above will happen

- likewise if the submodule exists only in the worktree, like for Allan.

In "next" there is 74d4731d (submodule--helper: replace
connect-gitdir-workingtree by ensure-core-worktree, 2018-08-13) which
uses robust helpers in C that handle this much better.  I think we
should revert e98317508c0 in "master" (for 2.19) and keep making use
of that 'second try' in "next" (for 2.20).

I'll try to pin down a reproduction case and send a revert + testsuite
patch.

Thanks again,
Jonathan

Re: Old submodules broken in 2.19rc1 and 2.19rc2

From: Allan Sandfeld Jensen <hidden>
Date: 2018-09-07 22:38:04

On Freitag, 7. September 2018 19:08:43 CEST Stefan Beller wrote:
On Fri, Sep 7, 2018 at 2:53 AM Allan Sandfeld Jensen [off-list ref] 
wrote:
quoted
Submodules checked out with older versions of git not longer works in the
latest 2.19 releases. A "git submodule update --recursive" command wil
fail
for each submodule with a line saying "fatal: could not open
'<submodule>/.git' for writing> Is a directory.
Can you run the update again with

    GIT_TRACE=1 git submodule update ....

and post the output?

I have the suspicion that e98317508c0 (submodule:
ensure core.worktree is set after update, 2018-06-18)
might be the offender.

Could you try reverting that commit and check as well?

    git clone https://github.com/git/git && cd git
    git revert e98317508c0
    make install # installs to you home dir at ~/bin

and then try again, as well?
(though bisection may be more fruitful if this doesn't pan out)
Okay. I had the issue on my workstation at work which I won't be back to until 
friday next week, but I managed to reproduce the exact same issue on separate 
machine running Ubuntu, and a freshly built git from git master, on another 
roughly one year old checkout of qt5.git with submodules

Re: Old submodules broken in 2.19rc1 and 2.19rc2

From: Stefan Beller <hidden>
Date: 2018-09-07 22:45:38

 I think we
should revert e98317508c0 in "master" (for 2.19) and keep making use
of that 'second try' in "next" (for 2.20).
Actually I'd rather revert the whole topic leading up to
7e25437d35a (Merge branch 'sb/submodule-core-worktree', 2018-07-18)
as the last patch in there doesn't work well without e98317508c0 IIRC.

And having only the first patch would bring an inconsistent state as
then different commands behave differently w.r.t. setting core.worktree.

So for this release we'd

  git revert 984cd77ddbf0 e98317508c 4fa4f90ccd85

and then can

  git cherry-pick 4fa4f90ccd8 984cd77ddbf0

on top of sb/submodule-update-in-c, as that re-instates the behavior
going forward.

Thoughts?

Thanks,
Stefan

[PATCH] Revert "Merge branch 'sb/submodule-core-worktree'" (was Re: Old submodules broken in 2.19rc1 and 2.19rc2)

From: Jonathan Nieder <hidden>
Date: 2018-09-08 00:09:51

Subject: Revert "Merge branch 'sb/submodule-core-worktree'"

This reverts commit 7e25437d35a70791b345872af202eabfb3e1a8bc, reversing
changes made to 00624d608cc69bd62801c93e74d1ea7a7ddd6598.

v2.19.0-rc0~165^2~1 (submodule: ensure core.worktree is set after
update, 2018-06-18) assumes an "absorbed" submodule layout, where the
submodule's Git directory is in the superproject's .git/modules/
directory and .git in the submodule worktree is a .git file pointing
there.  In particular, it uses $GIT_DIR/modules/$name to find the
submodule to find out whether it already has core.worktree set, and it
uses connect_work_tree_and_git_dir if not, resulting in

	fatal: could not open sub/.git for writing

The context behind that patch: v2.19.0-rc0~165^2~2 (submodule: unset
core.worktree if no working tree is present, 2018-06-12) unsets
core.worktree when running commands like "git checkout
--recurse-submodules" to switch to a branch without the submodule.  If
a user then uses "git checkout --no-recurse-submodules" to switch back
to a branch with the submodule and runs "git submodule update", this
patch is needed to ensure that commands using the submodule directly
are aware of the path to the worktree.

It is late in the release cycle, so revert the whole 3-patch series.
We can try again later for 2.20.

Reported-by: Allan Sandfeld Jensen <redacted>
Helped-by: Stefan Beller [off-list ref]
Signed-off-by: Jonathan Nieder <redacted>
---
Stefan Beller wrote:
Jonathan Nieder wrote:
quoted
I think we
should revert e98317508c0 in "master" (for 2.19) and keep making use
of that 'second try' in "next" (for 2.20).
Actually I'd rather revert the whole topic leading up to
7e25437d35a (Merge branch 'sb/submodule-core-worktree', 2018-07-18)
as the last patch in there doesn't work well without e98317508c0 IIRC.

And having only the first patch would bring an inconsistent state as
then different commands behave differently w.r.t. setting core.worktree.
Like this (generated using "git revert -m1)?

 builtin/submodule--helper.c | 26 --------------------------
 git-submodule.sh            |  5 -----
 submodule.c                 | 14 --------------
 submodule.h                 |  2 --
 t/lib-submodule-update.sh   |  5 ++---
 t/t7400-submodule-basic.sh  |  5 -----
 6 files changed, 2 insertions(+), 55 deletions(-)
diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index b56028ba9d..f6fb8991f3 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -1123,8 +1123,6 @@ static void deinit_submodule(const char *path, const char *prefix,
 		if (!(flags & OPT_QUIET))
 			printf(format, displaypath);
 
-		submodule_unset_core_worktree(sub);
-
 		strbuf_release(&sb_rm);
 	}
 
@@ -2005,29 +2003,6 @@ static int check_name(int argc, const char **argv, const char *prefix)
 	return 0;
 }
 
-static int connect_gitdir_workingtree(int argc, const char **argv, const char *prefix)
-{
-	struct strbuf sb = STRBUF_INIT;
-	const char *name, *path;
-	char *sm_gitdir;
-
-	if (argc != 3)
-		BUG("submodule--helper connect-gitdir-workingtree <name> <path>");
-
-	name = argv[1];
-	path = argv[2];
-
-	strbuf_addf(&sb, "%s/modules/%s", get_git_dir(), name);
-	sm_gitdir = absolute_pathdup(sb.buf);
-
-	connect_work_tree_and_git_dir(path, sm_gitdir, 0);
-
-	strbuf_release(&sb);
-	free(sm_gitdir);
-
-	return 0;
-}
-
 #define SUPPORT_SUPER_PREFIX (1<<0)
 
 struct cmd_struct {
@@ -2041,7 +2016,6 @@ static struct cmd_struct commands[] = {
 	{"name", module_name, 0},
 	{"clone", module_clone, 0},
 	{"update-clone", update_clone, 0},
-	{"connect-gitdir-workingtree", connect_gitdir_workingtree, 0},
 	{"relative-path", resolve_relative_path, 0},
 	{"resolve-relative-url", resolve_relative_url, 0},
 	{"resolve-relative-url-test", resolve_relative_url_test, 0},
diff --git a/git-submodule.sh b/git-submodule.sh
index f7fd80345c..1cb2c0a31b 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -580,11 +580,6 @@ cmd_update()
 			die "$(eval_gettext "Unable to find current \${remote_name}/\${branch} revision in submodule path '\$sm_path'")"
 		fi
 
-		if ! $(git config -f "$(git rev-parse --git-common-dir)/modules/$name/config" core.worktree) 2>/dev/null
-		then
-			git submodule--helper connect-gitdir-workingtree "$name" "$sm_path"
-		fi
-
 		if test "$subsha1" != "$sha1" || test -n "$force"
 		then
 			subforce=$force
diff --git a/submodule.c b/submodule.c
index 50cbf5f13e..a2b266fbfa 100644
--- a/submodule.c
+++ b/submodule.c
@@ -1534,18 +1534,6 @@ int bad_to_remove_submodule(const char *path, unsigned flags)
 	return ret;
 }
 
-void submodule_unset_core_worktree(const struct submodule *sub)
-{
-	char *config_path = xstrfmt("%s/modules/%s/config",
-				    get_git_common_dir(), sub->name);
-
-	if (git_config_set_in_file_gently(config_path, "core.worktree", NULL))
-		warning(_("Could not unset core.worktree setting in submodule '%s'"),
-			  sub->path);
-
-	free(config_path);
-}
-
 static const char *get_super_prefix_or_empty(void)
 {
 	const char *s = get_super_prefix();
@@ -1711,8 +1699,6 @@ int submodule_move_head(const char *path,
 
 			if (is_empty_dir(path))
 				rmdir_or_warn(path);
-
-			submodule_unset_core_worktree(sub);
 		}
 	}
 out:
diff --git a/submodule.h b/submodule.h
index 7d476cefa7..e452919aa4 100644
--- a/submodule.h
+++ b/submodule.h
@@ -127,8 +127,6 @@ int submodule_move_head(const char *path,
 			const char *new_head,
 			unsigned flags);
 
-void submodule_unset_core_worktree(const struct submodule *sub);
-
 /*
  * Prepare the "env_array" parameter of a "struct child_process" for executing
  * a submodule by clearing any repo-specific environment variables, but
diff --git a/t/lib-submodule-update.sh b/t/lib-submodule-update.sh
index 5b56b23166..016391723c 100755
--- a/t/lib-submodule-update.sh
+++ b/t/lib-submodule-update.sh
@@ -235,7 +235,7 @@ reset_work_tree_to_interested () {
 	then
 		mkdir -p submodule_update/.git/modules/sub1/modules &&
 		cp -r submodule_update_repo/.git/modules/sub1/modules/sub2 submodule_update/.git/modules/sub1/modules/sub2
-		# core.worktree is unset for sub2 as it is not checked out
+		GIT_WORK_TREE=. git -C submodule_update/.git/modules/sub1/modules/sub2 config --unset core.worktree
 	fi &&
 	# indicate we are interested in the submodule:
 	git -C submodule_update config submodule.sub1.url "bogus" &&
@@ -709,8 +709,7 @@ test_submodule_recursing_with_args_common() {
 			git branch -t remove_sub1 origin/remove_sub1 &&
 			$command remove_sub1 &&
 			test_superproject_content origin/remove_sub1 &&
-			! test -e sub1 &&
-			test_must_fail git config -f .git/modules/sub1/config core.worktree
+			! test -e sub1
 		)
 	'
 	# ... absorbing a .git directory along the way.
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index 7d3d984210..c0ffc1022a 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -984,11 +984,6 @@ test_expect_success 'submodule deinit should remove the whole submodule section
 	rmdir init
 '
 
-test_expect_success 'submodule deinit should unset core.worktree' '
-	test_path_is_file .git/modules/example/config &&
-	test_must_fail git config -f .git/modules/example/config core.worktree
-'
-
 test_expect_success 'submodule deinit from subdirectory' '
 	git submodule update --init &&
 	git config submodule.example.foo bar &&
-- 
2.19.0.rc2.392.g5ba43deb5a
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help