problem with backslash in directory name

Hi all,

I'm having a problem with submodules that reside in directories that 
(unwisely) contain a backslash in their name.


Transcript:

### Arrange

$ git init main
Initialized empty Git repository in /tmp/test/main/.git/

$ git init sub\\with\\backslash
Initialized empty Git repository in /tmp/test/sub\with\backslash/.git/
# This looks okay: the shell interpreted \\ as \,
# so we get sub\with\backslash

# Create a log entry in sub\with\backslash
# (it can't be added as a submodule otherwise)
# ((actually I think it's a misfeature, my current use case would be
# easier if git didn't insist on having a log in submodules))
$ touch sub\\with\\backslash/empty.file
$ git -C sub\\with\\backslash add empty.file
$ git -C sub\\with\\backslash commit -m "Added empty.file"
[master (root-commit) a27a485] Added empty.file
  1 file changed, 0 insertions(+), 0 deletions(-)
  create mode 100644 empty.file

### Act/Assert

$ git -C main submodule add ../sub\\with\\backslash
fatal: repository '/tmp/test/sub\witackslash' does not exist
fatal: clone of '/tmp/test/sub\witackslash' into submodule path 
'sub\with\backslash' failed
# The first "fatal:" line talks about "witackslash"
# Um... "ackslash"? Now that's a nice nickname for a CVE :-)

# Okay, let's see what's actually in that message
$ git -C main submodule add ../sub\\with\\backslash 2>&1 | xxd
00000000: 6661 7461 6c3a 2072 6570 6f73 6974 6f72  fatal: repositor
00000010: 7920 272f 746d 702f 7465 7374 2f73 7562  y '/tmp/test/sub
00000020: 5c77 6974 6808 6163 6b73 6c61 7368 2720  \with.ackslash'
00000030: 646f 6573 206e 6f74 2065 7869 7374 0a66  does not exist.f
00000040: 6174 616c 3a20 636c 6f6e 6520 6f66 2027  atal: clone of '
00000050: 2f74 6d70 2f74 6573 742f 7375 625c 7769  /tmp/test/sub\wi
00000060: 7468 0861 636b 736c 6173 6827 2069 6e74  th.ackslash' int
00000070: 6f20 7375 626d 6f64 756c 6520 7061 7468  o submodule path
00000080: 2027 7375 625c 7769 7468 5c62 6163 6b73   'sub\with\backs
00000090: 6c61 7368 2720 6661 696c 6564 0a         lash' failed.

# Yeah, there's a 0x08 at offset 0x25.
# It's pretty strange that it is eliding the w following the \b,
# not the h preceding it.


So... something inside "git submodule add" is replacing the \b with a 
backspace control code.


Next I tried something nasty:

$ mv sub\\with\\backslash 'sub: $(bc)'
git -C main submodule add '../sub: $(bc)'
Cloning into ' $(bc)'...
done.

Whatever that "something" is, it is not doing shell expansion, otherwise 
it would have started an interactive calculator session.
Phew :-)
I'm still a bit uneasy because I don't know what other escape sequences 
might get interpreted, and what their effects are.

On Fri, Apr 07, 2017 at 08:12:49AM +0200, Joachim Durchholz wrote:

So... something inside "git submodule add" is replacing the \b with a
backspace control code.
[...]
Whatever that "something" is, it is not doing shell expansion, otherwise it
would have started an interactive calculator session.

Probably it's "read" which does backslash expansion, but nothing else.
Just grepping git-submodule.sh, some of the "read" calls should probably
be "read -r" (I also don't know how some of those loops would cope with
a submodule name that needed quoting).

-Peff

Am 07.04.2017 um 08:30 schrieb Jeff King:

Probably it's "read" which does backslash expansion, but nothing else.
Just grepping git-submodule.sh, some of the "read" calls should probably
be "read -r"

http://wiki.bash-hackers.org/commands/builtin/read has this to say:

Essentially all you need to know about -r is to ALWAYS use it. The

 > exact behavior you get without -r is completely useless even for weird
 > purposes. It basically allows the escaping of input which matches
 > something in IFS, and also escapes line continuations. It's explained
 > pretty well in the POSIX read[1] spec.

[1] 
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/read.html#tag_20_109

(That's the kind of stuff that makes me shy of using shell scripts - 
always yet another surprise in waiting...)

Am 07.04.2017 um 08:30 schrieb Jeff King:

I also don't know how some of those loops would cope with
a submodule name that needed quoting).

"git submodule add" worked fine with most of the following names:

     "sub"
     # potentially confusing the shell
     "sub with blanks",
     "sub with\nnewline",
     "sub with'single quote",
     "sub with\"double quote",
     "sub with\\backslash",
     "sub with\bbackspace",
     "sub with\thorizontal tab",
     # potentially confusing git's configuration format
     "sub with #",
     "sub with ="

(That's Python 3 literals in case somebody is wondering. I'm using 
Python to unit test a shell script, just so I can catch this sort of 
stuff...)

On 04/07, Jeff King wrote:

On Fri, Apr 07, 2017 at 08:12:49AM +0200, Joachim Durchholz wrote:

quoted

So... something inside "git submodule add" is replacing the \b with a
backspace control code.
[...]
Whatever that "something" is, it is not doing shell expansion, otherwise it
would have started an interactive calculator session.

Probably it's "read" which does backslash expansion, but nothing else.
Just grepping git-submodule.sh, some of the "read" calls should probably
be "read -r" (I also don't know how some of those loops would cope with
a submodule name that needed quoting).

So I blindly converted all "read" calls to "read -r" and tested against
the case Joachim ran into and it seems to solve the issues.  All test
still pass too (though that may not mean too much).

-- 
Brandon Williams

quoted

Probably it's "read" which does backslash expansion, but nothing else.
Just grepping git-submodule.sh, some of the "read" calls should probably
be "read -r" (I also don't know how some of those loops would cope with
a submodule name that needed quoting).

So I blindly converted all "read" calls to "read -r" and tested against
the case Joachim ran into and it seems to solve the issues.  All test
still pass too (though that may not mean too much).

... because we may not have tests with weird names in submodule path.
Thanks for the conversion!

Stefan

When attempting to add a submodule with backslashes in its name 'git
submodule' fails in a funny way.  We can see that some of the
backslashes are expanded resulting in a bogus path:

git -C main submodule add ../sub\\with\\backslash
fatal: repository '/tmp/test/sub\witackslash' does not exist
fatal: clone of '/tmp/test/sub\witackslash' into submodule path

To solve this, convert calls to 'read' to 'read -r' in git-submodule.sh
in order to prevent backslash expantion in submodule names.

Reported-by: Joachim Durchholz <redacted>
Signed-off-by: Brandon Williams <redacted>
---
 git-submodule.sh           | 14 +++++++-------
 t/t7400-submodule-basic.sh | 14 ++++++++++++++
 2 files changed, 21 insertions(+), 7 deletions(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 6ec35e5fc..c0d0e9a4c 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh

@@ -332,7 +332,7 @@ cmd_foreach()
 		git submodule--helper list --prefix "$wt_prefix" ||
 		echo "#unmatched" $?
 	} |
-	while read mode sha1 stage sm_path
+	while read -r mode sha1 stage sm_path
 	do
 		die_if_unmatched "$mode" "$sha1"
 		if test -e "$sm_path"/.git

@@ -441,7 +441,7 @@ cmd_deinit()
 		git submodule--helper list --prefix "$wt_prefix" "$@" ||
 		echo "#unmatched" $?
 	} |
-	while read mode sha1 stage sm_path
+	while read -r mode sha1 stage sm_path
 	do
 		die_if_unmatched "$mode" "$sha1"
 		name=$(git submodule--helper name "$sm_path") || exit

@@ -605,7 +605,7 @@ cmd_update()
 		"$@" || echo "#unmatched" $?
 	} | {
 	err=
-	while read mode sha1 stage just_cloned sm_path
+	while read -r mode sha1 stage just_cloned sm_path
 	do
 		die_if_unmatched "$mode" "$sha1"
 

@@ -847,7 +847,7 @@ cmd_summary() {
 	# Get modified modules cared by user
 	modules=$(git $diff_cmd $cached --ignore-submodules=dirty --raw $head -- "$@" |
 		sane_egrep '^:([0-7]* )?160000' |
-		while read mod_src mod_dst sha1_src sha1_dst status sm_path
+		while read -r mod_src mod_dst sha1_src sha1_dst status sm_path
 		do
 			# Always show modules deleted or type-changed (blob<->module)
 			if test "$status" = D || test "$status" = T

@@ -873,7 +873,7 @@ cmd_summary() {
 	git $diff_cmd $cached --ignore-submodules=dirty --raw $head -- $modules |
 	sane_egrep '^:([0-7]* )?160000' |
 	cut -c2- |
-	while read mod_src mod_dst sha1_src sha1_dst status name
+	while read -r mod_src mod_dst sha1_src sha1_dst status name
 	do
 		if test -z "$cached" &&
 			test $sha1_dst = 0000000000000000000000000000000000000000

@@ -1020,7 +1020,7 @@ cmd_status()
 		git submodule--helper list --prefix "$wt_prefix" "$@" ||
 		echo "#unmatched" $?
 	} |
-	while read mode sha1 stage sm_path
+	while read -r mode sha1 stage sm_path
 	do
 		die_if_unmatched "$mode" "$sha1"
 		name=$(git submodule--helper name "$sm_path") || exit

@@ -1100,7 +1100,7 @@ cmd_sync()
 		git submodule--helper list --prefix "$wt_prefix" "$@" ||
 		echo "#unmatched" $?
 	} |
-	while read mode sha1 stage sm_path
+	while read -r mode sha1 stage sm_path
 	do
 		die_if_unmatched "$mode" "$sha1"
 

diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index cf77a3a35..c2706fe47 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh

@@ -273,6 +273,20 @@ test_expect_success 'submodule add with ./, /.. and // in path' '
 	test_cmp empty untracked
 '
 
+test_expect_success 'submodule add with \\ in path' '
+	test_when_finished "rm -rf parent sub\\with\\backslash" &&
+
+	# Initialize a repo with a backslash in its name
+	git init sub\\with\\backslash &&
+	touch sub\\with\\backslash/empty.file &&
+	git -C sub\\with\\backslash add empty.file &&
+	git -C sub\\with\\backslash commit -m "Added empty.file" &&
+
+	# Add that repository as a submodule
+	git init parent &&
+	git -C parent submodule add ../sub\\with\\backslash
+'
+
 test_expect_success 'submodule add in subdirectory' '
 	echo "refs/heads/master" >expect &&
 	>empty &&

-- 
2.12.2.715.g7642488e1d-goog

Thanks!

On Fri, Apr 07, 2017 at 10:23:06AM -0700, Brandon Williams wrote:

When attempting to add a submodule with backslashes in its name 'git
submodule' fails in a funny way.  We can see that some of the
backslashes are expanded resulting in a bogus path:

git -C main submodule add ../sub\\with\\backslash
fatal: repository '/tmp/test/sub\witackslash' does not exist
fatal: clone of '/tmp/test/sub\witackslash' into submodule path

To solve this, convert calls to 'read' to 'read -r' in git-submodule.sh
in order to prevent backslash expantion in submodule names.

This looks sane overall, without digging into the individual read calls.

The reason I mentioned escaping earlier is I wondered what would happen
when the submodule starts with a double-quote, or has a newline in the
name. Git's normal quoting would include backslash escape sequences, and
I wondered if we might be relying on any of these "read" calls to
interpret them. But I don't think so, for two reasons.

One, because that quoting also puts double-quotes around the name. So
plain "read" would not be sufficient to de-quote for us anyway.

And two, because these are being fed from "submodule--helper", which
does not seem to quote in the first place.

So I think your patch is fine there. But it does raise a few concerns.
It looks like git-submodule does not cope well with exotic filenames:

  $ git submodule add /some/repo "$(printf 'sub with\nnewline')"
  Cloning into '/home/peff/tmp/sub with
  newline'...
  done.
  error: invalid key (newline): submodule.sub with
  newline.url
  error: invalid key (newline): submodule.sub with
  newline.path
  Failed to register submodule 'sub with
  newline'

I'm not too worried about that. It's a nonsense request, and our config
format has no syntactic mechanism to represent that key. So tough luck.
But what I am more worried about is:

  $ git submodule--helper list
  160000 576053ed5ad378490974fabe97e4bd59633d2d1e 0	sub with
  newline

That's obviously nonsense that git-submodule.sh is going to choke on.
But what happens when the filename is:

  foo\n16000 <sha1> 0\t../../escaped

or something. Can a malicious repository provoke git-submodule.sh to
look at or modify files outside the repository?

-Peff

Am 08.04.2017 um 12:59 schrieb Jeff King:

The reason I mentioned escaping earlier is I wondered what would happen
when the submodule starts with a double-quote, or has a newline in the
name.

I have tested newlines within the name, these work fine.

I also tested double and single quotes within the name, but not at 
beginning or end.

So I think your patch is fine there. But it does raise a few concerns.
It looks like git-submodule does not cope well with exotic filenames:

  $ git submodule add /some/repo "$(printf 'sub with\nnewline')"
  Cloning into '/home/peff/tmp/sub with
  newline'...
  done.
  error: invalid key (newline): submodule.sub with
  newline.url
  error: invalid key (newline): submodule.sub with
  newline.path
  Failed to register submodule 'sub with
  newline'

Strange. I'm running essentially the same kind of request, and things 
work fine.
Might be due to me using Python3 instead of bash, or maybe due to 
different versions of git.

If anybody is interested, I can publish my test code on github, it was 
scheduled to land there anyway.

I'm not too worried about that.  It's a nonsense request, and our config
format has no syntactic mechanism to represent that key.

Oh. I've been thinking that the quoted format is exactly for that kind 
of stuff.
Though it might be prone to eol conversion if a submodule name contains 
crlf sequences.

Also, funny behavour. Experience has taught me that funny behaviour, if 
it isn't exploitable today, may combine with some new funny behaviour in 
a future version of the same software. So I'm worried even with that.

This is starting to look like a can of worms to me... one way to "close 
the lid" would be if git
* defined what's a valid submodule name,
* rejected invalid submodule names, and
* documented validity rules in the git-submodule docs.
YMMV, just my 2 cents :-)

Regards,
Jo