Re: git smart-http do not authent to allow git ls-remote to be called anonymously

[+cc git@vger; please keep discussion on the list]

On Sun, Oct 28, 2012 at 01:26:51PM +0800, 乙酸鋰 wrote:

quoted

POST /git/Cat1/SubCat1/xsp.git/git-upload-pack HTTP/1.1

User-Agent: git/1.8.0
Host: localhost
Accept-Encoding: gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Length: 190

* The requested URL returned error: 401
* Closing connection #0
Username for 'http://localhost': user
Password for 'http://user@localhost':
fatal: The remote end hung up unexpectedly

OK, I see what is going on. The code in b81401c to retry POST requests
does not handle gzipped contents, and upload-pack tends to gzip what it
sends.

Your apache configuration is not really something that we ever intended
to support, and I am a little dubious of the security tradeoff being
made. But it is actually pretty easy for us to support, and it
eliminates a special case from the code, so I am tempted to do so.

The following patch series (on top of the current master, as they
require some cleanup that did not make it into 1.8.0) seems to fix it
for me.

  [1/2]: remote-curl: hoist gzip buffer size to top of post_rpc
  [2/2]: remote-curl: retry failed requests for auth even with gzip

-Peff

When we gzip the post data for a smart-http rpc request, we
compute the gzip body and its size inside the "use_gzip"
conditional. We keep track of the body after the conditional
ends, but not the size. Let's remember both, which will
enable us to retry failed gzip requests in a future patch.

Signed-off-by: Jeff King <redacted>
---
This is a tiny change conceptually, but the name change makes the diff
quite noisy. Thus I pulled it into a separate patch from 2/2.

 remote-curl.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/remote-curl.c b/remote-curl.c
index aefafd3..10cd47d 100644
--- a/remote-curl.c
+++ b/remote-curl.c

@@ -413,6 +413,7 @@ static int post_rpc(struct rpc_state *rpc)
 	struct curl_slist *headers = NULL;
 	int use_gzip = rpc->gzip_request;
 	char *gzip_body = NULL;
+	size_t gzip_size;
 	int err, large_request = 0;
 
 	/* Try to load the entire request, if we can fit it into the

@@ -478,19 +479,18 @@ retry:
 		 * we can try to deflate it ourselves, this may save on.
 		 * the transfer time.
 		 */
-		size_t size;
 		git_zstream stream;
 		int ret;
 
 		memset(&stream, 0, sizeof(stream));
 		git_deflate_init_gzip(&stream, Z_BEST_COMPRESSION);
-		size = git_deflate_bound(&stream, rpc->len);
-		gzip_body = xmalloc(size);
+		gzip_size = git_deflate_bound(&stream, rpc->len);
+		gzip_body = xmalloc(gzip_size);
 
 		stream.next_in = (unsigned char *)rpc->buf;
 		stream.avail_in = rpc->len;
 		stream.next_out = (unsigned char *)gzip_body;
-		stream.avail_out = size;
+		stream.avail_out = gzip_size;
 
 		ret = git_deflate(&stream, Z_FINISH);
 		if (ret != Z_STREAM_END)

@@ -500,16 +500,16 @@ retry:
 		if (ret != Z_OK)
 			die("cannot deflate request; zlib end error %d", ret);
 
-		size = stream.total_out;
+		gzip_size = stream.total_out;
 
 		headers = curl_slist_append(headers, "Content-Encoding: gzip");
 		curl_easy_setopt(slot->curl, CURLOPT_POSTFIELDS, gzip_body);
-		curl_easy_setopt(slot->curl, CURLOPT_POSTFIELDSIZE, size);
+		curl_easy_setopt(slot->curl, CURLOPT_POSTFIELDSIZE, gzip_size);
 
 		if (options.verbosity > 1) {
 			fprintf(stderr, "POST %s (gzip %lu to %lu bytes)\n",
 				rpc->service_name,
-				(unsigned long)rpc->len, (unsigned long)size);
+				(unsigned long)rpc->len, (unsigned long)gzip_size);
 			fflush(stderr);
 		}
 	} else {

-- 
1.8.0.207.gdf2154c

Commit b81401c taught the post_rpc function to retry the
http request after prompting for credentials. However, it
did not handle two cases:

  1. If we have a large request, we do not retry. That's OK,
     since we would have sent a probe (with retry) already.

  2. If we are gzipping the request, we do not retry. That
     was considered OK, because the intended use was for
     push (e.g., listing refs is OK, but actually pushing
     objects is not), and we never gzip on push.

This patch teaches post_rpc to retry even a gzipped request.
This has two advantages:

  1. It is possible to configure a "half-auth" state for
     fetching, where the set of refs and their sha1s are
     advertised, but one cannot actually fetch objects.

     This is not a recommended configuration, as it leaks
     some information about what is in the repository (e.g.,
     an attacker can try brute-forcing possible content in
     your repository and checking whether it matches your
     branch sha1). However, it can be slightly more
     convenient, since a no-op fetch will not require a
     password at all.

  2. It future-proofs us should we decide to ever gzip more
     requests.

Signed-off-by: Jeff King <redacted>
---
I doubt we would ever want to gzip push requests. The bulk of the data
is objects, which are already compressed. In theory we could gzip the
non-pack parts, but that would first mean splitting them into a separate
request. The tiny bit of savings are almost certainly not worth the
complexity.

But the future-proofing might still be valuable if we ever modify the
protocol to have a new phase.

 remote-curl.c           | 11 ++++++++++-
 t/lib-httpd/apache.conf |  7 +++++++
 t/t5551-http-fetch.sh   | 15 +++++++++++++++
 3 files changed, 32 insertions(+), 1 deletion(-)

diff --git a/remote-curl.c b/remote-curl.c
index 10cd47d..fac2bef 100644
--- a/remote-curl.c
+++ b/remote-curl.c

@@ -474,6 +474,15 @@ retry:
 			fflush(stderr);
 		}
 
+	} else if (gzip_body) {
+		/*
+		 * If we are looping to retry authentication, then the previous
+		 * run will have set up the headers and gzip buffer already,
+		 * and we just need to send it.
+		 */
+		curl_easy_setopt(slot->curl, CURLOPT_POSTFIELDS, gzip_body);
+		curl_easy_setopt(slot->curl, CURLOPT_POSTFIELDSIZE, gzip_size);
+
 	} else if (use_gzip && 1024 < rpc->len) {
 		/* The client backend isn't giving us compressed data so
 		 * we can try to deflate it ourselves, this may save on.

@@ -530,7 +539,7 @@ retry:
 	curl_easy_setopt(slot->curl, CURLOPT_FILE, rpc);
 
 	err = run_slot(slot);
-	if (err == HTTP_REAUTH && !large_request && !use_gzip)
+	if (err == HTTP_REAUTH && !large_request)
 		goto retry;
 	if (err != HTTP_OK)
 		err = -1;

diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf
index ec8618d..15a3c71 100644
--- a/t/lib-httpd/apache.conf
+++ b/t/lib-httpd/apache.conf

@@ -96,6 +96,13 @@ SSLEngine On
 	Require valid-user
 </LocationMatch>
 
+<LocationMatch "^/auth-fetch/.*/git-upload-pack$">
+	AuthType Basic
+	AuthName "git-auth"
+	AuthUserFile passwd
+	Require valid-user
+</LocationMatch>
+
 <IfDefine DAV>
 	LoadModule dav_module modules/mod_dav.so
 	LoadModule dav_fs_module modules/mod_dav_fs.so

diff --git a/t/t5551-http-fetch.sh b/t/t5551-http-fetch.sh
index 7380f2a..5f174da 100755
--- a/t/t5551-http-fetch.sh
+++ b/t/t5551-http-fetch.sh

@@ -129,6 +129,21 @@ test_expect_success 'clone from auth-only-for-push repository' '
 	test_cmp expect actual
 '
 
+test_expect_success 'clone from auth-only-for-objects repository' '
+	echo two >expect &&
+	set_askpass user@host &&
+	git clone --bare "$HTTPD_URL/auth-fetch/smart/repo.git" half-auth &&
+	expect_askpass both user@host &&
+	git --git-dir=half-auth log -1 --format=%s >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'no-op half-auth fetch does not require a password' '
+	set_askpass wrong &&
+	git --git-dir=half-auth fetch &&
+	expect_askpass none
+'
+
 test -n "$GIT_TEST_LONG" && test_set_prereq EXPENSIVE
 
 test_expect_success EXPENSIVE 'create 50,000 tags in the repo' '

-- 
1.8.0.207.gdf2154c