Re: Cloning from sites with 404 overridden

If someone feels ambitious, you can detect this condition automatically
by searching for a file that you know won't be there and seeing if you
get a 404 response to that.

To avoid punishing good servers, it would be nice to defer the test
until reciving the first corrupted object.

I'm not sure what the best "object that's not supposed to be there" is.
It could just be a random hash, or would a malformed object file name
be better?  Any fixed name has a finite chance of being created by
someone somewhere, but generating 160-bit random numbers is a PITA on
non-freenix platforms.


(As an aside, I suspect this is all caused by Microsoft's "friendly HTML
error messages" invention.)

'0' x 40.  :-) There's some places already in the GIT source
which would have ``issues'' if they got an object with this hash.
Not sure if it is actually an entirely impossible hash or just one
that is highly improbable.

My own website has this problem and its because I'm using WordPress
to handle all URLs on the site; I haven't yet found a way to
configure WordPress to return a proper 404 when the URL can't be
mapped to something on the server.  Note that 404 status codes can
in fact return pretty HTML content for the user, and many websites
do this and many browsers display that pretty HTML.  But a bot can
then also recognize the status code and DTRT.

The webservers are just plain broken, mine included.  I think the
best option is to delay corrupt object reporting to the end of
the download process if you get only one corrupt object and that
corrupt object was actually attainable from a pack.  And in this
case its just a minor warning:

	Warning: The server appears to not return proper HTTP status
	codes on missing files.  The files were found in one or
	more packs so the download is OK, but the server administrator
	should really fix their server.  If you know the server
	administrator you might want to prod them to do so.

But that's already been suggested and I thought someone worked up
a patch based on that idea?  If not I could try to do so since my
own damn server has the problem.  :-)

linux@horizon.com wrote:

If someone feels ambitious, you can detect this condition automatically
by searching for a file that you know won't be there and seeing if you
get a 404 response to that.

To avoid punishing good servers, it would be nice to defer the test
until reciving the first corrupted object.

I'm not sure what the best "object that's not supposed to be there" is.
It could just be a random hash, or would a malformed object file name
be better?  Any fixed name has a finite chance of being created by
someone somewhere, but generating 160-bit random numbers is a PITA on
non-freenix platforms.


(As an aside, I suspect this is all caused by Microsoft's "friendly HTML
error messages" invention.)

-- 
Shawn.

On Tue, 21 Mar 2006, Shawn Pearce wrote:

'0' x 40.  :-) There's some places already in the GIT source
which would have ``issues'' if they got an object with this hash.
Not sure if it is actually an entirely impossible hash or just one
that is highly improbable.

The all-zeroes hash is as improbable as any other one, and finding a 
"collision" (ie a "real object") with that hash is as improbable as any 
other collision, ie we can (and do) depend on it beign a unique identifier 
for "does not exist".

			Linus

On 21 Mar 2006 21:59:21 -0500, linux@horizon.com [off-list ref] wrote:

If someone feels ambitious, you can detect this condition automatically
by searching for a file that you know won't be there and seeing if you
get a 404 response to that.

Perhaps I am proposing a total idiocy, I don't know git-fetch
internals, but wouldn't be better to avoid trying to download a non
existing object? So to fix the problem at the origin?

I don't know if it is possible to list contents before try to download
so to avoid asking for a non existing object.

Marco

linux@horizon.com wrote:

If someone feels ambitious, you can detect this condition automatically
by searching for a file that you know won't be there and seeing if you
get a 404 response to that.

To avoid punishing good servers, it would be nice to defer the test
until reciving the first corrupted object.

I'm not sure what the best "object that's not supposed to be there" is.

.git/objects/00/hoping-for-a-404-or-webadmin-should-fix

It has the right number of chars so it should fit in wherever a real 
object name does but is obviously bogus anyways.

It could just be a random hash, or would a malformed object file name
be better?

A malformed object name is infinitely better. Otherwise we'd end up with 
a wild guess that hits home some day, to much surprise and a bug-report 
I wouldn't want to track. Not to mention the embarrassment when 
explaining why that object-name was chosen.

(As an aside, I suspect this is all caused by Microsoft's "friendly HTML
error messages" invention.)

The body of the 404-page has absolutely nothing to do with it.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

On Tue, Mar 21, 2006 at 09:59:21PM -0500, linux@horizon.com wrote:

If someone feels ambitious, you can detect this condition automatically
by searching for a file that you know won't be there and seeing if you
get a 404 response to that.

It might be feasible to detect this condition using the Content-Type:
header in the server response.  So far, all the GIT repositories I've
tried return text/plain for loose objects and a special 404 page will
likely be text/html.

-- 
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.

On Wed, Mar 22, 2006 at 09:22:27AM -0800, Nick Hengeveld wrote:

It might be feasible to detect this condition using the Content-Type:
header in the server response.  So far, all the GIT repositories I've
tried return text/plain for loose objects and a special 404 page will
likely be text/html.

Something like this:

http_fetch: report text/html responses for loose objects

Some HTTP server environments return a 200 status and text/html error
document or a redirect to one rather than a 404 status if a loose
object does not exist.  This patch detects and reports this condition
to differentiate between a misconfigured server and an actual corrupt
object on the server.

Signed-off-by: Nick Hengeveld <redacted>


---

 http-fetch.c |   19 ++++++++++++++++++-
 1 files changed, 18 insertions(+), 1 deletions(-)

61069cc348640fef2b8c503b8b8f00f689872cab

diff --git a/http-fetch.c b/http-fetch.c
index dc67218..ee5b585 100644
--- a/http-fetch.c
+++ b/http-fetch.c

@@ -41,6 +41,7 @@ struct object_request
 	CURLcode curl_result;
 	char errorstr[CURL_ERROR_SIZE];
 	long http_code;
+	char *content_type;
 	unsigned char real_sha1[20];
 	SHA_CTX c;
 	z_stream stream;

@@ -258,9 +259,15 @@ static void finish_object_request(struct
 
 static void process_object_response(void *callback_data)
 {
+	char *content_type;
 	struct object_request *obj_req =
 		(struct object_request *)callback_data;
 
+	curl_easy_getinfo(obj_req->slot->curl, CURLINFO_CONTENT_TYPE,
+			  &content_type);
+	if (content_type)
+		obj_req->content_type = strdup(content_type);
+
 	obj_req->curl_result = obj_req->slot->curl_result;
 	obj_req->http_code = obj_req->slot->http_code;
 	obj_req->slot = NULL;

@@ -298,6 +305,8 @@ static void release_object_request(struc
 			entry->next = entry->next->next;
 	}
 
+	if (obj_req->content_type)
+		free(obj_req->content_type);
 	free(obj_req->url);
 	free(obj_req);
 }

@@ -340,6 +349,7 @@ void prefetch(unsigned char *sha1)
 	memcpy(newreq->sha1, sha1, 20);
 	newreq->repo = alt;
 	newreq->url = NULL;
+	newreq->content_type = NULL;
 	newreq->local = -1;
 	newreq->state = WAITING;
 	snprintf(newreq->filename, sizeof(newreq->filename), "%s", filename);

@@ -836,7 +846,14 @@ static int fetch_object(struct alt_base 
 				    obj_req->http_code, hex);
 	} else if (obj_req->zret != Z_STREAM_END) {
 		corrupt_object_found++;
-		ret = error("File %s (%s) corrupt", hex, obj_req->url);
+		if (obj_req->content_type &&
+		    !strcmp(obj_req->content_type, "text/html")) {
+			ret = error("text/html response for file %s (%s)",
+				    sha1_to_hex(obj_req->sha1), obj_req->url);
+		} else {
+			ret = error("File %s (%s) corrupt",
+				    sha1_to_hex(obj_req->sha1), obj_req->url);
+		}
 	} else if (memcmp(obj_req->sha1, obj_req->real_sha1, 20)) {
 		ret = error("File %s has bad hash", hex);
 	} else if (obj_req->rename < 0) {

-- 
1.2.4.gb1bc1d-dirty

Andreas Ericsson [off-list ref] wrote:

quoted

I'm not sure what the best "object that's not supposed to be there" is.

.git/objects/00/hoping-for-a-404-or-webadmin-should-fix

If .git/objects/00/00000000000000000000000000000000000000 exists, the
repository has big problems already.

(Aside: `C-u 38 0' doesn't work because Emacs hears `C-u 380' and waits
for a key.  `M-: (insert-char ?0 38) RET' does the right thing, but is
ugly.  Any better suggestions?)

-- [mdw]

On Fri, 24 Mar 2006, Mark Wooding wrote:

(Aside: `C-u 38 0' doesn't work because Emacs hears `C-u 380' and waits
for a key.  `M-: (insert-char ?0 38) RET' does the right thing, but is
ugly.  Any better suggestions?)

I don't do GNU emacs, but the way to do it in some other editors that do
repeats somewhat similarly is to do the action that starts with a number
as a macro, and do that macro 37 more times. 

On uemacs: ^X '(' '0' ^X ')' ESC '3' '7' ^X 'E'

(Of course, the easier way is to just do '0' LEFT ^K to put the 0 in the
buffer, and than ESC '3' '8' ^Y to yank it 38 times, but the macro trick
is generic, even if it's a few more keystrokes). 

		Linus "teaching people the one true editor" Torvalds

(Aside: `C-u 38 0' doesn't work because Emacs hears `C-u 380' and waits
for a key.  `M-: (insert-char ?0 38) RET' does the right thing, but is
ugly.  Any better suggestions?)

There's a million ways to skin that cat.

ESC 38 C-q 60 RET

[Octal 060 == '0']

M.

Mark Wooding wrote:

Andreas Ericsson [off-list ref] wrote:

quoted

quoted

I'm not sure what the best "object that's not supposed to be there" is.

.git/objects/00/hoping-for-a-404-or-webadmin-should-fix

If .git/objects/00/00000000000000000000000000000000000000 exists, the
repository has big problems already.

Indeed. I'm off sobriety again, it being friday and all, but I'm 
assuming there are more than 18 zeroes there, yes? The "feature" of the 
above line is that it will fit in any buffer that already exists, and 
will match any third argument to send(2) that already exists.

(Aside: `C-u 38 0' doesn't work because Emacs hears `C-u 380' and waits
for a key.  `M-: (insert-char ?0 38) RET' does the right thing, but is
ugly.  Any better suggestions?)

This I happily don't understand at all. I'm also happy ignorant of what 
it has to do with the issue at hand.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231