Re: Cloning from sites with 404 overridden

"Marco Costalba" [off-list ref] writes:

http://digilander.libero.it /mcostalba/scm/qgit.git/objects/8d/ea03519e75f47d

Git does not understand object is missing and thinks what site sends
_is_ the requested
object and then founds that is (of course) corrupted.

To be fair, the site is _not_ missing anything from HTTP
protocol perspective, because when git asks 8d/ea0351... file,
the server responds with a regular "HTTP/1.0 200 OK" response.
So it is _your_ repository that is corrupt -- instead of
correctly _lacking_ the file you should have removed with
prune-packed, it has a garbage file.

Having said that, I agree that it would be nicer if we support
such a site, in the same spirit that we already bend backwards
to support really dumb hosted http servers that do not give
directory index by using objects/info/packs and info/refs.

I think it wouldn't be too much a hassle to add logic to
http-fetch.c (perhaps with an additional "--no-404" option or
somesuch) to fall back on pack transfer upon seeing a corrupt
loose object.  We do the falling back when getting 404 error to
a request for a loose object, so the new code would essentially
do the same and you might be OK.

On 3/19/06, Junio C Hamano [off-list ref] wrote:

"Marco Costalba" [off-list ref] writes:

quoted

http://digilander.libero.it /mcostalba/scm/qgit.git/objects/8d/ea03519e75f47d

Git does not understand object is missing and thinks what site sends
_is_ the requested
object and then founds that is (of course) corrupted.

To be fair, the site is _not_ missing anything from HTTP
protocol perspective, because when git asks 8d/ea0351... file,
the server responds with a regular "HTTP/1.0 200 OK" response.
So it is _your_ repository that is corrupt -- instead of
correctly _lacking_ the file you should have removed with
prune-packed, it has a garbage file.

Currently my git repo layout is as follow
$ pwd
<local master copy>/qgit.git/.git
$ ls
branches/  description  HEAD    index  objects/   refs/
config     FETCH_HEAD   hooks/  info/  ORIG_HEAD  remotes/
$ ls objects
2c/  32/  53/  5c/  6a/  info/  pack/

The host copy should be the exact mirror of the local copy (I use
sitecopy to sync
host). I have also verified this directly accessing the host with ftp.

So the 8d/ea0351... file is really not existent. BTW I have run git
prune and git-prune-packed
also.

Finally accessing the missing object with a browser

http://digilander.libero.it/mcostalba/
scm/qgit.git/objects/8d/ea03519e75f47da91108330dde3043defddd60

gives a pre-canned (in italian) 'Sorry page not found' stuff.

So I really think the site "HTTP/1.0 200 OK" response it's a fake.
Perhaps security related to avoid sniffing (just a guess because I have
absolutely zero competence in security related things).


Marco

"Marco Costalba" [off-list ref] writes:

Finally accessing the missing object with a browser

http://digilander.libero.it/mcostalba/
scm/qgit.git/objects/8d/ea03519e75f47da91108330dde3043defddd60

gives a pre-canned (in italian) 'Sorry page not found' stuff.

So I really think the site "HTTP/1.0 200 OK" response it's a fake.
Perhaps security related to avoid sniffing (just a guess because I have
absolutely zero competence in security related things).

I think you are just rephrasing what I said.  From the HTTP
protocol perspective, you _do_ have that 8d/3a0351 thing on that
server, because you do not correctly say "No we donot have it"
using 404 response.

Your inability to produce 404 is a different matter -- often the
hosting server is not under your control.  But that does not
change the fact that the repository observed by your clients is
"broken".  That is why a workaround flag like I suggested may be
needed for such a setup.

This is totally untested, but maybe something like this?

---

diff --git a/http-fetch.c b/http-fetch.c
index 7de818b..d523798 100644
--- a/http-fetch.c
+++ b/http-fetch.c

@@ -8,6 +8,7 @@
 #define RANGE_HEADER_SIZE 30
 
 static int got_alternates = -1;
+static int unreliable_404 = 0;
 
 static struct curl_slist *no_pragma_header;
 

@@ -822,12 +823,18 @@ static int fetch_object(struct alt_base 
 		close(obj_req->local); obj_req->local = -1;
 	}
 
+	
+
 	if (obj_req->state == ABORTED) {
 		ret = error("Request for %s aborted", hex);
-	} else if (obj_req->curl_result != CURLE_OK &&
-		   obj_req->http_code != 416) {
+	} else if ((obj_req->curl_result != CURLE_OK &&
+		    obj_req->http_code != 416)  ||
+		   (unreliable_404 &&
+		    obj_req->curl_result == CURLE_OK &&
+		    obj_req->zret != Z_STREAM_END)) {
 		if (obj_req->http_code == 404 ||
-		    obj_req->curl_result == CURLE_FILE_COULDNT_READ_FILE)
+		    obj_req->curl_result == CURLE_FILE_COULDNT_READ_FILE ||
+		    unreliable_404)
 			ret = -1; /* Be silent, it is probably in a pack. */
 		else
 			ret = error("%s (curl_result = %d, http_code = %ld, sha1 = %s)",

@@ -966,6 +973,8 @@ int main(int argc, char **argv)
 			arg++;
 		} else if (!strcmp(argv[arg], "--recover")) {
 			get_recover = 1;
+		} else if (!strcmp(argv[arg], "--unreliable-404")) {
+			unreliable_404 = 1;
 		}
 		arg++;
 	}

On 3/20/06, Junio C Hamano [off-list ref] wrote:

Your inability to produce 404 is a different matter -- often the
hosting server is not under your control.  But that does not
change the fact that the repository observed by your clients is
"broken".  That is why a workaround flag like I suggested may be
needed for such a setup.

This is totally untested, but maybe something like this?

It works for me. Just some trailing white space warning when applying.

I didn't found a way to pass '--unreliable-404' flag from git-clone,
perhaps my bad,
I have tested forcing the flag in sources.


Marco

Junio C Hamano wrote:

"Marco Costalba" [off-list ref] writes:

quoted

http://digilander.libero.it /mcostalba/scm/qgit.git/objects/8d/ea03519e75f47d

To be fair, the site is _not_ missing anything from HTTP
protocol perspective, because when git asks 8d/ea0351... file,
the server responds with a regular "HTTP/1.0 200 OK" response.
So it is _your_ repository that is corrupt -- instead of
correctly _lacking_ the file you should have removed with
prune-packed, it has a garbage file.

Actually, it sends a 302 redirect. 

Perhaps a repository config option to treat a 302 as a 404?

/Lukas Sandström

Dear diary, on Mon, Mar 20, 2006 at 07:29:02PM CET, I got a letter
where Lukas Sandström [off-list ref] said that...

Junio C Hamano wrote:

quoted

"Marco Costalba" [off-list ref] writes:

quoted

http://digilander.libero.it /mcostalba/scm/qgit.git/objects/8d/ea03519e75f47d

To be fair, the site is _not_ missing anything from HTTP
protocol perspective, because when git asks 8d/ea0351... file,
the server responds with a regular "HTTP/1.0 200 OK" response.
So it is _your_ repository that is corrupt -- instead of
correctly _lacking_ the file you should have removed with
prune-packed, it has a garbage file.

Actually, it sends a 302 redirect. 

Perhaps a repository config option to treat a 302 as a 404?

I think that would be too ugly _and_ specific a workaround for the
particular site. It's reasonable to keep it generalized for all the
broken repositories when already doing it.

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Right now I am having amnesia and deja-vu at the same time.  I think
I have forgotten this before.

On Mon, Mar 20, 2006 at 07:29:02PM +0100, Lukas Sandström wrote:

Perhaps a repository config option to treat a 302 as a 404?

FWIW, it used to work that way and was modified to follow redirects back at
commit 66c9ec25553ce7332c46e2017b9c4d7c26310fff.

-- 
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.