Re: git-fetch per-repository speed issues

Linus Torvalds [off-list ref] writes:

Ok, a "git fetch" really shouldn't take any longer than a single 
connection. However, the fact that you have 32 heads, and it takes pretty 
close to _exactly_ 32 times 0.410 seconds (32*0.410s = 13.1s) makes me 
suspect that "git fetch" is just broken and fetches one branch at a time. 

Which would be just stupid.

But look as I might, I see only that one "git-fetch-pack" in git-fetch.sh 
that should trigger. Once. Not 32 times. But your timings sure sound like 
it's doing a _lot_ more than it should.

Junio, any ideas?

Isn't that because the repository have 32 subprojects, totally
unrelated content-wise?  If you have real stuff to pull from
there your pack generation needs to do 32 time as much work as
you would for a single head in that case.

If you are discussing "peek-remote runs, find out the 32 heads
are all up to date and no pack is generated" case, then you are
right.  There is one single fetch-pack to grab the specified
heads, and after that, an optional single ls-remote and
fetch-pack runs only once to follow all new tags.

On Mon, 3 Jul 2006, Junio C Hamano wrote:

Isn't that because the repository have 32 subprojects, totally
unrelated content-wise?  If you have real stuff to pull from
there your pack generation needs to do 32 time as much work as
you would for a single head in that case.

No, Keith said this was for the case where the fetching repository is 
already totally up-to-date:

    "And, it's painfully slow, even when the repository is up to date"

and gave a 17-second time.

			Linus

On Mon, 2006-07-03 at 20:40 -0700, Linus Torvalds wrote:

    "And, it's painfully slow, even when the repository is up to date"

and gave a 17-second time.

It's faster this evening, down to 8 seconds using ssh and 4 seconds
using git. I clearly need to force use of the git protocol. Anyone else
like the attached patch?

---
 connect.c |   18 ++++++++++++++----
 1 files changed, 14 insertions(+), 4 deletions(-)

diff --git a/connect.c b/connect.c
index 9a87bd9..e74eddc 100644
--- a/connect.c
+++ b/connect.c

@@ -303,6 +303,7 @@ enum protocol {
 	PROTO_LOCAL = 1,
 	PROTO_SSH,
 	PROTO_GIT,
+	PROTO_GIT_SSH,
 };
 
 static enum protocol get_protocol(const char *name)

@@ -312,9 +313,9 @@ static enum protocol get_protocol(const 
 	if (!strcmp(name, "git"))
 		return PROTO_GIT;
 	if (!strcmp(name, "git+ssh"))
-		return PROTO_SSH;
+		return PROTO_GIT_SSH;
 	if (!strcmp(name, "ssh+git"))
-		return PROTO_SSH;
+		return PROTO_GIT_SSH;
 	die("I don't handle protocol '%s'", name);
 }
 

@@ -572,6 +573,14 @@ static void git_proxy_connect(int fd[2],
 	close(pipefd[1][0]);
 }
 
+/* returns whether the specified command can be interpreted by the

daemon */
+int git_is_daemon_command (const char *prog) 
+{
+	if (!strcmp("git-upload-pack", prog))
+		return 1;
+	return 0;
+}
+
 /*
  * Yeah, yeah, fixme. Need to pass in the heads etc.
  */

@@ -641,7 +650,8 @@ int git_connect(int fd[2], char *url, co
 		*ptr = '\0';
 	}
 
-	if (protocol == PROTO_GIT) {
+	if (protocol == PROTO_GIT || 
+	    (protocol == PROTO_GIT_SSH && git_is_daemon_command (prog))) {
 		/* These underlying connection commands die() if they
 		 * cannot connect.
 		 */

@@ -678,7 +688,7 @@ int git_connect(int fd[2], char *url, co
 		close(pipefd[0][1]);
 		close(pipefd[1][0]);
 		close(pipefd[1][1]);
-		if (protocol == PROTO_SSH) {
+		if (protocol == PROTO_SSH || protocol == PROTO_GIT_SSH) {
 			const char *ssh, *ssh_basename;
 			ssh = getenv("GIT_SSH");
 			if (!ssh) ssh = "ssh";

-- 
1.4.1.g8fced-dirty

-- 
keith.packard@intel.com

Keith Packard wrote:

On Mon, 2006-07-03 at 20:40 -0700, Linus Torvalds wrote:

quoted

   "And, it's painfully slow, even when the repository is up to date"

and gave a 17-second time.

It's faster this evening, down to 8 seconds using ssh and 4 seconds
using git. I clearly need to force use of the git protocol. Anyone else
like the attached patch?

Since it changes the current meaning of ssh+git, I'm not exactly 
thrilled. However, "git/ssh" or "ssh/git" would work fine for me. The 
slash-separator could be used to say "fetch over this, push over that", 
so we can end up with any valid protocol to use for fetches and another 
one to push over.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

* Andreas Ericsson (ae@op5.se) wrote:

Keith Packard wrote:

quoted

On Mon, 2006-07-03 at 20:40 -0700, Linus Torvalds wrote:

quoted

  "And, it's painfully slow, even when the repository is up to date"

and gave a 17-second time.

It's faster this evening, down to 8 seconds using ssh and 4 seconds
using git. I clearly need to force use of the git protocol. Anyone else
like the attached patch?

Since it changes the current meaning of ssh+git, I'm not exactly 
thrilled. However, "git/ssh" or "ssh/git" would work fine for me. The 
slash-separator could be used to say "fetch over this, push over that", 
so we can end up with any valid protocol to use for fetches and another 
one to push over.

If we would do such a thing, we would be probably better off
allowing different URLs for pushing and pulling, because the git and
ssh URLs will only be the same, if the git repositories are located
in the root folder and I suspect that's almost never the case.

	Matthias

Matthias Kestenholz wrote:

* Andreas Ericsson (ae@op5.se) wrote:

quoted

Keith Packard wrote:

quoted

On Mon, 2006-07-03 at 20:40 -0700, Linus Torvalds wrote:

quoted

 "And, it's painfully slow, even when the repository is up to date"

and gave a 17-second time.

It's faster this evening, down to 8 seconds using ssh and 4 seconds
using git. I clearly need to force use of the git protocol. Anyone else
like the attached patch?

Since it changes the current meaning of ssh+git, I'm not exactly 
thrilled. However, "git/ssh" or "ssh/git" would work fine for me. The 
slash-separator could be used to say "fetch over this, push over that", 
so we can end up with any valid protocol to use for fetches and another 
one to push over.

If we would do such a thing, we would be probably better off
allowing different URLs for pushing and pulling, because the git and
ssh URLs will only be the same, if the git repositories are located
in the root folder and I suspect that's almost never the case.

True. We use relative paths where I work, so for us either way would 
work. Your way is better though.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231