Re: dumb transports not being welcomed..

Junio C Hamano [off-list ref] writes:

Sam Ravnborg [off-list ref] writes:

quoted

Whats wrong using cogito?
In other words. Why does you feel like that when we use cogito to do
cg-update.

Using cogito is not a problem at all.  The mechanism to prepare
trees to serve wider audience not being used widely is.

I need to clarify what I meant by 'not welcoming dumb transport'
a bit better.  Namely, those (~80 - 23) = ~57 repositories lack
support for 'git ls-remote' over http, which means you cannot
discover what refs the repository has.

Some people argued that it can be done via recursive wget on
refs/ hierarchy.  Here is what you would get if you do that
against kernel.org:

  $ wget -r -np -nH --cut-dirs=4 http://kernel.org/pub/scm/git/git.git/refs/.
  $ ls -R refs
  refs:
  ./      index.html          index.html?C=N;O=A  index.html?C=S;O=D
  ../     index.html?C=M;O=A  index.html?C=N;O=D  tags/
  heads/  index.html?C=M;O=D  index.html?C=S;O=A

  refs/heads:
  ./          index.html?C=M;O=A  index.html?C=N;O=D  master  todo
  ../         index.html?C=M;O=D  index.html?C=S;O=A  pu
  index.html  index.html?C=N;O=A  index.html?C=S;O=D  rc

  refs/tags:
  ./                  index.html?C=M;O=D  index.html?C=S;O=D  v0.99.2  v0.99.6
  ../                 index.html?C=N;O=A  junio-gpg-pub       v0.99.3
  index.html          index.html?C=N;O=D  v0.99               v0.99.4
  index.html?C=M;O=A  index.html?C=S;O=A  v0.99.1             v0.99.5

Of course, I do not have a branch called index.html there, and
this also means I will not be able to have a branch with that
name even if I wanted to.

Also some webservers are configured not to even allow directory
index, and they may use different formatting for directory index
even when they do support it, so excluding anything that matches
index.html* would work well but that is only heuristics.

The file $GIT_DIR/info/refs was introduced to solve this by
listing the available refs for discovery, and hooks/post-update,
when enabled, runs update-server-info to update the file (among
other things) whenever you push into the repository.  info/refs
is not strictly necessary for repositories at kernel.org because
people tend to know what refs are available for pulling and you
can always visit there via gitweb to find it out.

I just felt that it is a good habit to get into to prepare your
repositories in a shape usable even when served by an HTTP
server that is less forgiving than what kernel.org runs -- that
was what I felt "discouraging" about.

Another thing is that the missing info/refs file means the
repository is not prepared with update-server-info, so it is
likely that it lacks objects/info/packs to describe what packs
are in the object database.  I believe cogito uses git-http-pull
after you tell which ref to pull, and this step would break if
the repository is packed, objects/info/packs is not available,
and if the downloader does not have an object that is already
prune-packed in the repository.  This means either people are
not packing their repository (hence nobody complained), or
public are pulling over rsync transport (which slurps everything
in sight).  Both are good reasons to feel discouraged about.

Junio C Hamano wrote:

The file $GIT_DIR/info/refs was introduced to solve this by
listing the available refs for discovery, and hooks/post-update,
when enabled, runs update-server-info to update the file (among
other things) whenever you push into the repository.

This is helpful.  I'll run git-update-server-info before each push, now.

	Jeff

On Tue, 13 Sep 2005, Junio C Hamano wrote:

I need to clarify what I meant by 'not welcoming dumb transport'
a bit better.  Namely, those (~80 - 23) = ~57 repositories lack
support for 'git ls-remote' over http, which means you cannot
discover what refs the repository has.

You do realize that up until a week ago (six days, to be exact),
kernel.org was running git-0.99.4, which I don't think actually
implemented any of the info stuff?

So out of the 57 repositories, how many haven't been updated in a week?

I suspect that explains a large portion of it.

Also, I really do think that the dumb transports are oversold, and 
git-daemon is undersold. I know all about firewalls, but I also think that 
if people used the smart protocols more, that's a problem that would 
largely solve itself. 

Dumb protocols can never do really well. That's just very fundamental. 

		Linus

Hi,

On Tue, 13 Sep 2005, Linus Torvalds wrote:

Also, I really do think that the dumb transports are oversold, and 
git-daemon is undersold.

My tests confirm that a single git-pull via git-daemon brings a small 
machine to its knees. Which means that multiple git-pull's bring a nice 
big machine like kernel.org to its knees.

IMHO the culprit is git-rev-list, which takes ages and ages for big 
repositories (beware: this could be my Darwin client which might be 
incapable to stop the rev enumeration in time; but if that can be done 
unintentionally, this can be intentionally, too!).

Did anybody think about using the information which helps the dumb 
transports for intelligent transports, too? (A sort of cache for 
git-rev-list would do wonders...) This could at least help the CPU load on 
the server.

(In retrospect it might have been a mistake to make the call to 
git-update-server-info optional: maybe an environment variable should be 
set to _inhibit_ the behaviour for those which absolutely cannot live with 
the performance hit.)

Ciao,
Dscho

On Wed, 14 Sep 2005, Johannes Schindelin wrote:

IMHO the culprit is git-rev-list, which takes ages and ages for big 
repositories (beware: this could be my Darwin client which might be 
incapable to stop the rev enumeration in time; but if that can be done 
unintentionally, this can be intentionally, too!).

Packed too?

git-rev-list will take a long time if the tree is unpacked and not in the 
cache. It's all disk seeks. That's _especially_ true of a full clone 
(which will walk the whole way down).

But I have tons of memory in my machines, and I haven't looked at how 
badly it does if you don't have that. I know that master.kernel.org is 
certainly not having any trouble at all with me pulling from lots of 
trees.. Maybe git-rev-list uses up lots of your memory.

I'm seeing 14 seconds of CPU-time for a _full_ kernel history, with
"--objects". Yes, it's not exactly cheap, and maybe I should optimize it
(it's all in the "--objects" handling and probably a large portion of it
is because trees actually pack very well indeed, so it's actually
unpacking a lot of trees), but considering that that is preparing the
metadata for pulling down a hundred megs of stuff..

That said, I do think that --objects handling is _very_ CPU-hungry. The 
offender is this old commit of mine:

	4311d328fee11fbd80862e3c5de06a26a0e80046
	Author: Linus Torvalds [off-list ref]
	Date:   Sat Jul 23 10:01:49 2005 -0700

	    Be more aggressive about marking trees uninteresting
	...

which is much better about avoiding objects in old trees, but it does so 
at the expense of being _horribly_ CPU-inefficient. It will walk through 
every tree of every commit that we decided was uninteresting.

You can try to just undo that one commit - it will make pack-files have a 
few extraneous objects, but I think it will make a huge difference in the 
CPU cost of "small pulls" (it won't matter at all for the "git clone" 
case: for that case we just always have to walk the whole object tree).

		Linus

On Tue, 13 Sep 2005, Linus Torvalds wrote:

That said, I do think that --objects handling is _very_ CPU-hungry. The 
offender is this old commit of mine:

No, never mind. Even without that, we end up walking a _lot_ of really
uninterestng "internal" trees (ie trees where all parents were
uninteresting, and they were parsed just because we had to parse a lot of 
commits to determine what they reached). 

To explain it a bit better, let's see a common case:


HEAD:		a
	       / \
	      b   \
	     / \   \
	    c   d   \
	   /   / \   \
	  e   f   g   x
	   \ /   /   /
	    h   i   /
	     \ /   /
	      j   /
	       \ /
Old history:    k


Now, imagine that we do 

	git-rev-list b..a

which results in just two commits: 'x' and 'a' (everything else is
reachable from 'b'). This is actually not that uncommon. However, in order 
to realize that, we had to walk through _all_ of a..k and x before we saw 
that 'b'..'k' were all uninteresting, and there was nothing else reachable 
that migt be interesting.

Now, that's pretty cheap per se. git-rev-list is optimized for this case, 
and hey, it's usually just a few hundred objects. Not a big deal - 
generating the commit list takes a small fraction of a second.

However, now the true cost of "--objects" is clear: we will walk the two
"positive" trees ('a' and 'x') and look up all their objects (about 35,000
of them) interesting. So far so good. Just another fraction of a second. 

HOWEVER, then we walk _every_single_uninteresting_commit_ and walk _their_
objects to say "we've got this already". And the uninteresting commits are
often many more than the interesting ones - we might have had to go
several weeks back to list them all. The above example is not at all
extreme: we might have something like 20 interesting commits, and several
hundreds of the uninteresting ones.

Now, the way to optimize things is to realize that there are two "classes" 
of uninteresting commits. There are the uninteresting commits that are 
adjacent to an interesting one (in the above example, they are "b" and 
"k"), and there are the uninteresting commits that are only reachable from 
-other- uninteresting commits ('c'..'j'). Let's call the latter class 
"doubly uninteresting commits", and the former class "uninteresting edge 
commits".

And we really don't need to walk the "doubly uninteresting" trees. But we
do. Because we don't have another phase to discover the edge (we can't do
that during the initial discovery phase, because we don't know if a commit
is going to end up interesting in the end - we migth have another commit
that we haven't seen yet that might be the parent of a commit that _looks_
interesting right now, but ends up being uninteresting because that
eventually seen parent ended up being uninteresting).

In other words: I bet I could make "git-rev-list --objects" go from ten
seconds to a single second if I did that edge discovery for most small
incremental updates. Instead, I'm lazy, and I'm describing the problem on 
the list as an "educational experience", and am callously hoping that 
somebody will see it as an interesting challenge ;)

Btw, the above is definitely not made up. If I did my statistics right,
doing "git-rev-list v2.6.14-rc1.." with the current tree results in 178
"interesting" commits, and 6251 "uninteresting" ones. And I bet 99% of
those uninteresting ones are "doubly uninteresting" - and we're just
wasting CPU time looking at what objects are reachable from them..

		Linus

Hi,

On Tue, 13 Sep 2005, Linus Torvalds wrote:

On Wed, 14 Sep 2005, Johannes Schindelin wrote:

quoted

IMHO the culprit is git-rev-list, which takes ages and ages for big 
repositories (beware: this could be my Darwin client which might be 
incapable to stop the rev enumeration in time; but if that can be done 
unintentionally, this can be intentionally, too!).

Packed too?

Yes. Almost all of it.

git-rev-list will take a long time if the tree is unpacked and not in the 
cache. It's all disk seeks. That's _especially_ true of a full clone 
(which will walk the whole way down).

That could be the case, but my test case is a CVS project I track on one 
side, and I fetch on the other side. Therefore, your diagram from your 
other mail does not really apply. My history looks more or less like this:

a b c d origin
          |
          |
          |
          |
\ \ \ \   |
 ---------|

So, the origin is a linear CVS project. With many, many, many commits. At 
one stage I broke off new git branches. I kept tracking the CVS project, 
though.

What I see when fetching all heads (thanks to Junio, this is one call to 
git-fetch now), where all but origin are up to date, is that it takes a 
very long time. Swapping kicks in, and top tells me that 26.6% of the 
memory is occupied by git-rev-list (The server has 128M, with 1G swap, and 
I am unfortunately not the only user of this machine).

I fail to see why it should need those amounts of memory. (I tested this 
over the ssh protocol, which should essentially do the same as git-daemon, 
right?) After all, the merge point between the branches should be marked 
uninteresting after one single step from each of my private branches.

But I have tons of memory in my machines, and I haven't looked at how 
badly it does if you don't have that. I know that master.kernel.org is 
certainly not having any trouble at all with me pulling from lots of 
trees.. Maybe git-rev-list uses up lots of your memory.

That certainly is the case.

As for master.kernel.org: Unfortunately, you will not be the only puller. 
And if your process needs just 5% of the RAM, then 21 pullers will be too 
many.

That said, I do think that --objects handling is _very_ CPU-hungry.

In my experience, before the swapping started, the process did not get 
more than 20% CPU.

Nevertheless, I still think that it would be a good idea to reuse the 
files created for the dumb transport for the intelligent transport. 
Especially for a project which is more often fetched than uploaded.

I also see other strange things like packing 0 objects, and packing >0 
objects after just having fetched from that repository. Hopefully I will 
have time to look into that (and understand the code to begin with).

Ciao,
Dscho

On Tue, Sep 13, 2005 at 03:11:42PM -0700, Junio C Hamano wrote:

The file $GIT_DIR/info/refs was introduced to solve this by
listing the available refs for discovery, and hooks/post-update,
when enabled, runs update-server-info to update the file (among
other things) whenever you push into the repository.  

It doesn't help that update-server-info crashes if you run
it for the first time on an old repo.
Maybe it should create the appropriate directory structure on the fly,
but the patch below at least checks whether new rev-cache could
be created.

skimo
--

write_rev_cache: check whether new cache could be created.

---
commit d30b87459c690ff68e65dfe8ecdc585dab64323a
tree 51127c1af00f8fd63e7b996384e86d7d31ad5562
parent 2ba6c47be1762726ad0c1d5779064c489150d789
author Sven Verdoolaege [off-list ref] Wed, 14 Sep 2005 12:40:28 +0200
committer Sven Verdoolaege [off-list ref] Wed, 14 Sep 2005 12:40:28 +0200

 rev-cache.c   |    8 +++++++-
 rev-cache.h   |    2 +-
 server-info.c |    7 ++++---
 3 files changed, 12 insertions(+), 5 deletions(-)

diff --git a/rev-cache.c b/rev-cache.c
--- a/rev-cache.c
+++ b/rev-cache.c

@@ -103,7 +103,7 @@ static void write_one_rev_cache(FILE *re
 		write_one_rev_cache(rev_cache_file, rle->ri);
 }
 
-void write_rev_cache(const char *newpath, const char *oldpath)
+int write_rev_cache(const char *newpath, const char *oldpath)
 {
 	/* write the following commit ancestry information in
 	 * $GIT_DIR/info/rev-cache.

@@ -131,6 +131,11 @@ void write_rev_cache(const char *newpath
 		size_t sz;
 		FILE *oldfp = fopen(oldpath, "r");
 		rev_cache_file = fopen(newpath, "w");
+		if (!rev_cache_file) {
+			if (oldfp)
+				fclose(oldfp);
+			return error("cannot open %s", newpath);
+		}
 		if (oldfp) {
 			while (1) {
 				sz = fread(buf, 1, sizeof(buf), oldfp);

@@ -161,6 +166,7 @@ void write_rev_cache(const char *newpath
 		write_one_rev_cache(rev_cache_file, ri);
 	}
 	fclose(rev_cache_file);
+	return 0;
 }
 
 static void add_parent(struct rev_cache *child,

diff --git a/rev-cache.h b/rev-cache.h
--- a/rev-cache.h
+++ b/rev-cache.h

@@ -24,6 +24,6 @@ struct rev_list_elem {
 extern int find_rev_cache(const unsigned char *);
 extern int read_rev_cache(const char *, FILE *, int);
 extern int record_rev_cache(const unsigned char *, FILE *);
-extern void write_rev_cache(const char *new, const char *old);
+extern int write_rev_cache(const char *new, const char *old);
 
 #endif

diff --git a/server-info.c b/server-info.c
--- a/server-info.c
+++ b/server-info.c

@@ -536,6 +536,7 @@ static int update_info_revs(int force)
 	char *path0 = strdup(git_path("info/rev-cache"));
 	int len = strlen(path0);
 	char *path1 = xmalloc(len + 2);
+	int errs = 0;
 
 	strcpy(path1, path0);
 	strcpy(path1 + len, "+");

@@ -548,11 +549,11 @@ static int update_info_revs(int force)
 	for_each_ref(record_rev_cache_ref);
 
 	/* update the rev-cache database */
-	write_rev_cache(path1, force ? "/dev/null" : path0);
-	rename(path1, path0);
+	errs = errs || write_rev_cache(path1, force ? "/dev/null" : path0);
+	errs = errs || rename(path1, path0);
 	free(path1);
 	free(path0);
-	return 0;
+	return errs;
 }
 
 /* public */

On Tue, 2005-09-13 at 17:11, Junio C Hamano wrote:

I just felt that it is a good habit to get into to prepare your
repositories in a shape usable even when served by an HTTP
server that is less forgiving than what kernel.org runs -- that
was what I felt "discouraging" about.

Well, that is sort of just it, too.  Why not make the
default, obvious, common repo prep mechanism do all
the necessary steps for proper presentation?  Having
to remember to do 6 steps just begs for an additional
layer of scripting.

  This means either people are
not packing their repository (hence nobody complained), or
public are pulling over rsync transport (which slurps everything
in sight).  Both are good reasons to feel discouraged about.

I confess, I've been using rsync as it is what appears
to be able to reliably get a repository that works.

jdl

On Wed, 14 Sep 2005, Johannes Schindelin wrote:

What I see when fetching all heads (thanks to Junio, this is one call to 
git-fetch now), where all but origin are up to date, is that it takes a 
very long time. Swapping kicks in, and top tells me that 26.6% of the 
memory is occupied by git-rev-list (The server has 128M, with 1G swap, and 
I am unfortunately not the only user of this machine).

Ok. As mentioned, I've not looked at memory usage. The machines I play 
with tend to have 2GB or more, simply because bk needed at least 1GB to be 
nice and cached on the kernel ;)

Git has needed less than bk, so I've not cared ;)

I fail to see why it should need those amounts of memory. (I tested this 
over the ssh protocol, which should essentially do the same as git-daemon, 
right?) After all, the merge point between the branches should be marked 
uninteresting after one single step from each of my private branches.

One of the issues is that git-rev-list will (for example) keep track of 
the commit messages too for every commit. That in itself can be a lot of 
stuff, depending on how active the tree is and how large the messages are.

Now, that should be easy enough to fix (parse_commit() normally saves the 
buffer it parses into "commit->buffer", so we'd just need to do something 
like

	if (!verbose_header && commit->buffer) {
		free(commit->buffer);
		commit->buffer = NULL;
	}

for each commit.

But for --objects, the bigger memory pressure is that it needs to track 
the "struct object" for every single object when it generates the 
reference tracking. And THAT tends to be expensive. The object lists are 
also not very space-efficient (ie one small allocation for each list 
entry).

We could probably make objects/lists more space-efficient.

I also see other strange things like packing 0 objects, and packing >0 
objects after just having fetched from that repository. Hopefully I will 
have time to look into that (and understand the code to begin with).

Well, the "packing 0 objects" should be normal. I'm surprised at the ">0" 
case after a fetch: the packign is _not_ guaranteed to be exact, but if 
you have the exact same state as (or a superset of) the other end, you 
should always see a zero.

		Linus