Re: Git is not scalable with too many refs/*

Julian Phillips [off-list ref] writes:

Back when I made that change, I failed to notice that get_ref_dir was
recursive for subdirectories ... sorry ...

Aha, I also was blind while I was watching this discussion from the
sideline, and I thought I re-read the codepath involved X-<. Indeed
we were sorting the list way too early and the patch looks correct.

Thanks.

get_ref_dir is called recursively for subdirectories, which means that
we were calling sort_ref_list for each directory of refs instead of
once for all the refs.  This is a massive wast of processing, so now
just call sort_ref_list on the result of the top-level get_ref_dir, so
that the sort is only done once.

In the common case of only a few different directories of refs the
difference isn't very noticable, but it becomes very noticeable when
you have a large number of direcotries containing refs (e.g. as
created by Gerrit).

Reported by Martin Fick.

Signed-off-by: Julian Phillips <redacted>
---

This time the typos are fixed too ... perhaps I wrote the original commit at 1am
too ... :$

 refs.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)

diff --git a/refs.c b/refs.c
index a615043..a49ff74 100644
--- a/refs.c
+++ b/refs.c

@@ -319,7 +319,7 @@ static struct ref_list *get_ref_dir(const char *submodule, const char *base,
 		free(ref);
 		closedir(dir);
 	}
-	return sort_ref_list(list);
+	return list;
 }
 
 struct warn_if_dangling_data {

@@ -361,11 +361,13 @@ static struct ref_list *get_loose_refs(const char *submodule)
 	if (submodule) {
 		free_ref_list(submodule_refs.loose);
 		submodule_refs.loose = get_ref_dir(submodule, "refs", NULL);
+		submodule_refs.loose = sort_ref_list(submodule_refs.loose);
 		return submodule_refs.loose;
 	}
 
 	if (!cached_refs.did_loose) {
 		cached_refs.loose = get_ref_dir(NULL, "refs", NULL);
+		cached_refs.loose = sort_ref_list(cached_refs.loose);
 		cached_refs.did_loose = 1;
 	}
 	return cached_refs.loose;

-- 
1.7.6.1

On 09/27/2011 02:00 AM, Julian Phillips wrote:

get_ref_dir is called recursively for subdirectories, which means that
we were calling sort_ref_list for each directory of refs instead of
once for all the refs.  This is a massive wast of processing, so now
just call sort_ref_list on the result of the top-level get_ref_dir, so
that the sort is only done once.

+1

I think this patch should also be considered for maint, since it is
noninvasive and fixes a bad performance regression.

Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu
http://softwareswirl.blogspot.com/