Re: Git is not scalable with too many refs/*

On Mon, 26 Sep 2011 14:01:38 -0600, Martin Fick wrote:
-- snip --

So, maybe you are correct, maybe my repo is the corner case?
Is a repo which needs to be gced considered a corner case?
Should git be able to detect that the repo is so in
desperate need of gcing?  Is it normal for git to need to gc
right after a clone and then fetching ~100K refs?

Were you 100k refs packed before the gc?  If not, perhaps your refs are 
causing a lot of trouble for the merge sort?  They will be written out 
sorted to the packed-refs file, so the merge sort won't have to do any 
real work when loading them after that...

I am not sure what is right here, if this patch makes a repo
which needs gcing degrade 5 to 10 times worse than the
benefit of this patch, it still seems questionable to me.

Well - it does this _for your repo_, that doesn't automatically mean 
that it does generally, or frequently.  For instance, none of my normal 
repos that have a lot of refs are Gerrit ones, and I wouldn't be 
surprised if they benefitted from the merge sort (assuming that I am 
right that the merge sort is taking a long time on your gerrit refs).

Besides, you would be better off running gc, and thus getting the 
benefit too.

quoted

Random thought.  What happens to the with compression
case if you leave the commit in, but add a sleep(15) to
the end of sort_refs_list?

Why, what are you thinking?  Hmm, I am trying this on the
non gced repo and it doesn't seem to be completing (no cpu
usage)!  It appears that perhaps it is being called many
times (the sleeping would explain no cpu usage)?!?  This
could be a real problem, this should only get called once
right?

I was just wondering if the time taken to get the refs was changing the 
interaction with something else.  Not very likely, but ...

I added a print statement, and it was called four times when I had 
unpacked refs, and once with packed.  So, maybe you are hitting some 
nasty case with unpacked refs.  If you use a print statement instead of 
a sleep, how many times does sort_refs_lists get called in your unpacked 
case?  It may well also be worth calculating the time taken to do the 
sort.

-- 
Julian