Re: Git is not scalable with too many refs/*

On Monday, September 26, 2011 03:39:33 pm Martin Fick wrote:

On Monday, September 26, 2011 02:28:53 pm Julian Phillips
wrote:

quoted

quoted

quoted

Random thought.  What happens to the with
compression case if you leave the commit in, but
add a sleep(15) to the end of sort_refs_list?

Why, what are you thinking?  Hmm, I am trying this on
the non gced repo and it doesn't seem to be
completing (no cpu usage)!  It appears that perhaps
it is being called many times (the sleeping would
explain no cpu usage)?!?  This could be a real
problem, this should only get called once right?

I was just wondering if the time taken to get the refs
was changing the interaction with something else.  Not
very likely, but ...

I added a print statement, and it was called four times
when I had unpacked refs, and once with packed.  So,
maybe you are hitting some nasty case with unpacked
refs.  If you use a print statement instead of a sleep,
how many times does sort_refs_lists get called in your
unpacked case?  It may well also be worth calculating
the time taken to do the sort.

In my case it was called 18785 times!  Any other tests I
should run?

Gerrit stores the changes in directories under refs/changes 
named after the last 2 digits of the change.  Then under 
each change it stores each patchset.  So it looks like this:   
refs/changes/dd/change_num/ps_num

I noticed that:

 ls refs/changes/* | wc -l 
 -> 18876

somewhat close, but not super close to 18785,  I am not sure 
if that is a clue.  It's almost like each change is causing 
a re-sort,

 
-Martin

-- 
Employee of Qualcomm Innovation Center, Inc. which is a 
member of Code Aurora Forum