Re: Git is not scalable with too many refs/*

On Monday, September 26, 2011 12:56:04 pm Christian Couder 

wrote:

After "git pack-refs --all" I get:

OK.   So many great improvements in ref scalability, thanks 
everyone!

It is getting so good, that I had to take a step back and 
re-evaluate what we consider good/bad.  On doing so, I can't 
help but think that fetches still need some improvement.

Fetches had the worst regression of all > 8days, so the 
massive fix to bring it down to 7.5mins was awesome.  
7-8mins sounded pretty good 2 weeks ago, especially when a 
checkout took 5+ mins!  but now that almost every other 
operation has been sped up, that is starting to feel a bit 
on the slow side still.  My spidey sense tells me something 
is still not quite right in the fetch path.

Here is some more data to backup my spidey sense: after all 
the improvements, a noop fetch of all the changes (noop 
meaning they are all already uptodate) takes around 
3mins with a non gced (non packed refs) case.  That same 
noop only takes ~12s in the gced (packed ref case)!

I dug into this a bit further.  I took a non gced and non 
packed refs repo and this time instead of gcing it to get 
packedrefs, I only ran the above git pack-refs --all so that
objects did not get gced.  With this, the noop fetch was 
also only around 12s.  This confirmed that the non gced 
objects are not interfering with the noop fetch, the problem 
really is just the unpacked refs.  Just to confirm that the 
FS is not horribly slow, I did a "find .git/refs" and it 
only takes about .4s for about 80Kresults!

So, while I understand that a full fetch will actually have 
to transfer quite a bit of data, the noop fetch seems like 
it is still suffering in the non gced (non packed ref case).  
If that time were improved, I suspect that the full fetch 
will improve at least by an equivalent amount, if not more.

Any thoughts?

-Martin


-- 
Employee of Qualcomm Innovation Center, Inc. which is a 
member of Code Aurora Forum