Re: Git is not scalable with too many refs/*

On Monday, September 26, 2011 06:41:04 am Christian Couder 
wrote:

On Sun, Sep 25, 2011 at 10:43 PM, Martin Fick 

[off-list ref] wrote:
...

quoted

 git checkout

can also take rather long periods of time > 3 mins when
run on a repo with ~100K refs.

...

quoted

 So, I bisected this issue also, and it seems that the
"offending" commit is

...

quoted

commit 680955702990c1d4bfb3c6feed6ae9c6cb5c3c07
Author: Christian Couder [off-list ref]

   replace_object: add mechanism to replace objects
found in "refs/replace/"

...

I don't think there is an obvious problem with it, but it
would be nice if you could dig a bit deeper.

The first thing that could take a lot of time is the call
to for_each_replace_ref() in this function:

+static void prepare_replace_object(void)
+{
+       static int replace_object_prepared;
+
+       if (replace_object_prepared)
+               return;
+
+       for_each_replace_ref(register_replace_ref, NULL);
+       replace_object_prepared = 1;
+}

The time was actually spent in for_each_replace_ref()
which calls get_loose_refs() which has the recursive bug 
that Julian Phillips fixed 2 days ago.  Good to see that 
this fix helps other use cases too.

So with that bug fixed, the thing taking the most time now 
for a git checkout with ~100K refs seems to be the orphan 
check as Thomas predicted.  The strange part with this, is 
that the orphan check seems to take only about ~20s in the 
repo where the refs aren't packed.  However, in the repo 
where they are packed, this check takes at least 5min!  This 
seems a bit unusual, doesn't it?  Is the filesystem that 
much better at indexing refs than git's pack mechanism?  
Seems unlikely, the unpacked refs take 312M in the FS, the 
packed ones only take about 4.3M.  I suspect their is 
something else unexpected going on here in the packed ref 
case.  

Any thoughts?  I will dig deeper...

-Martin

-- 
Employee of Qualcomm Innovation Center, Inc. which is a 
member of Code Aurora Forum