Re: another packed-refs race

On Mon, May 06, 2013 at 02:12:29PM +0200, Michael Haggerty wrote:

quoted

I think that would be correct (modulo that step 1 cannot happen for
enumeration). But we would like to avoid loading all loose refs if we
can. Especially on a cold cache, it can be quite slow, and you may not
even care about those refs for the current operation (I do not recall
the exact original motivation for the lazy loading, but it was something
along those lines).

Lazy loading was first inspired by the observation that effectively
every git invocation was loading *all* loose references to do an
iteration over refs/replace/ (which I've never even used!)  This was
absolutely killing the performance of filter-branch, which creates a lot
of loose references and invokes git many times--even though the cache
was warm.

Yeah, obviously we want to avoid that. I _think_ we can even keep the
lazy loading, as long as keep the ordering as:

  1. Load a chunk of loose refs (whatever we need for the current
     iteration request).

  2. Double-check that our packed-refs cache is up to date, and reload
     if necessary.

  3. Return the results to the caller.

It would perhaps increase latency to getting results to the caller
(since otherwise we can start feeding answers to the caller as we walk
the tree), but should not increase the total amount of work (just the
extra stat() of packed-refs, once per for_each_ref, which is not much).

I'll see if I can cook up a patch.

-Peff