Re: [RFC PATCH] index-pack: improve performance on NFS

On Wed, Nov 07 2018, Geert Jansen wrote:

On Mon, Oct 29, 2018 at 07:27:39PM -0400, Jeff King wrote:

quoted

On Mon, Oct 29, 2018 at 08:36:07PM +0100, Ævar Arnfjörð Bjarmason wrote:

quoted

 * Re-roll my 4 patch series to include the patch you have in
   [ref]

I don't think it's quite ready for inclusion as-is. I hope to brush it
up a bit, but I have quite a backlog of stuff to review, as well.

We're still quite keen to get this patch included. Is there anything I can do
to help?

Also I just re-read your comments on maximum cache size. I think you were
arguing both sides of the equation and I wasn't sure where you'd ended up. :)
A larger cache size potentially takes more time to fill up especially on NFS
while a smaller cache size obviously would less effective. That said a small
cache is still effective for the "clone" case where the repo is empty.

It also occurred to me that as a performance optimization your patch could read
the the loose object directories in parallel using a thread pool. At least on
Amazon EFS this should result in al almost linear performance increase. I'm not
sure how much this would help for local file systems. In any case this may be
best done as a follow-up patch (that I'd be happy to volunteer for).

I'm planning to re-submit mine with some minor changes after the great
Documentation/config* move lands.

As noted in
https://public-inbox.org/git/87bm7clf4o.fsf@evledraar.gmail.com/ and
https://public-inbox.org/git/87h8gq5zmc.fsf@evledraar.gmail.com/ I think
it's regardless of Jeff's optimization is. O(nothing) is always faster
than O(something), particularly (as explained in that E-Mail) on NFS.

You didn't answer my question in
https://public-inbox.org/git/20181030024925.GC8325@amazon.com/ about
whether for your purposes you're interested in this for something where
it needs to work out of the box on some random Amazon's customer's
"git", or if it's something in-house and you just don't want to turn off
collision checking. That would be useful to know.

I've turned on core.checkCollisions=false in production at our
site. Cloning for some large repositories went from ~200 minutes to ~5m,
and some pushes from ~5 minutes to ~10 seconds. Those numbers will be
very similar (but slightly higher, maybe 1-5 seconds higher in the
latter case) with Jeff's (depending on the push).