Re: [PATCH v2 0/4] Speed up unpack_trees()

On 7/31/2018 11:31 AM, Duy Nguyen wrote:

On Mon, Jul 30, 2018 at 8:10 PM Ben Peart [off-list ref] wrote:

quoted

I ran "git checkout" on a large repo and averaged the results of 3 runs.
   This clearly demonstrates the benefit of the optimized unpack_trees()
as even the final "diff-index" is essentially a 3rd call to unpack_trees().

baseline        new
----------------------------------------------------------------------
0.535510167     0.556558733     s: read cache .git/index
0.3057373       0.3147105       s: initialize name hash
0.0184082       0.023558433     s: preload index
0.086910967     0.089085967     s: refresh index
7.889590767     2.191554433     s: unpack trees
0.120760833     0.131941267     s: update worktree after a merge
2.2583504       2.572663167     s: repair cache-tree
0.8916137       0.959495233     s: write index, changed mask = 28
3.405199233     0.2710663       s: unpack trees
0.000999667     0.0021554       s: update worktree after a merge
3.4063306       0.273318333     s: diff-index
16.9524923      9.462943133     s: git command:
'c:\git-sdk-64\usr\src\git\git.exe' checkout

The first call to unpack_trees() saves 72%
The 2nd and 3rd call save 92%

By the 3rd I guess you meant "diff-index" line. I think it's the same
with the second call. diff-index triggers the second unpack-trees but
there's no indent here and it's misleading to read this as diff-index
and unpack-trees execute one after the other.

quoted

Total time savings for the entire command was 44%

Wow.. I guess you have more trees since I could only save 30% on gcc.git.

Yes, with over 500K trees, this optimization really pays off for us.  I can't wait to see how this works out in the wild (vs my "lab" based performance testing).

Thank you!  I definitely owe you lunch. :)

quoted

In the performance game of whack-a-mole, that call to repair cache-tree
is now looking quite expensive...

Yeah and I think we can whack that mole too. I did some measurement.
Best case possible, we just need to scan through two indexes (one with
many good cache-tree, one with no cache-tree), compare and copy
cache-tree over. The scanning takes like 1% time of current repair
step and I suspect it's the hashing that takes most of the time. Of
course real world won't have such nice numbers, but I guess we could
maybe half cache-tree update/repair time.

I have some great profiling tools available so will take a look at this next and see exactly where the time is being spent.