Re: [PATCH v4 5/5] unpack-trees: reuse (still valid) cache-tree from src_index

On 8/13/2018 12:25 PM, Duy Nguyen wrote:

On Mon, Aug 13, 2018 at 6:05 PM Ben Peart [off-list ref] wrote:

quoted

I was part way through writing a patch that would copy the valid parts
of the cache-tree from the source index to the dest index

Yeah sorry about that. I make bad judgements all the time, unfortunately.

If it's sort of working though, please post to the list anyway to
archive it. Who knows, some time down the road we might actually need
it again.

quoted

I run some tests on a large repo and the results look very promising.

base    new     diff    % saved
0.55    0.52    0.02    4.32%   s:  read cache .git/index
0.31    0.30    0.01    2.98%   s:  initialize name hash
0.03    0.02    0.00    9.98%   s:  preload index
0.09    0.09    0.00    4.86%   s:  refresh index
5.93    1.19    4.74    79.95%  s:   traverse_trees
0.12    0.13    -0.01   -4.15%  s:   check_updates
2.14    0.00    2.14    100.00% s:   cache_tree_update
10.63   4.29    6.33    59.59%  s:  unpack_trees

There's a big gap here, I think. unpack_trees() takes 4s but the sum
of traverse_trees, check_updates and cache_tree_update is 1.5s top. I
guess that's sparse checkout and stuff? It's either that or there's
another big hidden thing we should pay attention to ;-)

Yes, there are additional costs associated with the sparse-checkout and 
excludes logic.  We've sped that up significantly by converting it to a 
hashmap but it still has measurable cost as we have to compute the hash 
of the cache entry name before looking it up.

Name                                            Inc %	     Inc
+ git!unpack_trees                      	 50.9	   4,575
|+ git!clear_ce_flags_1                 	 16.5	   1,479
||+ git!is_included_in_virtualfilesystem	 15.9	   1,430
|| + git!check_includes_hashmap         	 15.8	   1,418
|+ git!traverse_trees                   	 16.3	   1,468
|+ git!cache_tree_fully_valid           	 11.7	   1,055
|+ git!check_updates                    	  1.9	     169
|+ git!discard_index                    	  1.8	     162
|+ git!apply_sparse_checkout            	  0.2	      15
|+ git!next_cache_entry                 	  0.0	       3