Re: [PATCH 1/2] diffcore-rename: support rename cache

On 11/8/08, Jeff King [off-list ref] wrote:

On Sat, Nov 08, 2008 at 10:24:10AM +0100, Yann Dirson wrote:

 > Well, it could be that an out-of-pathspec pair would have a better
 > score than an in-pathspec one.  Maybe cache recording should be turned
 > off when doing pathspec limitation ?


One thing I notice is that the cache works at the level of "here is the
 best rename for this commit." Maybe it could go down a level and say
 "here is the inexact rename score between these blobs". Then you would
 still find the best score between two blobs each time, but save the
 really computationally intensive part (which is comparing the actual
 _content_ of the blobs).
 That should work in the face of path limiting or any other option,
 because it is caching something immutable: this is the similarity score
 between two pieces of content. And then you get arbitrary tree-to-tree
 speedups for free, since such a cache would be valid for every commit.

I did that and realized the cost was not from each diff, in
--find-copies-harder case, but from the number of diffs you had to do.
Even with exact matching on linux-2.6.git, it could take significant
time (it was about 5 minutes in no-cache case, 1 minute without exact
match cache, and less than 1 sec if everything is cached).

 The downsides are:

  - your cache is potentially bigger, since you are caching the score of
   every pair you look at, instead of just "good" pairs (OTOH, you are
   not doing a per-commit cache, which helps reduce the size)

It is huge if you accidentially add --find-copies-harder to your
command, considering that every new file will be compared against
every files in tree (about 25k).

  - you can still "lie" about a score to pre-seed imported SVN renames,
   but such lying will actually apply to all commits.

-- 
Duy