Re: [PATCH/RFC 0/3] faster inexact rename handling

On 10/30/07, Jeff King [off-list ref] wrote:

On Tue, Oct 30, 2007 at 08:38:24AM -0700, Linus Torvalds wrote:

quoted

quoted

with the old and new code. Pairs like Documentation/git-add-script.txt
-> Documentation/git-add.txt are not found, because the file is composed
almost entirely of boilerplate.

Ok, that does imply to me that we cannot just drop boilerplate text,
because the fact is, lots of files contain boilerplate, but people still
think they are "similar".

Well, the problem is that instead of just "dropping" boilerplate text,
we fail to count it as a similarity, but it still counts towards the
file size. It may be that just dropping it totally is the right thing
(in which case those renames _will_ turn up, because they will be filled
with identical non-boilerplate goodness).

Right, in the demo I make an extra pass after the inverted indexing
step to prune the index -- which means eliminating the common lines
*entirely* from the index (so they don't get attributed to a random
file) *and* decrementing all the file sizes by 1.  That way the
similarity scores shouldn't get skewed.

And as you mentioned we could bump the threshold from 1 to some other
small integer.  Intuitively I guess you could say it is common to copy
a file to 2 places or 3 places, and you don't want all the lines to
get thrown out because of that.  But usually you don't copy a file to
10 or 50 places.

Andy