From: Shawn O. Pearce <hidden> Date: 2016-06-15 22:43:17
So Govind Salinas has found an interesting case in the rename
detection code:
$ git clone git://repo.or.cz/Widgit.git
$ git diff -M --raw -r 192e^ 192e | grep .resx
:100755 000000 4c8ab79... 0000000... D Form1.resx
:100755 100755 9e70146... 9e70146... R100 CommitViewer.resx UI/CommitViewer.resx
:100755 100755 90929fd... b40ff98... C091 RepoManager.resx UI/Form1.resx
:100755 100755 90929fd... 90929fd... C100 PreferencesEditor.resx UI/PreferencesEditor.resx
:100755 100755 90929fd... 90929fd... R100 PreferencesEditor.resx UI/RepoManager.resx
:100755 100755 90929fd... 8535007... R097 RepoManager.resx UI/RepoTreeView.resx
In this case several files had identical old images, and some
kept that old image during the rename. Unfortunately because of
the ordering of the files in the tree Git has decided to "rename"
the PreferencesEditor.resx file to UI/RepoManager.resx, rather than
renaming RepoManager.resx to UI/RepoManager.resx. Go Git.
I'm wondering if we shouldn't play the game of trying to match
delete/add pairs up by not only similarity, but also by path
basename. In the case above its exactly what Govind thought should
happen; he moved the file from one directory to another, and didn't
even change its content during the move. But Git decided "better"
to use a totally different file in the "rename".
--
Shawn.
I'm wondering if we shouldn't play the game of trying to match
delete/add pairs up by not only similarity, but also by path
basename.
I think we should just consider the basename as an "added
similarity bonus".
IOW, we currently sort purely by data similarity, but how about just
adding a small increment for "same base name".
We could make it actually use the similarity of the filename itself as the
basis for the increment, which would be even better, but the trivial thing
is to do something like
--- a/diffcore-rename.c
+++ b/diffcore-rename.c
@@ -186,8 +186,11 @@ static int estimate_similarity(struct diff_filespec *src,
*/
if (!dst->size)
score = 0; /* should not happen */
- else
+ else {
score = (int)(src_copied * MAX_SCORE / max_size);
+ if (basename_same(src, dst))
+ score++;
+ }
return score;
}
and just implement that "basename_same()" function.
Or something.
I do agree that the filename logically can and probably _should_ count
towards the "similarity". The filename _is_ part of the data in the global
notion of "content", after all. It's the "index" to the data.
Linus
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:43:17
When there are several candidates for a rename source, and one of them
has an identical basename to the rename target, take that one.
Noticed by Govind Salinas, posted by Shawn O. Pearce, partial patch
by Linus Torvalds.
Signed-off-by: Johannes Schindelin <redacted>
---
On Wed, 20 Jun 2007, Linus Torvalds wrote:
> I think we should just consider the basename as an "added
> similarity bonus".
>
> IOW, we currently sort purely by data similarity, but how about
> just adding a small increment for "same base name".
>
> [patch suggestion snipped, since it is identical what is below]
How 'bout this?
diffcore-rename.c | 33 ++++++++++++++++++++++++++++++++-
t/t4001-diff-rename.sh | 13 +++++++++++++
2 files changed, 45 insertions(+), 1 deletions(-)
@@ -119,6 +119,21 @@ static int is_exact_match(struct diff_filespec *src,return0;}+staticintbasename_same(structdiff_filespec*src,structdiff_filespec*dst)+{+intsrc_len=strlen(src->path),dst_len=strlen(dst->path);+while(src_len&&dst_len){+charc1=src->path[--src_len];+charc2=dst->path[--dst_len];+if(c1!=c2)+return0;+if(c1=='/')+return1;+}+return(!src_len||src->path[src_len-1]=='/')&&+(!dst_len||dst->path[dst_len-1]=='/');+}+structdiff_score{intsrc;/* index in rename_src */intdst;/* index in rename_dst */
@@ -186,8 +201,11 @@ static int estimate_similarity(struct diff_filespec *src,*/if(!dst->size)score=0;/* should not happen */-else+else{score=(int)(src_copied*MAX_SCORE/max_size);+if(basename_same(src,dst))+score++;+}returnscore;}
@@ -295,9 +313,22 @@ void diffcore_rename(struct diff_options *options)if(rename_dst[i].pair)continue;/* dealt with an earlier round */for(j=0;j<rename_src_nr;j++){+intk;structdiff_filespec*one=rename_src[j].one;if(!is_exact_match(one,two,contents_too))continue;++/* see if there is a basename match, too */+for(k=j;k<rename_src_nr;k++){+one=rename_src[k].one;+if(basename_same(one,two)&&+is_exact_match(one,two,+contents_too)){+j=k;+break;+}+}+record_rename_pair(i,j,(int)MAX_SCORE);rename_count++;break;/* we are done with this entry */
@@ -64,4 +64,17 @@ test_expect_success \'validate the output.'\'compare_diff_patch current expected'+test_expect_success'favour same basenames over different ones''+cppath1another-path&&+gitaddanother-path&&+gitcommit-m1&&+gitrmpath1&&+mkdirsubdir&&+gitmvanother-pathsubdir/path1&&+gitrunstatus|grep"renamed: .*path1 -> subdir/path1"'++test_expect_success'favour same basenames even with minor differences''+gitshowHEAD:path1|sed"s/15/16/">subdir/path1&&+gitrunstatus|grep"renamed: .*path1 -> subdir/path1"'+ test_done
From: Jeff King <hidden> Date: 2016-06-15 22:43:17
On Thu, Jun 21, 2007 at 12:52:11PM +0100, Johannes Schindelin wrote:
When there are several candidates for a rename source, and one of them
has an identical basename to the rename target, take that one.
That's a reasonable heuristic, but it unfortunately won't match simple
things like:
i386_widget.c -> arch/i386/widget.c
You really don't care about "is this a good match" as much as providing
an order to potential matches. I think something like a Levenshtein
distance between the full pathnames would give good results, and would
cover almost every situation that the basename heuristic would (there
are a few exceptions, like getting "file.c" from either "file2.c" or
"foo/file.c", but that seems kind of pathological).
Sorry to post without a patch, but I don't have time right this second.
I'll add it to the end of my (ever-growing) todo list if you think it's
a good idea and don't do it yourself. :)
-Peff
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:43:17
Hi,
On Thu, 21 Jun 2007, Jeff King wrote:
On Thu, Jun 21, 2007 at 12:52:11PM +0100, Johannes Schindelin wrote:
quoted
When there are several candidates for a rename source, and one of them
has an identical basename to the rename target, take that one.
That's a reasonable heuristic, but it unfortunately won't match simple
things like:
i386_widget.c -> arch/i386/widget.c
That's right. But every heuristic falls down eventually. Personally, I
think basename_same() is good enough, even if the technical challenge to
implement a small enough Levenshtein, which still respects directory
boundaries somehow (and not just throws them away).
Besides, Levenshtein would introduce a ranking, not a boolean value like
basename_same(). And that complicates the code.
All in all, I'd say Levenshtein is not worth the _result_.
Ciao,
Dscho
On Thu, Jun 21, 2007 at 12:52:11PM +0100, Johannes Schindelin wrote:
quoted
When there are several candidates for a rename source, and one of them
has an identical basename to the rename target, take that one.
That's a reasonable heuristic, but it unfortunately won't match simple
things like:
i386_widget.c -> arch/i386/widget.c
We'e also had things like
arch/i386/kernel/pci-pc.c -> arch/i386/kernel/pci/common.c
so it's not always the ending of a file that is unchanged, but you still
often have some "similarity" of the name (ie the "pci" substring is still
common there).
So I agree that we can be even better about the heuristics. I don't know
how much it *matters* in practice.
I do agree with the people who argue that you simply shouldn't depend on
these kinds of things, and if you have identical files, and move them
around, you really are getting behaviour that doesn't matter.
The files are *identical* for christ sake! Following their history, it
doesn't matter *which* base you follow, since regardless, they've come to
the same point!
So in that sense, the current git behaviour is actually perfectly fine.
At the same time, I'll argue from a totally theoretical point that the
"filename" is obviously part of the data in the tree, and as such, a
similarity comparison that takes only the data into account is a bit
limited. So while I don't think a user should really care, I also think
that keeping the filename as part of the similarity analysis is actually
a perfectly logical and valid thing to do withing the git policy of
"content is king".
The filename *is* part of the content, and it's doubly so when you think
about a rename or copy operation, where the whole point of the exercise is
as much about the filename as about the data inside the file.
Linus
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:43:17
Hi,
On Thu, 21 Jun 2007, Jeff King wrote:
I think something like a Levenshtein distance between the full pathnames
would give good results, and would cover almost every situation that the
basename heuristic would (there are a few exceptions, like getting
"file.c" from either "file2.c" or "foo/file.c", but that seems kind of
pathological).
Well, now you only have to test if it makes sense:
-- snipsnap --
[PATCH] diffcore-rename: replace basename_same() heuristics by Levenshtein
Instead of insisting on identical basenames, try the levenshtein
distance.
Basically, if there are multiple rename source candidates, take the
one with the smallest Levenshtein distance.
Signed-off-by: Johannes Schindelin <redacted>
---
The dangerous thing is that the score can get negative now.
Makefile | 4 ++--
diffcore-rename.c | 42 +++++++++++++++---------------------------
levenshtein.c | 39 +++++++++++++++++++++++++++++++++++++++
levenshtein.h | 6 ++++++
4 files changed, 62 insertions(+), 29 deletions(-)
create mode 100644 levenshtein.c
create mode 100644 levenshtein.h
@@ -4,6 +4,7 @@#include"cache.h"#include"diff.h"#include"diffcore.h"+#include"levenshtein.h"/* Table of rename/copy destinations */
@@ -119,21 +120,6 @@ static int is_exact_match(struct diff_filespec *src,return0;}-staticintbasename_same(structdiff_filespec*src,structdiff_filespec*dst)-{-intsrc_len=strlen(src->path),dst_len=strlen(dst->path);-while(src_len&&dst_len){-charc1=src->path[--src_len];-charc2=dst->path[--dst_len];-if(c1!=c2)-return0;-if(c1=='/')-return1;-}-return(!src_len||src->path[src_len-1]=='/')&&-(!dst_len||dst->path[dst_len-1]=='/');-}-structdiff_score{intsrc;/* index in rename_src */intdst;/* index in rename_dst */
@@ -201,11 +187,9 @@ static int estimate_similarity(struct diff_filespec *src,*/if(!dst->size)score=0;/* should not happen */-else{-score=(int)(src_copied*MAX_SCORE/max_size);-if(basename_same(src,dst))-score++;-}+else+score=(int)(src_copied*MAX_SCORE/max_size)+-levenshtein(src->path,dst->path);returnscore;}
@@ -313,20 +297,24 @@ void diffcore_rename(struct diff_options *options)if(rename_dst[i].pair)continue;/* dealt with an earlier round */for(j=0;j<rename_src_nr;j++){-intk;+intk,distance;structdiff_filespec*one=rename_src[j].one;if(!is_exact_match(one,two,contents_too))continue;+distance=levenshtein(one->path,two->path);/* see if there is a basename match, too */for(k=j;k<rename_src_nr;k++){+intd2;one=rename_src[k].one;-if(basename_same(one,two)&&-is_exact_match(one,two,-contents_too)){-j=k;-break;-}+if(!is_exact_match(one,two,+contents_too))+continue;+d2=levenshtein(one->path,two->path);+if(d2>distance)+continue;+distance=d2;+j=k;}record_rename_pair(i,j,(int)MAX_SCORE);
From: Jeff King <hidden> Date: 2016-06-15 22:43:17
On Fri, Jun 22, 2007 at 02:14:43AM +0100, Johannes Schindelin wrote:
quoted hunk
@@ -313,20 +297,24 @@ void diffcore_rename(struct diff_options *options) if (rename_dst[i].pair) continue; /* dealt with an earlier round */ for (j = 0; j < rename_src_nr; j++) {- int k;+ int k, distance; struct diff_filespec *one = rename_src[j].one; if (!is_exact_match(one, two, contents_too)) continue;+ distance = levenshtein(one->path, two->path); /* see if there is a basename match, too */ for (k = j; k < rename_src_nr; k++) {
This loop can start at k = j+1, since otherwise we are just checking
rename_src[j] against itself.
+int levenshtein(const char *string1, const char *string2)
+{
+ int len1 = strlen(string1), len2 = strlen(string2);
+ int *row1 = xmalloc(sizeof(int) * (len2 + 1));
+ int *row2 = xmalloc(sizeof(int) * (len2 + 1));
+ int i, j;
+
+ for (j = 1; j <= len2; j++)
+ row1[j] = j;
This loop must start at j=0, not j=1; otherwise you have an undefined
value in row1[0], which gets read when setting row2[1], and you get
a totally meaningless distance (I got -1209667248 on my test case!).
-Peff
From: Johannes Sixt <hidden> Date: 2016-06-15 22:43:17
Johannes Schindelin wrote:
The dangerous thing is that the score can get negative now.
...
+ score = (int)(src_copied * MAX_SCORE / max_size)
+ - levenshtein(src->path, dst->path);
Does that also mean that you can't ever have a rename with a score of
100%?
(I haven't studied the algorithms and assume that levenshtein(a,b) == 0
only if a==b, and that without the -levenshtein(...) the score can grow
to 100%.)
-- Hannes
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:43:17
Hi,
On Fri, 22 Jun 2007, Jeff King wrote:
On Fri, Jun 22, 2007 at 02:14:43AM +0100, Johannes Schindelin wrote:
quoted
@@ -313,20 +297,24 @@ void diffcore_rename(struct diff_options *options) if (rename_dst[i].pair) continue; /* dealt with an earlier round */ for (j = 0; j < rename_src_nr; j++) {- int k;+ int k, distance; struct diff_filespec *one = rename_src[j].one; if (!is_exact_match(one, two, contents_too)) continue;+ distance = levenshtein(one->path, two->path); /* see if there is a basename match, too */ for (k = j; k < rename_src_nr; k++) {
This loop can start at k = j+1, since otherwise we are just checking
rename_src[j] against itself.
Right.
quoted
+int levenshtein(const char *string1, const char *string2)
+{
+ int len1 = strlen(string1), len2 = strlen(string2);
+ int *row1 = xmalloc(sizeof(int) * (len2 + 1));
+ int *row2 = xmalloc(sizeof(int) * (len2 + 1));
+ int i, j;
+
+ for (j = 1; j <= len2; j++)
+ row1[j] = j;
This loop must start at j=0, not j=1; otherwise you have an undefined
value in row1[0], which gets read when setting row2[1], and you get
a totally meaningless distance (I got -1209667248 on my test case!).
Sorry for that. I originally had an xcalloc in there, and did not look at
that loop afterwards.
And I completely forgot that on my laptop (on which I did this patch), I
had forgotten to add
ALL_CFLAGS += -DXMALLOC_POISON=1
to config.mak.
Ciao,
Dscho
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:43:17
Hi,
On Fri, 22 Jun 2007, Johannes Sixt wrote:
Johannes Schindelin wrote:
quoted
The dangerous thing is that the score can get negative now.
...
+ score = (int)(src_copied * MAX_SCORE / max_size)
+ - levenshtein(src->path, dst->path);
Does that also mean that you can't ever have a rename with a score of
100%?
(I haven't studied the algorithms and assume that levenshtein(a,b) == 0
only if a==b, and that without the -levenshtein(...) the score can grow
to 100%.)
There is a different code path for identical contents. So yes, you can
still hit 100%, but it is now much, much harder to hit a score close to
100% [*1*].
The obviously correct way to do this is to have a subscore, and use it
_strictly_ only when the score is identical.
I see two ways to do this properly:
- introduce a name_distance struct member, just below the score. This
means that estimate_similarity has to "return" two values instead of
one, and score_compare gets a bit more complex, too. Or
- change the score to unsigned long, and shift the score to higher bits,
adding a constant minus the Levenshtein distance. It is safe to assume
that the filenames are shorter than 16384 bytes (PATH_MAX is actually
much smaller than that), and even if two filenames of that length are
completely different, the distance can not be larger than twice that
number, i.e. 16384 deletions + 16384 insertions. Therefore, you could
pick 32768 as that constant.
However, I find both solutions ugly. Besides, I am not interested in the
feature myself, only the implementation of Levenshtein was interesting,
and I thought I just post the code here. So I did only the minimal stuff
on top of the interesting one to make it sort of work.
If somebody wants to pick up the ball, be my guest, because I am out of
that game.
Ciao,
Dscho
Footnote:
*1* Actually, it is not _that_ bad. The score is not a value between 0 and
100, IOW it is _not_ what you see in the output of "diff -M". It is an
unsigned short between 0 and MAX_SCORE, which is defined in
diffcore.h as 60000.0.
The Levenshtein distance between two filenames cannot be larger than
the sum of their lengths, so it should be relatively safe. That is, if
you don't have such insanely long paths as e.g. egit. But even there,
the paths share most of their directories, and therefore the distances
should be much, much smaller in real life.
From: David Kastrup <hidden> Date: 2016-06-15 22:43:17
Footnote:
*1* Actually, it is not _that_ bad. The score is not a value between 0 and
100, IOW it is _not_ what you see in the output of "diff -M". It is an
unsigned short between 0 and MAX_SCORE, which is defined in
diffcore.h as 60000.0.
The Levenshtein distance between two filenames cannot be larger than
the sum of their lengths, so it should be relatively safe. That is, if
you don't have such insanely long paths as e.g. egit. But even there,
the paths share most of their directories, and therefore the distances
should be much, much smaller in real life.
As a note aside: would it be possible to always round downwards when
computing similarities or converting between them?
I very much would like to see the 100% figure reserved for identity.
This is particularly relevant when interpreting the output of git-diff
--name-status with regard to R100, C100 and similar flags.
--
David Kastrup
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:43:17
Hi,
On Fri, 22 Jun 2007, David Kastrup wrote:
As a note aside: would it be possible to always round downwards when
computing similarities or converting between them?
I'd rather not. This would be counterintuitive. People expect rounded
values.
I very much would like to see the 100% figure reserved for identity.
This is particularly relevant when interpreting the output of git-diff
--name-status with regard to R100, C100 and similar flags.
You should never depend on the output of --name-status if you're
interested in identifying identical files, but on the object names.
Ciao,
Dscho
From: Andy Parkins <hidden> Date: 2016-06-15 22:43:17
On Thursday 2007 June 21, Linus Torvalds wrote:
The files are *identical* for christ sake! Following their history, it
doesn't matter *which* base you follow, since regardless, they've come to
the same point!
So in that sense, the current git behaviour is actually perfectly fine.
Perhaps not. (Please don't read this as meaning I disagree with your
favour-the-identical-filename patch at all - in fact I think that would
address the case I give below).
What if two files with different filenames and content converge at some point
in history, then diverge again? If git is tracking renames merely by content
and picks the wrong one, then the history of fileA suddenly becomes the
history of fileB.
Andy
--
Dr Andy Parkins, M Eng (hons), MIET
andyparkins@gmail.com
From: Johannes Schindelin <hidden> Date: 2016-06-15 22:43:17
Hi,
On Fri, 22 Jun 2007, Andy Parkins wrote:
What if two files with different filenames and content converge at some
point in history, then diverge again? If git is tracking renames merely
by content and picks the wrong one, then the history of fileA suddenly
becomes the history of fileB.
This is becoming highly ethereal. Like "I could imagine that some day in
future, some person could devise a device, that might allow you to do
something that I can not explain, because I have not even thought of it".
IOW show me a reasonable example, and we'll talk business.
Ciao,
Dscho
From: Aidan Van Dyk <hidden> Date: 2016-06-15 22:43:17
* Johannes Schindelin [off-list ref] [070622 13:34]:
Hi,
On Fri, 22 Jun 2007, Andy Parkins wrote:
quoted
What if two files with different filenames and content converge at some
point in history, then diverge again? If git is tracking renames merely
by content and picks the wrong one, then the history of fileA suddenly
becomes the history of fileB.
This is becoming highly ethereal. Like "I could imagine that some day in
future, some person could devise a device, that might allow you to do
something that I can not explain, because I have not even thought of it".
IOW show me a reasonable example, and we'll talk business.
The one time the "content-only" rename tracking bit me was the
after a merge, resulting in conflicts that were un-nessesary:
-*-*-*-*-A-B-C-D
\
*-E-*
At A, there were 2 files:
dir1/foo
dir2/foo
They were template files that happened to be the same in 2 themes.
In E, "foo" was renamed to "foo-bar" in all the template directories.
Git detected this not as 2 renames, but as:
dir1/foo-bar renamed from dir1/foo
dir2/foo-bar copied from dir1/foo
dir2/foo deleted
Meanwhile, work was happening in B, C, and D, changing foo in both
templates identically.
When the branch with E was merged back into ABCD, there was a merge
conflict with dir2/foo being deleted in one branch, and editit in the
other.
In this case, the simple "basename" comparison wouldn't have even been
enough.
But the merge was easy enough (because no edits were made in the E
branch to those files, just the renames) that I could resolve it easily.
I don't know if preventing this easy-to-fix merge conflict is worth the
necessary "likeness of names" necessary to avoid it...
a.
--
Aidan Van Dyk Create like a god,
aidan@highrise.ca command like a king,
http://www.highrise.ca/ work like a slave.