From: Clemens Buchacher <hidden> Date: 2016-06-15 22:49:34
Hi,
I finally came around to de-bashify my patch ID test script. Here we go.
[PATCH 1/3] add rebase patch id tests
[PATCH 2/3] do not search functions for patch ID
[PATCH 3/3] use cache for function names in hunk headers
The first patch adds correctness and (optional) performance tests for the patch
"hash binary sha1 into patch id"
http://thread.gmane.org/gmane.comp.version-control.git/153468/focus=155919 .
The test reveals a performance problem with the search for function names for
the hunk headers. This is fixed for patch ID computation by the second patch
and for diff in general by the third patch.
Clemens
---
diff.c | 2 +-
t/t3419-rebase-patch-id.sh | 109 ++++++++++++++++++++++++++++++++++++++++++++
xdiff/xemit.c | 44 +++++++++++++-----
3 files changed, 142 insertions(+), 13 deletions(-)
From: Clemens Buchacher <hidden> Date: 2016-06-15 22:49:34
Visual aids, such as the function name in the hunk
header, are not necessary for the purposes of
computing a patch ID.
This is a performance optimization.
Signed-off-by: Clemens Buchacher <redacted>
---
diff.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
@@ -0,0 +1,109 @@+#!/bin/bash++test_description='git rebase - test patch id computation'++../test-lib.sh++test_set_prereqNOT_EXPENSIVE+test-n"$GIT_PATCHID_TIMING_TESTS"&&test_set_prereqEXPENSIVE+test-x/usr/bin/time&&test_set_prereqUSR_BIN_TIME++count()+{+i=0+whiletest$i-lt$1+do+echo"$i"+i=$(($i+1))+done+}++scramble()+{+i=0+whilereadx+do+iftest$i-ne0+then+echo"$x"+fi+i=$(((i+1)%10))+done<"$1">"$1.new"+mv-f"$1.new""$1"+}++run()+{+echo\$"$@"+/usr/bin/time"$@">/dev/null+}++test_expect_success'setup''+gitcommit--allow-empty-minitial+gittagroot+'++do_tests()+{+pr=$1+nlines=$2++test_expect_success$pr"setup: $nlines lines""+rm-f.gitattributes&&+gitcheckout-q-fmaster&&+gitreset--hardroot&&+count$nlines>file&&+gitaddfile&&+gitcommit-q-minitial&&+gitbranch-fother&&++scramblefile&&+gitaddfile&&+gitcommit-q-m'change big file'&&++gitcheckout-qother&&+:>newfile&&+gitaddnewfile&&+gitcommit-q-m'add small file'&&++gitcherry-pickmaster>/dev/null2>&1+"++test_debug"+rungitdiffmaster^\!+"++test_expect_success$pr'setup attributes'"+echo'file binary'>.gitattributes+"++test_debug"+rungitformat-patch--stdoutmaster&&+rungitformat-patch--stdout--ignore-if-in-upstreammaster+"++test_expect_success$pr'detect upstream patch'"+gitcheckout-qmaster&&+scramblefile&&+gitaddfile&&+gitcommit-q-m'change big file again'&&+gitcheckout-qother^{}&&+gitrebasemaster&&+test_must_failtest-n\"\$(gitrev-listmaster...HEAD~)\"+"++test_expect_success$pr'do not drop patch'"+gitbranch-fsquashedmaster&&+gitcheckout-q-fsquashed&&+gitreset-q--softHEAD~2&&+gitcommit-q-msquashed&&+gitcheckout-qother^{}&&+test_must_failgitrebasesquashed&&+rm-rf.git/rebase-apply+"+}++do_testsNOT_EXPENSIVE500+do_testsEXPENSIVE50000++test_done
From: Clemens Buchacher <hidden> Date: 2016-06-15 22:49:34
For each hunk, xdl_find_func searches the preimage
for a function name until the beginning of the
file. If the file does not contain any function
names, this search has complexity O(n^2) in the
number of hunks n.
Instead of searching the entire file for each hunk
individually, cache and reuse the search result
from previous hunks.
Diff performance for the 50000 line test in t3419
before and after this patch:
2.78user 0.01system 0:02.82elapsed 99%CPU
0.05user 0.01system 0:00.06elapsed 96%CPU
Signed-off-by: Clemens Buchacher <redacted>
---
xdiff/xemit.c | 44 ++++++++++++++++++++++++++++++++------------
1 files changed, 32 insertions(+), 12 deletions(-)
@@ -85,8 +85,15 @@ static long def_ff(const char *rec, long len, char *buf, long sz, void *priv)return-1;}-staticvoidxdl_find_func(xdfile_t*xf,longi,char*buf,longsz,long*ll,-find_func_tff,void*ff_priv){+structxdl_find_func_cache{+charbuf[80];+longlen;+xdfile_t*xf;+intline;+};++staticvoidxdl_find_func(xdfile_t*xf,longline,find_func_tff,+void*ff_priv,structxdl_find_func_cache*cache){/**Bequitestupidaboutthisfornow.Findalineintheoldfile
@@ -96,13 +103,28 @@ static void xdl_find_func(xdfile_t *xf, long i, char *buf, long sz, long *ll,constchar*rec;longlen;+inti,l;-while(i-->0){-len=xdl_get_rec(xf,i,&rec);-if((*ll=ff(rec,len,buf,sz,ff_priv))>=0)+if(line<cache->line)+cache->xf=0;++i=line;+l=-1;+while(--i>=0&&l<0){+if(xf==cache->xf&&i<cache->line){+cache->line=line;return;+}++len=xdl_get_rec(xf,i,&rec);+l=ff(rec,len,cache->buf,sizeof(cache->buf),ff_priv);}-*ll=0;+if(l<0)+l=0;++cache->xf=xf;+cache->len=l;+cache->line=line;}
Very nice. It would improve the commit message if you could explain
what exactly it is this optimizes though, saving the reader from
having to read through t3419 to find out.
--
Cheers,
Sverre Rabbelier
Very nice. It would improve the commit message if you could explain
what exactly it is this optimizes though, saving the reader from
having to read through t3419 to find out.
Ok.
The test creates a file with 50000 lines and a one-line change
every 10 lines, i.e., about 5000 hunks. Since none of the lines
matches a function definition, previous to this optimization, the
file was searched 5000 times.
Clemens
Heya,
On Mon, Sep 20, 2010 at 19:36, Clemens Buchacher [off-list ref] wrote:
The test creates a file with 50000 lines and a one-line change
every 10 lines, i.e., about 5000 hunks. Since none of the lines
matches a function definition, previous to this optimization, the
file was searched 5000 times.
Ah, that makes sense, please add (something like) that to the commit message :).
--
Cheers,
Sverre Rabbelier
From: Clemens Buchacher <hidden> Date: 2016-06-15 22:49:36
For each hunk, xdl_find_func searches the preimage
for a function name until the beginning of the
file. If the file does not contain any function
names, this search has complexity O(n^2) in the
number of hunks n.
The timing test in t3419 creates a file with 50000
lines and a one-line change every 10 lines, i.e.,
about 5000 hunks. Since none of the lines matches
a function definition the file is searched 5000
times.
Instead of searching the entire file for each hunk
individually, cache and reuse the search result
from previous hunks.
Diff performance for the test described above
before and after this optimization:
2.78user 0.01system 0:02.82elapsed 99%CPU
0.05user 0.01system 0:00.06elapsed 96%CPU
Signed-off-by: Clemens Buchacher <redacted>
---
I have added the test description as requested by
Sverre. There are no code changes with respect to
the previous version of the patch.
The test scenario might seem quite obscure. But I
have this case in some text files which are not
edited directly by humans.
Clemens
xdiff/xemit.c | 44 ++++++++++++++++++++++++++++++++------------
1 files changed, 32 insertions(+), 12 deletions(-)
@@ -85,8 +85,15 @@ static long def_ff(const char *rec, long len, char *buf, long sz, void *priv)return-1;}-staticvoidxdl_find_func(xdfile_t*xf,longi,char*buf,longsz,long*ll,-find_func_tff,void*ff_priv){+structxdl_find_func_cache{+charbuf[80];+longlen;+xdfile_t*xf;+intline;+};++staticvoidxdl_find_func(xdfile_t*xf,longline,find_func_tff,+void*ff_priv,structxdl_find_func_cache*cache){/**Bequitestupidaboutthisfornow.Findalineintheoldfile
@@ -96,13 +103,28 @@ static void xdl_find_func(xdfile_t *xf, long i, char *buf, long sz, long *ll,constchar*rec;longlen;+inti,l;-while(i-->0){-len=xdl_get_rec(xf,i,&rec);-if((*ll=ff(rec,len,buf,sz,ff_priv))>=0)+if(line<cache->line)+cache->xf=0;++i=line;+l=-1;+while(--i>=0&&l<0){+if(xf==cache->xf&&i<cache->line){+cache->line=line;return;+}++len=xdl_get_rec(xf,i,&rec);+l=ff(rec,len,cache->buf,sizeof(cache->buf),ff_priv);}-*ll=0;+if(l<0)+l=0;++cache->xf=xf;+cache->len=l;+cache->line=line;}
From: René Scharfe <hidden> Date: 2016-06-15 22:49:37
Am 23.09.2010 09:04, schrieb Clemens Buchacher:
quoted hunk
For each hunk, xdl_find_func searches the preimage
for a function name until the beginning of the
file. If the file does not contain any function
names, this search has complexity O(n2) in the
number of hunks n.
The timing test in t3419 creates a file with 50000
lines and a one-line change every 10 lines, i.e.,
about 5000 hunks. Since none of the lines matches
a function definition the file is searched 5000
times.
Instead of searching the entire file for each hunk
individually, cache and reuse the search result
from previous hunks.
Diff performance for the test described above
before and after this optimization:
2.78user 0.01system 0:02.82elapsed 99%CPU
0.05user 0.01system 0:00.06elapsed 96%CPU
Signed-off-by: Clemens Buchacher <redacted>
---
I have added the test description as requested by
Sverre. There are no code changes with respect to
the previous version of the patch.
The test scenario might seem quite obscure. But I
have this case in some text files which are not
edited directly by humans.
Clemens
xdiff/xemit.c | 44 ++++++++++++++++++++++++++++++++------------
1 files changed, 32 insertions(+), 12 deletions(-)
@@ -85,8 +85,15 @@ static long def_ff(const char *rec, long len, char *buf, long sz, void *priv)return-1;}-staticvoidxdl_find_func(xdfile_t*xf,longi,char*buf,longsz,long*ll,-find_func_tff,void*ff_priv){+structxdl_find_func_cache{+charbuf[80];+longlen;+xdfile_t*xf;+intline;+};
Is xf needed? Does xdl_emit_diff() handle multiple files in one go?
If you inline xdl_find_func() the struct isn't needed anymore.
quoted hunk
+
+static void xdl_find_func(xdfile_t *xf, long line, find_func_t ff,
+ void *ff_priv, struct xdl_find_func_cache *cache) {
/*
* Be quite stupid about this for now. Find a line in the old file
@@ -96,13 +103,28 @@ static void xdl_find_func(xdfile_t *xf, long i, char *buf, long sz, long *ll, const char *rec; long len;+ int i, l;- while (i-- > 0) {- len = xdl_get_rec(xf, i, &rec);- if ((*ll = ff(rec, len, buf, sz, ff_priv)) >= 0)+ if (line < cache->line)+ cache->xf = 0;
NULL is preferred.
quoted hunk
+
+ i = line;
+ l = -1;
+ while (--i >= 0 && l < 0) {
+ if (xf == cache->xf && i < cache->line) {
+ cache->line = line;
return;
+ }
+
+ len = xdl_get_rec(xf, i, &rec);
+ l = ff(rec, len, cache->buf, sizeof(cache->buf), ff_priv);
}
- *ll = 0;
+ if (l < 0)
+ l = 0;
+
+ cache->xf = xf;
+ cache->len = l;
+ cache->line = line;
}
How about something like this? It also removes an outdated comment. The
inlining part should probably split out in its own patch..
xdiff/xemit.c | 38 ++++++++++++++------------------------
1 files changed, 14 insertions(+), 24 deletions(-)
From: Clemens Buchacher <hidden> Date: 2016-06-15 22:49:37
On Sun, Sep 26, 2010 at 06:26:56PM +0200, René Scharfe wrote:
Is xf needed? Does xdl_emit_diff() handle multiple files in one go?
Right now it does not.
If you inline xdl_find_func() the struct isn't needed anymore.
[...]
How about something like this?
Yes, that's better. Thanks.
- /*
- * Be quite stupid about this for now. Find a line in the old file
- * before the start of the hunk (and context) which starts with a
- * plausible character.
- */
It also removes an outdated comment.
Actually, in the default case, the comment is still correct and
helpful IMO.
The inlining part should probably split out in its own patch..
I am not sure what you mean here. Do you want to add the parameter
funclineprev to the function and remove the function in the next
page, or do you want to refactor the below into an inline function?
Clemens
From: René Scharfe <hidden> Date: 2016-06-15 22:49:37
Am 26.09.2010 22:43, schrieb Clemens Buchacher:
On Sun, Sep 26, 2010 at 06:26:56PM +0200, René Scharfe wrote:
quoted
- /*
- * Be quite stupid about this for now. Find a line in the old file
- * before the start of the hunk (and context) which starts with a
- * plausible character.
- */
It also removes an outdated comment.
Actually, in the default case, the comment is still correct and
helpful IMO.
The first sentence was probably written before the match function became
configurable. The second one could be moved to def_ff().
quoted
The inlining part should probably split out in its own patch..
I am not sure what you mean here. Do you want to add the parameter
funclineprev to the function and remove the function in the next
page, or do you want to refactor the below into an inline function?
I'd suggest having a first patch for folding xdl_find_func() into
xdl_emit_diff() without any functional change (manually inlining it) and
a second one for adding funclineprev etc..
René