Re: git-svn performance
From: Hin-Tak Leung <hidden>
Date: 2016-06-15 23:02:46
------------------------------ On Tue, Oct 21, 2014 10:00 BST Eric Wong wrote:
quoted hunk
Jakob Stoklund Olesen [off-list ref] wrote:quoted
Yes, but I think you can remove cached_mergeinfo_rev too.Thanks, pushed the patch at the bottom, too. Also started working on some memory reductions here: http://mid.gmane.org/20141021033912.GA27462@dcvr.yhbt.net But there seem to be more problems :< ----------------------------8<----------------------------- From: Eric Wong <redacted> Date: Tue, 21 Oct 2014 06:23:22 +0000 Subject: [PATCH] git-svn: remove mergeinfo rev caching This should further reduce memory usage from the new mergeinfo speedups without hurting performance too much, assuming reasonable latency to the SVN server. Cc: Hin-Tak Leung <redacted> Suggested-by: Jakob Stoklund Olesen <redacted> Signed-off-by: Eric Wong <redacted> --- perl/Git/SVN.pm | 30 +++++++++--------------------- 1 file changed, 9 insertions(+), 21 deletions(-)diff --git a/perl/Git/SVN.pm b/perl/Git/SVN.pm index f8a75b1..4364506 100644 --- a/perl/Git/SVN.pm +++ b/perl/Git/SVN.pm@@ -1710,32 +1710,20 @@ sub mergeinfo_changes {my %minfo = map {split ":", $_ } split "\n", $mergeinfo_prop; my $old_minfo = {}; - # Initialize cache on the first call. - unless (defined $self->{cached_mergeinfo_rev}) { - $self->{cached_mergeinfo_rev} = {}; - } - - my $cached_rev = $self->{cached_mergeinfo_rev}{$old_path}; - unless (defined $cached_rev && $cached_rev == $old_rev) { - my $ra = $self->ra; - # Give up if $old_path isn't in the repo. - # This is probably a merge on a subtree. - if ($ra->check_path($old_path, $old_rev) != $SVN::Node::dir) { - warn "W: ignoring svn:mergeinfo on $old_path, ", - "directory didn't exist in r$old_rev\n"; - return {}; - } - } - my (undef, undef, $props) = $self->ra->get_dir($old_path, $old_rev); + my $ra = $self->ra; + # Give up if $old_path isn't in the repo. + # This is probably a merge on a subtree. + if ($ra->check_path($old_path, $old_rev) != $SVN::Node::dir) { + warn "W: ignoring svn:mergeinfo on $old_path, ", + "directory didn't exist in r$old_rev\n"; + return {}; + } + my (undef, undef, $props) = $ra->get_dir($old_path, $old_rev); if (defined $props->{"svn:mergeinfo"}) { my %omi = map {split ":", $_ } split "\n", $props->{"svn:mergeinfo"}; $old_minfo = \%omi; } - $self->{cached_mergeinfo_rev}{$old_path} = $old_rev; - - # Cache the new mergeinfo. - $self->{cached_mergeinfo_rev}{$path} = $rev; my %changes = (); foreach my $p (keys %minfo) { -- EW
I'll have a look at the new changes at some point - I am still keeping the old
clone and the new clone and just fetching from time to time to keep them
in sync. I just tried that and fetching the same 50 commits on the old clone
took 1.7 GB memory vs 1.0 GB memory on the new. Details below.
This is just with the 2 earliest patches - I'll put the new 3 in at some point.
So I see some needs for retrospectively fixing old clones (maybe as part
of garbage collection?), since most would simply use an old clone through
the ages...
Comparing trunk of old and new, I see one difference - One short
commit message is missing in the *old* (the "Add checkPoFiles etc." part)
and so all the sha1 afterwards differed. Is that an old bug that's fixed
and therefore I should throw away the old clone?
Date: Wed Apr 25 18:21:29 2012 +0000
Add checkPoFiles etc.
git-svn-id: https://svn.r-project.org/R/trunk@59188
Here is the details of fetching old and new:
<---
$ /usr/bin/time -v git svn fetch --all
M doc/manual/R-admin.texi
r66784 = fc20374f26f8e03bb88c00933982e29138a6f929 (refs/remotes/trunk)
...
M configure
r66834 = d8d1876f6aa71b3fe3773cd28a760ff945d30bdf (refs/remotes/R-3-1-branch)
Command being timed: "git svn fetch --all"
User time (seconds): 1520.77
System time (seconds): 156.32
Percent of CPU this job got: 98%
Elapsed (wall clock) time (h:mm:ss or m:ss): 28:15.82
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 1738276
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 613
Minor (reclaiming a frame) page faults: 2039305
Voluntary context switches: 11243
Involuntary context switches: 181507
Swaps: 0
File system inputs: 658328
File system outputs: 754688
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
$ cd ../R-2/
[Hin-Tak@localhost R-2]$ /usr/bin/time -v git svn fetch --all
M doc/manual/R-admin.texi
r66784 = 6a08d94b456d33d85add914a1b780a972689443a (refs/remotes/trunk)
...
M configure
r66834 = 370a6484c2a65be78dfae184b50d8f08685d389c (refs/remotes/R-3-1-branch)
Command being timed: "git svn fetch --all"
User time (seconds): 1507.89
System time (seconds): 134.25
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 27:38.49
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 1026656
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 1110
Minor (reclaiming a frame) page faults: 1630150
Voluntary context switches: 10280
Involuntary context switches: 176444
Swaps: 0
File system inputs: 361472
File system outputs: 477912
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
---->