From: Sam Vilain <hidden> Date: 2016-08-11 20:44:15
From: <redacted>
A manual test that sets up a repository that looks like an SVK depot,
and then imports it to check that it looks like we mirrored the
'original' source.
There is also a minor modification to the git-svn test library shell
file which sets a variable for the subversion repository's filesystem
path.
---
t/lib-git-svn.sh | 3 -
t/t9107-git-svn-svk-mirrorpaths.sh | 92 ++++++++++++++++++++++++++++++++++++
2 files changed, 93 insertions(+), 2 deletions(-)
@@ -0,0 +1,92 @@+#!/bin/sh+#+# Copyright (c) 2006 Eric Wong+#++test_description='git-svn on SVK mirror paths'+../lib-git-svn.sh++iftest-n"$GIT_SVN_NO_LIB"&&test"$GIT_SVN_NO_LIB"-ne0+then+echo'Skipping: only implemented with SVN libraries'+test_done+exit0+fi++# ok, people who don't have SVK installed probably don't care about+# this test.++# we set up the repository manually, because even if SVK is installed+# it is difficult to use it in a way that is idempotent.++# we are not yet testing merge tickets..++uuid=b00bface-b1ff-c0ff-f0ff-b0bafe775e1e+url=https://really.slow.server.com/foobar++test_expect_success'initialize repo'"+echo'#!/bin/sh'>$rawsvnrepo/hooks/pre-revprop-change&&+echo'exit 0'>>$rawsvnrepo/hooks/pre-revprop-change&&+chmod+x$rawsvnrepo/hooks/pre-revprop-change&&++mkdirimport&&+cdimport&&+mkdirlocal&&+echohello>local/readme&&+svnimport-m'random local work'.$svnrepo&&+cd..&&++svnco$svnrepowc&&+cdwc&&+mkdir-pmirror/foobar&&+svnaddmirror&&+svnpssvm:source$urlmirror/foobar&&+svnpssvm:uuid$uuidmirror/foobar&&+svnpssvm:mirrormirror/foobar.&&+svncommit-m'setup mirror/foobar as mirror of upstream'&&+svnps-r2--revpropsvm:headrev$uuid:0$svnrepo&&++mkdirmirror/foobar/trunk+echohello,world>mirror/foobar/trunk/readme&&+svnaddmirror/foobar/trunk&&+svncommit-m'first upstream revision'&&+svnps-r3--revpropsvm:headrev$uuid:1$svnrepo&&++svnup&&+svnmkdirmirror/foobar/branches&&+svncpmirror/foobar/trunkmirror/foobar/branches/silly&&+svncommit-m'make branch for silliness'&&+svnps-r4--revpropsvm:headrev$uuid:2$svnrepo&&++svnup&&+echorandomuntestedfeature>>mirror/foobar/trunk/readme&&+svncommit-m'add a c00l feature to trunk'&&+svnps-r5--revpropsvm:headrev$uuid:3$svnrepo&&++svnup&&+echobugfix>>mirror/foobar/branches/silly/readme&&+svncommit-m'fix a bug'&&+svnps-r6--revpropsvm:headrev$uuid:4$svnrepo&&++svnmkdirmirror/foobar/tags&&+svncpmirror/foobar/branches/sillymirror/foobar/tags/blah-1.0&&+svncommit-m'make a release'&&+svnps-r7--revpropsvm:headrev$uuid:5$svnrepo&&++cd..+"++test_expect_success'multi-init an SVK mirror path'"git-svn multi-init -t tags -b branches $svnrepo/mirror/foobar"++test_expect_success'multi-fetch an SVK mirror path'"git-svn multi-fetch"++test_expect_success'got tag history OK'"test \`git-log --pretty=oneline remotes/tags/blah-1.0 | wc -l \` = 3"++test_expect_success're-wrote git-svn-id URL'"git-show HEAD | grep git-svn-id: | fgrep $url"+test_expect_success're-wrote git-svn-id UUID'"git-show HEAD | grep git-svn-id: | fgrep $uuid"+test_expect_success're-wrote git-svn-id revision'"git-show HEAD | grep git-svn-id: | fgrep '@3'"+test_expect_success're-wrote author e-mail domain UUID'"test \`git-show --pretty=fuller HEAD | grep '<.*@.*>' | fgrep $uuid | wc -l\` = 2"++test_debug'gitk --all &'++test_done
From: Sam Vilain <hidden> Date: 2016-08-11 19:30:33
Eric Wong wrote:
Upon further review, this would make 'git svn rebuild' behave
unexpectedly (it would make the git-svn metadata, including .rev_db
entries point to the original repo and not the SVK one). This may not
necessarily be a big deal, however.
Yes, that's the idea; a 'rebuild' should set it up to pull from the
original SVN repository directly. That probably needs some documentation...
Also, incremental fetches (or fetching more than 1k sequential
revisions) would probably fail. To fix this, read the offset of last
entry in .rev_db instead of git-svn-id: from the last commit to get the
last revision. But since rebuild won't work as expected; losing the
.rev_db file means you wouldn't be able to fetch from the SVK repo
anymore (but the original upstream one will be fine).
One last thing: feature should be made optional. I actually work
day-to-day on a repository that was created with svm/SVN::Mirror,
the original repository no longer exists; but the mirrored one
still has these properties (I suppose I could remove the props
server-side, but some people may not have the permissions).
ok, I'll work on that and the other issues you highlighted... possibly
the overhead of fetching the revprops during mirroring might hurt a
little for people not doing this, too. Thanks for reviewing the patch!
Sam.
From: Sam Vilain <hidden> Date: 2016-08-11 19:44:16
From: Sam Vilain <redacted>
Perhaps there is information in the "revision properties" (unversioned
metadata associated with commits) that will affect the way that we
save the revision. Collect them.
---
Sorry for the long lines.
git-svn.perl | 4 ++++
1 files changed, 4 insertions(+), 0 deletions(-)
From: Eric Wong <hidden> Date: 2016-08-11 19:49:18
do_switch works with the SVN Perl bindings after r22312 in the
Subversion trunk. Since no released version of SVN currently
supports it; we'll just autodetect it and enable its usage
when a user has a recent-enough version of SVN.
Signed-off-by: Eric Wong <redacted>
---
Eric Wong [off-list ref] wrote:
For the git-svn in master using the delta fetcher; there's no additional
overhead to fetch properties. I want to ditch the old non-delta
fetching code (it's only a mild performance benefit when using local
repositories) if I could get do_switch() working correctly.
I don't think I can ditch the old code anytime soon. I was tempted to
try using do_diff, but it appears SVN downloads the entire files (using
get_file) and generates the diffs locally, negating any bandwidth
saving it would have over the libsvn_fetch_full() path.
git-svn.perl | 46 +++++++++++++++++++++++++++++++++++++++-------
1 files changed, 39 insertions(+), 7 deletions(-)
@@ -72,7 +72,7 @@ my ($_revision,$_stdin,$_no_ignore_ext,$_no_stop_copy,$_help,$_rmdir,$_edit,$_username,$_config_dir,$_no_auth_cache,$_xfer_delta,$_pager,$_color);my(@_branch_from,%tree_map,%users,%rusers,%equiv);-my($_svn_co_url_revs,$_svn_pg_peg_revs);+my($_svn_co_url_revs,$_svn_pg_peg_revs,$_svn_can_do_switch);my@repo_path_split_cache;my%fc_opts=('no-ignore-externals'=>\$_no_ignore_ext,
@@ -2877,6 +2877,24 @@ sub libsvn_connect {return$ra;}+sublibsvn_can_do_switch{+unless(defined$_svn_can_do_switch){+my$pool=SVN::Pool->new;+my$rep=eval{+$SVN->do_switch(1,'',0,$SVN->{url},+SVN::Delta::Editor->new,$pool);+};+if($@){+$_svn_can_do_switch=0;+}else{+$rep->abort_report($pool);+$_svn_can_do_switch=1;+}+$pool->clear;+}+$_svn_can_do_switch;+}+sublibsvn_dup_ra{my($ra)=@_;SVN::Ra->new(map{$_=>$ra->{$_}}qw/configurl
@@ -3198,12 +3216,26 @@ sub libsvn_find_parent_branch {unlink$GIT_SVN_INDEX;printSTDERR"Found branch parent: ($GIT_SVN) $parent\n";sys(qw/git-read-tree/,$parent);-# I can't seem to get do_switch() to work correctly with-# the SWIG interface (TypeError when passing switch_url...),-# so we'll unconditionally bypass the delta interface here-# for now-returnlibsvn_fetch_full($parent,$paths,$rev,-$author,$date,$msg);+unless(libsvn_can_do_switch()){+returnlibsvn_fetch_full($parent,$paths,$rev,+$author,$date,$msg);+}+# do_switch works with svn/trunk >= r22312, but that is not+# included with SVN 1.4.2 (the latest version at the moment),+# so we can't rely on it.+my$ra=libsvn_connect("$url/$branch_from");+my$ed=SVN::Git::Fetcher->new({c=>$parent,q => $_q});+my$pool=SVN::Pool->new;+my$reporter=$ra->do_switch($rev,'',1,$SVN->{url},+$ed,$pool);+my@lock=$SVN::Core::VERSIONge'1.2.0'?(undef):();+$reporter->set_path('',$r0,0,@lock,$pool);+$reporter->finish_report($pool);+$pool->clear;+unless($ed->{git_commit_ok}){+die"SVN connection failed somewhere...\n";+}+returnlibsvn_log_entry($rev,$author,$date,$msg,[$parent]);}printSTDERR"Nope, branch point not imported or unknown\n";returnundef;
From: Eric Wong <hidden> Date: 2016-08-11 19:51:23
Sam Vilain [off-list ref] wrote:
quoted hunk
From: <redacted>
A manual test that sets up a repository that looks like an SVK depot,
and then imports it to check that it looks like we mirrored the
'original' source.
There is also a minor modification to the git-svn test library shell
file which sets a variable for the subversion repository's filesystem
path.
---
t/lib-git-svn.sh | 3 -
t/t9107-git-svn-svk-mirrorpaths.sh | 92 ++++++++++++++++++++++++++++++++++++
2 files changed, 93 insertions(+), 2 deletions(-)
@@ -0,0 +1,92 @@+#!/bin/sh+#+# Copyright (c) 2006 Eric Wong
Huh? I didn't write this test.
+test_description='git-svn on SVK mirror paths'
+. ./lib-git-svn.sh
+
+if test -n "$GIT_SVN_NO_LIB" && test "$GIT_SVN_NO_LIB" -ne 0
+then
+ echo 'Skipping: only implemented with SVN libraries'
+ test_done
+ exit 0
+fi
+
+# ok, people who don't have SVK installed probably don't care about
+# this test.
+
+# we set up the repository manually, because even if SVK is installed
+# it is difficult to use it in a way that is idempotent.
+
+# we are not yet testing merge tickets..
+
+uuid=b00bface-b1ff-c0ff-f0ff-b0bafe775e1e
+url=https://really.slow.server.com/foobar
I had to use --whitespace=strip with git-am above.
+test_expect_success 'multi-fetch an SVK mirror path' "git-svn multi-fetch"
I had to use the following patch to get the multi-fetch test to pass
with GIT_SVN_DELTA_FETCH=1. (I discovered it while running make -C t
full-svn-test)
Is it safe to assume that svk-mirrored URLs will _always_ be file://?
If so, then the delta fetching code should never be needed.
--- a/git-svn.perl+++ b/git-svn.perl
@@ -3017,8 +3017,7 @@ sub libsvn_fetch_delta {my$ed=SVN::Git::Fetcher->new({c=>$last_commit,q => $_q});my$reporter=$SVN->do_update($rev,'',1,$ed,$pool);my@lock=$SVN::Core::VERSIONge'1.2.0'?(undef):();-my(undef,$last_rev,undef)=cmt_metadata($last_commit);-$reporter->set_path('',$last_rev,0,@lock,$pool);+$reporter->set_path('',($rev-1),0,@lock,$pool);$reporter->finish_report($pool);$pool->clear;unless($ed->{git_commit_ok}){
Also, unconditionally setting ($rev - 1) in the above patch disables the
ability to do squashed history[1] imports (not sure if anybody cares about
them, though).
[1] - like shallow clone, but shallow in the middle of history
+test_expect_success 'got tag history OK' "test \`git-log --pretty=oneline remotes/tags/blah-1.0 | wc -l \` = 3"
From: Sam Vilain <hidden> Date: 2016-08-11 19:54:32
From: Sam Vilain <redacted>
If an SVN revision has a property, "svm:headrev", it is likely that
the revision was created by "svk sync". The property contains a
repository UUID and a revision. We want to make it look like we are
mirroring the original URL, so introduce a helper function that
returns the original identity trio, and use it when generating commit
messages and dummy e-mail domains.
---
git-svn.perl | 31 +++++++++++++++++++++++++++----
1 files changed, 27 insertions(+), 4 deletions(-)
@@ -2109,14 +2110,36 @@ sub get_svm_url {chomp($SVM_UUID=`git-repo-config --get svn.svkuuid`);}+subsvn_commit_id{+my$log_msg=shift;+my($url,$uuid,$rev)=($SVN_URL,$SVN_UUID,$log_msg->{revision});+my$svm_headrev=$log_msg->{revprops}{'svm:headrev'};+if($svm_headrev){+my($_uuid,$_rev)=split/:/,$svm_headrev;+chomp($_rev);+if(!$SVM_URL){+get_svm_url();+}+if($_uuidne$SVM_UUID){+warn"$uuid:$rev claims to be $_uuid:$_rev, but that's unknown";+}else{+($url,$uuid,$rev)=($SVM_URL,$SVM_UUID,$_rev);+}+}+($url,$uuid,$rev);+}+subset_commit_env{my($log_msg)=@_;my$author=$log_msg->{author};if(!defined$author||length$author==0){$author='(no author)';}-my($name,$email)=defined$users{$author}?@{$users{$author}}-:($author,"$author\@$SVN_UUID");+my($name,$email)=defined$users{$author}?@{$users{$author}}+:do{+my(undef,$uuid,undef)=svn_commit_id($log_msg);+($author,"$author\@$uuid")+};$ENV{GIT_AUTHOR_NAME}=$ENV{GIT_COMMITTER_NAME}=$name;$ENV{GIT_AUTHOR_EMAIL}=$ENV{GIT_COMMITTER_EMAIL}=$email;$ENV{GIT_AUTHOR_DATE}=$ENV{GIT_COMMITTER_DATE}=$log_msg->{date};
From: Sam Vilain <hidden> Date: 2016-08-11 19:56:04
From: Sam Vilain <redacted>
Allow an extra parameter to be passed to the libsvn_ls_fullurl
function to collect and return the properties of the URL being listed.
---
git-svn.perl | 7 ++++---
1 files changed, 4 insertions(+), 3 deletions(-)
@@ -3321,18 +3321,19 @@ sub libsvn_commit_cb {sublibsvn_ls_fullurl{my$fullurl=shift;+my$want_props=shift;my$ra=libsvn_connect($fullurl);-my@ret;+my(@ret,@props);my$pool=SVN::Pool->new;my$r=defined$_revision?$_revision:$ra->get_latest_revnum;-my($dirent,undef,undef)=$ra->get_dir('',$r,$pool);+my($dirent,undef,$props)=$ra->get_dir('',$r,$pool);foreachmy$d(keys%$dirent){if($dirent->{$d}->kind==$SVN::Node::dir){push@ret,"$d/";# add '/' for compat with cli svn}}$pool->clear;-return@ret;+return($want_props?(\@ret,$props):@ret);}
From: Eric Wong <hidden> Date: 2016-08-11 20:03:59
Sam Vilain [off-list ref] wrote:
Eric Wong wrote:
quoted
Also, incremental fetches (or fetching more than 1k sequential
revisions) would probably fail. To fix this, read the offset of last
entry in .rev_db instead of git-svn-id: from the last commit to get the
last revision. But since rebuild won't work as expected; losing the
.rev_db file means you wouldn't be able to fetch from the SVK repo
anymore (but the original upstream one will be fine).
One last thing: feature should be made optional. I actually work
day-to-day on a repository that was created with svm/SVN::Mirror,
the original repository no longer exists; but the mirrored one
still has these properties (I suppose I could remove the props
server-side, but some people may not have the permissions).
ok, I'll work on that and the other issues you highlighted... possibly
the overhead of fetching the revprops during mirroring might hurt a
little for people not doing this, too. Thanks for reviewing the patch!
For the git-svn in master using the delta fetcher; there's no additional
overhead to fetch properties. I want to ditch the old non-delta
fetching code (it's only a mild performance benefit when using local
repositories) if I could get do_switch() working correctly.
--
From: Eric Wong <hidden> Date: 2016-08-11 20:31:56
Sam Vilain [off-list ref] wrote:
From: Sam Vilain <redacted>
If an SVN revision has a property, "svm:headrev", it is likely that
the revision was created by "svk sync". The property contains a
repository UUID and a revision. We want to make it look like we are
mirroring the original URL, so introduce a helper function that
returns the original identity trio, and use it when generating commit
messages and dummy e-mail domains.
Upon further review, this would make 'git svn rebuild' behave
unexpectedly (it would make the git-svn metadata, including .rev_db
entries point to the original repo and not the SVK one). This may not
necessarily be a big deal, however.
Also, incremental fetches (or fetching more than 1k sequential
revisions) would probably fail. To fix this, read the offset of last
entry in .rev_db instead of git-svn-id: from the last commit to get the
last revision. But since rebuild won't work as expected; losing the
.rev_db file means you wouldn't be able to fetch from the SVK repo
anymore (but the original upstream one will be fine).
One last thing: feature should be made optional. I actually work
day-to-day on a repository that was created with svm/SVN::Mirror,
the original repository no longer exists; but the mirrored one
still has these properties (I suppose I could remove the props
server-side, but some people may not have the permissions).
--
From: Sam Vilain <hidden> Date: 2016-08-11 20:43:38
From: Sam Vilain <redacted>
If you use git-svn to import a mirror path within an SVK depot
directly (eg, file:///home/you/.svk/local/mirror/foo), then the URLs
and revisions in the generated commits will be of the wrong URL.
When we set up with git-svn multi-init, check whether the base URL is
(the root of) a mirror path, and store it for later. Set up a couple
of globals and helper functions for later use.
---
git-svn.perl | 21 +++++++++++++++++++++
1 files changed, 21 insertions(+), 0 deletions(-)
@@ -5,6 +5,7 @@ use warnings;usestrict;usevarsqw/$AUTHOR$VERSION$SVN_URL$SVN_INFO$SVN_WC$SVN_UUID+$SVM_URL$SVM_UUID$GIT_SVN_INDEX$GIT_SVN$GIT_DIR$GIT_SVN_DIR$REVDB/;$AUTHOR='Eric Wong <normalperson@yhbt.net>';
@@ -733,6 +734,21 @@ sub multi_init {init($_trunk);sys('git-repo-config','svn.trunk',$_trunk);}+if($url){+# check for the case of SVK mirror path+my($ents,$props)=libsvn_ls_fullurl($url,"1");+if(my$src=$props->{'svm:source'}){+$src=~s{!$}{};# don't know wtf a ! is there for+$src=~s{(^[a-z\+]*://)[^/@]*@}{$1};# username of no interest++# store the source as a repo-config item+sys('git-repo-config','svn.svkmirrorpath',$src);+my$uuid=$props->{'svm:uuid'};+$uuid=~m{^[0-9a-f\-]{36,}}+orcroak"doesn't look right - svm:uuid is '$uuid'";+sys('git-repo-config','svn.svkuuid',$uuid);+}+}complete_url_ls_init($url,$_branches,'--branches/-b','');complete_url_ls_init($url,$_tags,'--tags/-t','tags/');}