After the recent improvements to git-remote-mediawiki, I did a bit of
real-life testing, by importing https://ensiwiki.ensimag.fr/ and
https://git.wiki.kernel.org/. It turned out we had a few blocking bugs
and a grossly unoptimized algorithm on push. This patch serie makes
git-remote-mediawiki usable on these wikis (although it's still a bit
slow).
My initial patch serie also contained a patch "git-remote-mediawiki:
replace TODO-list in comment by appropriate link" that replaced the
TODO-list in the comments of the file with just
# Gateway between Git and MediaWiki.
# Documentation & bugtracker: https://github.com/moy/Git-Mediawiki/
but I'm holding it for now as there seems to be interest in changing
the homepage, so the link may not be appropriate.
Matthieu Moy (8):
git-remote-mediawiki: don't split namespaces with spaces
git-remote-mediawiki: actually send empty comment when they're empty
git-remote-mediawiki: make mediafiles export optional
git-remote-mediawiki: get rid of O(N^2) loop
git-remote-mediawiki: use --force when adding notes
git-remote-mediawiki: show progress information when listing pages
git-remote-mediawiki: show progress information when getting last
remote revision
git-remote-mediawiki: properly deal with invalid remote revisions
contrib/mw-to-git/git-remote-mediawiki | 80 +++++++++++++++++++++++++++-------
1 file changed, 64 insertions(+), 16 deletions(-)
--
1.7.11.1.30.g7e1baf9.dirty
@@ -51,6 +51,9 @@ use constant EMPTY_CONTENT => "<!-- empty page -->\n"; # used to reflect file creation or deletion in diff. use constant NULL_SHA1 => "0000000000000000000000000000000000000000";+# Used on Git's side to reflect empty edit messages on the wiki+use constant EMPTY_MESSAGE => '*Empty MediaWiki Message*';+ my $remotename = $ARGV[0]; my $url = $ARGV[1];
When notes are created to record a push, it normally doesn't exist yet.
However, when a push is interrupted and then restarted, it may happen
that a commit already has notes attached, and we want to reflect the newly
created remote revision, hence use 'git notes add -f' to override the
existing one
Signed-off-by: Matthieu Moy <redacted>
---
contrib/mw-to-git/git-remote-mediawiki | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
Initial phases of push and pull with git-remote-mediawiki can be long on
a large wiki. Let the user know what's going on.
Signed-off-by: Matthieu Moy <redacted>
---
contrib/mw-to-git/git-remote-mediawiki | 3 +++
1 file changed, 3 insertions(+)
@@ -371,6 +371,8 @@ sub get_mw_first_pages { sub get_mw_pages { mw_connect_maybe();+ print STDERR "Listing pages on remote wiki...\n";+ my %pages; # hash on page titles to avoid duplicates my $user_defined; if (@tracked_pages) {
The algorithm to find a path from the local revision to the remote one
was calling "git rev-list" and parsing its output N times. Run rev-list
only once, and fill a hashtable with the result to optimize the body of
the loop.
Signed-off-by: Matthieu Moy <redacted>
---
contrib/mw-to-git/git-remote-mediawiki | 25 ++++++++++++++++++-------
1 file changed, 18 insertions(+), 7 deletions(-)
@@ -1196,16 +1196,27 @@ sub mw_push_revision { if ($last_local_revid > 0) { my $parsed_sha1 = $remoteorigin_sha1; # Find a path from last MediaWiki commit to pushed commit+ print STDERR "Computing path from local to remote ...\n";+ my @local_ancestry = split(/\n/, run_git("rev-list --boundary --children $local ^$parsed_sha1"));+ my %local_ancestry;+ foreach my $line (@local_ancestry) {+ if (my ($parent, $child) = $line =~ m/^-?([a-f0-9]+) ([a-f0-9]+)/) {+ $local_ancestry{$parent} = $child;+ if ($parent eq $parsed_sha1 || $child eq $parsed_sha1) {+ print STDERR "$parent -> $child\n";+ }+ } elsif (!$line =~ m/^([a-f0-9]+)/) {+ die "Unexpected output from git rev-list: $line";+ }+ } while ($parsed_sha1 ne $HEAD_sha1) {- my @commit_info = grep(/^$parsed_sha1/, split(/\n/, run_git("rev-list --children $local")));- if (!@commit_info) {+ my $child = $local_ancestry{$parsed_sha1};+ if (!$child) {+ printf STDERR "Cannot find a path in history from remote commit to last commit\n"; return error_non_fast_forward($remote); }- my @commit_info_split = split(/ |\n/, $commit_info[0]);- # $commit_info_split[1] is the sha1 of the commit to export- # $commit_info_split[0] is the sha1 of its direct child- push(@commit_pairs, \@commit_info_split);- $parsed_sha1 = $commit_info_split[1];+ push(@commit_pairs, [$parsed_sha1, $child]);+ $parsed_sha1 = $child; } } else { # No remote mediawiki revision. Export the whole
@@ -1275,7 +1275,7 @@ sub get_mw_namespace_id { # Look at configuration file, if the record for that namespace is # already cached. Namespaces are stored in form: # "Name_of_namespace:Id_namespace", ex.: "File:6".- my @temp = split(/[ \n]/, run_git("config --get-all remote."+ my @temp = split(/[\n]/, run_git("config --get-all remote." . $remotename .".namespaceCache")); chomp(@temp); foreach my $ns (@temp) {
Some wiki, including https://git.wiki.kernel.org/ have invalid revision
numbers (i.e. the actual revision numbers are non-contiguous). Don't die
when encountering one.
Signed-off-by: Matthieu Moy <redacted>
---
contrib/mw-to-git/git-remote-mediawiki | 13 +++++++++----
1 file changed, 9 insertions(+), 4 deletions(-)
@@ -910,6 +910,10 @@ sub mw_import_revids { my $last_timestamp = 0; # Placeholer in case $rev->timestamp is undefined foreach my $pagerevid (@$revision_ids) {+ # Count page even if we skip it, since we display+ # $n/$total and $total includes skipped pages.+ $n++;+ # fetch the content of the pages my $query = { action => 'query',
@@ -924,6 +928,11 @@ sub mw_import_revids { die "Failed to retrieve modified page for revision $pagerevid"; }+ if (defined($result->{query}->{badrevids}->{$pagerevid})) {+ # The revision id does not exist on the remote wiki.+ next;+ }+ if (!defined($result->{query}->{pages})) { die "Invalid revision $pagerevid."; }
@@ -932,10 +941,6 @@ sub mw_import_revids { my $result_page = $result_pages[0]; my $rev = $result_pages[0]->{revisions}->[0];- # Count page even if we skip it, since we display- # $n/$total and $total includes skipped pages.- $n++;- my $page_title = $result_page->{title}; if (!exists($pages->{$page_title})) {
@@ -579,6 +579,8 @@ sub get_last_remote_revision { my $max_rev_num = 0;+ print STDERR "Getting last revision id on tracked pages...\n";+ foreach my $page (@pages) { my $id = $page->{pageid};
Changes since v1: more explanations in commit message for "make
mediafiles export optional", and use --parents instead of --children
in "get rid of O(N^2) loop".
Matthieu Moy (8):
git-remote-mediawiki: don't split namespaces with spaces
git-remote-mediawiki: actually send empty comment when they're empty
git-remote-mediawiki: make mediafiles export optional
git-remote-mediawiki: get rid of O(N^2) loop
git-remote-mediawiki: use --force when adding notes
git-remote-mediawiki: show progress information when listing pages
git-remote-mediawiki: show progress information when getting last
remote revision
git-remote-mediawiki: properly deal with invalid remote revisions
contrib/mw-to-git/git-remote-mediawiki | 79 +++++++++++++++++++++++++++-------
1 file changed, 63 insertions(+), 16 deletions(-)
--
1.7.11.2.258.g5ff3cdf.dirty
@@ -1275,7 +1275,7 @@ sub get_mw_namespace_id { # Look at configuration file, if the record for that namespace is # already cached. Namespaces are stored in form: # "Name_of_namespace:Id_namespace", ex.: "File:6".- my @temp = split(/[ \n]/, run_git("config --get-all remote."+ my @temp = split(/[\n]/, run_git("config --get-all remote." . $remotename .".namespaceCache")); chomp(@temp); foreach my $ns (@temp) {
Initial phases of push and pull with git-remote-mediawiki can be long on
a large wiki. Let the user know what's going on.
Signed-off-by: Matthieu Moy <redacted>
---
contrib/mw-to-git/git-remote-mediawiki | 3 +++
1 file changed, 3 insertions(+)
@@ -371,6 +371,8 @@ sub get_mw_first_pages { sub get_mw_pages { mw_connect_maybe();+ print STDERR "Listing pages on remote wiki...\n";+ my %pages; # hash on page titles to avoid duplicates my $user_defined; if (@tracked_pages) {
When notes are created to record a push, it normally doesn't exist yet.
However, when a push is interrupted and then restarted, it may happen
that a commit already has notes attached, and we want to reflect the newly
created remote revision, hence use 'git notes add -f' to override the
existing one
Signed-off-by: Matthieu Moy <redacted>
---
contrib/mw-to-git/git-remote-mediawiki | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
Some wiki, including https://git.wiki.kernel.org/ have invalid revision
numbers (i.e. the actual revision numbers are non-contiguous). Don't die
when encountering one.
Signed-off-by: Matthieu Moy <redacted>
---
contrib/mw-to-git/git-remote-mediawiki | 13 +++++++++----
1 file changed, 9 insertions(+), 4 deletions(-)
@@ -910,6 +910,10 @@ sub mw_import_revids { my $last_timestamp = 0; # Placeholer in case $rev->timestamp is undefined foreach my $pagerevid (@$revision_ids) {+ # Count page even if we skip it, since we display+ # $n/$total and $total includes skipped pages.+ $n++;+ # fetch the content of the pages my $query = { action => 'query',
@@ -924,6 +928,11 @@ sub mw_import_revids { die "Failed to retrieve modified page for revision $pagerevid"; }+ if (defined($result->{query}->{badrevids}->{$pagerevid})) {+ # The revision id does not exist on the remote wiki.+ next;+ }+ if (!defined($result->{query}->{pages})) { die "Invalid revision $pagerevid."; }
@@ -932,10 +941,6 @@ sub mw_import_revids { my $result_page = $result_pages[0]; my $rev = $result_pages[0]->{revisions}->[0];- # Count page even if we skip it, since we display- # $n/$total and $total includes skipped pages.- $n++;- my $page_title = $result_page->{title}; if (!exists($pages->{$page_title})) {
The algorithm to find a path from the local revision to the remote one
was calling "git rev-list" and parsing its output N times. Run rev-list
only once, and fill a hashtable with the result to optimize the body of
the loop.
Signed-off-by: Matthieu Moy <redacted>
---
contrib/mw-to-git/git-remote-mediawiki | 24 +++++++++++++++++-------
1 file changed, 17 insertions(+), 7 deletions(-)
@@ -1196,16 +1196,26 @@ sub mw_push_revision { if ($last_local_revid > 0) { my $parsed_sha1 = $remoteorigin_sha1; # Find a path from last MediaWiki commit to pushed commit+ print STDERR "Computing path from local to remote ...\n";+ my @local_ancestry = split(/\n/, run_git("rev-list --boundary --parents $local ^$parsed_sha1"));+ my %local_ancestry;+ foreach my $line (@local_ancestry) {+ if (my ($child, $parents) = $line =~ m/^-?([a-f0-9]+) ([a-f0-9 ]+)/) {+ foreach my $parent (split(' ', $parents)) {+ $local_ancestry{$parent} = $child;+ }+ } elsif (!$line =~ m/^([a-f0-9]+)/) {+ die "Unexpected output from git rev-list: $line";+ }+ } while ($parsed_sha1 ne $HEAD_sha1) {- my @commit_info = grep(/^$parsed_sha1/, split(/\n/, run_git("rev-list --children $local")));- if (!@commit_info) {+ my $child = $local_ancestry{$parsed_sha1};+ if (!$child) {+ printf STDERR "Cannot find a path in history from remote commit to last commit\n"; return error_non_fast_forward($remote); }- my @commit_info_split = split(/ |\n/, $commit_info[0]);- # $commit_info_split[1] is the sha1 of the commit to export- # $commit_info_split[0] is the sha1 of its direct child- push(@commit_pairs, \@commit_info_split);- $parsed_sha1 = $commit_info_split[1];+ push(@commit_pairs, [$parsed_sha1, $child]);+ $parsed_sha1 = $child; } } else { # No remote mediawiki revision. Export the whole
@@ -51,6 +51,9 @@ use constant EMPTY_CONTENT => "<!-- empty page -->\n"; # used to reflect file creation or deletion in diff. use constant NULL_SHA1 => "0000000000000000000000000000000000000000";+# Used on Git's side to reflect empty edit messages on the wiki+use constant EMPTY_MESSAGE => '*Empty MediaWiki Message*';+ my $remotename = $ARGV[0]; my $url = $ARGV[1];
@@ -579,6 +579,8 @@ sub get_last_remote_revision { my $max_rev_num = 0;+ print STDERR "Getting last revision id on tracked pages...\n";+ foreach my $page (@pages) { my $id = $page->{pageid};
It is possible to use git-remote-mediawiki on a tree with both .mw files
and other files. Before git-remote-mediawiki learnt how to export
mediafiles, such mixed trees allowed the user to maintain both the wiki
and other files for the same project in the same repository. With the
newly added support for exporting mediafiles, pushing such mixed trees
would upload unrelated files as mediafiles, which may not be desired.
Signed-off-by: Matthieu Moy <redacted>
---
contrib/mw-to-git/git-remote-mediawiki | 24 ++++++++++++++++++++++--
1 file changed, 22 insertions(+), 2 deletions(-)
These patches can be added to branch mm/mediawiki-usability in pu.
For the curious: I originally had patch 2/2 in my serie, but
mistakenly discarded it during a rebase. While recovering it and
testing it, I found a typo fixed by patch 1/2.
Matthieu Moy (2):
git-remote-mediawiki: fix incorrect test usage in test
git-remote-mediawiki: allow page names with a ':'
contrib/mw-to-git/git-remote-mediawiki | 49 +++++++++++++++++++----------
contrib/mw-to-git/t/push-pull-tests.sh | 2 +-
contrib/mw-to-git/t/t9362-mw-to-git-utf8.sh | 20 ++++++++++++
3 files changed, 54 insertions(+), 17 deletions(-)
--
1.7.11.2.258.g5ff3cdf.dirty
Traditionnally, pages named Foo:Bar are page 'Bar' in namespace 'Foo'.
However, it is also possible to call a page Foo:Bar if 'Foo' is not a
namespace. In this case, the actual name of the page is 'Foo:Bar', in the
main namespace. Since we can't tell with only the filename, query the
wiki for a namespace 'Foo' in these cases, but deal with the case where
no such namespace is found.
Signed-off-by: Matthieu Moy <redacted>
---
contrib/mw-to-git/git-remote-mediawiki | 49 +++++++++++++++++++----------
contrib/mw-to-git/t/t9362-mw-to-git-utf8.sh | 20 ++++++++++++
2 files changed, 53 insertions(+), 16 deletions(-)
@@ -955,8 +955,11 @@ sub mw_import_revids { # Differentiates classic pages and media files. my ($namespace, $filename) = $page_title =~ /^([^:]*):(.*)$/; my %mediafile;- if ($namespace && get_mw_namespace_id($namespace) == get_mw_namespace_id("File")) {- %mediafile = get_mw_mediafile_for_page_revision($filename, $rev->{timestamp});+ if ($namespace) {+ my $id = get_mw_namespace_id($namespace);+ if ($id && $id == get_mw_namespace_id("File")) {+ %mediafile = get_mw_mediafile_for_page_revision($filename, $rev->{timestamp});+ } } # If this is a revision of the media page for new version # of a file do one common commit for both file and media page.
@@ -1306,7 +1309,11 @@ sub get_mw_namespace_id { chomp(@temp); foreach my $ns (@temp) { my ($n, $id) = split(/:/, $ns);- $namespace_id{$n} = $id;+ if ($id eq 'notANameSpace') {+ $namespace_id{$n} = {is_namespace => 0};+ } else {+ $namespace_id{$n} = {is_namespace => 1, id => $id};+ } $cached_mw_namespace_id{$n} = 1; } }
@@ -1324,28 +1331,38 @@ sub get_mw_namespace_id { while (my ($id, $ns) = each(%{$result->{query}->{namespaces}})) { if (defined($ns->{id}) && defined($ns->{canonical})) {- $namespace_id{$ns->{canonical}} = $ns->{id};+ $namespace_id{$ns->{canonical}} = {is_namespace => 1, id => $ns->{id}}; if ($ns->{'*'}) { # alias (e.g. french Fichier: as alias for canonical File:)- $namespace_id{$ns->{'*'}} = $ns->{id};+ $namespace_id{$ns->{'*'}} = {is_namespace => 1, id => $ns->{id}}; } } } }- my $id = $namespace_id{$name};+ my $ns = $namespace_id{$name};+ my $id;- if (defined $id) {- # Store explicitely requested namespaces on disk- if (!exists $cached_mw_namespace_id{$name}) {- run_git("config --add remote.". $remotename- .".namespaceCache \"". $name .":". $id ."\"");- $cached_mw_namespace_id{$name} = 1;- }- return $id;- } else {- die "No such namespace $name on MediaWiki.";+ unless (defined $ns) {+ print STDERR "No such namespace $name on MediaWiki.\n";+ $ns = {is_namespace => 0};+ $namespace_id{$name} = $ns;+ }++ if ($ns->{is_namespace}) {+ $id = $ns->{id};+ }++ # Store "notANameSpace" as special value for inexisting namespaces+ my $store_id = ($id || 'notANameSpace');++ # Store explicitely requested namespaces on disk+ if (!exists $cached_mw_namespace_id{$name}) {+ run_git("config --add remote.". $remotename+ .".namespaceCache \"". $name .":". $store_id ."\"");+ $cached_mw_namespace_id{$name} = 1; }+ return $id; } sub get_mw_namespace_id_for_page {
@@ -169,6 +169,26 @@ test_expect_failure 'special character at the begining of file name from mw to gtest_path_is_filemw_dir_11/[char_2'+test_expect_success'Pull page with title containing ":" other than namespace separator''+wiki_editpageFoo:Barcontentfalse&&+(+cdmw_dir_11&&+gitpull+)&&+test_path_is_filemw_dir_11/Foo:Bar.mw+'++test_expect_success'Push page with title containing ":" other than namespace separator''+(+cdmw_dir_11&&+echocontent>NotANameSpace:Page.mw&&+gitaddNotANameSpace:Page.mw&&+gitcommit-m"add page with colon"&&+gitpush+)&&+wiki_page_existNotANameSpace:Page+'+ test_expect_success'test of correct formating for file name from mw to git''wiki_reset&&gitclonemediawiki::'"$WIKI_URL"'mw_dir_12&&
From: Dan Johnson <hidden> Date: 2016-06-15 22:54:18
On Tue, Jul 17, 2012 at 10:06 AM, Matthieu Moy [off-list ref] wrote:
Traditionnally, pages named Foo:Bar are page 'Bar' in namespace 'Foo'.
However, it is also possible to call a page Foo:Bar if 'Foo' is not a
namespace. In this case, the actual name of the page is 'Foo:Bar', in the
main namespace. Since we can't tell with only the filename, query the
wiki for a namespace 'Foo' in these cases, but deal with the case where
no such namespace is found.
Might not be worth fixing, and it's just a typo in the commit message, but:
s/Traditionnally/Traditionally/
?
--
-Dan