Git fast-import : Warning duplicated ref: refs/remotes/origin/master

Hi,
I am currently working on the git-mediawiki gateway project.
This project uses fast-import to get data from mediawiki. For each
revision from mediawiki, we generate a chunk of fast-export stream
looking like this :

commit refs/remotes/origin/master
mark :<int>
commiter <user> <address> <timestamp> +0000
data <sizeofcomment>
<comment>
M 644 inline <title>.wiki
data <sizeoffile>
<content>

At the end of a git clone mediawiki::http://ourwikitest, everything
looks perfectly fine, including the git log, but a "Warning duplicated
ref: refs/remotes/origin/master" is thrown. And indeed, it is
duplicated in .git/packedrefs

A git gc solves everything, but I wondered if there was a way to fix
this or if this was a known bug.

Thank you,
Regards,

--
Jérémie Nikaes

Hi Jérémie,

Jérémie NIKAES wrote:

At the end of a git clone mediawiki::http://ourwikitest, everything
looks perfectly fine

I just wanted to say, thank you for working on this[1].

This code answers some questions I had about the remote helper
protocol, among other things.

a "Warning duplicated
ref: refs/remotes/origin/master" is thrown.

To end the suspense: I assume this was fixed by 9055039c (Bug fix for
pulls with multiple revisions pulled, 2011-05-29).

Cheers,
Jonathan

[1] git://github.com/Bibzball/Git-Mediawiki.git

Jonathan,

I just wanted to say, thank you for working on this[1].

This code answers some questions I had about the remote helper
protocol, among other things.

Glad we can help !

quoted

a "Warning duplicated
ref: refs/remotes/origin/master" is thrown.

To end the suspense: I assume this was fixed by 9055039c (Bug fix for
pulls with multiple revisions pulled, 2011-05-29).

No, I actually still have the problem. The version commited on github
is pulling revisions straight to refs/heads/master which is gross
After discussing this matter with our teacher Matthieu Moy, I wanted
to change this to refs/remotes/origin/master but then this warning
gets thrown.

Does anyone have an idea ?

Thanks,
Cheers,
-- 
Jérémie Nikaes

Jérémie NIKAES wrote:

quoted

Jérémie NIKAES wrote:

quoted

quoted

a "Warning duplicated
ref: refs/remotes/origin/master" is thrown.

[...]

No, I actually still have the problem. The version commited on github
is pulling revisions straight to refs/heads/master which is gross
After discussing this matter with our teacher Matthieu Moy, I wanted
to change this to refs/remotes/origin/master but then this warning
gets thrown.

Could you push a branch exhibiting the problem?  The reason I ask
is that searching with

	git grep -F -e 'duplicated ref' origin/master

in git.git yields no hits for me, so I wanted to reproduce this to get
the exact error message or figure out which program is printing it.

Thanks.

Could you push a branch exhibiting the problem?  The reason I ask
is that searching with

       git grep -F -e 'duplicated ref' origin/master

in git.git yields no hits for me, so I wanted to reproduce this to get
the exact error message or figure out which program is printing it.

Thanks.

Ok I just committed my changes to the master branch, feel free to pull
and try it out.

Thanks for your help
-- 
Jérémie Nikaes

Jérémie NIKAES wrote:

Ok I just committed my changes to the master branch, feel free to pull
and try it out.

Nice.  Now I get

| $ git fetch --all
[...]
| Fetching & writing export data...
| 1/2: Revision n°7 of New_page
| 2/2: Revision n°8 of New_page
| 
| error: Ref refs/remotes/origin/master is at d94a46270250454f1fc6c1fb47abfde31a2196c9 but expected dfb79bbc658333d5c9b0427b71f6b1bc48629949
| From mediawiki::http://localhost/mediawiki
|  ! dfb79bb...c57c15b master     -> origin/master  (unable to update local ref)
| error: Could not fetch origin

which means that the transport machinery thought the helper was going
to be fetching directly to "master".  I suspect you will want a
'refspec' capability like

	refspec refs/heads/*:refs/mediawiki/${remotename}/*

to fix this.

Cc-ing Daniel who invented v1.7.0-rc0~62^2~19 (Allow helper to map
private ref names into normal names, 2009-11-18).  What namespace
should a helper use when asked to fetch to FETCH_HEAD without a remote
name, like

	git fetch mediawiki::testwiki

?

-- >8 --
Subject: Documentation: do not misinterpret refspecs as bold text

In v1.7.3.3~2 (Documentation: do not misinterpret pull refspec as bold
text, 2010-12-03) many uses of asterisks in expressions like
"refs/heads/*:refs/svn/origin/branches/*" were escaped as {asterisk}
to avoid being treated as delimiters for bold text, but these two were
missed.

Signed-off-by: Jonathan Nieder <redacted>
Signed-off-by: Junio C Hamano <redacted>
---
 Documentation/git-remote-helpers.txt |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/Documentation/git-remote-helpers.txt b/Documentation/git-remote-helpers.txt
index 87cd11f..58f6ad4 100644
--- a/Documentation/git-remote-helpers.txt
+++ b/Documentation/git-remote-helpers.txt

@@ -181,11 +181,11 @@ CAPABILITIES
 	When using the import command, expect the source ref to have
 	been written to the destination ref. The earliest applicable
 	refspec takes precedence. For example
-	"refs/heads/*:refs/svn/origin/branches/*" means that, after an
-	"import refs/heads/name", the script has written to
+	"refs/heads/{asterisk}:refs/svn/origin/branches/{asterisk}" means
+	that, after an "import refs/heads/name", the script has written to
 	refs/svn/origin/branches/name. If this capability is used at
 	all, it must cover all refs reported by the list command; if
-	it is not used, it is effectively "*:*"
+	it is not used, it is effectively "{asterisk}:{asterisk}"
 
 REF LIST ATTRIBUTES
 -------------------

-- 
1.7.5.3

Jonathan Nieder wrote:

| error: Ref refs/remotes/origin/master is at d94a46270250454f1fc6c1fb47abfde31a2196c9 but expected dfb79bbc658333d5c9b0427b71f6b1bc48629949
| From mediawiki::http://localhost/mediawiki
|  ! dfb79bb...c57c15b master     -> origin/master  (unable to update local ref)
| error: Could not fetch origin

which means that the transport machinery thought the helper was going
to be fetching directly to "master".  I suspect you will want a
'refspec' capability like

refspec refs/heads/*:refs/mediawiki/${remotename}/*

to fix this.

Cc-ing Daniel who invented v1.7.0-rc0~62^2~19 (Allow helper to map
private ref names into normal names, 2009-11-18).  What namespace
should a helper use when asked to fetch to FETCH_HEAD without a remote
name, like

git fetch mediawiki::testwiki

?

Actually cc-ing this time.  Sorry for the noise.

Jérémie NIKAES wrote:

No, I actually still have the problem. The version commited on github
is pulling revisions straight to refs/heads/master which is gross
After discussing this matter with our teacher Matthieu Moy, I wanted
to change this to refs/remotes/origin/master but then this warning
gets thrown.

The following fixes it for me.

-- >8 --
Subject: use refs/mediawiki/<remote> namespace for private refs

Unless the "refspec" capability is used, the remote helper machinery
expects remote helpers to write remote refs/heads/master ref to
refs/heads/master.  That's not very good since it scribbles on top of
whatever local changes the user might have.

So specify a refspec that puts the refs elsewhere.  For now this
only works when fetching from a named remote, like so:

	git remote add origin mediawiki::testwiki
	git fetch origin

Attempting to fetch directly from a URL with a command like
"git fetch mediawiki::testwiki" produces an "Invalid refspec"
error.
---
 git-remote-mediawiki |   10 ++++++----
 1 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/git-remote-mediawiki b/git-remote-mediawiki
index af66b58..7a3fdd3 100755
--- a/git-remote-mediawiki
+++ b/git-remote-mediawiki

@@ -7,6 +7,7 @@ use Storable qw(freeze thaw);
 use DateTime::Format::ISO8601;
 use Encode qw(encode_utf8);
 
+my $remotename = $ARGV[0];
 my $url = $ARGV[1];
 
 print STDERR "$url\n";

@@ -62,6 +63,7 @@ while ($loop) {
 
 sub mw_capabilities {
 #	print STDOUT "fetch\n";
+	print STDOUT "refspec refs/heads/*:refs/mediawiki/$remotename/*\n";
 	print STDOUT "import\n";
 	print STDOUT "list\n";
 	print STDOUT "option\n";

@@ -91,7 +93,7 @@ sub mw_import {
 	
 	sub get_last_revision {
 		# Get last commit sha1
-		my $commit_sha1 = `git rev-parse refs/remotes/origin/master 2>/dev/null`;
+		my $commit_sha1 = `git rev-parse refs/mediawiki/$remotename/master 2>/dev/null`;
 
 		# Get note regarding that commit
 		chomp($commit_sha1);

@@ -202,15 +204,15 @@ sub mw_import {
 		
 		# If it's a clone and the first revision, initalize this ref
 		if ($fetch_from == 1 && $n == 1) {
-			#print "reset refs/remotes/origin/master\n";
+			#print "reset refs/mediawiki/$remotename/master\n";
 		}
-		print "commit refs/remotes/origin/master\n";
+		print "commit refs/mediawiki/$remotename/master\n";
 		print "mark :$n\n";
 		print "committer $user <$user\@$wiki_name> ", $dt->epoch, " +0000\n";
 		print "data ", bytes::length(encode_utf8($comment)), "\n", encode_utf8($comment);
 		# If it's not a clone, needs to know where to start from
 		if ($fetch_from != 1 && $n == 1) {
-			print "from refs/remotes/origin/master^0\n";
+			print "from refs/mediawiki/$remotename/master^0\n";
 		}
 		print "M 644 inline $title.wiki\n";
 		print "data ", bytes::length(encode_utf8($content)), "\n", encode_utf8($content);

-- 
1.7.5.3

On Mon, 30 May 2011, Jonathan Nieder wrote:

Jonathan Nieder wrote:

quoted

| error: Ref refs/remotes/origin/master is at d94a46270250454f1fc6c1fb47abfde31a2196c9 but expected dfb79bbc658333d5c9b0427b71f6b1bc48629949
| From mediawiki::http://localhost/mediawiki
|  ! dfb79bb...c57c15b master     -> origin/master  (unable to update local ref)
| error: Could not fetch origin

which means that the transport machinery thought the helper was going
to be fetching directly to "master".  I suspect you will want a
'refspec' capability like

refspec refs/heads/*:refs/mediawiki/${remotename}/*

to fix this.

Cc-ing Daniel who invented v1.7.0-rc0~62^2~19 (Allow helper to map
private ref names into normal names, 2009-11-18).  What namespace
should a helper use when asked to fetch to FETCH_HEAD without a remote
name, like

git fetch mediawiki::testwiki

?

The main purpose of having the private ref names is to support incremental 
imports, where you obviously need to identify the sha1 of the last thing 
you imported, and you may have included sufficient machine-readable 
information in your imported commit messages to know what the foreign 
state was that generated that sha1. (And you need a namespace because you 
may have multiple refs that you imported.) I'm not sure that it makes much 
sense to do incremental imports without a remote name, since that's 
generally an operation you're not planning do repeatedly.

But the purpose of the namespacing is to be able to continue the correct 
incremental import, so it would make sense to do some arbitrary 
transformation to make your url be a valid ref directory, and use that.

It would make sense to add support for a namespace where fast-import can 
write whatever it wants, and it'll get discarded after the fetch is done, 
if it's the case that anyone can stand not having incremental imports.

	-Daniel
*This .sig left intentionally blank*

Heya,

On Mon, May 30, 2011 at 12:43, Daniel Barkalow [off-list ref] wrote:

But the purpose of the namespacing is to be able to continue the correct
incremental import, so it would make sense to do some arbitrary
transformation to make your url be a valid ref directory, and use that.

It would make sense to add support for a namespace where fast-import can
write whatever it wants, and it'll get discarded after the fetch is done,
if it's the case that anyone can stand not having incremental imports.

I remember a patch series by someone else a lot more that _disabled_
creating of refs during fast-import, and instead allowed for
specifying what refs to update with which values at the end of the
import.

-- 
Cheers,

Sverre Rabbelier