[RFC] git-remote

Subsystems: kernel build + files below scripts/ (unless maintained elsewhere), the rest

8 messages, 7 authors, 2016-06-15 · open the first message on its own page

[RFC] git-remote

From: Junio C Hamano <hidden>
Date: 2016-06-15 22:42:47

It might be handy to have a single command that helps you manage
your configuration that relates to downloading from remote
repositories.  This currently does only about 20% of what I want
it to do.

	$ git remote

shows the list of 'remotes' you have defined somewhere, and

	$ git remote origin

shows the details about the named remote (in this case
"origin").  How the branches are tracked, if you have a 
tracking branch that is stale, etc.

	$ git add another git://git.kernel.org/pub/...

defines the default remote.another.url and remote.another.fetch
entries just like a clone does; you can say "git fetch another"
afterwards.

For it to be useful, I think it should be enhanced to:

 - check overlaps of tracking branches and warn;

 - offer to remove stale tracking branches in one go;

 - offer ways to remove or rename remote;

 - give different levels of verbosity in its 'show' command,
   especially if ls-remote is run;

 - offer ways to update an existing remote, perhaps have an
   interactive mode;

Other enhancements might be also possible, but I do not think of
anything that is absolutely necessary other than the above right
now.

Signed-off-by: Junio C Hamano <redacted>
---
 Makefile        |    2 +-
 git-remote.perl |  260 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 261 insertions(+), 1 deletions(-)
diff --git a/Makefile b/Makefile
index fa1a022..ea07634 100644
--- a/Makefile
+++ b/Makefile
@@ -179,7 +179,7 @@ SCRIPT_SH = \
 SCRIPT_PERL = \
 	git-add--interactive.perl \
 	git-archimport.perl git-cvsimport.perl git-relink.perl \
-	git-cvsserver.perl \
+	git-cvsserver.perl git-remote.perl \
 	git-svnimport.perl git-cvsexportcommit.perl \
 	git-send-email.perl git-svn.perl
 
diff --git a/git-remote.perl b/git-remote.perl
new file mode 100755
index 0000000..b0ec838
--- /dev/null
+++ b/git-remote.perl
@@ -0,0 +1,260 @@
+#!/usr/bin/perl -w
+
+use Git;
+my $git = Git->repository();
+
+sub add_remote_config {
+	my ($hash, $name, $what, $value) = @_;
+	if ($what eq 'url') {
+		if (exists $hash->{$name}{'URL'}) {
+			print STDERR "Warning: more than one remote.$name.url\n";
+		}
+		$hash->{$name}{'URL'} = $value;
+	}
+	elsif ($what eq 'fetch') {
+		$hash->{$name}{'FETCH'} ||= [];
+		push @{$hash->{$name}{'FETCH'}}, $value;
+	}
+	if (!exists $hash->{$name}{'SOURCE'}) {
+		$hash->{$name}{'SOURCE'} = 'config';
+	}
+}
+
+sub add_remote_remotes {
+	my ($hash, $file, $name) = @_;
+
+	if (exists $hash->{$name}) {
+		$hash->{$name}{'WARNING'} = 'ignored due to config';
+		return;
+	}
+
+	my $fh;
+	if (!open($fh, '<', $file)) {
+		print STDERR "Warning: cannot open $file\n";
+		return;
+	}
+	my $it = { 'SOURCE' => 'remotes' };
+	$hash->{$name} = $it;
+	while (<$fh>) {
+		chomp;
+		if (/^URL:\s*(.*)$/) {
+			# Having more than one is Ok -- it is used for push.
+			if (! exists $it->{'URL'}) {
+				$it->{'URL'} = $1;
+			}
+		}
+		elsif (/^Push:\s*(.*)$/) {
+			; # later
+		}
+		elsif (/^Pull:\s*(.*)$/) {
+			$it->{'FETCH'} ||= [];
+			push @{$it->{'FETCH'}}, $1;
+		}
+		elsif (/^\#/) {
+			; # ignore
+		}
+		else {
+			print STDERR "Warning: funny line in $file: $_\n";
+		}
+	}
+	close($fh);
+}
+
+sub list_remote {
+	my ($git) = @_;
+	my %seen = ();
+
+	for ($git->command(qw(repo-config --get-regexp), '^remote\.')) {
+		if (/^remote\.([^.]*)\.(\S*)\s+(.*)$/) {
+			add_remote_config(\%seen, $1, $2, $3);
+		}
+	}
+
+	my $dir = $git->repo_path() . "/remotes";
+	if (opendir(my $dh, $dir)) {
+		local $_;
+		while ($_ = readdir($dh)) {
+			chomp;
+			next if (! -f "$dir/$_" || ! -r _);
+			add_remote_remotes(\%seen, "$dir/$_", $_);
+		}
+	}
+
+	return \%seen;
+}
+
+sub add_branch_config {
+	my ($hash, $name, $what, $value) = @_;
+	if ($what eq 'remote') {
+		if (exists $hash->{$name}{'REMOTE'}) {
+			print STDERR "Warning: more than one branch.$name.remote\n";
+		}
+		$hash->{$name}{'REMOTE'} = $value;
+	}
+	elsif ($what eq 'merge') {
+		$hash->{$name}{'MERGE'} ||= [];
+		push @{$hash->{$name}{'MERGE'}}, $value;
+	}
+}
+
+sub list_branch {
+	my ($git) = @_;
+	my %seen = ();
+	for ($git->command(qw(repo-config --get-regexp), '^branch\.')) {
+		if (/^branch\.([^.]*)\.(\S*)\s+(.*)$/) {
+			add_branch_config(\%seen, $1, $2, $3);
+		}
+	}
+
+	return \%seen;
+}
+
+my $remote = list_remote($git);
+my $branch = list_branch($git);
+
+sub update_ls_remote {
+	my ($harder, $info) = @_;
+
+	return if (($harder == 0) ||
+		   (($harder == 1) && exists $info->{'LS_REMOTE'}));
+
+	my @ref = map {
+		s|^[0-9a-f]{40}\s+refs/heads/||;
+		$_;
+	} $git->command(qw(ls-remote --heads), $info->{'URL'});
+	$info->{'LS_REMOTE'} = \@ref;
+}
+
+sub show_wildcard_mapping {
+	my ($forced, $ours, $ls) = @_;
+	my %refs;
+	for (@$ls) {
+		$refs{$_} = 01; # bit #0 to say "they have"
+	}
+	for ($git->command('for-each-ref', "refs/remotes/$ours")) {
+		chomp;
+		next unless (s|^[0-9a-f]{40}\s[a-z]+\srefs/remotes/$ours/||);
+		next if ($_ eq 'HEAD');
+		$refs{$_} ||= 0;
+		$refs{$_} |= 02; # bit #1 to say "we have"
+	}
+	my (@new, @stale, @tracked);
+	for (sort keys %refs) {
+		my $have = $refs{$_};
+		if ($have == 1) {
+			push @new, $_;
+		}
+		elsif ($have == 2) {
+			push @stale, $_;
+		}
+		elsif ($have == 3) {
+			push @tracked, $_;
+		}
+	}
+	if (@new) {
+		print "  New remote branches (next fetch will store in remotes/$ours)\n";
+		print "    @new\n";
+	}
+	if (@stale) {
+		print "  Stale tracking branches in remotes/$ours (you'd better remove them)\n";
+		print "    @stale\n";
+	}
+	if (@tracked) {
+		print "  Tracked remote branches\n";
+		print "    @tracked\n";
+	}
+}
+
+sub show_mapping {
+	my ($name, $info) = @_;
+	my $fetch = $info->{'FETCH'};
+	my $ls = $info->{'LS_REMOTE'};
+	my (@stale, @tracked);
+
+	for (@$fetch) {
+		next unless (/(\+)?([^:]+):(.*)/);
+		my ($forced, $theirs, $ours) = ($1, $2, $3);
+		if ($theirs eq 'refs/heads/*' &&
+		    $ours =~ /^refs\/remotes\/(.*)\/\*$/) {
+			# wildcard mapping
+			show_wildcard_mapping($forced, $1, $ls);
+		}
+		elsif ($theirs =~ /\*/ || $ours =~ /\*/) {
+			print STDERR "Warning: unrecognized mapping in remotes.$name.fetch: $_\n";
+		}
+		elsif ($theirs =~ s|^refs/heads/||) {
+			if (!grep { $_ eq $theirs } @$ls) {
+				push @stale, $theirs;
+			}
+			elsif ($ours ne '') {
+				push @tracked, $theirs;
+			}
+		}
+	}
+	if (@stale) {
+		print "  Stale tracking branches in remotes/$name (you'd better remove them)\n";
+		print "    @stale\n";
+	}
+	if (@tracked) {
+		print "  Tracked remote branches\n";
+		print "    @tracked\n";
+	}
+}
+
+sub show_remote {
+	my ($name, $ls_remote) = @_;
+	if (!exists $remote->{$name}) {
+		print STDERR "No such remote $name\n";
+		return;
+	}
+	my $info = $remote->{$name};
+	update_ls_remote($ls_remote, $info);
+
+	print "* remote $name\n";
+	print "  URL: $info->{'URL'}\n";
+	for my $branchname (sort keys %$branch) {
+		next if ($branch->{$branchname}{'REMOTE'} ne $name);
+		my @merged = map {
+			s|^refs/heads/||;
+			$_;
+		} split(' ',"@{$branch->{$branchname}{'MERGE'}}");
+		next unless (@merged);
+		print "  Remote branch(es) merged with 'git pull' while on branch $branchname\n";
+		print "    @merged\n";
+	}
+	if ($info->{'LS_REMOTE'}) {
+		show_mapping($name, $info);
+	}
+}
+
+sub add_remote {
+	my ($name, $url) = @_;
+	if (exists $remote->{$name}) {
+		print STDERR "remote $name already exists.\n";
+		exit(1);
+	}
+	$git->command('repo-config', "remote.$name.url", $url);
+	$git->command('repo-config', "remote.$name.fetch",
+		      "+refs/heads/*:refs/remotes/$name/*");
+}
+
+if (!@ARGV) {
+	for (sort keys %$remote) {
+		print "$_\n";
+	}
+}
+elsif ($ARGV[0] eq 'show') {
+	if (@ARGV != 2) {
+		print STDERR "Usage: git remote show <remote>\n";
+		exit(1);
+	}
+	show_remote($ARGV[1], 1);
+}
+elsif ($ARGV[0] eq 'add') {
+	if (@ARGV != 3) {
+		print STDERR "Usage: git remote show <name> <url>\n";
+		exit(1);
+	}
+	add_remote($ARGV[1], $ARGV[2]);
+}
+
-- 
1.5.0.rc0.gcc26

Re: [RFC] git-remote

From: Jakub Narebski <hidden>
Date: 2016-06-15 22:42:47

Junio C Hamano wrote:
+elsif ($ARGV[0] eq 'add') {
+       if (@ARGV != 3) {
+               print STDERR "Usage: git remote show <name> <url>\n";
I think you meant to write here "Usage: git remote add <name> <url>\n";
("add" instead of "show").
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: [RFC] git-remote

From: Shawn O. Pearce <hidden>
Date: 2016-06-15 22:42:47

Junio C Hamano [off-list ref] wrote:
It might be handy to have a single command that helps you manage
your configuration that relates to downloading from remote
repositories.  This currently does only about 20% of what I want
it to do.

	$ git remote
This is pretty cool.  It would be nice if something like it was
in 1.5.0.  ;-)

-- 
Shawn.

Re: [RFC] git-remote

From: Santi Béjar <hidden>
Date: 2016-06-15 22:42:47

+1. And much better than my RFC for "git clone --add".

Just some comments. I know it's in its early shape, tell me if you
want this kind of comments later.

* I think it is more coherent to list the tracked branches first and
second the "branch merges".

* In "git remote add <name> <remote>": git could use the remote url to
deduce a <name>, like what git-clone does.

* If a branch does not have a branch.<name>.remote git-remote does not
default it to "origin" and sends an:

Use of uninitialized value in string ne at
/home/santi/usr/stow/git/bin/git-remote line 222.

Maybe, in addition to this, git should require a branch.<name>.remote.

* With the config:

[remote "origin"]
        url = git://git2.kernel.org/pub/scm/git/git.git
        fetch = master:refs/remotes/origin/master
        fetch = next:refs/remotes/origin/next
        fetch = +refs/heads/pu:refs/remotes/origin/pu
        fetch = refs/heads/*:refs/remotes/origin/*

[branch "next"]
        merge=next

It outputs something as:
* remote origin
  URL: git://git2.kernel.org/pub/scm/git/git.git
  Tracked remote branches
    html maint man master next pu todo
  Tracked remote branches
    pu

* The first "Tracked ..." is for the wildcards and the second for the
explicit fetch. Maybe it should join the two or mark different as:

  Implicit tracked ...
  Tracked ...

* The next and master branches are missing because they are written
without the refs/heads/ prefix. And for this reason there is no:

Remote branch(es) merged with 'git pull' while on branch next
  next

(and the absence of branch.next.remote).

* In addition I would reformat this as:

Merges with 'git pull' while on branch:
  "next" merges "next"
  "topic" merges "master"
  ...

That's all for today :)

Santi

Re: [RFC] git-remote

From: Johannes Schindelin <hidden>
Date: 2016-06-15 22:42:47

Hi,

On Fri, 5 Jan 2007, Santi Béjar wrote:
* In "git remote add <name> <remote>": git could use the remote url to 
deduce a <name>, like what git-clone does.
That does not make any sense. For example, I track 
"git://git.kernel.org/.../git.git" and "192.168.0.128:gits/git.git". 
Something very similar applies to the host name: if you track multiple 
Linux repos, chances are that most of them are on git.kernel.org.

I guess _if_ you have more than one upstream you are tracking (which is 
not the most common case, but hey, git-remote is for exactly that case) it 
is not uncommon to have similar urls.

IMHO Junio's proposal is as good as it gets.

Ciao,
Dscho

Re: [RFC] git-remote

From: Santi Béjar <hidden>
Date: 2016-06-15 22:42:47

On 1/5/07, Johannes Schindelin [off-list ref] wrote:
Hi,

On Fri, 5 Jan 2007, Santi Béjar wrote:
quoted
* In "git remote add <name> <remote>": git could use the remote url to
deduce a <name>, like what git-clone does.
That does not make any sense. For example, I track
"git://git.kernel.org/.../git.git" and "192.168.0.128:gits/git.git".
Something very similar applies to the host name: if you track multiple
Linux repos, chances are that most of them are on git.kernel.org.

I guess _if_ you have more than one upstream you are tracking (which is
not the most common case, but hey, git-remote is for exactly that case) it
is not uncommon to have similar urls.

IMHO Junio's proposal is as good as it gets.
I was talking about the default name, so you could do:

$ git clone\
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
$ cd linux-2.6
$ git remote add \
git://git.kernel.org/pub/scm/linux/kernel/git/jgarzik/libata-2.6.git
$ git remote show libata-2.6

Santi

Re: [RFC] git-remote

From: Carl Worth <hidden>
Date: 2016-06-15 22:42:48

On Fri, 5 Jan 2007 20:53:22 +0100, "=?ISO-8859-1?Q?Santi_B=E9jar?=" wrote:
I was talking about the default name, so you could do:

$ git clone\
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
$ cd linux-2.6
$ git remote add \
git://git.kernel.org/pub/scm/linux/kernel/git/jgarzik/libata-2.6.git
$ git remote show libata-2.6
Yes.

I'd go one further and say that it'd be really nice if git-clone could
be seen as equivalent, (and basically equivalent internally), to a
simple sequence of steps that could be performed manually. Something
like:

	name=$(basename $url)
	mkdir $name
	cd $name
	git init-db
	git remote add $url
	git pull $name

I haven't played with the new git-remote stuff at all yet, but the
rest of the above does work already, (but for a "warning: no common
commits" message). The only thing missing compared to git-clone is the
remotes configuration. So if the new git-remote can provide behavior
equivalent to what git clone provides, that would be excellent.

If for one have often wanted the ability to easily add a second remote
as if from git-clone.

-Carl

Re: [RFC] git-remote

From: J. Bruce Fields <hidden>
Date: 2016-06-15 22:42:48

On Wed, Jan 03, 2007 at 01:40:56PM -0800, Junio C Hamano wrote:
	$ git add another git://git.kernel.org/pub/...

defines the default remote.another.url and remote.another.fetch
entries just like a clone does; you can say "git fetch another"
afterwards.
Nifty.

Would it make sense for "git add" to do the initial fetch as well?

That'd also help catch any typos in the URL early--the first time I used
this I mistyped the URL, then had to delete the configuration by hand
after I found the problem....

How about this as a man page?

--b.

Documentation: add git-remote man page

Add a preliminary man page for git-remote.

Signed-off-by: "J. Bruce Fields" <redacted>
---
 Documentation/git-remote.txt |   76 ++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 76 insertions(+), 0 deletions(-)
diff --git a/Documentation/git-remote.txt b/Documentation/git-remote.txt
new file mode 100644
index 0000000..7d6663e
--- /dev/null
+++ b/Documentation/git-remote.txt
@@ -0,0 +1,76 @@
+git-remote(1)
+============
+
+NAME
+----
+git-remote - manage set of tracked repositories
+
+
+SYNOPSIS
+--------
+[verse]
+'git-remote'
+'git-remote' add <name> <url>
+'git-remote' show <name>
+
+DESCRIPTION
+-----------
+
+Manage the set of repositories ("remotes") whose branches you track.
+
+With no arguments, shows a list of existing remotes.
+
+In the second form, adds a remote named <name> for the repository at
+<url>.  The command `git fetch <name>` can then be used to create and
+update remote-tracking branches <name>/<branch>.
+
+In the third form, gives some information about the remote <name>.
+
+The remote configuration is achieved using the `remote.origin.url` and
+`remote.origin.fetch` configuration variables.  (See
+gitlink:git-repo-config[1]).
+
+Examples
+--------
+
+Add a new remote, fetch, and check out a branch from it:
+
+------------
+$ git remote
+origin
+$ git branch -r
+origin/master
+$ git remote add linux-nfs git://linux-nfs.org/pub/nfs-2.6.git
+$ git remote
+linux-nfs
+origin
+$ git fetch
+* refs/remotes/linux-nfs/master: storing branch 'master' ...
+  commit: bf81b46
+$ git branch -r
+origin/master
+linux-nfs/master
+$ git checkout -b nfs linux-nfs/master
+...
+------------
+
+See Also
+--------
+gitlink:git-fetch[1]
+gitlink:git-branch[1]
+gitlink:git-repo-config[1]
+
+Author
+------
+Written by Junio Hamano 
+
+
+Documentation
+--------------
+Documentation by J. Bruce Fields and the git-list <git@vger.kernel.org>.
+
+
+GIT
+---
+Part of the gitlink:git[7] suite
+
-- 
1.4.4.4.g4083
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help