Re: [PATCH 1/8] Make the "traditionally-supported" URLs a special case

Daniel Barkalow [off-list ref] writes:

It turns out that the method used to form URLs that use a helper doesn't 
generalize well to other cases, because it interferes with the ssh-style 
locations. Instead, some different mechanism needs to be made up to handle 
arbitrary handlers that git doesn't know about. Since we want to keep 
supporting "http://something", that'll have to be a special case anyway, 
and so we might as well handle it by having git know what helpers to use 
for things that we've always supported, and use a single descriptive name 
for the helper that handles that collection of URLs.

As of this version, the idea is that there will be three ways helpers get 
selected:

 - git selects a helper based on the URL being something traditionally 
   supported internally; that is, git recognizes the URL and knows what to 
   run, if possible, to handle it

 - git uses the "vcs" option if it is set

 - something with the URL that we don't understand well enough yet to 
   design, but which doesn't seem to be possible to fit in as a single 
   rule with the first item.

Thanks for a clear description.

I do not see that there is much difference between the above description
and what Dscho is advocating, and I do not see anything to get excited
about as Dscho seems to do.  In his world, hg:// or any URL that begins
with <unknown>:// wants to be a short-hand to name the helper, and the
third rule whose detail is unspecified in the above list could be
something like:

 - With an explicit <prefix-separator>, i.e.

        <helper-name> <prefix-separator> <any-string>

   tells the named helper git-remote-<helper-name> to interact with
   repository that it can find using <any-string>.  We do not interpret,
   nor guess from, what <any-string> is, in this case.

 - When all else fails, and the URL looks like <unknown>://<any-string>,
   we see if git-remote-<unknown> is available and give it the whole
   string (including the <unknown>::// part).

which means that what Dscho wants is already a subset of the future
direction planned for this series.

As to the "curl" indirection, if you consider the possiblity of someday
adding the transparently backward compatible cgi based server with updated
clients Gitney talked about, I am reasonably sure that we would want to
have a new helper, say http-cgi, and have interested people invoke it
using the "more explicit" escape hatch:

    $ git clone http-cgi::http://repo.or.cz/w/alt-git.git/

while others can continue using the walker via a plain http://repo.or.cz/
URL.  When http-cgi helper proves to be successful and everybody's server
upgrades, we might choose to swap the default, say in git 1.10.0 release,
while leaving the door open for people to choose the old helper via an
explicit curl::http://repo.or.cz/ URL.

In short, from where I sit, I do not see much disagreement in the
semantics and in the future direction between what Dscho is saying (unless
I again misunderstood what he said) and what this round wants to bring.

The only slight difference is that having an explicit excape hatch as the
foundation, that usually does not have to be spelled out but does allow
you to, keeps the concept cleaner, while keeping the usability of the end
result.

Heya,

On Fri, Sep 4, 2009 at 19:23, Junio C Hamano[off-list ref] wrote:

In short, from where I sit, I do not see much disagreement in the
semantics and in the future direction between what Dscho is saying (unless
I again misunderstood what he said) and what this round wants to bring.

I think Dscho's main worry matches what I asked about earlier, will we
be able to say "hg://example.org" or not.

-- 
Cheers,

Sverre Rabbelier

Hi,

On Fri, 4 Sep 2009, Junio C Hamano wrote:

Daniel Barkalow [off-list ref] writes:

quoted

It turns out that the method used to form URLs that use a helper 
doesn't generalize well to other cases, because it interferes with the 
ssh-style locations. Instead, some different mechanism needs to be 
made up to handle arbitrary handlers that git doesn't know about. 
Since we want to keep supporting "http://something", that'll have to 
be a special case anyway, and so we might as well handle it by having 
git know what helpers to use for things that we've always supported, 
and use a single descriptive name for the helper that handles that 
collection of URLs.

As of this version, the idea is that there will be three ways helpers 
get selected:

 - git selects a helper based on the URL being something traditionally 
   supported internally; that is, git recognizes the URL and knows 
   what to run, if possible, to handle it

 - git uses the "vcs" option if it is set

 - something with the URL that we don't understand well enough yet to 
   design, but which doesn't seem to be possible to fit in as a single 
   rule with the first item.

Thanks for a clear description.

I do not see that there is much difference between the above description
and what Dscho is advocating, and I do not see anything to get excited
about as Dscho seems to do.

I mainly take exception at complicating things with a "vcs" config 
variable.

The way you describe it, I like it, as I do not see any mention of said 
config variable there.

If you allow "git clone <URL>" for foreign vcs URLs, you do not need the 
"vcs" variable.  If you require that variable, you cannot allow an easy 
clone, and you will earn my opposition.

Ciao,
Dscho

On Fri, 4 Sep 2009, Johannes Schindelin wrote:

Hi,

On Fri, 4 Sep 2009, Junio C Hamano wrote:

quoted

Daniel Barkalow [off-list ref] writes:

quoted

It turns out that the method used to form URLs that use a helper 
doesn't generalize well to other cases, because it interferes with the 
ssh-style locations. Instead, some different mechanism needs to be 
made up to handle arbitrary handlers that git doesn't know about. 
Since we want to keep supporting "http://something", that'll have to 
be a special case anyway, and so we might as well handle it by having 
git know what helpers to use for things that we've always supported, 
and use a single descriptive name for the helper that handles that 
collection of URLs.

As of this version, the idea is that there will be three ways helpers 
get selected:

 - git selects a helper based on the URL being something traditionally 
   supported internally; that is, git recognizes the URL and knows 
   what to run, if possible, to handle it

 - git uses the "vcs" option if it is set

 - something with the URL that we don't understand well enough yet to 
   design, but which doesn't seem to be possible to fit in as a single 
   rule with the first item.

Thanks for a clear description.

I do not see that there is much difference between the above description
and what Dscho is advocating, and I do not see anything to get excited
about as Dscho seems to do.

I mainly take exception at complicating things with a "vcs" config 
variable.

The way you describe it, I like it, as I do not see any mention of said 
config variable there.

If you allow "git clone <URL>" for foreign vcs URLs, you do not need the 
"vcs" variable.  If you require that variable, you cannot allow an easy 
clone, and you will earn my opposition.

Some foreign vcses, including the only one I ever personally use, do not 
have URLs, and require a bunch of options and paths to specify a 
repository. I don't want to have to use:

	url = p4://rsh:ssh+-q+-a+-x+-l+p4ssh+-q+-x+perforce+%2Fbin%2Ftrue//projects/foo/bar-1.0/...,//projects/foo/bar-1.1/...

(actually, I don't even know what the normal thing is for a URL for 
something that's split between multiple locations, or how URLs handle 
"servers" that are arbitrary commands including options which make a 
connection to the server)

For cases where the foreign vcs has something to put in the "url" spot, 
you don't need to set "vcs". In fact, you are only allowed to set one or 
the other of "vcs" and "url" with my current version. What you're 
interested in is explicitly left for later, when we have a prototype 
helper for such a foreign vcs and can try it out with potential users.

	-Daniel
*This .sig left intentionally blank*

Heya,

On Fri, Sep 4, 2009 at 21:05, Daniel Barkalow[off-list ref] wrote:

Some foreign vcses, including the only one I ever personally use, do not
have URLs, and require a bunch of options and paths to specify a
repository. I don't want to have to use:

       url = p4://rsh:ssh+-q+-a+-x+-l+p4ssh+-q+-x+perforce+%2Fbin%2Ftrue//projects/foo/bar-1.0/...,//projects/foo/bar-1.1/...

Btw, doesn't p4 have these config files that you can download that
contain the configuration? In that case
'p4://example.org/p4/main-development.configfile' would be very
convenient.

Regardless, I do think there should be some way to specify all this
outside of the url, but to me that's secondary. I think the primary
usecase is/should be cloning from some url in the form of
'hg://example.org/foo', rather than 'http://example.org/some-hg-repo'
or 'p4://.......', since those are both exceptions (the former being
an ambiguous url, and the latter being a non-url). Now I do understand
if you don't want to spend your time on implementing the specialized
url support since it doesn't scratch your itch, but at least your
series shouldn't impend supporting that in the near future.

For cases where the foreign vcs has something to put in the "url" spot,
you don't need to set "vcs". In fact, you are only allowed to set one or
the other of "vcs" and "url" with my current version. What you're
interested in is explicitly left for later, when we have a prototype
helper for such a foreign vcs and can try it out with potential users.

I need to hurry up and get working on that hg implementation then :).

-- 
Cheers,

Sverre Rabbelier

On Fri, 4 Sep 2009, Sverre Rabbelier wrote:

Heya,

On Fri, Sep 4, 2009 at 21:05, Daniel Barkalow[off-list ref] wrote:

quoted

Some foreign vcses, including the only one I ever personally use, do not
have URLs, and require a bunch of options and paths to specify a
repository. I don't want to have to use:

       url = p4://rsh:ssh+-q+-a+-x+-l+p4ssh+-q+-x+perforce+%2Fbin%2Ftrue//projects/foo/bar-1.0/...,//projects/foo/bar-1.1/...

Btw, doesn't p4 have these config files that you can download that
contain the configuration? In that case
'p4://example.org/p4/main-development.configfile' would be very
convenient.

The only thing I know of which you might be thinking of is "client 
specifications", which are like git superprojects. They're almost certain 
to only specify one of the multiple locations that you want to have in the 
same repository; the multiple locations are the paths you want to treat 
as branches, and the client picks one branch of each project and places 
it in some non-branch-specific location relative to other projects. (Of 
course, someday I might want to support importing a client specification 
as a git project with submodules, but it's got the same issues as 
svn::externals without revision specifications seems to).

In any case, p4 doesn't have any easy generic way to specify how to 
contact the server, and doesn't have anything client-side.

Regardless, I do think there should be some way to specify all this
outside of the url, but to me that's secondary. I think the primary
usecase is/should be cloning from some url in the form of
'hg://example.org/foo', rather than 'http://example.org/some-hg-repo'
or 'p4://.......', since those are both exceptions (the former being
an ambiguous url, and the latter being a non-url). Now I do understand
if you don't want to spend your time on implementing the specialized
url support since it doesn't scratch your itch, but at least your
series shouldn't impend supporting that in the near future.

I'm pretty sure that this series makes your primary usecase slightly 
simpler to support, because it no longer is expected to handle the 
ambiguous "http://" class of URLs.

	-Daniel
*This .sig left intentionally blank*

Hi,

On Fri, 4 Sep 2009, Sverre Rabbelier wrote:

On Fri, Sep 4, 2009 at 21:05, Daniel Barkalow[off-list ref] 
wrote:

quoted

Some foreign vcses, including the only one I ever personally use, do 
not have URLs, and require a bunch of options and paths to specify a 
repository. I don't want to have to use:

       url = p4://rsh:ssh+-q+-a+-x+-l+p4ssh+-q+-x+perforce+%2Fbin%2Ftrue//projects/foo/bar-1.0/...,//projects/foo/bar-1.1/...

Btw, doesn't p4 have these config files that you can download that 
contain the configuration? In that case 
'p4://example.org/p4/main-development.configfile' would be very 
convenient.

If that's how p4 users initialize their working directories, then that is 
the way to go.

And I cannot start to believe that the complicated way you described is 
the common way to initialize p4 working directories, as that would tempt 
the intelligence/enthusiasm of the average programmer.

quoted

For cases where the foreign vcs has something to put in the "url" 
spot, you don't need to set "vcs". In fact, you are only allowed to 
set one or the other of "vcs" and "url" with my current version. What 
you're interested in is explicitly left for later, when we have a 
prototype helper for such a foreign vcs and can try it out with 
potential users.

I need to hurry up and get working on that hg implementation then :).

Indeed you do.  If only to prove that _this_ and the likes are something 
to optimize for, not some obscure vcs config variable that only introduces 
a little-exercized code path that's _prone_ to break and does not help 
anybody.

Ciao,
Dscho

On Fri, 4 Sep 2009, Johannes Schindelin wrote:

Hi,

On Fri, 4 Sep 2009, Sverre Rabbelier wrote:

quoted

On Fri, Sep 4, 2009 at 21:05, Daniel Barkalow[off-list ref] 
wrote:

quoted

Some foreign vcses, including the only one I ever personally use, do 
not have URLs, and require a bunch of options and paths to specify a 
repository. I don't want to have to use:

       url = p4://rsh:ssh+-q+-a+-x+-l+p4ssh+-q+-x+perforce+%2Fbin%2Ftrue//projects/foo/bar-1.0/...,//projects/foo/bar-1.1/...

Btw, doesn't p4 have these config files that you can download that 
contain the configuration? In that case 
'p4://example.org/p4/main-development.configfile' would be very 
convenient.

If that's how p4 users initialize their working directories, then that is 
the way to go.

And I cannot start to believe that the complicated way you described is 
the common way to initialize p4 working directories, as that would tempt 
the intelligence/enthusiasm of the average programmer.

Perforce is probably the single most popular system for git to import from 
because it is such a monumental pain to use for anything at all that it's 
easier to learn git, write a git importer, and use your git importer than 
it is to actually use Perforce directly.

Of course, it's not really beyond the average programmer to get a p4 
working directory, because whoever is running the server will have 
provided a file to copy and instructions on setting an environment 
variable. They don't know what the magic formula means; they just use it. 
And they only work on one branch until that branch is done with,
and then they throw away that working directory, get a new working 
directory, and never look at the other branch's history again (and 
certainly never track anything across branches). Also, they have p4 
experts who deal with merging branches so that stuff doesn't get lost when 
moving to a new branch. And the experts have scripts built into the 
release process that attempt to insure that things don't get lost. The 
reason that my helper can't have a single location for a repository is 
that the branches of a single project are strewn randomly about the 
namespace, and a proper git import needs to know what to stitch into a 
single repository.

For the matter of where the server is, Perforce supports just having a 
"server:port" value, but if the organization uses this, there's no 
authentication of users possible. Instead, organizations set up an ad hoc 
collection of ssh proxies and give people a string which is the command to 
go through those proxies, because Perforce only knows how to use rsh or a 
command you provide that acts like rsh.

	-Daniel
*This .sig left intentionally blank*

Hi,

On Fri, 4 Sep 2009, Daniel Barkalow wrote:

On Fri, 4 Sep 2009, Johannes Schindelin wrote:

quoted

Hi,

On Fri, 4 Sep 2009, Sverre Rabbelier wrote:

quoted

On Fri, Sep 4, 2009 at 21:05, Daniel Barkalow[off-list ref] 
wrote:

quoted

Some foreign vcses, including the only one I ever personally use, do 
not have URLs, and require a bunch of options and paths to specify a 
repository. I don't want to have to use:

       url = p4://rsh:ssh+-q+-a+-x+-l+p4ssh+-q+-x+perforce+%2Fbin%2Ftrue//projects/foo/bar-1.0/...,//projects/foo/bar-1.1/...

Btw, doesn't p4 have these config files that you can download that 
contain the configuration? In that case 
'p4://example.org/p4/main-development.configfile' would be very 
convenient.

If that's how p4 users initialize their working directories, then that is 
the way to go.

And I cannot start to believe that the complicated way you described is 
the common way to initialize p4 working directories, as that would tempt 
the intelligence/enthusiasm of the average programmer.

Perforce is probably the single most popular system for git to import 
from because it is such a monumental pain to use for anything at all 
that it's easier to learn git, write a git importer, and use your git 
importer than it is to actually use Perforce directly.

Of course, it's not really beyond the average programmer to get a p4 
working directory, because whoever is running the server will have > 
provided a file to copy and instructions on setting an environment 
variable.

That is what we need to optimize for, then.

They don't know what the magic formula means; they just use it. And they 
only work on one branch until that branch is done with, and then they 
throw away that working directory, get a new working directory, and 
never look at the other branch's history again (and certainly never 
track anything across branches). Also, they have p4 experts who deal 
with merging branches so that stuff doesn't get lost when moving to a 
new branch. And the experts have scripts built into the release process 
that attempt to insure that things don't get lost. The reason that my 
helper can't have a single location for a repository is that the 
branches of a single project are strewn randomly about the namespace, 
and a proper git import needs to know what to stitch into a single 
repository.

And why not having the different branches which are strewn randomly about 
the namespace as separate remotes for a Git repository?  After all, the 
average p4 user will be wanting to work on _one_ branch, as you so aptly 
described.

For the matter of where the server is, Perforce supports just having a 
"server:port" value, but if the organization uses this, there's no 
authentication of users possible. Instead, organizations set up an ad 
hoc collection of ssh proxies and give people a string which is the 
command to go through those proxies, because Perforce only knows how to 
use rsh or a command you provide that acts like rsh.

That explains a tiny part of the long path you provided, but certainly not 
all (I am especially curious what /bin/true thinks it's doing in that 
URL).

If what you said about ssh is true, then it should be the same type of 
invocation everywhere, and it should certainly be very easy to provide a 
shortcut for that URL; no need for the _user_ (who could not care less how 
ssh happens to be called) to remember.

Something like "git clone p4::ssh://p4ssh@projects/foo/bar-1.0/..." should 
become a very easy and intuitive way for the average programmer to clone a 
p4 branch into a Git repository.

Should the developer ever need to work with another branch of the same 
project, very easy:

	$ git remote add -f bar-1.1 p4::ssh://p4ssh@projects/foo/bar-1.1/...
	$ git checkout -b my-1.1 bar-1.1/master

Now, I am not married to having more than one remote for multiple 
branches, but there is _no_ reason why this has to be done at clone time, 
if the average p4 user does not do that either.  You can always teach 
git-remote-p4 to behave sensibly and ask the user to

	$ git config --add remote.origin.fetch \
		+/foo/bar-1.1:refs/remotes/origin/bar-1.1

Note, these are two alternative suggestions.  I am not trying to decide 
what is better here, but I am convinced that both options are more 
intuitive than the "vcs" variable.

Ciao,
Dscho