On Wed, Oct 3, 2012 at 9:13 PM, Junio C Hamano [off-list ref] wrote:
Ævar Arnfjörð Bjarmason [off-list ref] writes:
quoted
I'm creating a system where a lot of remotes constantly fetch from a
central repository for deployment purposes, but I've noticed that even
with a remote.$name.fetch configuration to only get certain refs a
"git fetch" will still call git-upload pack which will provide a list
of all references.
It has been observed that the sender has to advertise megabytes of
refs because it has to speak first before knowing what the receiver
wants, even when the receiver is interested in getting updates from
only one of them, or worse yet, when the receiver is only trying to
peek the ref it is interested has been updated.
Has anyone started working on a next-gen Git protocol as a result of
this discussion? If not I thought I'd give it a shot if/when I have
time.
The current protocol is basically (S = Server, C = Client)
S: Spew out first ref
S: Advertisement of capabilities
S: Dump of all our refs
C/S: Declare wanted refs, negotiate with server
S: Send pack to client, if needed
And I thought I'd basically turn it into:
C: Connect to server, declare what protocol we understand
C: Advertisement of capabilities
S: Advertisement of capabilities
C/S: Negotiate what we want
C/S: Same as v1, without the advertisement of capabilities, and maybe
don't dump refs at all
Basically future-proofing it by having the client say what it supports
to begin with along with what it can handle (like in HTTP).
Then in the negotiation phase the client & server would go back &
forth about what they want & how they want it. I'd planned to
implement something like:
C: want_refs refs/heads/*
S: OK to that
C: want_refs refs/tags/*
S: OK to that
Or:
C: want_refs refs/heads/master
S: OK to that
C: want_refs refs/tags/v*
S: OK to that
As a proof of concept (and also something that'll solve the issue I
had), but by adding an initial negotiation phase the protocol should
be open to any future extensions without making assumptions about the
client wanting to know about all of the server's refs, unlike the
current protocol.
On Sun, Oct 07, 2012 at 09:57:56PM +0200, Ævar Arnfjörð Bjarmason wrote:
Has anyone started working on a next-gen Git protocol as a result of
this discussion? If not I thought I'd give it a shot if/when I have
time.
Unfortunately, client signaling the version is nasty to do in ways that
wouldn't cause current servers to hang up or do other undesirable things.
git://: Git-daemon will hang up[1] if it receives command it won't
understand (and one can't add arguments either).
ssh://: Commands are NAKed in non-standard ways (e.g. Gitolite vs. shell)
and one can't add arguments.
file://: That's easy.
CONNECT: The helper needs to be told that v2 is supported (helper doing
the rest).
Maybe with git://, one could hack the stuff in similar way as virtual
hosting was added. But that won't work with SSH (nor one can use environment
with SSH).
:-/
[1] And there is no guarantee that the server end of git:// is git-daemon.
There's at least one git:// server implemetation that responds to unknown
commands by ERR packet followed by hangup.
-Ilari
On Sun, Oct 07, 2012 at 09:57:56PM +0200, Ævar Arnfjörð Bjarmason wrote:
Has anyone started working on a next-gen Git protocol as a result of
this discussion? If not I thought I'd give it a shot if/when I have
time.
I haven't, and don't really plan on it soon (I have a few smaller things
I'm working on, then I'd like to look into the EWAH bitmap stuff from
Shawn next).
The current protocol is basically (S = Server, C = Client)
S: Spew out first ref
S: Advertisement of capabilities
S: Dump of all our refs
C/S: Declare wanted refs, negotiate with server
S: Send pack to client, if needed
In the "C" portion there, there is also "client acknowledges a
subset of capabilities shown by server" while it is declaring wanted
refs.
And I thought I'd basically turn it into:
C: Connect to server, declare what protocol we understand
C: Advertisement of capabilities
S: Advertisement of capabilities
The capability negotiation right now is that the server offers and the
client accepts. Are you swapping that so that the client offers and the
server accepts? Or are you thinking that they would be sent
simultaneously here? That could drop one round-trip (it's probably not
that important for git-over-tcp, but smart-http cares a lot about round
trips). But it also introduces a complexity with future additions (one
side may not know how to present its capabilities until understanding
what the other side can do).
C/S: Negotiate what we want
Refs we want, or capabilities we want?
C/S: Same as v1, without the advertisement of capabilities, and maybe
don't dump refs at all
Basically future-proofing it by having the client say what it supports
to begin with along with what it can handle (like in HTTP).
I feel like this "maybe..." bit needs more fleshed out before designing
the first part. I like the idea of future-proofing first and then adding
new features second, but what does the "don't advertise all refs"
protocol look like? Presumably the client is going to say "I'm
interested in refs/heads/* and refs/tags/*" or something. Does that come
with the capabilities? Or is it a new protocol phase?
I think we need to know what the second half of the two-step process
will look like to be sure the first half will accommodate it (and the
answer may be as simple as saying "they're not sending capabilities,
they're sending arbitrary key/value items, with the knowledge that the
other side may not understand particular keys, and we have to be
prepared to handle both cases).
Then in the negotiation phase the client & server would go back &
forth about what they want & how they want it. I'd planned to
implement something like:
C: want_refs refs/heads/*
S: OK to that
C: want_refs refs/tags/*
S: OK to that
Or:
C: want_refs refs/heads/master
S: OK to that
C: want_refs refs/tags/v*
S: OK to that
That seems simple. But how will it work over smart-http? Are we adding a
round-trip to do want_refs negotiation?
-Peff
On 10/07/2012 09:57 PM, Ævar Arnfjörð Bjarmason wrote:
On Wed, Oct 3, 2012 at 9:13 PM, Junio C Hamano [off-list ref] wrote:
quoted
Ævar Arnfjörð Bjarmason [off-list ref] writes:
quoted
I'm creating a system where a lot of remotes constantly fetch from a
central repository for deployment purposes, but I've noticed that even
with a remote.$name.fetch configuration to only get certain refs a
"git fetch" will still call git-upload pack which will provide a list
of all references.
It has been observed that the sender has to advertise megabytes of
refs because it has to speak first before knowing what the receiver
wants, even when the receiver is interested in getting updates from
only one of them, or worse yet, when the receiver is only trying to
peek the ref it is interested has been updated.
Has anyone started working on a next-gen Git protocol as a result of
this discussion? If not I thought I'd give it a shot if/when I have
time.
The current protocol is basically (S = Server, C = Client)
S: Spew out first ref
S: Advertisement of capabilities
S: Dump of all our refs
C/S: Declare wanted refs, negotiate with server
S: Send pack to client, if needed
And I thought I'd basically turn it into:
C: Connect to server, declare what protocol we understand
C: Advertisement of capabilities
S: Advertisement of capabilities
C/S: Negotiate what we want
C/S: Same as v1, without the advertisement of capabilities, and maybe
don't dump refs at all
Basically future-proofing it by having the client say what it supports
to begin with along with what it can handle (like in HTTP).
Then in the negotiation phase the client & server would go back &
forth about what they want & how they want it. I'd planned to
implement something like:
C: want_refs refs/heads/*
S: OK to that
C: want_refs refs/tags/*
S: OK to that
Or:
C: want_refs refs/heads/master
S: OK to that
C: want_refs refs/tags/v*
S: OK to that
You'll want that to be a single "wants" message to avoid incurring
insane amounts of roundtrip latency with lots of refs. github and
other hosted services are quite popular, but with my 120ms ping
rtt I'd be spending half a minute just telling the other side what
I want when I fetch from a repo with 250 refs.
It's a flagday and a half to change the protocol though, so I expect
it'll have to wait for 2.0, unless the current client-side part of
it is dumb and ignores existing refs when requesting its "wants", in
which case the server can just stop advertising existing refs and
most of the speedup is already done.
--
Andreas Ericsson andreas.ericsson@op5.se
OP5 AB www.op5.se
Tel: +46 8-230225 Fax: +46 8-230231
Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.