9 messages, 5 authors, 2016-06-15 · open the first message on its own page
From: Junio C Hamano <hidden>
Date: 2016-06-15 22:45:15
"H. Peter Anvin" [off-list ref] writes:
We should be able to detect either inconsistency, or lack of forward progress, but as long as there is forward progress made there doesn't seem to be a strong need to terminate.
Yeah, what I wanted to say was that it would be tricky to detect (any/lack of) forward progress without having server side state and without trusting the client.
From: Shawn O. Pearce <hidden>
Date: 2016-06-15 22:45:15
Yet another draft follows. I believe that I have covered all
comments with this draft. But I welcome any additional ones,
as thus far it has been a very constructive process.
The updated protocol looks more like the current native protocol
does. This should make it easier to reuse code between the two
protocol implementations.
--8<--
Smart HTTP transfer protocols
=============================
Git supports two HTTP based transfer protocols. A "dumb" protocol
which requires only a standard HTTP server on the server end of the
connection, and a "smart" protocol which requires a Git aware CGI
(or server module). This document describes the "smart" protocol.
As a design feature smart clients can automatically translate and
upgrade "dumb" protocol URLs. This permits all users to have the
same published URL, with the peers automatically choosing to use
the most efficient transport available to them.
HTTP Transport
--------------
All requests are encoded as HTTP POST requests to the smart service
URL, "$url/backend.git-http/$service".
All responses are encoded as 200 Ok responses, even if the server
side has "failed" the request. Service specific success/failure
codes are embedded in the content.
Authentication
--------------
Standard HTTP authentication is used if authentication is required
to access a repository, and must be configured and enforced by the
HTTP server software itself.
Stateless
---------
The protocol, much like its underlying HTTP, is stateless, from the
perspective of the HTTP server side. All state must be retained and
managed by the client. This permits round-robin load-balancing on
the server side, among many other implementation details.
Content Type
------------
All requests/responses use "application/x-git" as the content type.
Action specific subtypes are specified by the parameter "service",
e.g. "application/x-git; service=upload-pack".
HTTP Redirects
--------------
If a POST request results in an HTTP 302 or 303 redirect response
clients should retry the request by updating the URL and POSTing
the same request to the new location. Subsequent requests should
still be sent to the original URL.
This redirect behavior is unrelated to the in-payload redirect
that is described below in "Service show-ref".
Detecting Smart Servers
-----------------------
HTTP clients can detect a smart Git-aware server by sending
a request to service "show-ref".
A Git-aware server will respond with a valid response. Clients
must check the following properties to prevent being fooled by
misconfigured servers:
* HTTP status code is 200.
* Content-Type is "application/x-git; service=show-ref"
* The body can be parsed without errors. The length of
each pkt-line must be 4 valid hex digits.
A dumb server will respond with a non-200 HTTP status code.
A misconfigured server may respond with a normal 200 status
code, but an incorrect content type, or an invalid leading
4 byte sequence for a pkt-line (e.g. "<htm" or "<!DO" are
not valid lengths).
pkt-line Format
---------------
Much of the payload is described around pkt-lines.
A pkt-line is a variable length binary string. The first four bytes
of the line indicates the total length of the line, in hexadecimal.
The total length includes the 4 bytes used to denote the length. A
line is usually terminated by an LF, which must be included in the
total length if present.
Binary data is permitted within a pkt-line so implementors should
ensure their pkt-line parsing/formatting routines are 8-bit clean.
The maximum length of a pkt-line's data is 65532 bytes (65536 - 4).
Examples (as C-style strings):
pkt-line actual value
---------------------------------
"0006a\n" "a\n"
"0005a" "a"
"000bfoobar\n" "foobar\n"
"0004" ""
A pkt-line with a length of 0 ("0000") is a special case and is
treated as a break or terminator in the payload.
Service show-ref
----------------
Obtains the available refs from the remote repository.
URL: $url/backend.git-http/show-ref
Content-Type: application/x-git; service=show-ref
The request is an empty body.
The response is a pkt-line with "refs", followed by zero
or more ref pkt-lines ("$id $name"), and a final pkt-line
with a length of 0:
S: 0009refs
S: 003295dcfa3633004da0049d3d0fa03f80589cbcaf31 HEAD
S: 003e95dcfa3633004da0049d3d0fa03f80589cbcaf31 refs/heads/maint
S: 003fd049f6c27a2244e12041955e262a404c7faba355 refs/heads/master
S: 003b2cb58b79488a98d2721cea644875a8dd0026b115 refs/heads/pu
S: 0000
The response may begin with an optional redirect to a new service
URL for the repository:
S: 0028redirect http://s1.example.com/git/
S: 0009refs
S: 003295dcfa3633004da0049d3d0fa03f80589cbcaf31 HEAD
S: 003fd049f6c27a2244e12041955e262a404c7faba355 refs/heads/master
S: 0000
or be composed of only a redirect:
S: 0028redirect http://s1.example.com/git/
S: 0000
If a redirect is returned the client should update itself
to use the new URL as the location for future requests.
A server may use the redirect to request that the client
"pin" itself to a particular server for the remainder of
the current transaction.
The URL listed in any redirect should be the base URL
without any query args. The client will automatically
append "/backend.git-http/$service" as it makes each
future request.
If no "refs" line was received in the response, but
a "redirect" was received, the client should retry
its request at the new location before giving up.
Service upload-pack
-------------------
Prepares an estimated minimal pack to transfer new objects to the
client.
URL: $url/backend.git-http/upload-pack
Content-Type: application/x-git; service=upload-pack
The computation to select the minimal pack proceeds as follows
(c = client, s = server):
init step:
(c) Use show-ref to obtain the advertised refs.
(c) Place any object seen in show-ref into set ADVERTISED.
(c) Build a set, WANT, of the objects from ADVERTISED the client
wants to fetch, based on what it saw from show-ref.
(c) Build an empty set, COMMON, to hold the objects that are later
determined to be on both ends.
(c) Start a queue, C_PENDING, ordered by commit time (popping newest
first). Add all client refs. When a commit is popped from the
queue its parents should be automatically inserted back. Commits
should only enter the queue once.
one compute step:
(c) Send an upload-pack request:
C: 001bcapability include-tag
C: 0019capability thin-pack
....
C: 0032want <WANT #1>...............................
C: 0032want <WANT #2>...............................
....
C: 0034common <COMMON #1>.............................
C: 0034common <COMMON #2>.............................
....
C: 0032have <HAVE #1>...............................
C: 0032have <HAVE #2>...............................
....
C: 0000
The stream is organized into "commands", with each command
appearing by itself in a pkt-line. Within a command line
the text leading up to the first space is the command name,
and the remainder of the line to the first LF is the value.
Command lines are terminated with an LF as the last byte of
the pkt-line value.
Servers must ignore commands which they do not recognize.
This permits newer clients to transmit additional data to
an unknown server, in case the server is new enough to use
the additional information.
Commands must appear in the following order, if they appear
at all in the request stream:
* capability
* want
* common
* have
* give-up
The stream is terminated by a pkt-line flush ("0000").
The "capability" command requests a single protocol feature
to be enabled by the server. Typically these are used to
describe aspects of the pack that will be returned. See
below for more details on the current capabilities.
A single "want", "common", or "have" command has one hex
formatted SHA-1 as its value. Multiple SHA-1s can be sent
by sending multiple commands.
The HAVE list is created by popping the first 64 commits
from C_PENDING. Less can be supplied if C_PENDING empties.
If the client has sent 256 HAVE commits and has not yet
received one of those back from S_COMMON, or the client
has emptied C_PENDING it should include a "give-up"
command to let the server know it won't proceed:
C: 000cgive-up
(s) Parse the upload-pack request:
Verify all objects in WANT are reachable from refs. As
this may require walking backwards through history to
the very beginning on invalid requests the server may
use a reasonable limit of commits (e.g. 1000) walked
beyond any ref tip before giving up.
If no WANT objects are received, send an error:
S: 0019status error no want
If any WANT object is not reachable, send an error:
S: 001estatus error invalid want
Create an empty list, S_COMMON.
If 'common' was sent:
Load all objects into S_COMMON. If an object appears in
'common' but the server does not have the object locally
an error should be returned:
S: 001estatus error invalid common
If 'have' was sent:
Loop through the objects in the order supplied by the client.
For each object, if the server has the object reachable from
a ref, add it to S_COMMON. If a commit is added to S_COMMON,
do not add any ancestors, even if they also appear in HAVE.
(s) Send the upload-pack response:
If the server has found a closed set of objects to pack or the
request contains "give-up", it replies with the pack and the
enabled capabilities. The set of enabled capabilities is limited
to the intersection of what the client requested and what the
server supports.
S: 0010status pack
C: 001bcapability include-tag
C: 0019capability thin-pack
S: 000c.PACK...
The returned stream is the side-band-64k protocol supported
by the git-upload-pack service, and the pack is embedded into
stream 1. Progress messages from the server side may appear
in stream 2.
Here a "closed set of objects" is defined to have at least
one path from every WANT to at least one COMMON object.
If the server needs more information, it replies with a
status continue response:
S: 0014status continue
S: 0034common <S_COMMON #1>...........................
S: 0034common <S_COMMON #2>...........................
...
S: 0000
The stream formatting rules are the same as the request.
The "common" command details the contents of S_COMMON,
that is all objects from HAVE that the server also has.
(c) Parse the upload-pack response:
If the status pkt-line is "status pack:"
Process the pack stream and update the local refs.
If the status pkt-line is "status continue":
Reset COMMON to the items in S_COMMON. The new S_COMMON
should be a superset of the existing COMMON set.
Remove all items in S_COMMON, and all of their ancestors,
from PENDING.
Do another compute step.
Capability include-tag
~~~~~~~~~~~~~~~~~~~~~~
When packing an object that an annotated tag points at, include
the tag object too. Clients can request this if they want to
fetch tags, but don't know which tags they will need until after
they receive the branch data. By enabling include-tag an entire
call to upload-pack can be avoided.
Capability thin-pack
~~~~~~~~~~~~~~~~~~~~
When packing a deltified object the base is not included if the
base is reachable from an object listed in the COMMON set by the
client. This reduces the bandwidth required to transfer, but it
does slightly increase processing time for the client to save the
pack to disk.
Service receive-pack
--------------------
Uploads a pack and updates refs.
URL: $url/backend.git-http/receive-pack
Content-Type: application/x-git; service=receive-pack
The start of the stream is the commands to update the refs and
the remainder of the stream is the pack file itself. See
git-receive-pack and its network protocol in pack-protocol.txt,
as this is essentially the same.
C: 006395dcfa3633004da0049d3d0fa03f80589cbcaf31 d049f6c27a2244e12041955e262a404c7faba355 refs/heads/maint
C: 0000
C: PACK...
S: 0005
S: ...<output of receive-pack>...
The capabilities are handled exactly as in the fetch protocol,
however the server may reject a pack and its associated commands
if an invalid capability request is made by the client, or the
client has assumed a pack capability that the server does not
have support for. In the latter case the server must still send
the capabilities key in the response so the client can correct
itself and try again.
--
Shawn.
From: Nicolas Pitre <hidden>
Date: 2016-06-15 22:45:15
On Fri, 29 Aug 2008, Shawn O. Pearce wrote:
Yet another draft follows. I believe that I have covered all comments with this draft. But I welcome any additional ones, as thus far it has been a very constructive process. The updated protocol looks more like the current native protocol does. This should make it easier to reuse code between the two protocol implementations.
[...]
pkt-line Format --------------- Much of the payload is described around pkt-lines. A pkt-line is a variable length binary string. The first four bytes of the line indicates the total length of the line, in hexadecimal. The total length includes the 4 bytes used to denote the length. A line is usually terminated by an LF, which must be included in the total length if present. Binary data is permitted within a pkt-line so implementors should ensure their pkt-line parsing/formatting routines are 8-bit clean. The maximum length of a pkt-line's data is 65532 bytes (65536 - 4).
Shouldn't that be 65531, since you cannot represent 65536 with 4 hex digits?
C: 001bcapability include-tag C: 0019capability thin-pack ....
[...]
The "capability" command requests a single protocol feature
to be enabled by the server. Typically these are used to
describe aspects of the pack that will be returned. See
below for more details on the current capabilities.Why not having all capabilities listed at once on a single line instead? That's more or less what the current protocol does already. Nicolas
From: Tarmigan <hidden>
Date: 2016-06-15 22:45:16
On Fri, Aug 29, 2008 at 10:39 AM, Shawn O. Pearce [off-list ref] wrote:
Yet another draft follows. I believe that I have covered all comments with this draft. But I welcome any additional ones, as thus far it has been a very constructive process.
Sorry I'm jumping into this a bit late, but something just occurred to me.
The updated protocol looks more like the current native protocol does. This should make it easier to reuse code between the two protocol implementations. --8<-- Smart HTTP transfer protocols
[...]
HTTP Redirects -------------- If a POST request results in an HTTP 302 or 303 redirect response clients should retry the request by updating the URL and POSTing the same request to the new location. Subsequent requests should still be sent to the original URL. This redirect behavior is unrelated to the in-payload redirect that is described below in "Service show-ref".
I just want to see smart http could support a new feature (please yell if git:// already supports this and I am not aware of it). The idea is from http://lkml.org/lkml/2008/8/21/347, the relevant portion being: Greg KH wrote:
David Vrabel wrote:quoted
Or you can pull the changes from the uwb branch of git://pear.davidvrabel.org.uk/git/uwb.git (Please don't clone the entire tree from here as I have very limited bandwidth.)If this is an issue, I think you can use the --reference option to git-clone when creating the tree to reference an external tree (like Linus's). That way you don't have the whole tree on your server for stuff like this.
I do not believe that the server (either git:// or http://) can
currently be setup with --reference to redirect to another server for
certain refs, but perhaps with smart http and the POST 302/303
redirect responses, this would now be possible as a way to reduce
bandwidth for people's home servers? I have also seen similar
requests before ("don't pull the whole kernel from me, just add my
repo as a remote after you've cloned linus-2.6"), so for larger
projects, it might be a nice feature. Would that be something
desirable to support?
Would the current proposal be able to support this kind of partial
redirect? I don't quite see how it would, but it seems very close.
Perhaps if the show-ref redirect could appear partway through the
show-ref response and then the client could go off, fetch the some
refs from that server and then return to the original server for the
remainder? Or maybe in the upload-pack negotiations, there could be a
special redirect command as part of the "status continue" response
that told the client to run off and look for a specific sha at another
url? Something like
status continue
S: 0014status continue
S: 0034common <S_COMMON #1>...........................
S: 0034common <S_COMMON #2>...........................
...
Otherwise, it looks very cool, but I have a few more minor questions
to help my general understanding...
If the client has sent 256 HAVE commits and has not yet
received one of those back from S_COMMON, or the client
has emptied C_PENDING it should include a "give-up"
command to let the server know it won't proceed:
C: 000cgive-upWhat does the server do after a 000cgive-up ? Does the server send back a complete pack (like a new clone) or if not, how does clone work over smart http? Does that mean that if I fall more than 256 commits behind, I have to redownload the whole repo? Or am I missing something about the the C_PENDING commits being sparse and doing some kind of smart back-off (I'm not at all familiar with the existing receive-pack/upload-pack)?
(s) Parse the upload-pack request:
Verify all objects in WANT are reachable from refs. As
this may require walking backwards through history to
the very beginning on invalid requests the server may
use a reasonable limit of commits (e.g. 1000) walked
beyond any ref tip before giving up.
If no WANT objects are received, send an error:
S: 0019status error no want
If any WANT object is not reachable, send an error:
S: 001estatus error invalid wantSo again, if the client falls more than 1000 commits behind (not hard to do for example during the linux merge window), and then the client WANTs HEAD^1001, what happens? Does the get nothing from the server, or does the client essentially reclone, or I am missing something?
(s) Send the upload-pack response:
If the server has found a closed set of objects to pack or the
request contains "give-up", it replies with the pack and the
enabled capabilities. The set of enabled capabilities is limited
to the intersection of what the client requested and what the
server supports.
S: 0010status pack
C: 001bcapability include-tag
C: 0019capability thin-pack
S: 000c.PACK...Should these be all S: ... ? Thanks, Tarmigan
From: Tarmigan <hidden>
Date: 2016-06-15 22:45:16
(Oops, hit send too early by mistake, so some of my thoughts were incomplete) On Mon, Sep 1, 2008 at 9:05 AM, Tarmigan [off-list ref] wrote:
On Fri, Aug 29, 2008 at 10:39 AM, Shawn O. Pearce [off-list ref] wrote:quoted
Yet another draft follows. I believe that I have covered all comments with this draft. But I welcome any additional ones, as thus far it has been a very constructive process.Sorry I'm jumping into this a bit late, but something just occurred to me.quoted
The updated protocol looks more like the current native protocol does. This should make it easier to reuse code between the two protocol implementations. --8<-- Smart HTTP transfer protocols[...]quoted
HTTP Redirects -------------- If a POST request results in an HTTP 302 or 303 redirect response clients should retry the request by updating the URL and POSTing the same request to the new location. Subsequent requests should still be sent to the original URL. This redirect behavior is unrelated to the in-payload redirect that is described below in "Service show-ref".I just want to see smart http could support a new feature (please yell if git:// already supports this and I am not aware of it). The idea is from http://lkml.org/lkml/2008/8/21/347, the relevant portion being: Greg KH wrote:quoted
David Vrabel wrote:quoted
Or you can pull the changes from the uwb branch of git://pear.davidvrabel.org.uk/git/uwb.git (Please don't clone the entire tree from here as I have very limited bandwidth.)If this is an issue, I think you can use the --reference option to git-clone when creating the tree to reference an external tree (like Linus's). That way you don't have the whole tree on your server for stuff like this.I do not believe that the server (either git:// or http://) can currently be setup with --reference to redirect to another server for certain refs, but perhaps with smart http and the POST 302/303 redirect responses, this would now be possible as a way to reduce bandwidth for people's home servers? I have also seen similar requests before ("don't pull the whole kernel from me, just add my repo as a remote after you've cloned linus-2.6"), so for larger projects, it might be a nice feature. Would that be something desirable to support? Would the current proposal be able to support this kind of partial redirect? I don't quite see how it would, but it seems very close. Perhaps if the show-ref redirect could appear partway through the show-ref response and then the client could go off, fetch the some refs from that server and then return to the original server for the remainder? Or maybe in the upload-pack negotiations, there could be a special redirect command as part of the "status continue" response that told the client to run off and look for a specific sha at another url? Something like status continue S: 0014status continue S: 0034common <S_COMMON #1>........................... S: 0034common <S_COMMON #2>........................... ...
I meant to write:
S: 0014status continue
S: 0034common <S_COMMON #1>...........................
S: 0034common <S_COMMON #2>...........................
S: 00xxredirect <WILL_BE_COMMON> <REMOTE_URL>
and then the client could go try the remote url, fetch that SHA and
ancestors, and then resume the upload pack negotiations with
<WILL_BE_COMMON> among the <COMMON> commits. Obviously it's still
somewhat of a half baked idea, and would probably need some kind of
fallback, but does that seem like a reasonable thing to do and a
reasonable way to do it?
Otherwise, it looks very cool, but I have a few more minor questions to help my general understanding...quoted
If the client has sent 256 HAVE commits and has not yet received one of those back from S_COMMON, or the client has emptied C_PENDING it should include a "give-up" command to let the server know it won't proceed: C: 000cgive-upWhat does the server do after a 000cgive-up ? Does the server send back a complete pack (like a new clone) or if not, how does clone work over smart http? Does that mean that if I fall more than 256 commits behind, I have to redownload the whole repo? Or am I missing something about the the C_PENDING commits being sparse and doing some kind of smart back-off (I'm not at all familiar with the existing receive-pack/upload-pack)?quoted
(s) Parse the upload-pack request: Verify all objects in WANT are reachable from refs. As this may require walking backwards through history to the very beginning on invalid requests the server may use a reasonable limit of commits (e.g. 1000) walked beyond any ref tip before giving up. If no WANT objects are received, send an error: S: 0019status error no want If any WANT object is not reachable, send an error: S: 001estatus error invalid wantSo again, if the client falls more than 1000 commits behind (not hard to do for example during the linux merge window), and then the client WANTs HEAD^1001, what happens? Does the get nothing from the server, or does the client essentially reclone, or I am missing something?quoted
(s) Send the upload-pack response: If the server has found a closed set of objects to pack or the request contains "give-up", it replies with the pack and the enabled capabilities. The set of enabled capabilities is limited to the intersection of what the client requested and what the server supports. S: 0010status pack C: 001bcapability include-tag C: 0019capability thin-pack S: 000c.PACK...Should these be all S: ... ?
Thanks, Tarmigan
From: Shawn O. Pearce <hidden>
Date: 2016-06-15 22:45:16
Tarmigan [off-list ref] wrote:
On Fri, Aug 29, 2008 at 10:39 AM, Shawn O. Pearce [off-list ref] wrote: I just want to see smart http could support a new feature (please yell if git:// already supports this and I am not aware of it). The idea is from http://lkml.org/lkml/2008/8/21/347, the relevant portion being: Greg KH wrote:quoted
David Vrabel wrote:quoted
Or you can pull the changes from the uwb branch of git://pear.davidvrabel.org.uk/git/uwb.git (Please don't clone the entire tree from here as I have very limited bandwidth.)If this is an issue, I think you can use the --reference option to git-clone when creating the tree to reference an external tree (like Linus's). That way you don't have the whole tree on your server for stuff like this.I do not believe that the server (either git:// or http://) can currently be setup with --reference to redirect to another server for certain refs,
Correct. Today _none_ of the transport protocols allow the server
to force the client to use some sort of reference repository for an
initial clone. There are likely two reasons for this:
*) Its a lot simpler to program to just get everything from
one location.
*) If you really are forking an open source project then in
some cases you may need to distribute the full source,
not your delta. You may just as well distribute the full
source and call it a day.
The dumb http:// currently supports getting packs from a remote HTTP
server via its objects/info/http-alternates. But the native and
rsync protocols don't support that. The logic behind http-alternates
isn't to allow moving load onto a different server, but to make
the locally available alternate repository available through the
same web server. The path on the UNIX filesystem that is used in
objects/info/alternates may not be the same path used in the web
server's namespace.
but perhaps with smart http and the POST 302/303
redirect responses, this would now be possible as a way to reduce
bandwidth for people's home servers? I have also seen similar
requests before ("don't pull the whole kernel from me, just add my
repo as a remote after you've cloned linus-2.6"), so for larger
projects, it might be a nice feature. Would that be something
desirable to support?I think this isn't a bad idea, but I'd rather have the server say "In order to talk to me you need at least these objects in common with me: ...". If you don't have those then the user should go find it on their own, rather than forcing them to a particular URL and automatically following it. I'm a little concerned about a US user putting a US mirror site of kernel.org into the server and forcing a user in India to do a full clone over the Atlantic links when they could have just used a more local mirror for that initial "linus-2.6" clone.
Otherwise, it looks very cool, but I have a few more minor questions to help my general understanding...quoted
If the client has sent 256 HAVE commits and has not yet received one of those back from S_COMMON, or the client has emptied C_PENDING it should include a "give-up" command to let the server know it won't proceed: C: 000cgive-upWhat does the server do after a 000cgive-up ? Does the server send back a complete pack (like a new clone) or if not, how does clone work over smart http?
When the server receives a "give-up" it needs to create a pack based on "git rev-list --objects-boundary $WANT --not $COMMON". If the set $COMMON is non-empty then its a partial pack; if $COMMON is empty then its a full clone. This is what the native protocol does when the client gives up.
Does that mean that if I fall more than 256 commits behind, I have to redownload the whole repo?
You are thinking the wrong way. If you have more than 256 commits that the other side doesn't have you may give up too early. For that to be true you need to create 256 commits locally that aren't on the remote peer and whose timestamps are all ahead of the commits you last fetched from the remote peer. Yes, it can happen. But its less likely than you think because we're talking about you doing 256 commits worth of development and not picking up any new commits from remote peers in the middle of that time period. Get just one and it resets the counter back to 0 and allows it to try another 256 commits before giving up. I should amend this section to talk about what giving up here really means. If we have nothing sent in common yet or maybe very little sent in common we may have existing remote refs tied to this URL in .git/config that can send, and we may have one or more annotated tags that we know for a fact are in common as both peers have the same tag name pointing to the same tag object. A smart(er) client might try to toss some recently dated annotated tags at the server before throwing a give-up if it would otherwise throw a give-up. Its likely to narrow the result set, and doesn't hurt if it doesn't.
quoted
(s) Parse the upload-pack request: Verify all objects in WANT are reachable from refs. As this may require walking backwards through history to the very beginning on invalid requests the server may use a reasonable limit of commits (e.g. 1000) walked beyond any ref tip before giving up. If no WANT objects are received, send an error: S: 0019status error no want If any WANT object is not reachable, send an error: S: 001estatus error invalid wantSo again, if the client falls more than 1000 commits behind (not hard to do for example during the linux merge window), and then the client WANTs HEAD^1001, what happens? Does the get nothing from the server, or does the client essentially reclone, or I am missing something?
Oh, this is a live-lock condition. If the client grabs the list of refs from the server, then has to wait 100 ms to get back to the server and start upload-pack (due to latency) and in that 100ms window Linus shoves a 1001 commit merge into his tree then yes, the server may abort and tell the client "error invalid want". At which point the client may try to restart from the beginning, or just plain give up and tell the end user try again later. This condition of 1000 is just some aribtrary limit to allow the client to still continue with an in-progress download if right in the middle of the client's RPCs the remote was modified by its owner.
quoted
(s) Send the upload-pack response: If the server has found a closed set of objects to pack or the request contains "give-up", it replies with the pack and the enabled capabilities. The set of enabled capabilities is limited to the intersection of what the client requested and what the server supports. S: 0010status pack C: 001bcapability include-tag C: 0019capability thin-pack S: 000c.PACK...Should these be all S: ... ?
Yes, thanks. I will make the correction. Damn copy and paste. -- Shawn.
From: "H. Peter Anvin" <hpa@zytor.com>
Date: 2016-06-15 22:45:16
Shawn O. Pearce wrote:
Correct. Today _none_ of the transport protocols allow the server
to force the client to use some sort of reference repository for an
initial clone. There are likely two reasons for this:
*) Its a lot simpler to program to just get everything from
one location.
*) If you really are forking an open source project then in
some cases you may need to distribute the full source,
not your delta. You may just as well distribute the full
source and call it a day.3) it encourages single points of failure. -hpa
From: Shawn O. Pearce <hidden>
Date: 2016-06-15 22:45:16
"H. Peter Anvin" [off-list ref] wrote:
Shawn O. Pearce wrote:quoted
Correct. Today _none_ of the transport protocols allow the server to force the client to use some sort of reference repository for an initial clone. There are likely two reasons for this: *) Its a lot simpler to program to just get everything from one location. *) If you really are forking an open source project then in some cases you may need to distribute the full source, not your delta. You may just as well distribute the full source and call it a day.3) it encourages single points of failure.
Or bad network usage, as I pointed out later about an India user unknowingly being forced into a US based mirror when another was closer to them. I didn't make it clear in my response but I'm really against our protocol having this sort of explicit redirect. I'd rather put a requirement in that says "Unless you have X,Y,Z in common with me (directly or indirectly) I'm just not going to give you a pack". FWIW that fixes an issue for me at day-job that people will be cursing about later this year in public. Not my fault. We would all rather just publish the entire repository. Instead we have to publish something that requires the user to clone it from another source first, and use fetch or "clone --reference" to get our updates. *sigh* -- Shawn.
From: Tarmigan <hidden>
Date: 2016-06-15 22:45:16
On Mon, Sep 1, 2008 at 11:06 PM, Shawn O. Pearce [off-list ref] wrote:
quoted
What does the server do after a 000cgive-up ? Does the server send back a complete pack (like a new clone) or if not, how does clone work over smart http?When the server receives a "give-up" it needs to create a pack based on "git rev-list --objects-boundary $WANT --not $COMMON". If the set $COMMON is non-empty then its a partial pack; if $COMMON is empty then its a full clone. This is what the native protocol does when the client gives up.
OK, that makes sense now.
quoted
Does that mean that if I fall more than 256 commits behind, I have to redownload the whole repo?You are thinking the wrong way. If you have more than 256 commits that the other side doesn't have you may give up too early. For that to be true you need to create 256 commits locally that aren't on the remote peer and whose timestamps are all ahead of the commits you last fetched from the remote peer. Yes, it can happen. But its less likely than you think because we're talking about you doing 256 commits worth of development and not picking up any new commits from remote peers in the middle of that time period. Get just one and it resets the counter back to 0 and allows it to try another 256 commits before giving up. I should amend this section to talk about what giving up here really means. If we have nothing sent in common yet or maybe very little sent in common we may have existing remote refs tied to this URL in .git/config that can send, and we may have one or more annotated tags that we know for a fact are in common as both peers have the same tag name pointing to the same tag object. A smart(er) client might try to toss some recently dated annotated tags at the server before throwing a give-up if it would otherwise throw a give-up. Its likely to narrow the result set, and doesn't hurt if it doesn't.
Yes, throwing in tags and remotes as a last resort sounds like a good idea.
quoted
So again, if the client falls more than 1000 commits behind (not hard to do for example during the linux merge window), and then the client WANTs HEAD^1001, what happens? Does the get nothing from the server, or does the client essentially reclone, or I am missing something?Oh, this is a live-lock condition. If the client grabs the list of refs from the server, then has to wait 100 ms to get back to the server and start upload-pack (due to latency) and in that 100ms window Linus shoves a 1001 commit merge into his tree then yes, the server may abort and tell the client "error invalid want".
Ahh, now I get it. Somehow I forgot that the WANTs were only boundary commits and not a list of all the commits that the client wants. On Mon, Sep 1, 2008 at 11:13 PM, Shawn O. Pearce [off-list ref] wrote:
"H. Peter Anvin" [off-list ref] wrote:quoted
Shawn O. Pearce wrote:quoted
Correct. Today _none_ of the transport protocols allow the server to force the client to use some sort of reference repository for an initial clone. There are likely two reasons for this: *) Its a lot simpler to program to just get everything from one location. *) If you really are forking an open source project then in some cases you may need to distribute the full source, not your delta. You may just as well distribute the full source and call it a day.3) it encourages single points of failure.Or bad network usage, as I pointed out later about an India user unknowingly being forced into a US based mirror when another was closer to them. I didn't make it clear in my response but I'm really against our protocol having this sort of explicit redirect. I'd rather put a requirement in that says "Unless you have X,Y,Z in common with me (directly or indirectly) I'm just not going to give you a pack".
OK, this all makes sense. http:// and git:// are probably the wrong protocols to reduce bandwidth for the server for new clones. Long term, maybe gittorrent will be the right solution... Thanks, Tarmigan