From: Junio C Hamano <hidden> Date: 2016-06-15 22:53:54
Colby Ranger [off-list ref] writes:
Git over HTTPS has a high request startup latency, since the SSL
negotiation can take up to a second. In order to reduce this latency,
connections should be left open to the Git server across requests
(or invocations of the git commandline).
Reduce SSL startup latency by running a daemon job that keeps
connections open to a Git server. The daemon job
(git-remote-persistent-https--proxy) is started on the first request
through the client binary (git-remote-persistent-https) and remains
running for 24 hours after the last request, or until a new daemon
binary is placed in the PATH. The client determines the daemon's
HTTP address by communicating over a UNIX socket with the daemon.
quoted
From there, the rest of the Git protocol work is delegated to the
"git-remote-http" binary, with the environment's http_proxy set to
the daemon.
Signed-off-by: Colby Ranger <redacted>
Clever. Do you have some numbers?
If this persistent proxy weren't in the picture, the git client would
directly delegate its communication with the origin server to
git-remote-https, and git-remote-https would interact with the credential
API to handle authentication for "https://origin.server.xz/repo".
How does the persistent proxy sitting in between your git client and the
origin server and/or the real proxy you have at the perimeter of your
network interact with respect to authentication/authorization to access
the repository or the real proxy? They will talk with the persistent
proxy, and the persistent proxy will talk with the git client via
git-remote-http, and I am assuming that this last git-remote-http will be
the one that uses the credential API. Would it ask credential for
"https://origin.server.xz/repo"? "http://origin.server.xz/repo"? Or
would it be "persistent-https://origin.server.xz.repo"?
I do not mind carrying this in the contrib/ area (I am assuming that
distributing Apache licensed software that does not link with GPLv2 core
is OK). It may be just me, but a file called COPYING that does not have
GPL text in it was a bit surprising. I wonder if it is more customary to
call it either LICENSE (or perhaps LICENSE-2.0)?
On Thu, May 24, 2012 at 11:17 AM, Junio C Hamano [off-list ref] wrote:
Colby Ranger [off-list ref] writes:
quoted
Git over HTTPS has a high request startup latency, since the SSL
negotiation can take up to a second. In order to reduce this latency,
connections should be left open to the Git server across requests
(or invocations of the git commandline).
Reduce SSL startup latency by running a daemon job that keeps
connections open to a Git server. The daemon job
(git-remote-persistent-https--proxy) is started on the first request
through the client binary (git-remote-persistent-https) and remains
running for 24 hours after the last request, or until a new daemon
binary is placed in the PATH. The client determines the daemon's
HTTP address by communicating over a UNIX socket with the daemon.
quoted
From there, the rest of the Git protocol work is delegated to the
"git-remote-http" binary, with the environment's http_proxy set to
the daemon.
Signed-off-by: Colby Ranger <redacted>
real 0m0.208s
real 0m0.085s
real 0m0.079s
real 0m0.067s
real 0m0.059s
That initial 208ms run was the local proxy starting, and the
connection being established. Its similar to the normal case.
Subsequent executions have lower latency.
If this persistent proxy weren't in the picture, the git client would
directly delegate its communication with the origin server to
git-remote-https, and git-remote-https would interact with the credential
API to handle authentication for "https://origin.server.xz/repo".
Yes.
How does the persistent proxy sitting in between your git client and the
origin server and/or the real proxy you have at the perimeter of your
network interact with respect to authentication/authorization to access
the repository or the real proxy? They will talk with the persistent
proxy, and the persistent proxy will talk with the git client via
git-remote-http, and I am assuming that this last git-remote-http will be
the one that uses the credential API.
IIRC it asks for "http://origin.server.xz/repo".
The persistent-https code tells the git credential helper the
connection is "secure" (that is the proxy will use SSL as it exits the
local machine) by setting GIT_GOOGLE_CREDENTIAL_CORPSSO_ENABLE=1 in
the environment. This leaked from our internal version of the proxy,
Colby was supposed to scrub this string before open sourcing. :-)
So now everyone knows $DAYJOB = Google, we have a credential helper,
and it supports some sort of corporate single sign on. Whee.
I do not mind carrying this in the contrib/ area (I am assuming that
distributing Apache licensed software that does not link with GPLv2 core
is OK). It may be just me, but a file called COPYING that does not have
GPL text in it was a bit surprising. I wonder if it is more customary to
call it either LICENSE (or perhaps LICENSE-2.0)?
Agreed, LICENSE or LICENSE-2.0 is the right name for this file, not COPYING.
real 0m0.208s
real 0m0.085s
real 0m0.079s
real 0m0.067s
real 0m0.059s
Nice numbers. And as clever as I find this helper-wrapping-a-helper
solution, I wonder if the right layer for a fix isn't inside curl. It
already keeps an ssl session-id cache in memory; how hard would it be to
turn that into an on-disk cache?
I don't think that is grounds for rejecting this patch; obviously it is
working for you guys, and it is available right now, and it is only
going into contrib/ anyway. But a curl solution seems like a cleaner
long-term fix, and would benefit all curl users. It is even mentioned in
curl's doc/TODO file. :)
-Peff
real 0m0.208s
real 0m0.085s
real 0m0.079s
real 0m0.067s
real 0m0.059s
Nice numbers. And as clever as I find this helper-wrapping-a-helper
solution, I wonder if the right layer for a fix isn't inside curl. It
already keeps an ssl session-id cache in memory; how hard would it be to
turn that into an on-disk cache?
I don't think that is grounds for rejecting this patch; obviously it is
working for you guys, and it is available right now, and it is only
going into contrib/ anyway. But a curl solution seems like a cleaner
long-term fix, and would benefit all curl users. It is even mentioned in
curl's doc/TODO file. :)
Well, this helper "solution" also has the benefit of HTTP keep-alive
working across Git command invocations. Its common for servers to use
a 5 minute keep-alive on an HTTP 1.1 connection. Git-over-HTTP
commonly uses Transfer-Encoding: chunked on replies, so keep-alive
will generally just work, even though a pack stream's length isn't
known in advance. Because the helper is an external process holding
that connection open, we also benefit from being able to reuse an
existing TCP connection to the server.
But sure, it would be nice if libcurl was able to share SSL sessions
across Git command invocations without this black magic proxy thing.
From: Daniel Stenberg <hidden> Date: 2016-06-15 22:53:54
On Thu, 24 May 2012, Jeff King wrote:
Nice numbers. And as clever as I find this helper-wrapping-a-helper
solution, I wonder if the right layer for a fix isn't inside curl. It
already keeps an ssl session-id cache in memory; how hard would it be to
turn that into an on-disk cache?
I don't think that is grounds for rejecting this patch; obviously it is
working for you guys, and it is available right now, and it is only going
into contrib/ anyway. But a curl solution seems like a cleaner long-term
fix, and would benefit all curl users. It is even mentioned in curl's
doc/TODO file. :)
Yes, in the curl project we would certainly not object to being able to
export/import SSL-session-ids (which then could be used to implement a cache,
on disk or in memory). However, I don't know how easily done that is to
implement with the different TLS libraries in use.
--
/ daniel.haxx.se
real 0m0.208s
real 0m0.085s
real 0m0.079s
real 0m0.067s
real 0m0.059s
Nice numbers. And as clever as I find this helper-wrapping-a-helper
solution, I wonder if the right layer for a fix isn't inside curl. It
already keeps an ssl session-id cache in memory; how hard would it be to
turn that into an on-disk cache?
...
Well, this helper "solution" also has the benefit of HTTP keep-alive
working across Git command invocations.
Here is plaintext HTTP, where the benefit is from HTTP keep-alive:
(for i in {1..5}; do time git ls-remote
http://android.googlesource.com/tools/repo >/dev/null;done) 2>&1 |
grep real
real 0m0.098s
real 0m0.097s
real 0m0.106s
real 0m0.095s
real 0m0.105s
(for i in {1..5}; do time git ls-remote
persistent-http://android.googlesource.com/tools/repo >/dev/null;done)
2>&1 | grep real
real 0m0.134s
real 0m0.065s
real 0m0.063s
real 0m0.061s
real 0m0.067s
Notice we still save 30ms or so in this case. That is about the RTT
for my workstation to that server. :-)
From: Jeff King <hidden> Date: 2016-06-15 22:53:54
On Thu, May 24, 2012 at 01:46:36PM -0700, Shawn O. Pearce wrote:
quoted
quoted
Nice numbers. And as clever as I find this helper-wrapping-a-helper
solution, I wonder if the right layer for a fix isn't inside curl. It
already keeps an ssl session-id cache in memory; how hard would it be to
turn that into an on-disk cache?
...
quoted
Well, this helper "solution" also has the benefit of HTTP keep-alive
working across Git command invocations.
True. It also works even when the server does not support ssl id
caching.
Here is plaintext HTTP, where the benefit is from HTTP keep-alive:
(for i in {1..5}; do time git ls-remote
http://android.googlesource.com/tools/repo >/dev/null;done) 2>&1 |
grep real
real 0m0.098s
real 0m0.097s
real 0m0.106s
real 0m0.095s
real 0m0.105s
(for i in {1..5}; do time git ls-remote
persistent-http://android.googlesource.com/tools/repo >/dev/null;done)
2>&1 | grep real
real 0m0.134s
real 0m0.065s
real 0m0.063s
real 0m0.061s
real 0m0.067s
Notice we still save 30ms or so in this case. That is about the RTT
for my workstation to that server. :-)
Nice. I assumed most of your speedup was coming from dropping the SSL
setup, but it seems that a good chunk of it is just the TCP setup.
Thanks for providing numbers.
-Peff