Handle HTTP error 511 Network Authentication Required (standard secure proxy authentification/captive portal detection)

Since
http://code.google.com/p/chromium/issues/detail?id=7338 and
https://bugzilla.mozilla.org/show_bug.cgi?id=479880

there is no clean way for a proxy or captive portal to get a web client to
display an authentication dialog when user credentials expire while he is
browsing on an https url.

(to be sure, the previous methods were insecure and hackish but they existed
because nothing better was available)

The IETF finally set up to fix this problem and defined a standard HTTP error
that lets access control equipments tell the web client authentication or
re-authentication is needed and where the authentication form is located.

http://tools.ietf.org/id/draft-nottingham-http-new-status-04.txt

→ <http://www.rfc-editor.org/queue2.html#draft-nottingham-http-new-status> (the
spec is approved and in the queue for publication as RFC)

Please add error 511 handling in git, so git users that try to access external
git repositories over http can authenticate on the corporate proxy

On Sun, Feb 19, 2012 at 10:03:37PM +0100, Nicolas Mailhot wrote:

The IETF finally set up to fix this problem and defined a standard HTTP error
that lets access control equipments tell the web client authentication or
re-authentication is needed and where the authentication form is located.

http://tools.ietf.org/id/draft-nottingham-http-new-status-04.txt

→ <http://www.rfc-editor.org/queue2.html#draft-nottingham-http-new-status> (the
spec is approved and in the queue for publication as RFC)

Please add error 511 handling in git, so git users that try to access external
git repositories over http can authenticate on the corporate proxy

If I'm reading this right, the process works something like this:

  1. Git wants to make a request to http://example.com

  2. We make our request to a proxy server which is transparently
     proxying our traffic (i.e, a "captive portal").

  3. The proxy returns 511 along with some URL (e.g.,
     "http://login.corporatenetwork"), indicating that we need
     to go to that URL to complete some authentication.

As a non-browser client, what should git do? We can't make sense of the
content at http://login.corporatenetwork, which is most likely an HTML
form asking for credentials (or even money, if the captive portal is
something like a public wireless provider). The best we can probably do
is die and say "apparently you need to go http://login.corporatenetwork
in a browser before making your request".

Reading that rfc draft, the man impetus for non-browser clients seems
not to get them to do anything useful, but rather to return them a code
that is clearly not from the actual site (if it were a redirect, for
example, then we would think that example.com is redirecting is, which
is simply not true).

-Peff

Le Lun 20 février 2012 02:06, Jeff King a écrit :

On Sun, Feb 19, 2012 at 10:03:37PM +0100, Nicolas Mailhot wrote:

quoted

The IETF finally set up to fix this problem and defined a standard HTTP
error
that lets access control equipments tell the web client authentication or
re-authentication is needed and where the authentication form is located.

http://tools.ietf.org/id/draft-nottingham-http-new-status-04.txt

→ <http://www.rfc-editor.org/queue2.html#draft-nottingham-http-new-status>
(the
spec is approved and in the queue for publication as RFC)

Please add error 511 handling in git, so git users that try to access
external
git repositories over http can authenticate on the corporate proxy

If I'm reading this right, the process works something like this:

  1. Git wants to make a request to http://example.com

  2. We make our request to a proxy server which is transparently
     proxying our traffic (i.e, a "captive portal").

  3. The proxy returns 511 along with some URL (e.g.,
     "http://login.corporatenetwork"), indicating that we need
     to go to that URL to complete some authentication.

As a non-browser client, what should git do? We can't make sense of the
content at http://login.corporatenetwork, which is most likely an HTML
form asking for credentials (or even money, if the captive portal is
something like a public wireless provider). The best we can probably do
is die and say "apparently you need to go http://login.corporatenetwork
in a browser before making your request".

Actually, the best would be to launch something capable of interpreting html
forms on the url given by the error. But short of that, that depends on how
good git is at resuming work later. Error 511 can occur at any time, not just
on initial connection (because credentials can expire at any time). So pausing
may be better than dying.

However without going there: the portal page will usually be pretty simple, a
standard basic auth form, can't git handle this? If simple web clients such as
git have specific constrains on what can appear or not on this page, can you
not define them and send them ietf-side so they can document them in a later
rfc revision?

Reading that rfc draft, the man impetus for non-browser clients seems
not to get them to do anything useful, but rather to return them a code
that is clearly not from the actual site (if it were a redirect, for
example, then we would think that example.com is redirecting is, which
is simply not true).

The main impetus from my point of view is that captive portal/proxy auth is a
mess, because they try to trick web clients into displaying something they are
not prepared to and don't want to do, and this spec is replacing trick with
explicit request, which is nice.

Best regards,

-- 
Nicolas Mailhot

On Mon, Feb 20, 2012 at 06:38:54AM +0100, Nicolas Mailhot wrote:

quoted

As a non-browser client, what should git do? We can't make sense of the
content at http://login.corporatenetwork, which is most likely an HTML
form asking for credentials (or even money, if the captive portal is
something like a public wireless provider). The best we can probably do
is die and say "apparently you need to go http://login.corporatenetwork
in a browser before making your request".

Actually, the best would be to launch something capable of interpreting html
forms on the url given by the error.

Doing that portably is near impossible (keep in mind that git runs on
things like antique versions of Solaris). Not to mention that git is
often running without a user present, or on a remote system connected
to the user only by ssh. So our best bet would probably be a
configuration option to tell git how to run a browser. Distributions
which ship binary packages could ship a sane default for their system.

But short of that, that depends on how good git is at resuming work
later. Error 511 can occur at any time, not just on initial connection
(because credentials can expire at any time). So pausing may be better
than dying.

It can, but I doubt it is a big problem with smart-http. We will make a
handful of quick negotiation requests at the beginning, and then the
bulk of the data will come over a single http request.

However without going there: the portal page will usually be pretty simple, a
standard basic auth form, can't git handle this? If simple web clients such as
git have specific constrains on what can appear or not on this page, can you
not define them and send them ietf-side so they can document them in a later
rfc revision?

Git handles http basic auth. But my experience has been that captive
portals almost _never_ do basic auth. Instead, they give you an html
page with a bunch of form fields. And possibly some javascript required
to submit it. Git does not understand either of those things, and nor
should it; spawning a browser is the right thing to do there.

I don't think the IETF can or should mandate what goes on such a page.
Some portals will want login/password. Some will want billing
information. Some will even want other things (at some airports, I have
seen captive portals offer the option to take a short survey in return
for net access for a day). So it is not a matter of what git wants, but
what the captive portals want.

The main impetus from my point of view is that captive portal/proxy auth is a
mess, because they try to trick web clients into displaying something they are
not prepared to and don't want to do, and this spec is replacing trick with
explicit request, which is nice.

Yeah, and I think it is an improvement over the current state. Right now
git will properly fail with a 511 (which is what the designers
intended). I'd rather hold off on something more advanced until somebody
comes forward with a concrete case and says "hey, my network gives a 511
in this instance, and here's what would be the best thing for git to
do". Preferably with patches, of course. :)

-Peff

Le Lun 20 février 2012 14:56, Jeff King a écrit :

On Mon, Feb 20, 2012 at 06:38:54AM +0100, Nicolas Mailhot wrote:

quoted

quoted

As a non-browser client, what should git do? We can't make sense of the
content at http://login.corporatenetwork, which is most likely an HTML
form asking for credentials (or even money, if the captive portal is
something like a public wireless provider). The best we can probably do
is die and say "apparently you need to go http://login.corporatenetwork
in a browser before making your request".

Actually, the best would be to launch something capable of interpreting html
forms on the url given by the error.

Doing that portably is near impossible (keep in mind that git runs on
things like antique versions of Solaris).

Can't the you let the user specify a browser command (firefox, elinks w3m) to
auto-feed the portal page to when needed ?

The main problem with captive portals is when they shut down the connection
and the user has no idea how to restore it (and error 511 is intended to fix
this, but that won't do a lot of good if the user does is not shown the
captive portal url transmitted with the error)

Regards,

-- 
Nicolas Mailhot

On Mon, Feb 20, 2012 at 04:34:19PM +0100, Nicolas Mailhot wrote:

quoted

quoted

Actually, the best would be to launch something capable of interpreting html
forms on the url given by the error.

Doing that portably is near impossible (keep in mind that git runs on
things like antique versions of Solaris).

Can't the you let the user specify a browser command (firefox, elinks w3m) to
auto-feed the portal page to when needed ?

Yes, that's why I said "we could add a configuration option" in the part
that you snipped. But doing it out of the box is not going to be
portable.

The main problem with captive portals is when they shut down the connection
and the user has no idea how to restore it (and error 511 is intended to fix
this, but that won't do a lot of good if the user does is not shown the
captive portal url transmitted with the error)

In my experience, the captive portal process usually goes like this:

  1. Connect to network.

  2. Try some non-browser command. Wonder why in the world it isn't
     working.

  3. Open a browser and say "Ah, I see. A captive portal".

The 511 proposal makes step 2 a lot better if the protocol is http[1].
But it pretty much makes it better even without non-browser client
support, because at least you will get a 511 error instead of having git
complain that the remote repository is corrupted (which happens if the
captive portal returns a redirect to an html page).

We should already be doing that. Adding more support could make step 3 a
little nicer, but like I said, I'd be more interested in seeing a real
case first. It may even be a feature that would be more appropriate to
curl (which git builds on for http access).

-Peff

[1] Of course it doesn't help at all for git:// or ssh:// (which are
    usually even worse off in the first place, as many captive portals
    will simply drop the packets, making it look like the remote server
    is down).

Le Lun 20 février 2012 16:44, Jeff King a écrit :

In my experience, the captive portal process usually goes like this:

  1. Connect to network.

  2. Try some non-browser command. Wonder why in the world it isn't
     working.

  3. Open a browser and say "Ah, I see. A captive portal".

The 511 proposal makes step 2 a lot better if the protocol is http[1].
But it pretty much makes it better even without non-browser client
support, because at least you will get a 511 error instead of having git
complain that the remote repository is corrupted (which happens if the
captive portal returns a redirect to an html page).

We should already be doing that. Adding more support could make step 3 a
little nicer, but like I said, I'd be more interested in seeing a real
case first. It may even be a feature that would be more appropriate to
curl (which git builds on for http access).

Step 3 is a quite less obvious on a corporate network, where Internet access
is gated by a filtering proxy, that will let some sites pass transparently but
require credentials to let you access others. Worst case, there are several
load-balanced gateways on different physical sites (to avoid spofs in case of
planes falling on the wrong place), that do not share authentication (because
propagating auth across physical sites is hard). So no, just launching a
browser is not sufficient to find the captive portal, you need to actually
access the URL returned by error 511 in meta information. Git should at
minimum report this URL.

(and no this is not an hypothetical scenario and yes there are git users
trying to pass the gateways there)

-- 
Nicolas Mailhot

On Mon, Feb 20, 2012 at 08:06:46PM +0100, Daniel Stenberg wrote:

quoted

3. Open a browser and say "Ah, I see. A captive portal".

We should already be doing that. Adding more support could make
step 3 a little nicer, but like I said, I'd be more interested in
seeing a real case first. It may even be a feature that would be
more appropriate to curl (which git builds on for http access).

We're already discussing the 511 in the curl camp as well, but with
even more sighs and hands in the air. 511 is clearly intended for
HTML-understanding user agents and curl is not one of those. IMHO,
curl will remain to simply help users to figure out that it is 511
and leave it at that.

Thanks for the input. It sounds like our best bet is to just report the
URL from a 511 better, then. Do you have any idea yet how that
information will be available to curl library users?

As a git user, I would probably be very surprised if using 'git'
suddenly caused by browser to pop up a captive portal login. I would
prefer git to instead properly explain to me that is being the victim
of a 511 and what I should do to fix it.

I agree. Even if the "step 3" in my list is "then the user starts a
browser given the URL from git's error message", that is a huge
improvement over the current state. And it retains the principle of
least surprise.

-Peff

On Mon, Feb 20, 2012 at 07:27:08PM +0100, Nicolas Mailhot wrote:

Step 3 is a quite less obvious on a corporate network, where Internet access
is gated by a filtering proxy, that will let some sites pass transparently but
require credentials to let you access others. Worst case, there are several
load-balanced gateways on different physical sites (to avoid spofs in case of
planes falling on the wrong place), that do not share authentication (because
propagating auth across physical sites is hard). So no, just launching a
browser is not sufficient to find the captive portal, you need to actually
access the URL returned by error 511 in meta information. Git should at
minimum report this URL.

(and no this is not an hypothetical scenario and yes there are git users
trying to pass the gateways there)

This is exactly the sort of information I wanted to get from a
real-world scenario. From your initial messages, it sounded like a
purely hypothetical thing.

I think a good first step would be improving the error message for a
511, then. Unfortunately, it seems from the rfc draft you sent that
callers are expected to parse the link out of the HTML given in the body
of the response. It seems silly that there is not a Location field
associated with a 511, similar to redirects.

-Peff

On Mon, 20 Feb 2012, Jeff King wrote:

 3. Open a browser and say "Ah, I see. A captive portal".

We should already be doing that. Adding more support could make step 3 a 
little nicer, but like I said, I'd be more interested in seeing a real case 
first. It may even be a feature that would be more appropriate to curl 
(which git builds on for http access).

We're already discussing the 511 in the curl camp as well, but with even more 
sighs and hands in the air. 511 is clearly intended for HTML-understanding 
user agents and curl is not one of those. IMHO, curl will remain to simply 
help users to figure out that it is 511 and leave it at that.

As a git user, I would probably be very surprised if using 'git' suddenly 
caused by browser to pop up a captive portal login. I would prefer git to 
instead properly explain to me that is being the victim of a 511 and what I 
should do to fix it.

-- 

  / daniel.haxx.se

Le Lun 20 février 2012 20:15, Jeff King a écrit :

On Mon, Feb 20, 2012 at 07:27:08PM +0100, Nicolas Mailhot wrote:

quoted

Step 3 is a quite less obvious on a corporate network, where Internet access
is gated by a filtering proxy, that will let some sites pass transparently
but
require credentials to let you access others. Worst case, there are several
load-balanced gateways on different physical sites (to avoid spofs in case
of
planes falling on the wrong place), that do not share authentication
(because
propagating auth across physical sites is hard). So no, just launching a
browser is not sufficient to find the captive portal, you need to actually
access the URL returned by error 511 in meta information. Git should at
minimum report this URL.

(and no this is not an hypothetical scenario and yes there are git users
trying to pass the gateways there)

This is exactly the sort of information I wanted to get from a
real-world scenario. From your initial messages, it sounded like a
purely hypothetical thing.

I think a good first step would be improving the error message for a
511, then. Unfortunately, it seems from the rfc draft you sent that
callers are expected to parse the link out of the HTML given in the body
of the response. It seems silly that there is not a Location field
associated with a 511, similar to redirects.

The URL is not lost in the HTML text, it's in the url meta field

<meta http-equiv="refresh"
       content="0; url=https://login.example.net/">

As for while there is no Location field, I think it's because otherwise it
could behave like a redirect, and browser people made it plain they didn't
want redirects of https accesses (but I wasn't there when the spec was
written, and only skimmed the workgroup archives, so there may have been other
reasons for this choice. I'm pretty sure it's deliberate anyway).

Regards,

-- 
Nicolas Mailhot

On Mon, Feb 20, 2012 at 08:24:15PM +0100, Nicolas Mailhot wrote:

quoted

I think a good first step would be improving the error message for a
511, then. Unfortunately, it seems from the rfc draft you sent that
callers are expected to parse the link out of the HTML given in the body
of the response. It seems silly that there is not a Location field
associated with a 511, similar to redirects.

The URL is not lost in the HTML text, it's in the url meta field

<meta http-equiv="refresh"
       content="0; url=https://login.example.net/">

Sorry, but

  1. That is in the HTML in the body of the response (by body I don't
     mean the HTML <body>, but the body of the http request).

  2. I don't see anything in the rfc indicating that there must be a
     meta tag in the response. They use it in the example of the rfc,
     but they also have human-readable text with an <a> link.  Do we yet
     know what will be common among captive portals?

You said you have a non-hypothetical case. Can you show us the response?

As for while there is no Location field, I think it's because otherwise it
could behave like a redirect, and browser people made it plain they didn't
want redirects of https accesses (but I wasn't there when the spec was
written, and only skimmed the workgroup archives, so there may have been other
reasons for this choice. I'm pretty sure it's deliberate anyway).

Even if they didn't call it Location, it would be nice to have some
machine-readable format that is understood by non-browser agents that
don't know how to parse HTML. But I recognize that is not your decision,
so don't feel obligated to defend it.

-Peff

Le Lun 20 février 2012 20:30, Jeff King a écrit :

On Mon, Feb 20, 2012 at 08:24:15PM +0100, Nicolas Mailhot wrote:

quoted

quoted

I think a good first step would be improving the error message for a
511, then. Unfortunately, it seems from the rfc draft you sent that
callers are expected to parse the link out of the HTML given in the body
of the response. It seems silly that there is not a Location field
associated with a 511, similar to redirects.

The URL is not lost in the HTML text, it's in the url meta field

<meta http-equiv="refresh"
       content="0; url=https://login.example.net/">

Sorry, but

  1. That is in the HTML in the body of the response (by body I don't
     mean the HTML <body>, but the body of the http request).

  2. I don't see anything in the rfc indicating that there must be a
     meta tag in the response. They use it in the example of the rfc,
     but they also have human-readable text with an <a> link.  Do we yet
     know what will be common among captive portals?

You said you have a non-hypothetical case. Can you show us the response?

Not yet because it's currently non-standard custom redirection mess we're
repurposing to follow the ietf spec (got tired of being accused of running a
crap non-standard proxy by users, so now it's ll be a crap standard proxy)

The proxy response is totally configurable (a so there's no reason we won't
follow the new spec to the letter


-- 
Nicolas Mailhot