False negative authentication with multiple accounts on a SSH-GIT server

Hello,

I've noticed an edge case in Git when a user has two Git accounts on a single server - as might be common for a personal account and a work account on a Git forge (Github, GitLab, Bitbucket...)

When attempting SSH login, `ssh` and Git will eagerly connect with the first matching key. This may or may not be the key right key for the repository the user needs. As a result, Git clones, pulls, and pushes will fail with the `Repository not found` if the wrong key is tried first.

For example, the user `alice` has two accounts on the host `git-server.com`. `alice`'s accounts are `alice-work`, and `alice-personal`. `alice-work` has access to the `foo` repository, and `alice-personal` has access to the `bar` repository.

`alice` attempts to clone `foo` with both her `alice-work` and `alice-personal` keys in her SSH Agent. SSH Agent does not define which key it will attempt first, so SSH may connect successfully to `git-server.com` with her `alice-personal` keys, which do not have access to the `foo` repository.

I'd be interested in your opinions for fixes to this. I am willing to make a patch, although my knowledge of the Git codebase isn't perfect.

- Should Git servers provide distinctly different error messages for `access denied`, and `repository does not exist`? Currently the server immediately closes the connection in this case, so `transport.c:handshake()` with fail when attempting to `discover_version()` because the reader hits the EOF. Perhaps the server could send a hypothetical access denied packet here, and a more appropriate error generated? 

Can anyone point me to where in the Git codebase the daemon receives and responds to these requests? I haven't found it yet, if I wanted to patch this.

- Should Git provide a `-i` option to allow the user to choose an SSH key, which could be added to the SSH subprocess's command line?

- Should Git attempt to iterate over all keys in the SSH Agent when the connection is setup, testing the connection to check if each connected key has access to the target repository, before giving up and reporting an error? This may be difficult looking at the current behavior of `ssh` and `ssh-agent`. `ssh-add -l` no longer lists paths to files (which could be plugged into `ssh -i`), just the key signature. Does anyone know of any SSH/SSH-Agent tricks which might help with this?

All the best,



H Paterson.

Hi there,

you can use GIT_SSH_COMMAND or config 'core.sshCommand'.
For more flexibility use wrappers like ssh-ident.

Recommended read: https://superuser.com/a/1616186/557798

Hope this helps
Kind regards
Maddes


On 2021-01-19 07:17, Harley Paterson wrote:

Hello,

I've noticed an edge case in Git when a user has two Git accounts on a
single server - as might be common for a personal account and a work
account on a Git forge (Github, GitLab, Bitbucket...)

When attempting SSH login, `ssh` and Git will eagerly connect with the
first matching key. This may or may not be the key right key for the
repository the user needs. As a result, Git clones, pulls, and pushes
will fail with the `Repository not found` if the wrong key is tried
first.

For example, the user `alice` has two accounts on the host
`git-server.com`. `alice`'s accounts are `alice-work`, and
`alice-personal`. `alice-work` has access to the `foo` repository, and
`alice-personal` has access to the `bar` repository.

`alice` attempts to clone `foo` with both her `alice-work` and
`alice-personal` keys in her SSH Agent. SSH Agent does not define
which key it will attempt first, so SSH may connect successfully to
`git-server.com` with her `alice-personal` keys, which do not have
access to the `foo` repository.

I'd be interested in your opinions for fixes to this. I am willing to
make a patch, although my knowledge of the Git codebase isn't perfect.

- Should Git servers provide distinctly different error messages for
`access denied`, and `repository does not exist`? Currently the server
immediately closes the connection in this case, so
`transport.c:handshake()` with fail when attempting to
`discover_version()` because the reader hits the EOF. Perhaps the
server could send a hypothetical access denied packet here, and a more
appropriate error generated?

Can anyone point me to where in the Git codebase the daemon receives
and responds to these requests? I haven't found it yet, if I wanted to
patch this.

- Should Git provide a `-i` option to allow the user to choose an SSH
key, which could be added to the SSH subprocess's command line?

- Should Git attempt to iterate over all keys in the SSH Agent when
the connection is setup, testing the connection to check if each
connected key has access to the target repository, before giving up
and reporting an error? This may be difficult looking at the current
behavior of `ssh` and `ssh-agent`. `ssh-add -l` no longer lists paths
to files (which could be plugged into `ssh -i`), just the key
signature. Does anyone know of any SSH/SSH-Agent tricks which might
help with this?

All the best,



H Paterson.

On 2021-01-19 at 06:17:21, Harley Paterson wrote:

Hello,

I've noticed an edge case in Git when a user has two Git accounts on a
single server - as might be common for a personal account and a work
account on a Git forge (Github, GitLab, Bitbucket...)

When attempting SSH login, `ssh` and Git will eagerly connect with the
first matching key. This may or may not be the key right key for the
repository the user needs. As a result, Git clones, pulls, and pushes
will fail with the `Repository not found` if the wrong key is tried
first.

For example, the user `alice` has two accounts on the host
`git-server.com`. `alice`'s accounts are `alice-work`, and
`alice-personal`. `alice-work` has access to the `foo` repository, and
`alice-personal` has access to the `bar` repository.

Yes, this is because SSH authentication happens before any command is
run.  The server never knows what resource is being requested until the
user is already authenticated.

`alice` attempts to clone `foo` with both her `alice-work` and
`alice-personal` keys in her SSH Agent. SSH Agent does not define
which key it will attempt first, so SSH may connect successfully to
`git-server.com` with her `alice-personal` keys, which do not have
access to the `foo` repository.

I'd be interested in your opinions for fixes to this. I am willing to
make a patch, although my knowledge of the Git codebase isn't perfect.

We've documented how to deal with situation properly in the FAQ, which
you can see at gitfaq(7) or
https://git-scm.com/docs/gitfaq#Documentation/gitfaq.txt-HowdoIusemultipleaccountswiththesamehostingproviderusingSSH

Is there some reason that this doesn't work for you?

- Should Git servers provide distinctly different error messages for
  `access denied`, and `repository does not exist`? Currently the
  server immediately closes the connection in this case, so
  `transport.c:handshake()` with fail when attempting to
  `discover_version()` because the reader hits the EOF. Perhaps the
  server could send a hypothetical access denied packet here, and a
  more appropriate error generated?

Unfortunately, this leaks whether a repository exists.  If Company XYZ
has a repository for each of its clients, it then becomes easy to see if
Company XYZ is doing work for a particular client by trying to see if a
repository exists.  This would be bad and a huge violation of privacy,
so nobody is likely do to that.  I can tell you as one of the staff who
maintains the GitHub Git service, we're not likely to change the
behavior, and I suspect nobody else is, either.

Can anyone point me to where in the Git codebase the daemon receives
and responds to these requests? I haven't found it yet, if I wanted to
patch this.

- Should Git provide a `-i` option to allow the user to choose an SSH
  key, which could be added to the SSH subprocess's command line?

You can use GIT_SSH_COMMAND or `core.sshcommand` to set this.  We don't
provide such an option because the user is not necessarily using
OpenSSH and different SSH programs provide this feature differently.

- Should Git attempt to iterate over all keys in the SSH Agent when
  the connection is setup, testing the connection to check if each
  connected key has access to the target repository, before giving up
  and reporting an error? This may be difficult looking at the current
  behavior of `ssh` and `ssh-agent`. `ssh-add -l` no longer lists
  paths to files (which could be plugged into `ssh -i`), just the key
  signature. Does anyone know of any SSH/SSH-Agent tricks which might
  help with this?

For historical reasons, we retry SSH connections in Git LFS and users
frequently complain about being prompted for their password multiple
times before giving up, so I suspect this would not be a good experience
for the user.  Users may also have multiple keys and have configured
their agent to prompt them for a passphrase before using the key, and
I'm pretty sure those users will complain very loudly if we do this.
-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US

On Wed, Jan 20 2021, brian m. carlson wrote:

On 2021-01-19 at 06:17:21, Harley Paterson wrote:

quoted

Hello,

I've noticed an edge case in Git when a user has two Git accounts on a
single server - as might be common for a personal account and a work
account on a Git forge (Github, GitLab, Bitbucket...)

When attempting SSH login, `ssh` and Git will eagerly connect with the
first matching key. This may or may not be the key right key for the
repository the user needs. As a result, Git clones, pulls, and pushes
will fail with the `Repository not found` if the wrong key is tried
first.

For example, the user `alice` has two accounts on the host
`git-server.com`. `alice`'s accounts are `alice-work`, and
`alice-personal`. `alice-work` has access to the `foo` repository, and
`alice-personal` has access to the `bar` repository.

Yes, this is because SSH authentication happens before any command is
run.  The server never knows what resource is being requested until the
user is already authenticated.

quoted

`alice` attempts to clone `foo` with both her `alice-work` and
`alice-personal` keys in her SSH Agent. SSH Agent does not define
which key it will attempt first, so SSH may connect successfully to
`git-server.com` with her `alice-personal` keys, which do not have
access to the `foo` repository.

I'd be interested in your opinions for fixes to this. I am willing to
make a patch, although my knowledge of the Git codebase isn't perfect.

We've documented how to deal with situation properly in the FAQ, which
you can see at gitfaq(7) or
https://git-scm.com/docs/gitfaq#Documentation/gitfaq.txt-HowdoIusemultipleaccountswiththesamehostingproviderusingSSH

Is there some reason that this doesn't work for you?

quoted

- Should Git servers provide distinctly different error messages for
  `access denied`, and `repository does not exist`? Currently the
  server immediately closes the connection in this case, so
  `transport.c:handshake()` with fail when attempting to
  `discover_version()` because the reader hits the EOF. Perhaps the
  server could send a hypothetical access denied packet here, and a
  more appropriate error generated?

Unfortunately, this leaks whether a repository exists.  If Company XYZ
has a repository for each of its clients, it then becomes easy to see if
Company XYZ is doing work for a particular client by trying to see if a
repository exists.

I wonder how many hosting providers are confident that the code involved
in this isn't vulnerable to a timing attack.

This would be bad and a huge violation of privacy, so nobody is likely
do to that.  I can tell you as one of the staff who maintains the
GitHub Git service, we're not likely to change the behavior, and I
suspect nobody else is, either.

Let's be clear, it's a violation of privacy in the case that involves a
private repo.

But the fact that when you e.g. clone git@github.com:git/git.git with no
valid key and get a "Permission denied" is just an emergent effect of
how people string together openssh & their own auth solutions in the
wild.

You can just as well let the dialog proceed down to a shell without a
valid key or password, and then check that the user wants to run "git
upload-pack git/git.git", and then decide "sure, it's a public repo" and
proceed.

Indeed that's what e.g. GitHub does when you make up a username &
password to clone a public repo over https.

    # Under OpenSSH, See https://tools.ietf.org/html/rfc4252#section-5.2
    Match User git
        PermitEmptyPasswords yes
        AuthenticationMethods none

Why would a site like GitHub treat ssh differently than https here? I
think it just comes down to openssh's support for use-case being a bit
of a pain to configure.

I don't know of a way to do that *and* pass down to e.g. a login shell
wrapper script if/how the user was authenticated.

Hrm, maybe I take that back. I suppose you can make the default shell
for the "git" user "git-shell-public" and set the rw login shell to
"git-shell-private" via an Authorized{Keys,Principals}Command". But I
haven't tested it.

Anyway, just saying that there's bad error messages for good reasons,
and bad error messages for lazyness/not wanting to deal with the edge
case.

What Harley Paterson is complaining about is sometimes in the "good
reasons" bucket, but sometimes just server operator lazyness, or not
caring about this marginal edge case enough to provide a better error
message.

On Wed, Jan 20, 2021 at 01:42:56PM +0100, Ævar Arnfjörð Bjarmason wrote:

quoted

Unfortunately, this leaks whether a repository exists.  If Company XYZ
has a repository for each of its clients, it then becomes easy to see if
Company XYZ is doing work for a particular client by trying to see if a
repository exists.

I wonder how many hosting providers are confident that the code involved
in this isn't vulnerable to a timing attack.

I would say "not very certain" in the case of GitHub. I don't recall any
analysis of the timing ever having happened.

Why would a site like GitHub treat ssh differently than https here? I
think it just comes down to openssh's support for use-case being a bit
of a pain to configure.

GitHub doesn't use openssh at all; ssh sessions are terminated at a
custom load-balancing/proxy layer daemon.

I agree that servers in general probably could accept this notion of "no
valid user" for ssh sessions. I just think it's something that nobody
has asked for. I.e., if you are not planning to be an authenticated
user, then why use ssh in the first place? Certainly you could ask "why
not", but AFAIK this is the first time it has come up.

There is one protocol issue I'm not sure of, though. My understanding of
ssh auth is that the client offers keys in sequence, and the server says
"yes" or "no" on each. We'd want to continue saying "no" on keys that we
don't know about, so the client may keep trying other keys to which we
might say "yes". But after having seen all of the keys, what then?

We would want the server to say "OK, even though we don't like any of
your keys, let's start a session anyway". Is that possible within the
protocol? I think having the server say "OK, now try password auth" is
not a good idea. If the client asks for a dummy password, that is likely
to be either confusing, or disrupt sessions without a human present.

But I don't know the ssh protocol very well, so perhaps my understanding
of what is possible totally wrong. https has this much easier because
unauthenticated requests are totally normal, and the server sees the
auth along with the whole request at the same time.

-Peff

Repost: Accedentialy forgot to CC this to the mailing list.

quoted

Unfortunately, this leaks whether a repository exists.  If Company XYZ
has a repository for each of its clients, it then becomes easy to see if
Company XYZ is doing work for a particular client by trying to see if a
repository exists.

I wonder how many hosting providers are confident that the code involved
in this isn't vulnerable to a timing attack.

I would say "not very certain" in the case of GitHub. I don't recall any
analysis of the timing ever having happened.

I have to agree with this... It's certainly a risk, but if an attacker has enough
information to guess the repository names/URLs without triggering a rate
limiter, they already have a pretty good idea who Company XYZ is likely working
for, and what on.

HP.