Re: [RFC/PATCH 0/3] protocol v2

On Fri, Feb 27, 2015 at 3:05 PM, Junio C Hamano [off-list ref] wrote:

Junio C Hamano [off-list ref] writes:

quoted

I do not think v1 can be fixed by "send one ref with capability,
newer client may respond immediately so we can stop enumerating
remaining refs and older one will get stuck so we can have a timeout
to see if the connection is from the newer one, and send the rest
for the older client", because anything that involves such a timeout
would not reliably work over WAN.

Just for fun, I was trying to see if there is a hole in the current
protocol that allows a new client to talk a valid v1 protocol
exchange with existing, deployed servers without breaking, while
letting it to know a new server that it is a new client and it does
not want to get blasted by megabytes of ref advertisement.

The idea is to find a request that can be sent as the first
utterance by the client to an old server that is interpreted as a
no-op and can be recognised by a new server as such a "no-op probe".
If there is such a request, then the exchange can go like this with
(new client, old server) pair:

    - new client connects and sends that no-op.

    - old server starts blasting the ref advertisement

    - new client monitors and notices that the other side
      started speaking, and the ref advertisement lacks the
      capability bit for new protocol.

    - new client accepts the ref advertisement and does the v1
      protocol thing as a follow-up to what it already sent.

As long as the first one turns out to be no-op for old server, we
would be OK.  On the other hand, (new client, new server) pair
would go like this:

    - new client connects and sends that no-op.

    - new server notices that there is already a data from the
      client, and recognises the "no-op probe".

    - new server gives the first v2 protocol message with
      capability.

    - new client notices thqat the other side started speaking, and
      it is the first v2 protocol message.

    - both sides happily speak v2.

and (old client, new server) pair would go like this:

    - old client connects and waits.

    - new server notices that there is *no* data sent from the
      client and decides that the other side is a v1 client.  It
      starts blasting the ref advertisement.

    - both sides happily speak v1 from here on.

A misdetected case between (new client, new server) pair might go
like this:

    - new client connects and sends that no-op.

    - new server accepts the connection, but that no-op probe has
      not arrived yet.".  It misdetects the other side as a v1
      client and it starts blasting the ref advertisement.

    - new client notices that the ref advertisement has the
      capability bit and the server is capable of v2 protocol.  it
      waits until the server sends "sorry, I misdetected" message.

    - new server eventually notices the "no-op probe" while blasting
      the ref advertisement and it can stop in the middle.
      hopefully this can happen after only sending a few kilobytes
      among megabytes of ref advertisement data ;-).  The server
      sends "sorry, I misdetected" message to synchronise.

    - both sides happily speak v2 from here on.

So the topic of this exercise ("just for fun") is to see if there is
such a no-op request the client side can send as the first thing for
probing.

On the fetch side, the first response upload-pack expects are one
of:

  - "want " followed by an object name.
  - "shallow " followed by an object name.
  - "deepen " followed by a positive integer.

And there _is_ a hole ;-).  The parsing of "shallow " object name is
done in such a way that an object name that passes get_sha1_hex()
that results in a NULL return from parse_object() is _ignored_.  So
a new client can use "shallow 0{40}" as a no-op probe.

It appears that on the push side, there is a similar hole that can
be used. receive-pack expects either "shallow ", "push-cert" or the
refname updates (i.e. two "[0-9a-f]{40}" followed by a refname); the
parsing of "shallow " is not as loose as the fetch side in that
using a "shallow 0{40}" as a no-op probe will end up causing
prepare_shallow_info() sift the "0{40}" object name into "theirs",
but I think it will be ignored at the end as "unreachable cruft"
without causing harm.

I am _not_ proposing that we should go this route, at least not yet.
I am merely pointing out that an in-place sidegrade from v1 to a
protocol that avoids the megabyte-advertisement-at-the-beginning
seems to be possible, as a food for thought.

This is a fun thing indeed, though I'd personally feel uneasy with
such a probe as
a serious proposal. (Remember somebody 10 years from now wants to enjoy
reading the source code). So let's keep the idea around if we don't find another
solution.

As far as I can tell we have
* native git protocol (git daemon)
* ssh
* http(s)
* ftp (deprecated!)
* rsync(deprecated)

For both native git as well as ssh, Duy presented a solution at [1, 2]
a year ago,
which essentially presents the desired client capabilites 'out of
band' to the server
via an argument to the server.  So we'd only need to examine the
http(s) path how to
pass in arguments there.

Since 5 years an additional argument in the git protocol would cause
no harm, so it's
pretty safe to extend it with caution. We'd advertise the
client-may-ask-for-capabilities-out-of-band
per server and the client would eventually learn about it and only use
this for servers
which advertised they can do so. And nobody sane out there will
downgrade a version
of today to a git of 5 years ago.

I did rebase Duys' patches and I am trying to send out a patch series
today going that route
asking for comments.

Thanks a lot for the digging!
Stefan

[1] https://github.com/pclouds/git/commit/e26fa77c4d9ace06b9f2c80091af9eb7b63a1c95
[2] https://github.com/pclouds/git/commit/20d048e5fc650b20fdc7dd8bbe35cb8510ac9c50