From: Junio C Hamano <hidden> Date: 2016-06-15 22:56:04
Duy Nguyen [off-list ref] writes:
On Tue, Feb 5, 2013 at 5:29 PM, Michael Haggerty [off-list ref] wrote:
quoted
Hiderefs creates a "dark" corner of a remote git repo that can hold
arbitrary content that is impossible for anybody to discover but
nevertheless possible for anybody to download (if they know the name of
a hidden reference). In earlier versions of the patch series I believe
that it was possible to push to a hidden reference hierarchy, which made
it possible to upload dark content. The new version appears (from the
code) to prohibit adding references in a hidden hierarchy, which would
close the main loophole that I was worried about. But the documentation
and the unit tests only explicitly say that updates and deletes are
prohibited; nothing is said about adding references (unless "update" is
understood to include "add"). I think the true behavior should be
clarified and tested.
I was worried that somehow this "dark" content could be used for
malicious purposes; for example, pushing compromised code then
convincing somebody to download it by SHA1 with the implicit argument
"it's safe since it comes directly from the project's official
repository". If it is indeed impossible to populate the dark namespace
remotely then I can't think of a way to exploit it.
Or you can think hiderefs is the first step to addressing the
initial ref advertisment problem. The series says hidden refs are
to be fetched out of band, but that's not the only way.
Let me help unconfuse this thread.
I think the series as 8-patch series was poorly presented, and
separating it into two will help understanding what they are about.
The first three:
upload-pack: share more code
upload-pack: simplify request validation
upload/receive-pack: allow hiding ref hierarchies
is _the_ topic of the series. As far as I am concerned (I am not
speaking for Gerrit users, but am speaking as the Git maintainer),
the topic is solely about uncluttering. There may be refs that the
server end may need to keep for its operation, but that remote users
have _no_ business knowing about. Allowing the server to keep these
refs in the repository, while not showing these refs over the wire,
is the problem the series solves.
In other words, it is not about "these are *usually* not wanted by
clients, so do not show them by default". It is about "these are
not to be shown, ever".
OK?
Now, there may be some refs that are not *usually* wanted by clients
but there may be cases where clients want to
(1) learn about them via the same protocol; and/or
(2) fetch them over the protocol.
If you want to solve both of these two issues generally, the
solution has to involve a separate protocol from the today's
protocol. It would go like this:
* The upload-pack-2 service sits on a port different from today's,
waits for a ls-remote/fetch/clone client to connect to it, makes
a default advertisement that only includes the refs that are
usually wanted by clients with hints on what other refs the
initial advertisement omitted, to let the client know that it is
allowed to ask for them.
* An updated client, if it sees that some refs are omitted from the
initial advertisement *and* what the user told it to fetch or
list may be one of the omitted ones (this is why the server gives
hints in the previous step in the first step; when the server
says it did not omit anything, or when it says it omitted only
refs/pull/*, a client that wanted to fetch refs/heads/frotz will
know the request will fail without continuing this step), then
makes a "expand-refs" request to the server, asking for the refs
it did not see and the server could supply.
* When the server sees "expand-refs", it responds with additional
advertisement. "expand-refs refs/pull/*" may result in listing
of all refs in that hierarchy. "expand-refs refs/changes/1/1"
would result in listing that single ref. "expand-refs no-such"
may result in nothing, indicating an error.
* After the (possible) expand-refs exchange, the client knows
exactly the same and necessary information as the current
protocol gives it in order to go to the common ancestor discovery
step, and the protocol can continue the same way as the current
protocol.
Note that this cannot sit on the current port in general, as
existing clients will not be able to tell some refs are not
advertised, so unless you are hiding large and truly unused part of
the refspace, interoperability with older clients will render the
mechanism useless. You cannot use this to delay the refs/tags/
hierarchy with this mechanism and have older client come to the
updated service that by default does not advertise tags, for
example.
The above is what I called the "delayed advertisement" in the
discussion, which was brought up several months ago but nothing
materialized as the result. People who are interested in pursuing
this can volunteer and start discussing the design refinements now
and submit implementation for reviews.
But in the meantime, if there is a niche use case where a solution
to only the second problem is sufficient (and Gerrit and GitHub pull
requests could both be such use cases), the remainder of the series
can help, without waiting the solution to solve "usually not wanted
but may need to be learned" problem. That is the latter 4 patches
(the very last one is a demonstration to illustrate why allowing a
push to hidden ref hierarchy would not and should not work, and is
not for application):
parse_fetch_refspec(): clarify the codeflow a bit
fetch: use struct ref to represent refs to be fetched
upload-pack: optionally allow fetching from the tips of hidden refs
fetch: fetch objects by their exact SHA-1 object names
From: Jonathan Nieder <hidden> Date: 2016-06-15 22:56:04
Junio C Hamano wrote:
Duy Nguyen [off-list ref] writes:
quoted
On Tue, Feb 5, 2013 at 5:29 PM, Michael Haggerty [off-list ref] wrote:
quoted
quoted
Hiderefs creates a "dark" corner of a remote git repo
[...]
quoted
Or you can think hiderefs is the first step to addressing the
initial ref advertisment problem. The series says hidden refs are
to be fetched out of band, but that's not the only way.
Let me help unconfuse this thread.
I think the series as 8-patch series was poorly presented, and
separating it into two will help understanding what they are about.
The first three:
upload-pack: share more code
upload-pack: simplify request validation
upload/receive-pack: allow hiding ref hierarchies
is _the_ topic of the series. As far as I am concerned (I am not
speaking for Gerrit users, but am speaking as the Git maintainer),
the topic is solely about uncluttering. There may be refs that the
server end may need to keep for its operation, but that remote users
have _no_ business knowing about.
An obvious question when looking at that alone is, is there ever
actually need for such private refs? If the refs are not meant to be
shared with users *at all*, why are they even refs?
An answer is "because refs force gc to keep the corresponding
objects". For example, the sysadmin may want to keep refs/archived/
refs for dead branches that should not be advertised or accessible to
the user any more. Seems sane, though not especially exciting.
What is more exciting to me is that it is a first step toward
addressing the complicated problem of offering access to more refs
than can be efficiently presented in the current ref advertisement. I
think that's a harder problem but something like this would be needed
in order to support existing clients without performance degredation.
And in the meantime, it helps with the refs/archived case.
Thanks for explaining.
Jonathan
From: Michael Haggerty <hidden> Date: 2016-06-15 22:56:04
On 02/06/2013 08:17 PM, Junio C Hamano wrote:
Duy Nguyen [off-list ref] writes:
quoted
On Tue, Feb 5, 2013 at 5:29 PM, Michael Haggerty [off-list ref] wrote:
quoted
Hiderefs creates a "dark" corner of a remote git repo that can hold
arbitrary content that is impossible for anybody to discover but
nevertheless possible for anybody to download (if they know the name of
a hidden reference). In earlier versions of the patch series I believe
that it was possible to push to a hidden reference hierarchy, which made
it possible to upload dark content. The new version appears (from the
code) to prohibit adding references in a hidden hierarchy, which would
close the main loophole that I was worried about. But the documentation
and the unit tests only explicitly say that updates and deletes are
prohibited; nothing is said about adding references (unless "update" is
understood to include "add"). I think the true behavior should be
clarified and tested.
I was worried that somehow this "dark" content could be used for
malicious purposes; for example, pushing compromised code then
convincing somebody to download it by SHA1 with the implicit argument
"it's safe since it comes directly from the project's official
repository". If it is indeed impossible to populate the dark namespace
remotely then I can't think of a way to exploit it.
Or you can think hiderefs is the first step to addressing the
initial ref advertisment problem. The series says hidden refs are
to be fetched out of band, but that's not the only way.
Let me help unconfuse this thread.
I think the series as 8-patch series was poorly presented, and
separating it into two will help understanding what they are about.
The first three:
upload-pack: share more code
upload-pack: simplify request validation
upload/receive-pack: allow hiding ref hierarchies
is _the_ topic of the series. As far as I am concerned (I am not
speaking for Gerrit users, but am speaking as the Git maintainer),
the topic is solely about uncluttering. There may be refs that the
server end may need to keep for its operation, but that remote users
have _no_ business knowing about. Allowing the server to keep these
refs in the repository, while not showing these refs over the wire,
is the problem the series solves.
In other words, it is not about "these are *usually* not wanted by
clients, so do not show them by default". It is about "these are
not to be shown, ever".
OK?
Yes, the first three patches sound much more reasonable if this is the
goal. Do you know of users who want the feature defined by the first
three patches, or is it only a stepping stone towards an actually useful
feature? (I ask because I have trouble imagining a real-world scenario
where these alone would be useful.)
Now, there may be some refs that are not *usually* wanted by clients
but there may be cases where clients want to
(1) learn about them via the same protocol; and/or
(2) fetch them over the protocol.
If you want to solve both of these two issues generally, the
solution has to involve a separate protocol from the today's
protocol. It would go like this:
[... omitted clear explanation of how delayed advertisement could be
implemented via a new protocol ...]
But in the meantime, if there is a niche use case where a solution
to only the second problem is sufficient (and Gerrit and GitHub pull
requests could both be such use cases), the remainder of the series
can help, without waiting the solution to solve "usually not wanted
but may need to be learned" problem. That is the latter 4 patches
(the very last one is a demonstration to illustrate why allowing a
push to hidden ref hierarchy would not and should not work, and is
not for application):
Given that some people *do* want to fetch all pull requests, is this a
feature that any hosting service would really turn on? True, the
majority of users would be spared clutter, but at the cost of completely
preventing other users from fetching all pull requests, mirroring the
repository, etc.
In other words, I wonder whether your two incremental steps are useful
at all, in the real world, without yet-to-be-implemented future changes.
If not, then it doesn't make sense to merge them without at least
imagining the final goal and gaining confidence that they are not false
starts.
I think that a more useful interim solution would be to make it easy to
have two URLs accessing a single git repository, with different levels
of reference visibility applied to each. This is something that
providers could turn on without sacrificing any existing functionality.
And it would solve all three problems: clutter, bandwidth, and provenance.
Your first three patches would allow two-tier access to be implemented,
for example by setting GIT_CONFIG or GIT_CONFIG_PARAMETERS or
command-line parameters differently for the processes serving the two
URLs, like:
git upload-pack ...
vs.
GIT_CONFIG=config-with-hidden-refs git upload-pack ...
or
git -c transfer.hiderefs=refs/pull upload-pack ...
But this is a bit awkward because the admin would either have to
maintain two config files, or maintain the hiderefs configuration in the
script starting upload-pack rather than in the configuration file.
Therefore, I suggest a slight change to how hiderefs are configured to
make two-tier URLs easier to configure, such as
# Define one or more views:
[view "uncluttered"]
hiderefs = refs/pull
# This would set the default view for all services:
[transfer]
view = uncluttered
# Peff also wanted the possibility to configure each service
# independently which could be done like this:
[receive]
view = uncluttered
[uploadpack]
view = full
I also tentatively suggest that we add a git-level option "--view" and
an environment variable GIT_VIEW (similar to "--namespace" and
GIT_NAMESPACE) to override the default setting:
GIT_VIEW=uncluttered git upload-pack ...
This way whoever starts the process only needs to choose a particular
view name; the actual definition would reside in the config file.
I think these changes would make it easier to support two-tier URLs and
would also leave the way open to use the "view" concept for other things
in the future.
I've said my piece now and am gratified that there has been more
discussion about your proposal, which was my main goal. Therefore FWIW
I turn my -1 into a -0 and leave it up to the people experiencing more
clutter-induced pain to decide how to proceed.
Michael
--
Michael Haggerty
mhagger@alum.mit.edu
http://softwareswirl.blogspot.com/
On Wed, Feb 6, 2013 at 8:17 PM, Junio C Hamano [off-list ref] wrote:
Maybe this should be split up into a different thread, but:
The upload-pack-2 service sits on a port different from today's
[...].
I think there's a simpler way to do this, which is that:
* New clients supporting v2 of the protocol send some piece of data
that would break old servers.
* If that fails the new client goes "oh jeeze, I guess it's an old
server", and try again with the old protocol.
* The client then saves a date (or the version the server gave us)
indicating that it tried the new protocol on that remote, tries
again sometime later.
We already covered in previous discussions how this would be simpler
with the HTTP protocol, since you could just send an extra header
inviting the server to speak the new protocol.
But for the other transports we can just try the new protocol and
retry with the old one as a fallback if it doesn't work. That'll allow
us to gracefully migrate without needing to change the git:// port.
Besides, I think the vast majority of users are using Git via http://
or ssh://, where we can't just change the port, but even so making
people change the port when we could handle this more gracefully would
be a big PITA. Adding new firewall holes is often a big bureaucratic
nightmare in some organizations.
From: Jeff King <hidden> Date: 2016-06-15 22:56:04
On Wed, Feb 06, 2013 at 11:17:06AM -0800, Junio C Hamano wrote:
Let me help unconfuse this thread.
I think the series as 8-patch series was poorly presented, and
separating it into two will help understanding what they are about.
The first three:
upload-pack: share more code
upload-pack: simplify request validation
upload/receive-pack: allow hiding ref hierarchies
is _the_ topic of the series. As far as I am concerned (I am not
speaking for Gerrit users, but am speaking as the Git maintainer),
the topic is solely about uncluttering. There may be refs that the
server end may need to keep for its operation, but that remote users
have _no_ business knowing about. Allowing the server to keep these
refs in the repository, while not showing these refs over the wire,
is the problem the series solves.
In other words, it is not about "these are *usually* not wanted by
clients, so do not show them by default". It is about "these are
not to be shown, ever".
OK?
Right. I am not opposed to this series, as it does have a use-case. And
if it helps Gerrit folks or other users unclutter, great. The fact that
I could throw away the custom receive.hiderefs patch we use at GitHub is
a bonus. If people want fancier things, they can do them separately.
_But_. As a potential user of the feature (to hide refs/pull/*), I do
not think it is sufficiently flexible for me to use transfer.hiderefs
(or uploadpack.hiderefs). We use "fetch" internally to migrate objects
between forks and our alternates repos. And in that case, we really do
want to see all refs. In other words, all fetches are not the same: we
would want upload-pack to understand the difference between a client
fetch and an internal administrative fetch. But this feature does not
provide that lee-way. Even if you tried:
git fetch -u 'git -c uploadpack.hiderefs= upload-pack'
the list nature of the config variable means you cannot reset it.
This isn't a show-stopper for the series; it may just mean that it is
not a good fit for GitHub's use case, but others (like Gerrit) may
benefit. But since refs/pull is used as an example of where this could
be applied, I wanted to point out that it does not achieve that goal.
-Peff