dwh@linuxprogrammer.org writes:
I think Git should externalize the calculation of object digests just
like it externalizes the calcualtion of object digital signatures.
The hashing algorithms used to generate object names has
requirements fundamentally different from that of digital
signatures. I strongly suspect that that fact would change the
equation when you rethink what you said above.
We can "upgrade" digital signature algorithms fairly easily---nobody
would complain if you suddenly choose different signing algorithm
over a blob of data, as long as all project participants are aware
(and self-describing datastream helps here) and are capable of
grokking the new algorithm we are adopting. But because object
names are used by one object to refer to another, and most
importantly, we do not want a single object to have multiple names,
we cannot afford to introduce a new hashing algorithm every time we
feel like it. In other words, diversity of object naming algorithms
is to be avoided as much as possible, while diversity of signature
algorithms is naturally expected.
On 14.05.2021 06:03, Junio C Hamano wrote:
dwh@linuxprogrammer.org writes:
quoted
I think Git should externalize the calculation of object digests just
like it externalizes the calcualtion of object digital signatures.
The hashing algorithms used to generate object names has
requirements fundamentally different from that of digital
signatures. I strongly suspect that that fact would change the
equation when you rethink what you said above.
I agree with you. Object names are exactly that: names. Names for
resources/data must be persistent, as well as global in scope and
uniqueness, and autonomously assigned. What this means is that once an
object has a name, that name shall never change as long as the object
remains unchanged. The names must be unique in the scope of all objects
(e.g. all copies of a repo) and generated without coordination.
Calculating object names using a digest algorithm meets all of these
requirements. Choosing a strong digest algorithm creates a strong
cryptographic binding between the name and the object contents. Using
self-describing digests allows for a repo to switch digest algorithms at
arbitrary points in the history.
I think that objects named with SHA1 digests should remain named with
the SHA1 digest. I do *not* advocate going back and rewriting history
to change all of the object names to a digest with a different
algorithm. Git is a provenance log and history matters. I recommend
preserving all existing names, even if they were created with known-weak
digest algorithms, and making the change to a new algorithm at a
specific point in time (e.g. at a tag). Using self-describing digest
encoding and externalizing digest calculation future-proofs
repositories and allows for preservation of history while allowing
algorithm agility.
To illustrate my point, I envision that a repos could have a history
like this:
object 2923f6fa36614586ea09b4424b438915cc1b9b67 (naked SHA1)
|
<many objects named with SHA1>
|
object 5f167fb6b3e96273b564fff0b041fb94fee4d3de (naked SHA1)
|
<modify Git to ext. digest calculation and self-desc encoding>
|
object 98c2e1c0965e60b0f137577ac5dd0a5c96ce224d (naked SHA1)
|
<many objects named with SHA1>
|
<a project decides to switch to SHA2-256, maybe marked in a tag>
|
object IAOdLVxteOxQwKa-xn8yCBUkuPkjAqcuQ2V7fKAlao8o (self-desc.SHA2-256)
|
<many objects named with self-describing SHA2-256 digests>
|
<a project decices to switch to SHA3-256, maybe marked in a tag>
|
object EK832G0PFhBFf-Dfgr205UKpUMqmVXJX9ltLwQo4Awct (self-desc.SHA3-256)
|
<many objects named with self-descring SHA3-256 digests>
.
.
.
Neither decision to switch to SHA2-256 nor to SHA3-256 would require any
code changes. If we continue down the current SHA-256 road, we will have
to repeat that multi-year effort in the future to switch to SHA3 or
something else. Most importantly, the choice of digest algorithm would
be left up to the maintainers of a given repo and not limited to the
algorithms we have hard coded into Git.
Brian's work on the SHA-256 switch is valuable. We can leverage a lot of
it to switch to externalized digest calculation and self-describing
digests and never have to worry about doing that again.
Cheers!
Dave
On Fri, May 14 2021, Junio C Hamano wrote:
dwh@linuxprogrammer.org writes:
quoted
I think Git should externalize the calculation of object digests just
like it externalizes the calcualtion of object digital signatures.
The hashing algorithms used to generate object names has
requirements fundamentally different from that of digital
signatures. I strongly suspect that that fact would change the
equation when you rethink what you said above.
We can "upgrade" digital signature algorithms fairly easily---nobody
would complain if you suddenly choose different signing algorithm
over a blob of data, as long as all project participants are aware
(and self-describing datastream helps here) and are capable of
grokking the new algorithm we are adopting. But because object
names are used by one object to refer to another, and most
importantly, we do not want a single object to have multiple names,
we cannot afford to introduce a new hashing algorithm every time we
feel like it. In other words, diversity of object naming algorithms
is to be avoided as much as possible, while diversity of signature
algorithms is naturally expected.
I agree insofar that I don't see a good reason for us to support some
plethora of hash algorithms, but I wouldn't have objections to adding
more if people find them useful for some reason. See e.g. [1] for an
implementation.
But I really don't see how anything you've said would present a
technical hurdle once we have SHA-1<->SHA-256 interop in a good enough
state. At that point we'll support re-hashing on arrival of content
hashed with algorithm X into Y, with a local lookup table between X<=>Y.
So if somebody wants to maintain content hashed with algorithm Z locally
we should easily be able to support that. The "diversity of naming"
won't matter past that local repository, any mention of Z will be
translated to X or Y on fetch/push.
1. https://lore.kernel.org/git/20191222064809.35667-1-michaeljclark@mac.com/
On 14.05.2021 10:49, Ævar Arnfjörð Bjarmason wrote:
I agree insofar that I don't see a good reason for us to support some
plethora of hash algorithms, but I wouldn't have objections to adding
more if people find them useful for some reason. See e.g. [1] for an
implementation.
I think Git should not try to do any cryptographic operations at all and
rely on external tools that are implemented properly and hardended.
Implementing cryptography isn't just about translating the algorithm
into code but also getting memory security correct, file handling
correct, input security correct, control flow correct (equal cost
multi-path), etc, etc. Most of the cryptography libraries aren't
designed to be misuse resistant. The only one I know of that has that as
a top-line requirement is Hyperledger Ursa [1].
I would like to see us remove all cryptography code (e.g. digests,
digital signatures, etc) from Git and rely on external tools entirely.
If we store the cryptographic material in a self-describing format that
identifies the associated tool as well as the cryptographic data, then
Git can be completely agnostic.
But I really don't see how anything you've said would present a
technical hurdle once we have SHA-1<->SHA-256 interop in a good enough
state. At that point we'll support re-hashing on arrival of content
hashed with algorithm X into Y, with a local lookup table between X<=>Y.
So if somebody wants to maintain content hashed with algorithm Z locally
we should easily be able to support that. The "diversity of naming"
won't matter past that local repository, any mention of Z will be
translated to X or Y on fetch/push.
Using self-describing formats allows us to honor history and keep old
object names as they and eliminate all of this added complications you
describe. I think there is a lot of room for errors to creep in when
collaborators have copies of the same repo and they have local mappings
between different hashing algorithms. How is this not setting up for a
combinatorial explosion of data? If the canonical repo uses SHA1 and one
contributor uses SHA2-512, another uses Blake2b-256, and yet another
uses SHA3-384, won't they all have to maintain six different translation
tables for all objects? SHA1 <=> SHA2-512, SHA1 <=> Blake2b-256, SHA1
<=> SHA3-384, SHA2-512 <=> Blake2b-256, SHA2-512 <=> SHA3-384, and
Blake2b-256 <=> SHA3-384? I guess that's your motivation for not
allowing algorithmic agility.
The way around this is to use self-describing formats and external
tools. Git repo copies wouldn't be required to have only *one* algorithm
naming all objects, requiring the translation tables. Instead Git repos
would/could have heterogeneous object names, each one with a single name
generated with a different digest algorithm. Git would simply consider
those names as plain strings and validating those strings requires
talking to the correct external tool, sending the name string and the
object data and reading back the result.
I think this is a much better approach because:
1. It creates algorithmic agility in a way that isn't top-down and heavy
handed.
2. It eliminates the need for all of the translation tables and
round-tripping complexity.
3. It empowers maintainers to decide which algorithms can/must be used
when naming objcts in a given repo. Merge hooks, CI/CD checks and
etiquette guides can be used to enforce this.
4. Git's attack surface becomes smaller (a very good thing) and limited
to doing IPC to external tools correctly and securely (easy) instead of
trying to get cryptography client code correct (very difficult).
One other thing to consider is that there are new tools being developed
that do similar things as Git that do have algorithmic agility and use
self-describing cryptographic primitives. Late-binding trust is now a
best practice and has been for quite some time. Many people rely upon
Git and I think we should keep up with the best practices.
Cheers!
Dave