Re: [PATCH] Documentation: mark `--object-format=sha256` as experimental

On 2020-08-06 at 20:23:58, Martin Ågren wrote:

After eff45daab8 ("repository: enable SHA-256 support by default",
2020-07-29), vanilla builds of Git enable the user to run, e.g.,

  git init --object-format=sha256

and hack away. This can be a good way to gain experience with the
SHA-256 world, e.g., to find bugs that

  GIT_TEST_DEFAULT_HASH=sha256 make test

doesn't spot.

But it really is a separate world: Such SHA-256 repos will live entirely
separate from the (by now fairly large) set of SHA-1 repos. Interacting
across the border is possible in principle, e.g., through "diff + apply"
(or "format-patch + am"), but even that has its limitations: Applying a
SHA-256 diff in a SHA-1 repo works in the simple case, but if you need
to resort to `-3`, you're out of luck.

Similarly, "push + pull" should work, but you really will be operating
mostly offset from the rest of the world. That might be ok by the time
you initialize your repository, and it might be ok for several months
after that, but there might come a day when you're starting to regret
your use of `git init --object-format=sha256` and have dug yourself into
a fairly deep hole.

I do agree that they don't interoperate right now, and that we'd like it
to in the future.  But there are definitely people who can use SHA-256
support for new projects without problems.  I'm aware of certain
government agencies who very much do not want to use SHA-1 at all (and
at some point will be legally prohibited from doing so), and they will
be completely fine with the status quo.  Some of those same
organizations are unhappy about prohibited algorithms even being linked
into the binaries they use.  These folks can use a suitably new version
of Git everywhere and not care about the lack of backwards
compatibility.

I am, of course, in favor of abandoning SHA-1 as fast as practically
possible, but I understand that backwards compatibility is obviously a
concern.

Workflows aside, let's consider a more technical aspect. Pack index
files (pack-*.idx) exist in two flavours: v1 and v2. The hash transition
document foresees a v3, which we do not yet support (and for all we
know, the final v3 might end up different from the one sketched in the
hash transition document).

When the test suite is run with SHA-1 as the default hash algo, it
creates and consumes v2 pack files. But with SHA-256, we use an
undocumented, hybrid format where the header looks like v2, but where
the payload is not only "not correct SHA1", but where even the data
sizes are different. The trailing checksum is different, meaning no-one
(except us!) should/would try to interpret this file as a proper v2 pack
index.

We could certainly (re)define v2 to match our SHA-256 behavior, but we
do foresee v3 for a reason. And that would still just fix this specific
issue. And even when everything around SHA-256 is well-defined and we
have SHA-1--SHA-256 interoperability, there's a risk, at least
initially, that somewhere we'd be permeating buggy data that we'd then
feel responsible for and need to be able to handle for a long time to
come.

These are valid index v1 and v2 files, just with a different hash
algorithm.  v3 is there for the point where we do interoperate and need
to store hash values of multiple algorithms at once.  There's little to
no benefit to v3 if you don't need multiple algorithm support, other
than the fact that they declare the algorithms in them.

This is no different than saying that our commit or tree objects are in
a different form; they are syntactically identical, just with a
different hash algorithm.  That's how everything is in the .git
directory.

I'm fine with marking the functionality experimental for a few releases,
since it is possible we have bugs people haven't found, and adding a
note about interoperability after that point, since I think that's a
fair and valuable issue.  I think if we go a few releases without any
major issues, we can change this to the following:

  Note that a SHA-256 repository cannot yet share work with "regular"
  SHA-1 repositories.  Many tools do not yet understand SHA-256
  repositories, so users may wish to take this into account when
  creating new repositories.
-- 
brian m. carlson: Houston, Texas, US