Re: [PATCH v2 0/8] midx: prevent bitmap corruption when permuting pack order

On 12/20/2021 2:52 PM, Taylor Blau wrote:

On Mon, Dec 20, 2021 at 01:51:22PM -0500, Derrick Stolee wrote:

quoted

3. Should the chunk be made mandatory?

Unfortunately, the chunk format did not follow the index format's
example of making lowercase chunk IDs required. Instead, the chunks
that are necessary for v1 are necessary forever and all other chunks
are deemed optional. Changing this would require something more
drastic like updating the version number or giving some grace period
where released versions start treating lowercase chunk IDs as required
before creating a new "required" chunk.

This does mean that if there is a version incompatibility, the RIDX
chunk will just be ignored by the older version of Git.

In terms of making this a safe format upgrade, I think Taylor has
achieved that.

Thanks. And yeah, the chunk should (and is) mandatory when writing a
MIDX bitmap. But if we ran `git multi-pack-index write` without
`--bitmap`, then we would be free to not write the RIDX chunk (and
indeed that is what we do).

I just want to be careful of the language here, with respect to how
chunks are listed as required or optional in
Documentation/technical/pack-format.txt. "Required" means every MIDX
needs one or is invalid. "Optional" means that all versions of Git
should ignore the chunk if it does not recognize the ID.

So here, the new RIDX chunk will be ignored by older versions and
will be written only when we care about bitmaps.

(Nothing I say here is in conflict with what you said, but I
anticipate confusion with your use of the word "mandatory".)

quoted

The only thing I can think is that server operators might want to
deploy this version with GIT_TEST_MIDX_WRITE_REV=1 for a while, so
any need to downgrade would not suffer a performance penalty for a
missing .rev file. If that is a planned way to safely deploy this
change, then it might be worth adding a test that we safely delete
a .rev file after writing both a .rev file and a RIDX chunk. (The
RIDX chunk will be preferred, so maybe the previous .rev file hits
some logic that would skip its deletion?)

That logic (that we delete auxiliary files--including the .rev file--not
matching the checksum of the MIDX we just wrote) is unchanged. So I
think we should be good there since we have existing coverage.

Sounds good.

Thanks,
-Stolee