Taylor Blau [off-list ref] writes:
Here is a reroll of my series which fixes a serious problem with MIDX bitmaps by
which they can become corrupt when permuting their pack order.
This seems to depend on tb/cruft-packs that is not yet in 'next', so
I'll redo this topic branch by forking it at 'master', merging the
other topic in, and then queuing these 8 patches.
So I'm definitely open to suggestions there, but otherwise this series should go
a long ways towards fixing my design mistake of having the MIDX .rev file be
separate from the MIDX itself.
Yeah, a single file with different chunks is a good way to ensure
atomicity of update.
A note to reviewers.
We need to make sure that not just we can still read .rev in
existing repositories (and convert it to the new chunk) correctly,
but also decide what to do to older versions of Git once the
repository is touched by this new version. Would they be upset to
see no .rev files or is it just the performance thing (and it is
more correct to recompute the reverse index on the fly)? Should the
new chunk be made mandatory to cause them notice that they should
not muck with the repository, or is it optional? Things like that.
Thanks.
On Wed, Dec 15, 2021 at 11:46:16AM -0800, Junio C Hamano wrote:
Taylor Blau [off-list ref] writes:
quoted
Here is a reroll of my series which fixes a serious problem with MIDX bitmaps by
which they can become corrupt when permuting their pack order.
This seems to depend on tb/cruft-packs that is not yet in 'next', so
I'll redo this topic branch by forking it at 'master', merging the
other topic in, and then queuing these 8 patches.
Hmm. They shouldn't depend on that topic. My local copy depends on
abe6bb3905 (The first batch to start the current cycle, 2021-11-29), but
I'm happy to rebase things if basing on abe6bb3905 was a mistake.
quoted
So I'm definitely open to suggestions there, but otherwise this series should go
a long ways towards fixing my design mistake of having the MIDX .rev file be
separate from the MIDX itself.
Yeah, a single file with different chunks is a good way to ensure
atomicity of update.
A note to reviewers.
We need to make sure that not just we can still read .rev in
existing repositories (and convert it to the new chunk) correctly,
but also decide what to do to older versions of Git once the
repository is touched by this new version. Would they be upset to
see no .rev files or is it just the performance thing (and it is
more correct to recompute the reverse index on the fly)? Should the
new chunk be made mandatory to cause them notice that they should
not muck with the repository, or is it optional? Things like that.
An old client that is looking in a repository where the reverse index is
stored as a chunk inside of the MIDX (and does not appear as a separate
file on disk) will not be able to read the MIDX's bitmap. We could
compute the reverse index on the fly (as we did for a long time with
single packs) but it is slow, and older clients obviously would not know
how to do it.
But the failure is graceful(ish): we'll get a warning that says we
couldn't find a reverse index, and then carry on pretending that the
MIDX did not have a bitmap.
~/s/git [nand] (master) $ git.compile multi-pack-index write --bitmap
Selecting bitmap commits: 73008, done.
Building bitmaps: 100% (319/319), done.
~/s/git [nand] (master) $ rm -f .git/objects/pack/multi-pack-index-*.rev
~/s/git [nand] (master) $ git.compile rev-list --count --objects --use-bitmap-index HEAD
warning: multi-pack bitmap is missing required reverse index
warning: ignoring extra bitmap file:
.git/objects/pack/pack-00465554dc3f3d96ac89a0ecc73cb5f5abbb35a5.pack
warning: multi-pack bitmap is missing required reverse index
warning: ignoring extra bitmap file:
.git/objects/pack/pack-00465554dc3f3d96ac89a0ecc73cb5f5abbb35a5.pack
307913
As far as whether or not the chunk is necessary, it only *needs* to be
present in the MIDX if there is a corresponding MIDX bitmap. We could
generate it always, but this series keeps in the tradition of the .rev
file and only writes it when we are also writing a bitmap.
Thanks.
Thanks,
Taylor
On 12/15/2021 2:46 PM, Junio C Hamano wrote:
Taylor Blau [off-list ref] writes:
quoted
Here is a reroll of my series which fixes a serious problem with MIDX bitmaps by
which they can become corrupt when permuting their pack order.
...
quoted
So I'm definitely open to suggestions there, but otherwise this series should go
a long ways towards fixing my design mistake of having the MIDX .rev file be
separate from the MIDX itself.
Yeah, a single file with different chunks is a good way to ensure
atomicity of update.
I just reviewed this series for the first time. Sorry for being so
late getting to it.
I had a few minor recommendations but they are mostly nitpicks and
don't deserve holding up the series if there are no other major
comments.
A note to reviewers.
We need to make sure that not just we can still read .rev in
existing repositories (and convert it to the new chunk) correctly,
but also decide what to do to older versions of Git once the
repository is touched by this new version. Would they be upset to
see no .rev files or is it just the performance thing (and it is
more correct to recompute the reverse index on the fly)? Should the
new chunk be made mandatory to cause them notice that they should
not muck with the repository, or is it optional? Things like that.
1. Can we still read a .rev?
The new test script specifically verifies that existing repositories
will continue to read their .rev upon upgrade. Their .rev files will
be replaced with the chunk during the next write.
2. What if they downgrade after the RIDX chunk is in place?
The .rev file is missing and the repo has a performance issue because
they can't use bitmaps. Correctness is not a problem. Anyone using
.rev files for server use (where bitmaps are most useful) is hopefully
already careful about downgrading Git versions.
3. Should the chunk be made mandatory?
Unfortunately, the chunk format did not follow the index format's
example of making lowercase chunk IDs required. Instead, the chunks
that are necessary for v1 are necessary forever and all other chunks
are deemed optional. Changing this would require something more
drastic like updating the version number or giving some grace period
where released versions start treating lowercase chunk IDs as required
before creating a new "required" chunk.
This does mean that if there is a version incompatibility, the RIDX
chunk will just be ignored by the older version of Git.
In terms of making this a safe format upgrade, I think Taylor has
achieved that.
The only thing I can think is that server operators might want to
deploy this version with GIT_TEST_MIDX_WRITE_REV=1 for a while, so
any need to downgrade would not suffer a performance penalty for a
missing .rev file. If that is a planned way to safely deploy this
change, then it might be worth adding a test that we safely delete
a .rev file after writing both a .rev file and a RIDX chunk. (The
RIDX chunk will be preferred, so maybe the previous .rev file hits
some logic that would skip its deletion?)
Thanks,
-Stolee
On Mon, Dec 20, 2021 at 01:51:22PM -0500, Derrick Stolee wrote:
On 12/15/2021 2:46 PM, Junio C Hamano wrote:
quoted
Taylor Blau [off-list ref] writes:
quoted
Here is a reroll of my series which fixes a serious problem with MIDX bitmaps by
which they can become corrupt when permuting their pack order.
...
quoted
quoted
So I'm definitely open to suggestions there, but otherwise this series should go
a long ways towards fixing my design mistake of having the MIDX .rev file be
separate from the MIDX itself.
Yeah, a single file with different chunks is a good way to ensure
atomicity of update.
I just reviewed this series for the first time. Sorry for being so
late getting to it.
Thanks for your review. I think that the upgrade plan is sane, but I can
comment a little bit more about that below.
In the meantime, some thoughts on all of the combinations of the new
GIT_TEST_ variables:
- GIT_TEST_MIDX_READ_RIDX=0 GIT_TEST_MIDX_WRITE_REV=0: this means that
we won't use bitmaps at all, since we won't have a .rev file to
read, and we will pretend that the RIDX chunk does not exist.
- GIT_TEST_MIDX_READ_RIDX=0 GIT_TEST_MIDX_WRITE_REV=1: this is the
status-quo of how things work today.
- GIT_TEST_MIDX_READ_RIDX=1 GIT_TEST_MIDX_WRITE_REV=0: this is the
status-quo of of how things will work after this patch series.
- GIT_TEST_MIDX_READ_RIDX=1 GIT_TEST_MIDX_WRITE_REV=1: this is useful
for testing that the RIDX chunk is preferred over reading the .rev
file.
So all but the (0, 0) combination make sense. Perhaps we should ban that
combination entirely, since nobody would ever set it for a good reason.
But I think that none of these combinations would allow us to propagate
the corruption, since we will always *write* the new RIDX chunk, which
causes the MIDX's checksum to change when we changes its object order.
I kind of hate these runtime checks that are only useful for testing. If
anybody has better ideas of how we should go about this that still
provides comprehensive coverage of .rev files, then I'm all ears.
(The nuclear option might be to just stop supporting .rev files for
MIDX's altogether, since this is such a new feature. But that seems like
a pretty drastic step, and definitely isn't backwards compatible. So I'd
be wary of going in that direction).
1. Can we still read a .rev?
The new test script specifically verifies that existing repositories
will continue to read their .rev upon upgrade. Their .rev files will
be replaced with the chunk during the next write.
Yes, exactly.
2. What if they downgrade after the RIDX chunk is in place?
The .rev file is missing and the repo has a performance issue because
they can't use bitmaps. Correctness is not a problem. Anyone using
.rev files for server use (where bitmaps are most useful) is hopefully
already careful about downgrading Git versions.
Yes, and we gracefully degrade here (I think that I demonstrated this in
my previous response), so in this scenario the worst an operator would
encounter is a performance regression.
3. Should the chunk be made mandatory?
Unfortunately, the chunk format did not follow the index format's
example of making lowercase chunk IDs required. Instead, the chunks
that are necessary for v1 are necessary forever and all other chunks
are deemed optional. Changing this would require something more
drastic like updating the version number or giving some grace period
where released versions start treating lowercase chunk IDs as required
before creating a new "required" chunk.
This does mean that if there is a version incompatibility, the RIDX
chunk will just be ignored by the older version of Git.
In terms of making this a safe format upgrade, I think Taylor has
achieved that.
Thanks. And yeah, the chunk should (and is) mandatory when writing a
MIDX bitmap. But if we ran `git multi-pack-index write` without
`--bitmap`, then we would be free to not write the RIDX chunk (and
indeed that is what we do).
The only thing I can think is that server operators might want to
deploy this version with GIT_TEST_MIDX_WRITE_REV=1 for a while, so
any need to downgrade would not suffer a performance penalty for a
missing .rev file. If that is a planned way to safely deploy this
change, then it might be worth adding a test that we safely delete
a .rev file after writing both a .rev file and a RIDX chunk. (The
RIDX chunk will be preferred, so maybe the previous .rev file hits
some logic that would skip its deletion?)
That logic (that we delete auxiliary files--including the .rev file--not
matching the checksum of the MIDX we just wrote) is unchanged. So I
think we should be good there since we have existing coverage.
Thanks,
Taylor
On 12/20/2021 2:52 PM, Taylor Blau wrote:
On Mon, Dec 20, 2021 at 01:51:22PM -0500, Derrick Stolee wrote:
quoted
3. Should the chunk be made mandatory?
Unfortunately, the chunk format did not follow the index format's
example of making lowercase chunk IDs required. Instead, the chunks
that are necessary for v1 are necessary forever and all other chunks
are deemed optional. Changing this would require something more
drastic like updating the version number or giving some grace period
where released versions start treating lowercase chunk IDs as required
before creating a new "required" chunk.
This does mean that if there is a version incompatibility, the RIDX
chunk will just be ignored by the older version of Git.
In terms of making this a safe format upgrade, I think Taylor has
achieved that.
Thanks. And yeah, the chunk should (and is) mandatory when writing a
MIDX bitmap. But if we ran `git multi-pack-index write` without
`--bitmap`, then we would be free to not write the RIDX chunk (and
indeed that is what we do).
I just want to be careful of the language here, with respect to how
chunks are listed as required or optional in
Documentation/technical/pack-format.txt. "Required" means every MIDX
needs one or is invalid. "Optional" means that all versions of Git
should ignore the chunk if it does not recognize the ID.
So here, the new RIDX chunk will be ignored by older versions and
will be written only when we care about bitmaps.
(Nothing I say here is in conflict with what you said, but I
anticipate confusion with your use of the word "mandatory".)
quoted
The only thing I can think is that server operators might want to
deploy this version with GIT_TEST_MIDX_WRITE_REV=1 for a while, so
any need to downgrade would not suffer a performance penalty for a
missing .rev file. If that is a planned way to safely deploy this
change, then it might be worth adding a test that we safely delete
a .rev file after writing both a .rev file and a RIDX chunk. (The
RIDX chunk will be preferred, so maybe the previous .rev file hits
some logic that would skip its deletion?)
That logic (that we delete auxiliary files--including the .rev file--not
matching the checksum of the MIDX we just wrote) is unchanged. So I
think we should be good there since we have existing coverage.
Sounds good.
Thanks,
-Stolee