Re: reftable [v7]: new ref storage format

On Tue, Aug 15, 2017 at 7:48 PM, Shawn Pearce [off-list ref] wrote:

7th iteration of the reftable storage format.

You can read a rendered version of this here:
https://googlers.googlesource.com/sop/jgit/+/reftable/Documentation/technical/reftable.md

Changes from v6:
- Blocks are variable sized, and alignment is optional.
- ref index is required on variable sized multi-block files.

- restart_count/offsets are again at the end of the block.
- value_type = 0x3 is only for symbolic references.
- "other" files cannot be stored in reftable.

- object blocks are explicitly optional.
- object blocks use position (offset in bytes), not block id.
- removed complex log_chained format for log blocks

- Layout uses log, ref file extensions
- Described reader algorithm to obtain a snapshot

- back to the old "intra-block index is last"
  for all block types. ok.
- changed (only ref?) indexes to start char + 3 byte size:
  Which starting char do object/log indexes have?

"Unaligned files must include the ref index to support fast lookup."

Why this? I would imagine the client (which has ~5 branches),
would not need this, but only a ref block, that's it.

Ctrl-F for 'block_size' reveals nothing is computed
relative to the block_size in this format, yet we can
set it to an arbitrary number. If following the spec,
the reader at $DAY_JOB needs to be able to read
both aligned and unaligned reftables, despite our plan
to ever write aligned ref tables, what would the reader
use the block_size for? (I think we can omit that field
from the header/footer now, no?)