Re: reftable [v3]: new ref storage format

On Mon, Jul 24, 2017 at 4:00 PM, Shawn Pearce [off-list ref] wrote:

On Mon, Jul 24, 2017 at 3:22 PM, Stefan Beller [off-list ref] wrote:

quoted

quoted

#### index record

An index record describes the last entry in another block.
Index records are written as:

    varint( prefix_length )
    varint( (suffix_length << 3) | 0 )
    suffix
    varint( block_offset )

Index records use prefix compression exactly like `ref_record`.

Index records store `block_offset` after the suffix, specifying the
offset in bytes (from the start of the file) of the block that ends
with this reference.

Instead of hardcoding the "0" in the last 3 bits, maybe pick one
of the reserved bit patterns to be there? I would imagine this
makes debugging easier:

    0x5? Hah that must be an index block I have been
    looking at the wrong block!

This is an excellent suggestion. I'll include it in the next iteration.

I was thinking about this a bit more and wondering if this would
allow mixing all sorts of entries in the same block. Think about one
of the most common cases client side:

    git commit -m "just a lonely ref update, no fancy stuff"

For that we need (a) a single ref update and (b) now that the
format evolved a single reflog entry, maybe (c) an obj entry, too.

I just realize that in this case we'd crank down the block size
such that there is very little padding required, but I still wonder
if it would be possible to omit the second (and third) block,
but rather go for a "m"ixed block containing just these 3 entries.

Additionally by setting the block_size to a special value "0",
we could indicate the omission of a footer, such that a single
ref update reftable is super small.