Re: Mozilla .git tree

Shawn Pearce [off-list ref] writes:

What I did in fast-import was give inflate whatever was left in
the current mapping; then if I got a Z_OK or Z_BUF_ERROR back from
inflate I move the mapping to the next 128 MiB chunk and reset my
z_stream's next_in/avail_in accordingly, then recall inflate.

Makes sense.

But having the length or ending offset in the index will help with
copying the object during a repack as well as prevent us from needing
to guess during accesses.

Actually pack-objects already builds the reverse index without
the help from the updated .idx file format, so while having it
pre-built in .idx may help, it is not necessary for it.  With
your "feed until the end of the current window and if we need
more map in the next window" logic, we do not even need to know
the length of each entry at runtime either.

So I am inclined to chuck the previous patch that records the
next object number in each entry.  We could keep the 64-bit
offset, which would make an entry to be 28-byte instead of
32-byte.


-- 
VGER BF report: U 0.5

Junio C Hamano [off-list ref] wrote:

So I am inclined to chuck the previous patch that records the
next object number in each entry.  We could keep the 64-bit
offset, which would make an entry to be 28-byte instead of
32-byte.

I can agree with that.  :-)

Repacking is infrequent compared to searching for an object.
Asking the repack code to determine object lengths (especially if it
is going to deflate the entry anyway to verify it) isn't that much of
a performance hit.  As you pointed out its already happening today.

Using a 28 byte index entry instead of a 32 byte index entry means
the Mozilla historical pack index will only be 52.4 MiB rather than
the slightly larger 59.9 MiB.  1.9x10^6 objects will do that to you.

-- 
Shawn.