Re: [PATCH 10/19] pack-bitmap: add support for bitmap indexes

On Wed, Oct 30, 2013 at 04:47:57PM +0100, Vicent Martí wrote:

Yes, our pack writing and bitmap code takes enough precautions to
arrange the objects in the packfile in a way that can be partially
reused, so for any given bitmap file written from Git, I'd say we're
safe to always reuse the leader of the pack if this is possible.

Perhaps it is worth adding a flag to the bitmap index to say "my bases
always come before deltas".

Obviously I intend to keep the pack reuse on production because the
CPU savings are noticeable, but we can drop it from the public
patchset. Ideally, we'd have full pack reuse like JGit, but we cannot
reasonably do that in GitHub because splitting a pack for the network
root would double our disk usage for all the forks.

Ah, right, I was forgetting about the fork-network repositories in my
analysis.  This doesn't work out of the box with this series, because we
will have all of the commits for all of the forks, followed by all of
the trees, and so forth.

What the list hasn't seen yet is that we have another patch series that
organizes the objects based on refs, so you can put all of
"refs/remotes/X/*" in one "island", "refs/remotes/Y/*" in another
"island", and so forth (in our scheme, each remote represents a single
fork, but it's configurable).

This lets us avoid deltas to objects that are not in the same island,
because such deltas would need to be thrown out when cloning a single
fork. And it lets us tweak the write order so that we can put all of one
fork's objects at the beginning, making it available for pack-reuse (so
obviously you would want to pick the "root" fork to put at the
beginning).

I'm planning to send those patches once the bitmap code is merged. One
is not strictly dependent on the other, but they touch some of the same
areas, and rely on some of the same infrastructure, so I think it makes
sense to do them in series.

-Peff