Re: [PATCH] pack-format: correct multi-pack-index description

On Mon, 2020-02-10 at 10:02 -0500, Derrick Stolee wrote:

Git loads the multi-pack-index file, which includes a sorted list of
the packs it covers. It then scans the "pack" directory for pack-indexes
and checks if they are covered by the multi-pack-index. If not, then
Git will add them to the packed_git struct and use them as normal.
The hope is that this list of "uncovered" packs is small compared to
the data covered by the multi-pack-index.

This allows Git to continue functioning after an action like "git fetch"
that adds a new pack but may not want to rewrite the multi-pack-index.

Ah, ok.

So then perhaps I'll just make bup write the multi-pack-index file as
is. This is fine, there's no real need to have multiple, I just didn't
want to have to make sure the file was always consistent.

Or maybe just call git to do it, and only be able to read the resulting
file :-)

Our background maintenance essentially runs these commands:

 1. git multi-pack-index write
 2. git multi-pack-index expire
 3. git multi-pack-index repack

Step 1 ensures all packs are pulled into the multi-pack-index. Step 2
deletes any pack-files whose objects are contained in newer pack-files.
Step 3 creates a new pack-file containing all objects from a set of
small pack-files (using the --batch-size=X option). This process helps
incrementally reduce the size and number of packs. That may be helpful
for your backup took, too.

I'll have to look at this in more detail later, and understand exactly
what the steps do here. Evidently that modifies pack files, which I
hadn't expected for a type of "index" command :-)

Perhaps after an incremental multi-pack-index is added, then Git could
(optionally) have a mode that only checks the multi-pack-index to
avoid scanning the packs directory. It would require inserting a
multi-pack-index write into the index-pack logic so Git.

I guess you'd still want to read non-covered pack files just in case old
git was used or something though.

I'm not sure if that mode would be helpful, since the pack directory
scan is typically done once per command and is relatively fast.

Right.

quoted

quoted

That said: if someone wanted to contribute an incremental format,
then I would be happy to review it!

I might still get motivated to do so :-)

YOU CAN DO IT! (Did that help?)

:-)

Thanks,
johannes