Re: [RFC PATCH] index-pack: optionally allow duplicate objects

Taylor Blau [off-list ref] writes:

If we can find useful ways to combine the ideas above with Git's in-tree
implementation of upload-pack, one could imagine that Git itself may
eventually send packs containing duplicate copies of some object(s)
behind a capability. In other words, for clients that know how to
process such a pack, the server may wish to ask the client to do just
that in the name of saving some CPU cycles necessary to generate a pack
that doesn't have any duplicate objects.

I can live with such an extension as long as we teach the receiving
end to deduplicate the extra copy.  Leaving packs with duplicate
objects on disk is a completely different story, as it will become a
source of spreading such broken packs elsewhere, though.

But I would note that having packs containing duplicate objects is not a
new repository state for Git. Non-strict `index-pack` accepts duplicate
entries today, and shallow and filtered clones can store the same pack.

The same as what???

My series in 'tb/pack-with-duplicates' attempts to fix the known-broken
assumptions in reverse indexes, delta resolution, MIDX verification, and
bitmap reuse because those packs can already exist.

If duplicate entries are to be forbidden entirely in order for a pack to
be considered valid, then I think we should reject them in every
`index-pack` mode, including the shallow and filtered clone paths, and
diagnose existing packs in `fsck`.

Yup, I think that would be a sensible longer-term direction.  We may
need a bit more tool support to "fix" by reindexing at the receiving
end, though.  As you say, "cancatenate two packs, damn the duplicates"
may be a cheap way for server side to give union of objects contained
in these two packs, but doing so without even measuring how much they
are duplicating cannot go on forever unchecked.  Somebody needs to
remove these duplicates, and the time the downloader indexes the
incoming pack would be the best place to do so.  It needs to read
each and every object in the pack stream to make the .idx file out
of the stream anyway.