[PATCH 0/6] repack: don't lose objects to a ".keep" that appears mid-run

A concurrent push can make "git repack -d" delete a pack whose objects were
never copied anywhere, and exit 0. We hit this in production: a ref pointing
at a commit that no longer exists, on git 2.43, and it reproduces on master.

What happens:

 * repack scans for ".keep" files and decides which packs to delete, then
   spawns pack-objects with --honor-pack-keep, which scans again;
 * in between, a push of content identical to an earlier one finishes
   migrating its quarantine. Its pack is a duplicate and is dropped, but its
   ".keep" is linked into place, onto the old pack;
 * pack-objects sees that ".keep" and leaves the pack's objects out; repack
   deletes the pack by its earlier list, with force_delete.

Two things are wrong, and each is fixed on its own:

 * 1/6: receive-pack removes a ".keep" it never installed -- the one it
   linked onto somebody else's pack, or a foreign one when its own push was
   rejected before any migration. Only remove a ".keep" that carries our own
   message.
 * 6/6: repack and pack-objects each scan for ".keep" files. Hand
   pack-objects the snapshot repack took at startup instead.

Patches 2-5 are what 6/6 needs to be safe:

 * 2/6: under --stdin-packs=follow, a --keep-pack pack stops the traversal
   like a "^" pack; on-disk ".keep" packs never did.
 * 3/6: the cruft walk goes by a stale kept-pack cache, which
   --honor-pack-keep happened to mask. Pre-existing, reproducible today.
 * 4/6: look --keep-pack names up in a sorted list; it gets long.
 * 5/6: --keep-pack-from-file, since a repository can have more kept packs
   than fit on a command line (32K characters on Windows).

Every fix comes with a test that fails without it; the race itself is
reproduced in t7703 by having a ".keep" appear as pack-objects starts. The
full suite passes, and the series merges cleanly into next and seen.

Qin ShiCheng (6):
  odb: don't remove a ".keep" we never installed
  pack-objects: keep --keep-pack open when following
  pack-objects: reset kept-pack cache for cruft walk
  pack-objects: sort --keep-pack list for lookup
  pack-objects: add --keep-pack-from-file
  repack: tell pack-objects which packs are kept

 Documentation/git-pack-objects.adoc |  8 +++
 builtin/pack-objects.c              | 71 +++++++++++++++++----
 builtin/repack.c                    | 15 +++++
 object-file.c                       | 95 ++++++++++++++++++++++-------
 odb/source-packed.h                 |  3 +-
 packfile.c                          |  9 ++-
 packfile.h                          |  7 +++
 repack-filtered.c                   |  3 -
 repack.c                            | 34 ++++++++++-
 repack.h                            | 17 +++++-
 t/t5329-pack-objects-cruft.sh       | 40 ++++++++++++
 t/t5331-pack-objects-stdin.sh       | 87 ++++++++++++++++++++++++++
 t/t5547-push-quarantine.sh          | 52 ++++++++++++++++
 t/t7700-repack.sh                   | 43 +++++++++++++
 t/t7703-repack-geometric.sh         | 72 ++++++++++++++++++++++
 tempfile.c                          | 12 ++++
 tempfile.h                          |  9 +++
 17 files changed, 533 insertions(+), 44 deletions(-)


base-commit: 3cb9185f65410273787f74333cc027d2ea5daada
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2219%2Fqeesung%2Frepack-kept-packs-snapshot-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2219/qeesung/repack-kept-packs-snapshot-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/2219
-- 
gitgitgadget