[PATCH v3 00/13] odb: stop registering in-memory sources

Hi,

the object database has a list of sources that is used for two
different purposes:

  - We use it to track the list of alternates.

  - We use it to track temporary in-memory sources that we create for
    various purposes. Most importantly, this is used to link object
    database sources from submodules into the main store.

This dual-use is quite awkward, as it mixes two different levels of
concerns and thus as a consequence makes both harder to reason about.
It's also a source of bugs: we make assumptions about the ordering of
sources all over the place, and we furthermore assume in other places
that the sources only contain alternates in the first place. I don't
think this surfaces in the form of real bugs, but I've long disliked
this dual-use.

Furthermore, we want to migrate handling of alternates into the "files"
backend itself in a subsequent patch series. This is most importantly to
fix a performance regression by making the backend own all of its
alternates, but it also fixes a couple of longer-standing design issues
that I've been struggling with [1].

Most importantly though: this whole machinery is not even needed at all.
A couple years ago we have already refactored our codebase so that
submodule sources don't even have to be linked into the main object
database anymore. And all the other use cases where we link sources into
the main object database can be trivially converted, too.

So this patch series does exactly that: it removes the mechanism to link
ad-hoc sources into the object database entirely. This ensures that the
list of sources is exactly the list of alternates, and that makes it
easier to move them into the "files" backend in a subsequent patch
series.

There is one exception though: creating transactions still creates a
temporary quarantine directory. This mechanism is left as-is for now,
but as it's an implementation detail of the "files" backend anyway
that's not conflicting with our above stated goals.

This series is built on top of 1630431f32 (The 21st batch, 2026-08-31)
with ty/repository-fetch-if-missing at 508ec9837c (repository: move
fetch_if_missing into struct repository, 2026-08-15) merged into it.
There's still two merge conflicts, but these are trivial to resolve: in
"odb.c" and "odb.h" you simply remove both ours and theirs, and in
"builtin/multi-pack-index.c" you only need to munge the parameters a
bit.

Changes in v3:
  - Improve commit message clarity a tiny bit :)
  - Link to v2: https://patch.msgid.link/20260902-pks-odb-registering-in-memory-sources-v2-0-c6ca12fdea4d@pks.im

Changes in v2:
  - Adapt `cache_tree_fully_valid()` to take a `struct index_state` as
    input instead of taking both a repository and a cache tree, as
    suggested by Junio.
  - Link to v1: https://patch.msgid.link/20260901-pks-odb-registering-in-memory-sources-v1-0-97a312d5fa25@pks.im

Thanks!

Patrick

[1]: [ref]

---
Patrick Steinhardt (13):
      cache-tree: drop `the_repository` in `cache_tree_fully_valid()`
      cache-tree: remove dependency on `the_repository`
      submodule-config: remove uses of `the_repository`
      submodule-config: stop using `the_hash_algo`
      submodule-config: stop registering submodule sources
      builtin/grep: stop registering submodule ODB as source
      odb: remove infrastructure to register submodule sources
      tmp-objdir: drop unused function to register alternate
      odb/packed: fix memory leaks when freeing source
      builtin/multi-pack-index: refuse unknown sources with "--object-dir="
      t/helper: adapt read-midx to not link ad-hoc source anymore
      t/helper: stop registering alternates in "ref-store" command
      odb: remove the ability to link sources ad-hoc

 builtin/checkout.c                     |  2 +-
 builtin/commit.c                       |  2 +-
 builtin/fetch.c                        |  2 +-
 builtin/grep.c                         | 28 +++-------
 builtin/multi-pack-index.c             |  3 +-
 builtin/submodule--helper.c            |  8 +--
 cache-tree.c                           | 95 +++++++++++++++++++---------------
 cache-tree.h                           |  7 +--
 odb.c                                  | 42 ---------------
 odb.h                                  | 22 --------
 odb/source-packed.c                    |  1 +
 read-cache-ll.h                        |  5 +-
 read-cache.c                           |  9 ++--
 sequencer.c                            |  2 +-
 sparse-index.c                         |  2 +-
 submodule-config.c                     | 59 +++++++++++----------
 submodule-config.h                     | 12 +++--
 submodule.c                            |  2 +-
 t/README                               |  7 ---
 t/helper/test-read-midx.c              | 43 ++++++++++-----
 t/helper/test-ref-store.c              |  8 ---
 t/helper/test-submodule.c              |  4 +-
 t/t5319-multi-pack-index.sh            |  9 ++--
 t/t5526-fetch-submodules.sh            |  3 --
 t/t5531-deep-submodule-push.sh         |  3 --
 t/t5545-push-options.sh                |  3 --
 t/t5572-pull-submodule.sh              |  3 --
 t/t6437-submodule-merge.sh             |  3 --
 t/t7418-submodule-sparse-gitmodules.sh |  3 --
 t/t7814-grep-recurse-submodules.sh     |  3 --
 tmp-objdir.c                           |  5 --
 tmp-objdir.h                           |  6 ---
 unpack-trees.c                         |  9 ++--
 33 files changed, 168 insertions(+), 247 deletions(-)

Range-diff versus v2:

 1:  95ff726928 =  1:  a785e29859 cache-tree: drop `the_repository` in `cache_tree_fully_valid()`
 2:  2b2cf592c4 =  2:  002cca01eb cache-tree: remove dependency on `the_repository`
 3:  7727c40da8 =  3:  17ab3e4553 submodule-config: remove uses of `the_repository`
 4:  6c2168af46 =  4:  586a786dc6 submodule-config: stop using `the_hash_algo`
 5:  60c1c9558d !  5:  539dd9f9d5 submodule-config: stop registering submodule sources
    @@ Commit message
         submodule-config: stop registering submodule sources
     
         When reading the ".gitmodules" file from a blob in a repository other
    -    than `the_repository`, we register the repository's object database as
    +    than `the_repository`, we register that repository's object database as
         an in-memory source of `the_repository`'s object database. This call has
         its origins in d9b8b8f896 (submodule-config.c: use repo_get_oid for
         reading .gitmodules, 2019-04-16): back then, `config_with_options()` was
 6:  7f5354c2bf =  6:  11db0ac853 builtin/grep: stop registering submodule ODB as source
 7:  24e5a2a3b7 =  7:  5d4660657c odb: remove infrastructure to register submodule sources
 8:  c07db4254f =  8:  577e58d475 tmp-objdir: drop unused function to register alternate
 9:  e1022b26a6 =  9:  00bc8549a9 odb/packed: fix memory leaks when freeing source
10:  8c5c827595 = 10:  d66457c73a builtin/multi-pack-index: refuse unknown sources with "--object-dir="
11:  7d7cb53936 = 11:  7d13925a93 t/helper: adapt read-midx to not link ad-hoc source anymore
12:  537a58d286 = 12:  6a0092d870 t/helper: stop registering alternates in "ref-store" command
13:  d704faeb44 = 13:  4e446e8a51 odb: remove the ability to link sources ad-hoc

---
base-commit: e5d60560f61f520e9ea350645a6cc9770b0f1607
change-id: 20260811-pks-odb-registering-in-memory-sources-88648cd95735