Re: [RFC PATCH] upload_pack.c: make deepen-not more tree-ish

Re-send to the Git mailing-list as setting a font on gmail switched
plain-text to HTML and thus, got blocked by mailing-list.

On Sun, Feb 12, 2023 at 3:09 PM Son Luong Ngoc [off-list ref] wrote:

Hi Andrew,

On Sat, Feb 11, 2023 at 11:49 PM Andrew Wansink [off-list ref] wrote:

quoted

This unlocks `git clone --shallow-exclude=<commit-sha1>`

git-clone only accepts --shallow-excude arguments where
the argument is a branch or tag because upload_pack only
searches deepen-not arguments for branches and tags.

Make process_deepen_not search for commit objects if no
branch or tag is found then add them to the deepen_not
list.

Signed-off-by: Andrew Wansink <redacted>
---

At Uber we have a lot of patches in CI simultaneously,
the CI jobs will frequently clone the monorepo multiple
times for each patch.  They do this to calculate diffs
between a patch and its parent commit.

I used to manage a CI system that support monorepo use cases not so long ago.
We had several hosts(VM/Baremetal) on which we spin up containers for CI to run.

We maintain a bare copy of the monorepo on the host level (cron job / systemd / DaemonSet) and mount this as read-only into each of the CI containers.

Each of the CI containers would attempt to clone/fetch the monorepo with `--reference-if-able ./path/to/read-only-mount/repo.git` (1)
So that most of the needed objects are already on disk in the shared bare repo.


+-----------+  +-----------+  +-----------+
| container |  | container |  | container |
+-----------+  +-----------+  +-----------+
             \       |       /
      (mount) \      |      /
              +------------+                 +--------+
              | bare-repo  | <-------------- | Remote |
              +------------+   (git-fetch)   +--------+
                    |
                    | (maintain)
                    |
              +----------+
              | cron-job |
              +----------+

(forgive my horrible drawing)

With this setup, we did not have a need to shallow clone any longer,
and our git-clone in each container is simply a combination of git-ls-remote and a very light-weighted git-fetch.
In some cases, such as a job in the later stages of a CI pipeline,
the host would already download all the needed objects into the bare copy of the repository.
This lets us skip git-fetch entirely when the CI container executes.

Compared to the shallow clone approach,
our "local cache" approach sped up the clone speed drastically
while allowing developers to interact with git history inside tests a lot easier.

quoted

One optimisation in this flow is to clone only to a specific
depth, this may or may not work, depending on how old the
patch is.  In this case we have to --unshallow or discard
the shallow clone and fully clone the repo.

This patch would allow us to clone to exactly the depth we
need to find a patch's parent commit.

Hope it helps,
Son Luong.

(1): https://git-scm.com/docs/git-clone#Documentation/git-clone.txt---reference-if-ableltrepositorygt