Re: git fetch recursion problem
From: Glen Choo <hidden>
Date: 2023-06-01 22:40:42
Fraser Hanson [off-list ref] writes:
The git trace output looks like this (some server names redacted):
$ export GIT_TRACE=1
$ git fetch
07:41:19.325652 git.c:439 trace: built-in: git fetch
07:41:19.330118 run-command.c:655 trace: run_command:
GIT_DIR=.git git remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
07:41:19.337765 git.c:725 trace: exec:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
07:41:19.338220 run-command.c:655 trace: run_command:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
07:41:20.664527 run-command.c:655 trace: run_command: git -c
fetch.negotiationAlgorithm=noop fetch origin --no-tags
--no-write-fetch-head --recurse-submodules=no --filter=blob:none
--stdin
07:41:20.679814 git.c:439 trace: built-in: git fetch
origin --no-tags --no-write-fetch-head --recurse-submodules=no
--filter=blob:none --stdin
07:41:20.681735 run-command.c:655 trace: run_command: git
remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
07:41:20.692494 git.c:725 trace: exec:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
07:41:20.693132 run-command.c:655 trace: run_command:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
07:41:21.695677 run-command.c:655 trace: run_command: git -c
fetch.negotiationAlgorithm=noop fetch origin --no-tags
--no-write-fetch-head --recurse-submodules=no --filter=blob:none
--stdin
07:41:21.715621 git.c:439 trace: built-in: git fetch
origin --no-tags --no-write-fetch-head --recurse-submodules=no
--filter=blob:none --stdin
07:41:21.717230 run-command.c:655 trace: run_command: git
remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
07:41:21.727616 git.c:725 trace: exec:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
07:41:21.728337 run-command.c:655 trace: run_command:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
07:41:22.801117 run-command.c:655 trace: run_command: git -c
fetch.negotiationAlgorithm=noop fetch origin --no-tags
--no-write-fetch-head --recurse-submodules=no --filter=blob:none
--stdin
07:41:22.816026 git.c:439 trace: built-in: git fetch
origin --no-tags --no-write-fetch-head --recurse-submodules=no
--filter=blob:none --stdin
07:41:22.817972 run-command.c:655 trace: run_command: git
remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
07:41:22.828436 git.c:725 trace: exec:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
07:41:22.829219 run-command.c:655 trace: run_command:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
07:41:24.242220 run-command.c:655 trace: run_command: git -c
fetch.negotiationAlgorithm=noop fetch origin --no-tags
--no-write-fetch-head --recurse-submodules=no --filter=blob:none
--stdin
07:41:24.257211 git.c:439 trace: built-in: git fetch
origin --no-tags --no-write-fetch-head --recurse-submodules=no
--filter=blob:none --stdin
07:41:24.259264 run-command.c:655 trace: run_command: git
remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
07:41:24.269718 git.c:725 trace: exec:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
07:41:24.270612 run-command.c:655 trace: run_command:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
07:41:25.306129 run-command.c:655 trace: run_command: git -c
fetch.negotiationAlgorithm=noop fetch origin --no-tags
--no-write-fetch-head --recurse-submodules=no --filter=blob:none
--stdin
07:41:25.320842 git.c:439 trace: built-in: git fetch
origin --no-tags --no-write-fetch-head --recurse-submodules=no
--filter=blob:none --stdin
07:41:25.322801 run-command.c:655 trace: run_command: git
remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
07:41:25.332775 git.c:725 trace: exec:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
07:41:25.333522 run-command.c:655 trace: run_command:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
07:41:26.332580 run-command.c:655 trace: run_command: git -c
fetch.negotiationAlgorithm=noop fetch origin --no-tags
--no-write-fetch-head --recurse-submodules=no --filter=blob:none
--stdin
07:41:26.348349 git.c:439 trace: built-in: git fetch
origin --no-tags --no-write-fetch-head --recurse-submodules=no
--filter=blob:none --stdin
07:41:26.350225 run-command.c:655 trace: run_command: git
remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
07:41:26.361171 git.c:725 trace: exec:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
07:41:26.361969 run-command.c:655 trace: run_command:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
07:41:27.585641 run-command.c:655 trace: run_command: git -c
fetch.negotiationAlgorithm=noop fetch origin --no-tags
--no-write-fetch-head --recurse-submodules=no --filter=blob:none
--stdin
07:41:27.600316 git.c:439 trace: built-in: git fetch
origin --no-tags --no-write-fetch-head --recurse-submodules=no
--filter=blob:none --stdin
07:41:27.602370 run-command.c:655 trace: run_command: git
remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
07:41:27.611823 git.c:725 trace: exec:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
07:41:27.612607 run-command.c:655 trace: run_command:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
07:41:28.666477 run-command.c:655 trace: run_command: git -c
fetch.negotiationAlgorithm=noop fetch origin --no-tags
--no-write-fetch-head --recurse-submodules=no --filter=blob:none
--stdin
07:41:28.678956 git.c:439 trace: built-in: git fetch
origin --no-tags --no-write-fetch-head --recurse-submodules=no
--filter=blob:none --stdin
07:41:28.680665 run-command.c:655 trace: run_command: git
remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
07:41:28.690534 git.c:725 trace: exec:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
07:41:28.691277 run-command.c:655 trace: run_command:
git-remote-https origin
https://repomirror.eng.corporate.com/github-neovim/nvim-treesitter/nvim-treesitter-textobjects.git
^C
Random clues and experiments:
A `git clone` from our corporate mirrors always succeeds. Immediately
after a `git clone`, I am always able to `git fetch`, so far.
Something else must cause the repository to enter a broken state,
possibly the addition of upstream commits. I can't verify that since
I can't complete `git fetch.`
I can fix a broken repository by deleting these settings from `.git/config`:
promisor = true
partialclonefilter = blob:none
After this, `git fetch` succeeds.
Deleting just one of these settings is not sufficient, it must be both.
I copied one such broken repository out of our secure environment onto
a mac laptop that has access to both our mirror sites and to the
internet.
I verified that `git fetch` from our mirror is still broken on the mac.
Then I modified .git/config to point to the https://github.com/ url
instead of our mirror site, leaving the 'promisor' and
'partialclonefilter' settings in place.
Running `git fetch` in this state succeeded.
This suggests that our internal mirror site is returning something
different from github itself.
Here is the trace from successfully fetching from github.com:
$ git fetch
07:43:30.901275 git.c:439 trace: built-in: git fetch
07:43:30.904188 run-command.c:655 trace: run_command:
GIT_DIR=.git git remote-https origin
https://github.com/nvim-treesitter/nvim-treesitter-textobjects.git
07:43:30.917345 git.c:725 trace: exec:
git-remote-https origin
https://github.com/nvim-treesitter/nvim-treesitter-textobjects.git
07:43:30.918560 run-command.c:655 trace: run_command:
git-remote-https origin
https://github.com/nvim-treesitter/nvim-treesitter-textobjects.git
remote: Enumerating objects: 8, done.
remote: Counting objects: 100% (8/8), done.
remote: Compressing objects: 100% (5/5), done.
07:43:31.714337 run-command.c:655 trace: run_command: git
index-pack --stdin -v --fix-thin --promisor --pack_header=2,8
remote: Total 8 (delta 2), reused 5 (delta 1), pack-reused 0
07:43:31.729793 git.c:439 trace: built-in: git
index-pack --stdin -v --fix-thin --promisor --pack_header=2,8
Receiving objects: 100% (8/8), 3.36 KiB | 3.36 MiB/s, done.
Resolving deltas: 100% (2/2), completed with 1 local object.
07:43:31.738815 run-command.c:655 trace: run_command: git
rev-list --objects --stdin --exclude-promisor-objects --not --all
--quiet --alternate-refs
07:43:31.748286 git.c:439 trace: built-in: git
rev-list --objects --stdin --exclude-promisor-objects --not --all
--quiet --alternate-refs
From https://github.com/nvim-treesitter/nvim-treesitter-textobjects
9c5237b..95b76b9 master -> origin/master
9c5237b..11ae4c7 update-readme -> origin/update-readme
07:43:31.757504 run-command.c:1524 run_processes_parallel:
preparing to run up to 1 tasks
07:43:31.757515 run-command.c:1551 run_processes_parallel: done
07:43:31.757522 run-command.c:655 trace: run_command: git
maintenance run --auto --no-quiet
07:43:31.765282 git.c:439 trace: built-in: git
maintenance run --auto --no-quiet
The difference starts just a few lines in, my failed fetches never see this:
remote: Counting objects: 100% (8/8), done.
remote: Compressing objects: 100% (5/
Next I enabled both GIT_TRACE and GIT_CURL_VERBOSE to identify
differences in the HTTP conversation between a failed fetch versus a
successful one.
The successful run was made by deleting the `promisor` and
`partialclonefilter` settings from .git/config.
I'm having some some trouble attaching these to this email, I'll have
to include them in a follow up.This looks a lot like the partial clone fetch is recursing infinitely. This could happen if Git encountered a missing object, initiated a partial clone fetch to get the missing object, then in the process of that fetch it encountered a missing object, initiated a partial clone fetch to get the missing object, then (etc etc etc). I don't have good clues as to why that might be the case though. Wild speculation: Does this reproduce if you clone without submodules, or if you pass --no-recurse-submodules to "git fetch"? In a non-partial clone fetch, "git fetch" may read .gitmodules to get submodule information. Perhaps something similar is happening in the partial clone fetch.