Re: [RFH] Why do osx CI jobs so unreliable?
From: Michael Montalbo <hidden>
Date: 2026-06-20 15:33:26
Patrick Steinhardt [off-list ref] writes:
So I strongly suspect that it most be one of the t555* tests. [...] Maybe this is something that's specific to GitHub's environment...
I think you're right it's t5551/t5559. The runs Junio linked: osx-clang cancelled 360min osx-gcc cancelled 360min osx-reftable success 35min osx-meson success 61min All four run the same t5551/t5559 under EXPENSIVE. The two that finished differ in just two ways, which look like the levers: osx-reftable generates the 100k-ref advertisement in ~24ms vs ~1.2s for loose refs on macOS (so much less time mid-response), and osx-meson runs tests at nproc while the prove jobs hardcode --jobs=10 on a 3-core runner (over recent master/next the prove jobs hang ~40%, meson ~10%). When it is wedged the whole chain sits at 0% CPU. upload-pack is blocked in write() on the ls-refs advertisement, curl blocked in select(). So it looks like an HTTP/2 flow-control stall on the response side. The same stall resets itself after ~60-85s on my Linux box and on a bare-metal Mac, but not on the GitHub runner; I haven't pinned down why yet. On the chance those two levers are the fix, a branch off master: https://github.com/mmontalbo/git/tree/mm/macos-ci-hang-fix - pack the refs in t5551's enormous-ref-negotiation test (doesn't change what it checks on the wire, just avoids re-reading 100k loose files to advertise them, like reftable already does) - use the core count for $JOBS on the GitHub macOS path, matching the GitLab branch in the same ci/lib.sh and what meson does I ran the two macOS jobs under EXPENSIVE about eight times with these and they all finished in ~30-44min instead of hanging. Happy to send out a patch if it's helpful.