Re: git push --mirror hung indefinitely

On 2025-03-29 at 14:07:17, Akash S wrote:

We compared the GIT_TRACE_PACKET=1 output of successful and
unsuccessful pushes and noticed that the "git flush message 0000" is
not being received from the Azure DevOps Server during unsuccessful
push. 

-------------------------------------------------------------------------------------------------------------------------------------------------------

Successful git push:
11:29:09.032050 pkt-line.c:85           packet:          git< ok refs/tags/v3.9.1
11:29:09.034601 http.c:878              == Info: TLSv1.3 (IN), TLS app data, [no content] (0):
11:29:09.034630 pkt-line.c:85           packet:     sideband< \10000
11:29:09.034643 http.c:878              == Info: TLSv1.3 (IN), TLS app data, [no content] (0):
11:29:09.034664 pkt-line.c:85           packet:     sideband< 0000
11:29:09.034664 pkt-line.c:85           packet:          git< 0000
11:29:09.034699 pkt-line.c:85           packet:          git> 0000
11:29:09.035667 http.c:878              == Info: TLSv1.3 (IN), TLS app data, [no content] (0):
11:29:09.035701 http.c:878              == Info: Connection #0 to host dev.azure.com left intact
To https://dev.azure.com/ORGNAME/PRJNAME/_git/REPONAME

-------------------------------------------------------------------------------------------------------------------------------------------------------

Unsuccessful git push:
11:33:58.759005 pkt-line.c:85           packet:     sideband< \10018ok refs/tags/v3.1.7
11:33:58.759035 pkt-line.c:85           packet:          git< ok refs/tags/v3.1.7
11:33:58.760165 http.c:878              == Info: TLSv1.3 (IN), TLS app data, [no content] (0):
11:33:58.760197 http.c:878              == Info: Connection #0 to host dev.azure.com left intact
(process hung here)

-------------------------------------------------------------------------------------------------------------------------------------------------------

We noticed that "git push mirror" ultimately spawns "git send-pack"
with options "--stdin" and "--stateless-rpc" and from the git
send-pack documentation we confirmed that the line must end with a
flush packet which is ultimately not happening in the hung cases.
pstack of send-pack process also shows it is waiting on reading
packets as you mentioned earlier.

`--stateless-rpc` is used for HTTPS, so that's expected.

Can you please confirm if the hung issue is due to the missing message or packet from the Azure DevOps Server?

We don't have a full trace output here, so it's hard to say completely
for certain, but from what I know of the protocol off the top of my
head, a flush packet is always required at the end of the section of
message.  So I would say that it's very likely that's the cause here.

Note that it could be that you're _also_ missing one or more `ok`
responses from the ref updates you're doing here, and that it's hanging
_before_ it would normally send the flush packet.  That might point to a
different cause on the server side, but the effect would be the same.

If you're talking to the Azure DevOps people, I'd provide them a full
trace output of both working and non-working pushes so that they can
inspect it themselves, which will help them figure out where it might be
going wrong.
-- 
brian m. carlson (they/them)
Toronto, Ontario, CA