Git hangs at “Writing objects: 11%”

Hi, everyone,

I've run into a problem that I cannot debug.  I've talked with people at length on IRC, I've made ample use of Google.  I'm getting nowhere.  I'm not even getting any attention on stackexchange (where usually, questions are answered with surprising alacrity).  

This is not the common scenario of Windows and git-daemon that I have seen answered elsewhere. This is Linux to Linux, using ssh. On both the client and server, git version is 1.7.8.6.

I've tried doing the following:

	• Create a local empty repo and push to that -- works fine
	• Pull from local repo to local repo -- works fine
	• Clone the remote repo to the local machine -- works fine
	• git fsck and git gc on both server and client repos
	• Push to pre-existing remote repo -- hangs writing objects at 11%
	• Create empty remote repo and push to that -- hangs writing objects at 2%

There's an env variable I can set to make git produce debug messages. Nothing useful there.

Can anyone here help me figure out what's going wrong?


The post to stackexchange is at:  http://stackoverflow.com/questions/10740601/cannot-push-to-remote-git-repo-over-ssh-stuck-at-writing-objects-11


Thanks in advance for any advice you can give!

On Thu, May 24, 2012 at 05:40:41PM -0400, Timothy Normand Miller wrote:

I've run into a problem that I cannot debug.  I've talked with people
at length on IRC, I've made ample use of Google.  I'm getting nowhere.
I'm not even getting any attention on stackexchange (where usually,
questions are answered with surprising alacrity).  

This is not the common scenario of Windows and git-daemon that I have
seen answered elsewhere. This is Linux to Linux, using ssh. On both
the client and server, git version is 1.7.8.6.

I'd first start with trying to remove as many variables as possible.
Does the problem only happen over ssh, or does it also happen when
pushing across a pipe to a repo on the local machine? If so, does it
also happen during a fetch of the same data?

If you can reproduce it at will (which it sounds like you can), you
could also try some older or newer git versions to see if they work any
better. If you can find a working version, it might be worth trying to
bisect and find the commit that introduces the breakage.

If the problem still exists in the latest version, then I'd start by
stracing as much as possible. On the client side, you can use "strace
-f" to see what all of the processes are doing; you'll probably also
want to pass:

  --receive-pack='strace -f -o foo.out git-receive-pack'

to git-push to ask the remote side to strace. There's a reasonable
chance you'll simply see that the client side is waiting on the server
side for I/O, so you'll want to know what the server side is doing.

I see you posted an strace snippet of a process waiting in select() on
stack overflow. It's hard to tell what's going on from there, though,
because we can't see which processes are which (we see the pids, but we
don't know which programs they're running, or where the commands go). A
full strace log would help a lot (if it's long and you need a place to
post it, try something like https://gist.github.com).

And finally, if the repo is something you can make public, I can try to
reproduce on my machine. That might tell us if the problem is with your
repo, or something else about your machines or setup.

-Peff

On May 24, 2012, at 8:51 PM, Jeff King wrote:

On Thu, May 24, 2012 at 05:40:41PM -0400, Timothy Normand Miller wrote:

quoted

I've run into a problem that I cannot debug.  I've talked with people
at length on IRC, I've made ample use of Google.  I'm getting nowhere.
I'm not even getting any attention on stackexchange (where usually,
questions are answered with surprising alacrity).  

This is not the common scenario of Windows and git-daemon that I have
seen answered elsewhere. This is Linux to Linux, using ssh. On both
the client and server, git version is 1.7.8.6.

Thank you for helping!

I'd first start with trying to remove as many variables as possible.
Does the problem only happen over ssh, or does it also happen when
pushing across a pipe to a repo on the local machine? If so, does it
also happen during a fetch of the same data?

If the local repo I'm pushing to is given by pathname, it's fine.  If I'm pushing to another account via ssh on the localhost, it has the same hangup.

All fetches work fine.

If you can reproduce it at will (which it sounds like you can), you
could also try some older or newer git versions to see if they work any
better. If you can find a working version, it might be worth trying to
bisect and find the commit that introduces the breakage.

There's this one CentOS 6 machine where I know for a fact that git hasn't been updated since I was last able to push this particular repo.  That machine has git version 1.7.1.  That's where I first encountered the problem.  So I copied the repo to my Mac (version 1.7.8.6), but it freezes there at 11%.  Then I tried copying it to the same machine that hosts the repo (Gentoo Linux with git version 1.7.8.6), and push from there (still using ssh, but locally), and it also freezes at 11%.

So there's something wonky about the repo that git doesn't like, although it only has the problem over ssh.  I haven't tried git's native protocol.

If the problem still exists in the latest version, then I'd start by
stracing as much as possible. On the client side, you can use "strace
-f" to see what all of the processes are doing; you'll probably also
want to pass:

 --receive-pack='strace -f -o foo.out git-receive-pack'

First, I ran the push and then killed it, and I got this:	
http://www.cse.ohio-state.edu/~millerti/foo.out

But then I realized that you'd want to see where it hung up, so I can it again and then sent the file before killing the push:
http://www.cse.ohio-state.edu/~millerti/foo2.out

to git-push to ask the remote side to strace. There's a reasonable
chance you'll simply see that the client side is waiting on the server
side for I/O, so you'll want to know what the server side is doing.

I really don't know how to interpret the trace.

I see you posted an strace snippet of a process waiting in select() on
stack overflow. It's hard to tell what's going on from there, though,
because we can't see which processes are which (we see the pids, but we
don't know which programs they're running, or where the commands go). A
full strace log would help a lot (if it's long and you need a place to
post it, try something like https://gist.github.com).

Try this:
http://www.cse.ohio-state.edu/~millerti/revue_strace3.txt

This one was from earlier in the day.  It's from a push from the same machine that hosts the repo.

And finally, if the repo is something you can make public, I can try to
reproduce on my machine. That might tell us if the problem is with your
repo, or something else about your machines or setup.

That would make things a lot easier, but I'm not at liberty to share it.


Thanks again!

I did some more research and testing.  This is an ssh problem.  If I change the protocol to git://, then I can push remotely all I want.  Same for file://.  But using ssh, it hangs.  Since I want ssh encryption, I still need to figure out why ssh is going wrong.


On May 24, 2012, at 9:41 PM, Timothy Normand Miller wrote:

On May 24, 2012, at 8:51 PM, Jeff King wrote:

quoted

On Thu, May 24, 2012 at 05:40:41PM -0400, Timothy Normand Miller wrote:

quoted

I've run into a problem that I cannot debug.  I've talked with people
at length on IRC, I've made ample use of Google.  I'm getting nowhere.
I'm not even getting any attention on stackexchange (where usually,
questions are answered with surprising alacrity).  

This is not the common scenario of Windows and git-daemon that I have
seen answered elsewhere. This is Linux to Linux, using ssh. On both
the client and server, git version is 1.7.8.6.

Thank you for helping!

quoted

I'd first start with trying to remove as many variables as possible.
Does the problem only happen over ssh, or does it also happen when
pushing across a pipe to a repo on the local machine? If so, does it
also happen during a fetch of the same data?

If the local repo I'm pushing to is given by pathname, it's fine.  If I'm pushing to another account via ssh on the localhost, it has the same hangup.

All fetches work fine.

quoted

If you can reproduce it at will (which it sounds like you can), you
could also try some older or newer git versions to see if they work any
better. If you can find a working version, it might be worth trying to
bisect and find the commit that introduces the breakage.

There's this one CentOS 6 machine where I know for a fact that git hasn't been updated since I was last able to push this particular repo.  That machine has git version 1.7.1.  That's where I first encountered the problem.  So I copied the repo to my Mac (version 1.7.8.6), but it freezes there at 11%.  Then I tried copying it to the same machine that hosts the repo (Gentoo Linux with git version 1.7.8.6), and push from there (still using ssh, but locally), and it also freezes at 11%.

So there's something wonky about the repo that git doesn't like, although it only has the problem over ssh.  I haven't tried git's native protocol.

quoted

If the problem still exists in the latest version, then I'd start by
stracing as much as possible. On the client side, you can use "strace
-f" to see what all of the processes are doing; you'll probably also
want to pass:

--receive-pack='strace -f -o foo.out git-receive-pack'

First, I ran the push and then killed it, and I got this:	
http://www.cse.ohio-state.edu/~millerti/foo.out

But then I realized that you'd want to see where it hung up, so I can it again and then sent the file before killing the push:
http://www.cse.ohio-state.edu/~millerti/foo2.out

quoted

to git-push to ask the remote side to strace. There's a reasonable
chance you'll simply see that the client side is waiting on the server
side for I/O, so you'll want to know what the server side is doing.

I really don't know how to interpret the trace.

quoted

I see you posted an strace snippet of a process waiting in select() on
stack overflow. It's hard to tell what's going on from there, though,
because we can't see which processes are which (we see the pids, but we
don't know which programs they're running, or where the commands go). A
full strace log would help a lot (if it's long and you need a place to
post it, try something like https://gist.github.com).

Try this:
http://www.cse.ohio-state.edu/~millerti/revue_strace3.txt

This one was from earlier in the day.  It's from a push from the same machine that hosts the repo.

quoted

And finally, if the repo is something you can make public, I can try to
reproduce on my machine. That might tell us if the problem is with your
repo, or something else about your machines or setup.

That would make things a lot easier, but I'm not at liberty to share it.


Thanks again!

--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

In message [off-list ref], Timothy Normand Miller writes:

    I did some more research and testing.  This is an ssh problem.  If
    I change the protocol to git://, then I can push remotely all I
    want.  Same for file://.  But using ssh, it hangs.  Since I want
    ssh encryption, I still need to figure out why ssh is going wrong.

You could try smart-http with encryption, either as an experiment or
as a workaround.

http://sethrobertson.github.com/HowToPutGitOnTheWeb/ provides
information on how to put a git repository on the web, with encryption
and password/pam/ldap/public key authentication.

					-Seth Robertson

I tried uploading a file via sftp, and it stalls out at 160K.  So this is the cause of the problem.  SSH is broken.  Unfortunately, googling for ssh and stall isn't getting me anywhere.

On May 26, 2012, at 11:12 PM, Seth Robertson wrote:

In message [off-list ref], Timothy Normand Miller writes:

   I did some more research and testing.  This is an ssh problem.  If
   I change the protocol to git://, then I can push remotely all I
   want.  Same for file://.  But using ssh, it hangs.  Since I want
   ssh encryption, I still need to figure out why ssh is going wrong.

You could try smart-http with encryption, either as an experiment or
as a workaround.

http://sethrobertson.github.com/HowToPutGitOnTheWeb/ provides
information on how to put a git repository on the web, with encryption
and password/pam/ldap/public key authentication.

					-Seth Robertson
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Mon, May 28, 2012 at 08:48:32AM -0400, Timothy Normand Miller wrote:

I tried uploading a file via sftp, and it stalls out at 160K.  So this
is the cause of the problem.  SSH is broken.  Unfortunately, googling
for ssh and stall isn't getting me anywhere.

Thanks for following up. I hadn't had a chance to look at your strace
output until today, and now you have saved me the trouble. :)

So I think we can rule out a git bug at this point. I would say it is an
issue with network settings, but it sounds like you reproduced it while
sshing to the local host (though depending on your setup, you might
still be able to trigger firewall rules in that case, which could have
an impact).

At this point, I'd try upgrading ssh if possible, and otherwise submit a
bug report to the ssh vendor (either your distro, or openssh directly if
you can reproduce it on pristine source).

-Peff

On Mon, 28 May 2012, Timothy Normand Miller wrote:

I tried uploading a file via sftp, and it stalls out at 160K.  So this 
is the cause of the problem.  SSH is broken.  Unfortunately, googling 
for ssh and stall isn't getting me anywhere.

Not sure if this is the same issue you're seeing or not, but I ran into 
something similar a while ago - when connected from less reliable 
networks, connections (especially rsync or scp) over SSH stalled after a 
short while - interactive sessions worked fine.

In my case, the issue was a too pedantic iptables configuration. When 
communicating with the server from a less reliable network, packets could 
arrive out of order. If the server received packets that 
iptables/conntrack didn't think belonged to the session, my config sent 
ICMP unreachable packets as response (-j REJECT), basically terminating 
the connection.

I fixed it by adjusting the iptables config to not send ICMP unreachable 
for this case. An alternative solution was to set sysctl 
"net.netfilter.nf_conntrack_tcp_be_liberal = 1" - this made the out of 
order packets still be counted as part of the session, not triggering the 
bad behaviour of my iptables config.

Again, not sure if this is related to your issue or not, but it at least a 
similar symptom.

// Martin

The fix is to set "TcpRcvBufPoll no" in sshd_config.

http://forums.gentoo.org/viewtopic-t-925044-start-0-postdays-0-postorder-asc-highlight-.html
http://forums.gentoo.org/viewtopic-p-7036250.html


On May 28, 2012, at 3:36 PM, Martin Storsjö wrote:

On Mon, 28 May 2012, Timothy Normand Miller wrote:

quoted

I tried uploading a file via sftp, and it stalls out at 160K.  So this is the cause of the problem.  SSH is broken.  Unfortunately, googling for ssh and stall isn't getting me anywhere.

Not sure if this is the same issue you're seeing or not, but I ran into something similar a while ago - when connected from less reliable networks, connections (especially rsync or scp) over SSH stalled after a short while - interactive sessions worked fine.

In my case, the issue was a too pedantic iptables configuration. When communicating with the server from a less reliable network, packets could arrive out of order. If the server received packets that iptables/conntrack didn't think belonged to the session, my config sent ICMP unreachable packets as response (-j REJECT), basically terminating the connection.

I fixed it by adjusting the iptables config to not send ICMP unreachable for this case. An alternative solution was to set sysctl "net.netfilter.nf_conntrack_tcp_be_liberal = 1" - this made the out of order packets still be counted as part of the session, not triggering the bad behaviour of my iptables config.

Again, not sure if this is related to your issue or not, but it at least a similar symptom.

// Martin
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html