error: git-remote-https died of signal 13

Hi,

Using git version 1.9.2 I am getting this error:

[normal@laptop tmp]$ git clone https://github.com/mozilla/rust.git
Cloning into 'rust'...
remote: Reusing existing pack: 296648, done.
remote: Counting objects: 80, done.
remote: Compressing objects: 100% (77/77), done.
remote: Total 296728 (delta 22), reused 9 (delta 3)
Receiving objects: 100% (296728/296728), 110.68 MiB | 190.00 KiB/s, done.
Resolving deltas: 100% (238828/238828), done.
Checking connectivity... done.
error: git-remote-https died of signal 13

The repository appears to be cloned fine, I can clone other repositories
without error.

I appear to have the exact same symptoms as Stefan in November 2013 as
archived here:
http://thread.gmane.org/gmane.comp.version-control.git/238242/focus=238311.
In this case the cause was a curl error that was fixed in version curl
version 7.34...

I have curl version 7.36 though, in case some of the other output matters:

[normal@laptop tmp]$ curl --version
curl 7.36.0 (x86_64-unknown-linux-gnu) libcurl/7.36.0 OpenSSL/1.0.1g
zlib/1.2.8 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp
scp sftp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IPv6 Largefile NTLM NTLM_WB SSL libz
TLS-SRP

The curl command to reproduce the since fixed bug (curl --limit-rate 250k
-L http://github.com/raspberrypi/firmware/archive/4ade27942e.tar.gz >
/dev/null) runs without error and exits with a value of 0.

The first response to Stefan's bug report requested the output of the
command with GIT_TRANSPORT_HELPER_DEBUG=1 and strace -f, so I've included
that:

GIT_TRANSPORT...:
https://drive.google.com/file/d/0BxdiDTQp3MYXQWJJaVdydUstbWs/edit?usp=sharing
strace (33M download, 330M uncompressed):
https://drive.google.com/file/d/0BxdiDTQp3MYXb1VKWWtFUDdvbjQ/edit?usp=sharing

Thanks,
Greg

On Sun, Apr 20, 2014 at 08:42:15PM -0400, Greg M wrote:

Using git version 1.9.2 I am getting this error:

[normal@laptop tmp]$ git clone https://github.com/mozilla/rust.git
Cloning into 'rust'...
remote: Reusing existing pack: 296648, done.
remote: Counting objects: 80, done.
remote: Compressing objects: 100% (77/77), done.
remote: Total 296728 (delta 22), reused 9 (delta 3)
Receiving objects: 100% (296728/296728), 110.68 MiB | 190.00 KiB/s, done.
Resolving deltas: 100% (238828/238828), done.
Checking connectivity... done.
error: git-remote-https died of signal 13

Thanks for a thorough bug report. I looked through your strace output,
and this really does look like another case of OpenSSL getting SIGPIPE
while closing the connection.

It's odd, though, as your curl version has my patches, along with
similar extra fixes in 854aca5 (multi: ignore sigpipe internally,
2014-02-17). But I guess there may be a code path that needs similar
treatment.

The easiest way to find it is probably to attach a debugger to the
running git-remote-https, and get a backtrace when it dies from SIGPIPE.
You'll probably want to install your system's debug packages for curl,
too.

I have curl version 7.36 though, in case some of the other output matters:

[normal@laptop tmp]$ curl --version
curl 7.36.0 (x86_64-unknown-linux-gnu) libcurl/7.36.0 OpenSSL/1.0.1g
zlib/1.2.8 libssh2/1.4.3

Another possibility is that your curl binary is up-to-date, but you are
linking against an older version of libcurl that does not have the
SIGPIPE workarounds.

I'm not sure of the best way to check that, but a hacky way under Linux
is:

  $ ldd $(git --exec-path)/git-remote-https | grep libcurl
          libcurl.so.4 => /usr/lib/x86_64-linux-gnu/libcurl.so.4
  $ strings  /usr/lib/x86_64-linux-gnu/libcurl.so.4 | grep '7\.'
  CLIENT libcurl 7.36.0

-Peff

On Wed, Apr 23, 2014 at 2:59 AM, Jeff King [off-list ref] wrote:

On Sun, Apr 20, 2014 at 08:42:15PM -0400, Greg M wrote:

quoted

Using git version 1.9.2 I am getting this error:

[normal@laptop tmp]$ git clone https://github.com/mozilla/rust.git
Cloning into 'rust'...
remote: Reusing existing pack: 296648, done.
remote: Counting objects: 80, done.
remote: Compressing objects: 100% (77/77), done.
remote: Total 296728 (delta 22), reused 9 (delta 3)
Receiving objects: 100% (296728/296728), 110.68 MiB | 190.00 KiB/s, done.
Resolving deltas: 100% (238828/238828), done.
Checking connectivity... done.
error: git-remote-https died of signal 13

Thanks for a thorough bug report. I looked through your strace output,
and this really does look like another case of OpenSSL getting SIGPIPE
while closing the connection.

It's odd, though, as your curl version has my patches, along with
similar extra fixes in 854aca5 (multi: ignore sigpipe internally,
2014-02-17). But I guess there may be a code path that needs similar
treatment.

The easiest way to find it is probably to attach a debugger to the
running git-remote-https, and get a backtrace when it dies from SIGPIPE.
You'll probably want to install your system's debug packages for curl,
too.

The backtrace:

Program received signal SIGPIPE, Broken pipe.
0x00007fdcfd511a2d in write () from /usr/lib/libpthread.so.0
(gdb) bt
#0  0x00007fdcfd511a2d in write () from /usr/lib/libpthread.so.0
#1  0x00007fdcfd81a0f6 in sock_write () from /usr/lib/libcrypto.so.1.0.0
#2  0x00007fdcfd817edb in BIO_write () from /usr/lib/libcrypto.so.1.0.0
#3  0x00007fdcfc662902 in ssl3_write_pending () from /usr/lib/libssl.so.1.0.0
#4  0x00007fdcfc664b77 in ssl3_dispatch_alert () from /usr/lib/libssl.so.1.0.0
#5  0x00007fdcfc660822 in ssl3_shutdown () from /usr/lib/libssl.so.1.0.0
#6  0x00007fdcfd2e944e in Curl_ossl_close () from /usr/lib/libcurl.so.4
#7  0x00007fdcfd2b6459 in Curl_disconnect () from /usr/lib/libcurl.so.4
#8  0x00007fdcfd2c8131 in curl_multi_cleanup () from /usr/lib/libcurl.so.4
#9  0x000000000040764b in ?? ()
#10 0x0000000000404e4d in ?? ()
#11 0x00007fdcfcf0fb05 in __libc_start_main () from /usr/lib/libc.so.6
#12 0x000000000040552e in ?? ()

Another possibility is that your curl binary is up-to-date, but you are
linking against an older version of libcurl that does not have the
SIGPIPE workarounds.

I'm not sure of the best way to check that, but a hacky way under Linux
is:

  $ ldd $(git --exec-path)/git-remote-https | grep libcurl
          libcurl.so.4 => /usr/lib/x86_64-linux-gnu/libcurl.so.4
  $ strings  /usr/lib/x86_64-linux-gnu/libcurl.so.4 | grep '7\.'
  CLIENT libcurl 7.36.0

-Peff

Seems to actually be that version of libcurl:

[normal@laptop tmp]$ ldd $(git --exec-path)/git-remote-https | grep libcurl
    libcurl.so.4 => /usr/lib/libcurl.so.4 (0x00007ff1cce5e000)
[normal@laptop tmp]$ strings /usr/lib/libcurl.so.4 | grep '7\.'
7.36f
CLIENT libcurl 7.36.0
CLIENT libcurl 7.36.0
CLIENT libcurl 7.36.0
7.36.0
[normal@laptop tmp]$

Also I don't think I have any other versions to link against:

[normal@laptop tmp]$ ls -l /usr/lib/libcurl*
lrwxrwxrwx 1 root root     16 Mar 26 10:12 /usr/lib/libcurl.so ->
libcurl.so.4.3.0
lrwxrwxrwx 1 root root     16 Mar 26 10:12 /usr/lib/libcurl.so.4 ->
libcurl.so.4.3.0
-rwxr-xr-x 1 root root 443488 Mar 26 10:12 /usr/lib/libcurl.so.4.3.0

Greg

On Wed, Apr 23, 2014 at 07:49:09AM -0400, Greg M wrote:

quoted

The easiest way to find it is probably to attach a debugger to the
running git-remote-https, and get a backtrace when it dies from SIGPIPE.
You'll probably want to install your system's debug packages for curl,
too.

The backtrace:

Program received signal SIGPIPE, Broken pipe.
0x00007fdcfd511a2d in write () from /usr/lib/libpthread.so.0
(gdb) bt
#0  0x00007fdcfd511a2d in write () from /usr/lib/libpthread.so.0
#1  0x00007fdcfd81a0f6 in sock_write () from /usr/lib/libcrypto.so.1.0.0
#2  0x00007fdcfd817edb in BIO_write () from /usr/lib/libcrypto.so.1.0.0
#3  0x00007fdcfc662902 in ssl3_write_pending () from /usr/lib/libssl.so.1.0.0
#4  0x00007fdcfc664b77 in ssl3_dispatch_alert () from /usr/lib/libssl.so.1.0.0
#5  0x00007fdcfc660822 in ssl3_shutdown () from /usr/lib/libssl.so.1.0.0
#6  0x00007fdcfd2e944e in Curl_ossl_close () from /usr/lib/libcurl.so.4
#7  0x00007fdcfd2b6459 in Curl_disconnect () from /usr/lib/libcurl.so.4
#8  0x00007fdcfd2c8131 in curl_multi_cleanup () from /usr/lib/libcurl.so.4
#9  0x000000000040764b in ?? ()
#10 0x0000000000404e4d in ?? ()
#11 0x00007fdcfcf0fb05 in __libc_start_main () from /usr/lib/libc.so.6
#12 0x000000000040552e in ?? ()

Thanks, that's very helpful. I wasn't able to reproduce your problem
locally, but I suspect the curl patch below may fix it:

diff --git a/lib/multi.c b/lib/multi.c
index bc93264..72e4825 100644
--- a/lib/multi.c
+++ b/lib/multi.c

@@ -1804,10 +1804,13 @@ static void close_all_connections(struct Curl_multi *multi)
 
   conn = Curl_conncache_find_first_connection(multi->conn_cache);
   while(conn) {
+    SIGPIPE_VARIABLE(pipe_st);
     conn->data = multi->closure_handle;
 
+    sigpipe_ignore(conn->data, &pipe_st);
     /* This will remove the connection from the cache */
     (void)Curl_disconnect(conn, FALSE);
+    sigpipe_restore(&pipe_st);
 
     conn = Curl_conncache_find_first_connection(multi->conn_cache);
   }

Let me know if you need any pointers on getting curl built or tested
with git.

Daniel, I think the similar fix to curl_multi_cleanup in commit a900d45
missed this code path, and we need something like the above patch. I
know you were trying to keep the SIGPIPE mess at the entrance-points to
the library, and this works against that. But we need a SessionHandle to
pass to sigpipe_ignore to look at its "no_signal" flag, and of course in
the case of multi we may have several such handles. If there's a similar
flag we can check on the multi handle, we could just cover all of
curl_multi_cleanup with a single ignore/reset pair.

-Peff

On Thu, Apr 24, 2014 at 12:15 AM, Jeff King [off-list ref] wrote:

The patch fixes the problem,

Greg

On Thu, 24 Apr 2014, Jeff King wrote:

Thanks, that's very helpful. I wasn't able to reproduce your problem 
locally, but I suspect the curl patch below may fix it:

...

Daniel, I think the similar fix to curl_multi_cleanup in commit a900d45 
missed this code path, and we need something like the above patch. I know 
you were trying to keep the SIGPIPE mess at the entrance-points to the 
library, and this works against that. But we need a SessionHandle to pass to 
sigpipe_ignore to look at its "no_signal" flag, and of course in the case of 
multi we may have several such handles. If there's a similar flag we can 
check on the multi handle, we could just cover all of curl_multi_cleanup 
with a single ignore/reset pair.

Thanks a lot for this! I've taken this issue over to the libcurl mailing list 
and we'll work out the best way to address it over there... At least we know 
the patch works as intended.

-- 

  / daniel.haxx.se