From: Thomas Rast <hidden> Date: 2016-06-15 22:56:45
Sivaram Kannan [off-list ref] writes:
Hi,
I am using git with Gitlab/Gitolite configuration. Git version is
1.7.9.5 in Ubuntu 12.04. There has been a consistent git crash
recently and have attached the /var/crash/_usr_lib_git-core_
git.1001.crash file.
The crash output is pasted in the following link
http://pastebin.com/uAQS81BX
I removed some long binary information at the end of the file as
pastebin does not allow more than 500k pasting. The crash is
consistently happening and I am planning on for a debian deployment of
gitlab.
Can you tell us what command you ran, and also try to get a readable
backtrace from your installation?
It seems that the paste would have contained a core dump (you snipped
it9, but it would be pretty useless without the corresponding binary
anyway. Once you have the coredump in hand (as a file) you can use
gdb $(which git) the_coredump_file
and then in the GDB prompt, enter 'backtrace' and paste its output, to
give us an idea what is going on.
--
Thomas Rast
trast@{inf,student}.ethz.ch
Can you tell us what command you ran, and also try to get a readable
backtrace from your installation?
The crash is happening only when the users are trying to do a clone. I
was monitoring from the htop when triggering a clone operation, all
the cores of the processor hits 100% for some time before dropping.
The servers hw config is Intel Quad core processor with 8GB Ram.
It seems that the paste would have contained a core dump (you snipped
it9, but it would be pretty useless without the corresponding binary
anyway. Once you have the coredump in hand (as a file) you can use
gdb $(which git) the_coredump_file
and then in the GDB prompt, enter 'backtrace' and paste its output, to
give us an idea what is going on.
Output of coredump gdb:
gitadmin@gitserver:/var/crash/dump$ gdb git CoreDump
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html> This is free software: you are
free to change and redistribute it. There is NO WARRANTY, to the
extent permitted by law. Type "show copying" and "show warranty" for
details. This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
Reading symbols from /usr/bin/git...(no debugging symbols
found)...done. BFD: Warning: /var/crash/dump/CoreDump is truncated:
expected core file size >= 600195072, found: 1114112.
^^^ Try to issue the
$ ulimit -c unlimited
command in your shell before attempting the cloning -- this should
remove the upper limit on the core file size. And try look for the core
file in the current directory after the crash occurs. I'm not sure
Ubuntu's "crash interceptor" won't kick in, but just in case...
Have set the git user's crash limit to 1GB in
/etc/security/limits.conf and still getting the same error when
issuing gdb to the crash file.
command in your shell before attempting the cloning -- this should
remove the upper limit on the core file size. And try look for the core
file in the current directory after the crash occurs. I'm not sure
Ubuntu's "crash interceptor" won't kick in, but just in case...
You mean, /usr/bin/git? crash file for git is getting created each
time it crashes in /var/crash.
Can you please tell me what else I could try? Would upgrading to the
1.8.2.1 - latest in Ubuntu PPA would help?
./Siva.
Have set the git user's crash limit to 1GB in
/etc/security/limits.conf and still getting the same error when
issuing gdb to the crash file.
Yep, suppsedly in Ubuntu it's not that easy to just get a plain old
coredump file -- see below.
quoted
command in your shell before attempting the cloning -- this should
remove the upper limit on the core file size. And try look for the
core file in the current directory after the crash occurs. I'm not
sure Ubuntu's "crash interceptor" won't kick in, but just in case...
You mean, /usr/bin/git? crash file for git is getting created each
time it crashes in /var/crash.
Can you please tell me what else I could try?
Googling for "ubuntu+disable+crash" turns up that your Git crashes are
handled by a system-wide tool called "apport" [1].
Considering this, I would try to explore two routes:
* [1] Tells that apport has a special tool, apport-retrace, which is
said to be able to download available matching debug packages, if
any, and generate the stack traces. Basically this would do what
Thomas advised you to attempt to do using GDB.
* Try to disable apprort permanently and then crash Git normally,
so that apport does not interfere with the crash and the kernel is
able to generate a regular core file in your current directory.
Be sure to verify the core-file-size limit has a sensibly large
value in your shell before attempting to do that.
Would upgrading to the 1.8.2.1 - latest in Ubuntu PPA would help?
Yes, this is a viable way to try solving the problem.
*But* there's a downside: the crash you're experiencing might affect
later Git versions as well as yours. And if you just throw your hands
there, the bug will continue to be unfixed. Hence I urge you to be a
good F/OSS user and help the Git devs investigate the case.
1. https://wiki.ubuntu.com/Apport
Have set the git user's crash limit to 1GB in
/etc/security/limits.conf and still getting the same error when
issuing gdb to the crash file.
Yep, suppsedly in Ubuntu it's not that easy to just get a plain old
coredump file -- see below.
Got an proper dump from git this time. See whether it helps. I have
setup another machine with Ubuntu 12.04 and updated only git and
ported the repo there, after 200 clones no crash so far. Mostly will
be moving latest repo to that server to solve this issue.
gitadmin@gitserver:/var/crash/gitcash$ gdb git CoreDump
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
Reading symbols from /usr/bin/git...(no debugging symbols found)...done.
[New LWP 12823]
warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `git pack-objects --revs --all --stdout
--progress --delta-base-offset'.
Program terminated with signal 11, Segmentation fault.
#0 0x000000000044bb77 in ?? ()
(gdb) bt
#0 0x000000000044bb77 in ?? ()
#1 0x000000000044cb54 in ?? ()
#2 0x000000000044d4f1 in ?? ()
#3 0x0000000000405634 in ?? ()
#4 0x0000000000404a30 in ?? ()
#5 0x00007fe2a149876d in __libc_start_main (main=0x404980, argc=7,
ubp_av=0x7fffad58fb28, init=<optimized out>, fini=<optimized out>,
rtld_fini=<optimized out>,
stack_end=0x7fffad58fb18) at libc-start.c:226
#6 0x0000000000404e65 in ?? ()
#7 0x00007fffad58fb18 in ?? ()
#8 0x000000000000001c in ?? ()
#9 0x0000000000000007 in ?? ()
#10 0x00007fffad591c95 in ?? ()
#11 0x00007fffad591c99 in ?? ()
#12 0x00007fffad591ca6 in ?? ()
#13 0x00007fffad591cad in ?? ()
#14 0x00007fffad591cb3 in ?? ()
#15 0x00007fffad591cbc in ?? ()
#16 0x00007fffad591cc7 in ?? ()
#17 0x0000000000000000 in ?? ()
Thanks,
./Siva.
Probably not because there are no debugging symbols. Not sure how
ubuntu packages these symbols..
Would recompiling the source packages and debugging would give
different results?
Any chance you could publish the repository that causes the crash?
--
Duy
I don't think I can publish the repo online. But I am willing to try
any steps on the repo to get more info. Most likely I would restore
the latest repo to the stanby server I tested which works with any
crash. Let me know what I can do to get more information.
./Siva.
Hi,
The git crashed during one of the commits by a developer I think, the
remote is not even showing the working branch. The local branch of is
all right, but the remote repo is corrupted and could not git fsck
also. Is restoring the last night's backup is my only option??
$ git remote show origin
* remote origin
Fetch URL: git@gitserver:sggroup/sgrid.git
Push URL: git@gitserver:sggroup/sgrid.git
HEAD branch: master
Remote branches:
SGRID_5_5_0_BRANCH tracked
master tracked
refs/remotes/origin/4_4_Release_Branch stale (use 'git remote prune' to remo
ve)
Local branch configured for 'git pull':
4_4_Release_Branch merges with remote 4_4_Release_Branch
Thanks,
./Siva.
On Thu, Apr 18, 2013 at 5:02 PM, Sivaram Kannan [off-list ref] wrote:
Hi,
quoted
Probably not because there are no debugging symbols. Not sure how
ubuntu packages these symbols..
Would recompiling the source packages and debugging would give
different results?
quoted
Any chance you could publish the repository that causes the crash?
--
Duy
I don't think I can publish the repo online. But I am willing to try
any steps on the repo to get more info. Most likely I would restore
the latest repo to the stanby server I tested which works with any
crash. Let me know what I can do to get more information.
./Siva.
Hi
After some struggle I finally got apport work and got some
information. Please check whether it will be of any help. And you guys
are right, I have started to get the same crash again right now after
upgrading to latest git in Ubuntu.
Yes. Please do. The reason is a backtrace without symbols gives very
little information (at least for people who do not use Ubuntu).
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>.
Reading symbols from /tmp/tmp6_sJPS/usr/lib/git-core/git...(no
debugging symbols found)...done.
[New LWP 31469]
[New LWP 31468]
[New LWP 31466]
[New LWP 31464]
[New LWP 31467]
warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
warning: the debug information found in
"/tmp/tmp6_sJPS/usr/lib/debug//lib/x86_64-linux-gnu/ld-2.15.so" does
not match "/tmp/tmp6_sJPS/lib64/ld-linux-x86-64.so.2" (CRC mismatch).
Core was generated by `git pack-objects --revs --all --stdout
--delta-base-offset'.
Program terminated with signal 11, Segmentation fault.
#0 0x00000000004820f0 in ?? ()
(gdb) bt
#0 0x00000000004820f0 in ?? ()
#1 0x000000000044b3aa in ?? ()
#2 0x000000000044b641 in ?? ()
#3 0x00007f077973de9a in start_thread (arg=0x7f0747fff700)
at pthread_create.c:308
#4 0x00007f077946acbd in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:104
#5 0x0000000000000000 in ?? ()
(gdb)
./Siva.