From: Junio C Hamano <hidden> Date: 2016-06-15 22:42:54
Bill Lear [off-list ref] writes:
I then did a commit, and something went wrong:
% git commit -a -m "Nuke CVS Id strings"
error: Could not read ab66b31e390889e6bcbb2002111e2803c51f42b5
error: unable to read tree object HEAD
# On branch master
error: Could not read ab66b31e390889e6bcbb2002111e2803c51f42b5
error: unable to read tree object HEAD
Does 1.4.4 find that object? What's in HEAD (cat .git/HEAD)?
What does "git fsck-objects --full" report?
From: Bill Lear <hidden> Date: 2016-06-15 22:42:54
On Wednesday, February 14, 2007 at 09:15:08 (-0800) Junio C Hamano writes:
Bill Lear [off-list ref] writes:
quoted
I then did a commit, and something went wrong:
% git commit -a -m "Nuke CVS Id strings"
error: Could not read ab66b31e390889e6bcbb2002111e2803c51f42b5
error: unable to read tree object HEAD
# On branch master
error: Could not read ab66b31e390889e6bcbb2002111e2803c51f42b5
error: unable to read tree object HEAD
Does 1.4.4 find that object? What's in HEAD (cat .git/HEAD)?
What does "git fsck-objects --full" report?
% cat .git/HEAD
ref: refs/heads/master
% git --version
git version 1.5.0-dirty
% git fsck-objects --full
error: Packfile .git/objects/pack/pack-23d1a9af78b4b78d1f3750cf70f83cb91a20ba64.pack SHA1 mismatch with itself
fatal: failed to find delta-pack base object 90bad0d280a6d7c155bbd9582b35ffcf5e3bdd27
% /usr/bin/git --version
git version 1.4.4.1
% /usr/bin/git fsck-objects --full
error: Packfile .git/objects/pack/pack-23d1a9af78b4b78d1f3750cf70f83cb91a20ba64.pack SHA1 mismatch with itself
fatal: failed to find delta-pack base object 90bad0d280a6d7c155bbd9582b35ffcf5e3bdd27
So, all I did was try to do a commit with the new git ... haven't
recloned, or pulled from upstream...
Bill
This in itself could have been due to a historical buglet that shouldn't
matter (SHA1 on pack-files got miscomputed). However, that's probably NOT
the problem, since:
fatal: failed to find delta-pack base object 90bad0d280a6d7c155bbd9582b35ffcf5e3bdd27
implies that the pack really is corrupt.
Even a single-bit error will corrupt a pack in bad ways, which is one
reason why we're so careful with it and add its own SHA1 to the end.
The best way to proceed:
- MAKE A BACKUP ("tar" up everything). If for no other reason than
(a) then you don't have to worry about making things worse even by
mistake
(b) it might be interesting for others (if you can make those
pack-files available) to try to figure out what exactly the
corruption was. We've done it before, when it turned out to be a
single-bit error.
- if you have other git archives or just back-ups of everything, just use
them, and throw the corrupt one away entirely (but see above on why
it's nice to have an archive of the corruption for posterity anyway)
- if you don't, you can try "git unpack-objects -r". See the man-page on
why you need to first _move_ the pack-file away:
mv <bad-pack-file> .git/bad-pack.pack
mv <bad-pack-index> .git/bad-index.index
git unpack-objects -r < .git/bad-pack.pack
this will unpack as many objects into loose format as it can. Hopefully
you haven't lost much.
- after that, the ones that you *did* lose, you can hopefully find in
older git repos: even if you didn't have the *full* new repo anywhere
else, other git repositories may have the particular objects that got
corrupted. "git fsck" will tell you what is missing, and you can just
point your .git/info/alternates file at other repositories to "steal"
objects from automatically.
- if you aren't missing any objects after that, you can now repack the
repository, and then remove the alternates file:
git repack -a -d
rm .git/info/alternates
because the repack will steal all the objects you need, and thus you
don't need alternates any more.
Finally: it would be very interesting to hear if you do something strange
or unusual that could have made your chances of getting corruption higher.
Have you ever seen random SIGSEV's or strange oopses, which could be a
sign of memory corruption on your machine? Do you do a lot of things over
NFS? (which really can corrupt things, especially in circumstances with
dodgy ethernet chips: the UDP checksums are very weak, and some ethernet
cards do not do a good job of checking the ethernet CRC's!).
So, all I did was try to do a commit with the new git ... haven't
recloned, or pulled from upstream...
Yes, don't do anything more (certainly do *not* repack or anything) until
you have tarred up and saved the current state, and then only _after_ you
have a good safe archive to restart from, try to fix it up.
And if you can make the git history available to outsiders, I'd love to
see the corrupt tar-file (it doesn't have to be *public*, if you just can
trust me and perhaps a few other people with the data).
So far, as far as I can recall, we've certainly had people screw up their
own trees by mistake, but apart from that kind of "user error" things, the
only real corruption I recall was the single-bit error in a pack-file. We
were able to recover that, but in general, for safety, the best way to
protect your data is to replicate it across multiple independent machines
(something that git is _good_ at, happily).
Linus
And if you can make the git history available to outsiders, I'd love to
see the corrupt tar-file (it doesn't have to be *public*, if you just can
trust me and perhaps a few other people with the data).
Side note: one reason why this is nice - even if you don't care about the
corruption and can fix it other ways - is that the last time we had the
one-bit corruption is also the reason why we now have the "-r" option to
git-unpack-objects.
In other words, real-life corruption is not just a really nasty event,
it's also a good way for *us* to verify that our recovery tools do as good
a job as they possibly can. Maybe there are other things like that "-r"
option where we could possibly do even better.
The git data structures are designed to be extremely robust, but there's
nothing they can do about "corruption after the fact". The same way that a
logging filesystem doesn't help if the disk itself starts getting read
errors, the git data structures aren't going to guarantee that you can't
lose data if you have actual disk or memory corruption going on.
The things git can do is:
- detection. The SHA1's should basically guarantee that you will never
ever have an _undetectable_ corruption anywhere (which is really really
easy with just about any other SCM)
- make replication easy (so that once you've detected corruption, you
have mirrors you can trust).
- and finally: in the absense of replication, we can do our damndest to
try to figure out what the data was. But in many ways, the fact that we
are really really good at compressing data (people do love their small
repositories) also means that we have basically no redundancy anywhere,
because redundancy is what compression gets rid of (both delta- and
zlib compression do it - it's very fundamentally what any compression
is based on)
but it's always interesting to have real-life corruption cases to verify.
Linus
From: Bill Lear <hidden> Date: 2016-06-15 22:42:54
On Wednesday, February 14, 2007 at 10:19:53 (-0800) Linus Torvalds writes:
On Wed, 14 Feb 2007, Bill Lear wrote:
quoted
fatal: failed to find delta-pack base object 90bad0d280a6d7c155bbd9582b35ffcf5e3bdd27
implies that the pack really is corrupt.
...
(b) it might be interesting for others (if you can make those
pack-files available) to try to figure out what exactly the
corruption was. We've done it before, when it turned out to be a
single-bit error.
If you could tell me who I should contact about this, I will.
- if you have other git archives or just back-ups of everything, just use
them, and throw the corrupt one away entirely (but see above on why
it's nice to have an archive of the corruption for posterity anyway)
I would prefer to help straighten this out --- I have a company repo
to fall back on, I have git 1.4 and other repos to fall back on, so
I'm safe.
- if you don't, you can try "git unpack-objects -r". See the man-page on
why you need to first _move_ the pack-file away:
mv <bad-pack-file> .git/bad-pack.pack
mv <bad-pack-index> .git/bad-index.index
git unpack-objects -r < .git/bad-pack.pack
Since I can reproduce the error fairly readily, I can do this later.
Finally: it would be very interesting to hear if you do something strange
or unusual that could have made your chances of getting corruption higher.
Have you ever seen random SIGSEV's or strange oopses, which could be a
sign of memory corruption on your machine? Do you do a lot of things over
NFS? (which really can corrupt things, especially in circumstances with
dodgy ethernet chips: the UDP checksums are very weak, and some ethernet
cards do not do a good job of checking the ethernet CRC's!).
No NFS, but I checked /var/log/messages. I see segfaults from git,
that I missed somehow (don't remember seeing anything awry on the
terminal):
Feb 14 10:05:07 lisa kernel: git[21648]: segfault at 0000000000000000 rip 0000003f5eb709d0 rsp 0000007fbfffc158 error 4
Feb 14 10:05:43 lisa kernel: git[21710]: segfault at 0000000000000000 rip 0000003f5eb709d0 rsp 0000007fbfffc158 error 4
Feb 14 10:06:28 lisa kernel: git[21858]: segfault at 0000000000000000 rip 0000003f5eb709d0 rsp 0000007fbfffc158 error 4
Feb 14 10:10:04 lisa kernel: git[22385]: segfault at 0000000000000000 rip 0000003f5eb709d0 rsp 0000007fbfffc158 error 4
Feb 14 11:01:56 lisa kernel: git[24446]: segfault at 0000000000000000 rip 0000003f5eb709d0 rsp 0000007fbfffc888 error 4
Feb 14 11:02:34 lisa kernel: git[24479]: segfault at 0000000000000000 rip 0000003f5eb70a40 rsp 0000007fbfffc868 error 4
Feb 14 11:02:40 lisa kernel: git[24700]: segfault at 0000000000000000 rip 0000003f5eb709d0 rsp 0000007fbfffc868 error 4
Feb 14 11:07:51 lisa kernel: git[24844]: segfault at 0000000000000000 rip 0000003f5eb709d0 rsp 0000007fbfffc128 error 4
Feb 14 11:07:52 lisa kernel: git[24855]: segfault at 0000000000000000 rip 0000003f5eb709d0 rsp 0000007fbfffc128 error 4
Feb 14 11:08:01 lisa kernel: git[24886]: segfault at 0000000000000000 rip 0000003f5eb709d0 rsp 0000007fbfffc128 error 4
Feb 14 11:08:06 lisa kernel: git[24897]: segfault at 0000000000000000 rip 0000003f5eb709d0 rsp 0000007fbfffc118 error 4
Feb 14 11:08:09 lisa kernel: git[24908]: segfault at 0000000000000000 rip 0000003f5eb709d0 rsp 0000007fbfffc118 error 4
Feb 14 11:08:27 lisa kernel: git[24939]: segfault at 0000000000000000 rip 0000003f5eb709d0 rsp 0000007fbfffc158 error 4
10:05 is just before I posted my first note of this to the git list, and
the first instance of a segfault that I see.
And if you can make the git history available to outsiders, I'd love to
see the corrupt tar-file (it doesn't have to be *public*, if you just can
trust me and perhaps a few other people with the data).
Again, please let me know who to contact about helping on this.
Bill
No NFS, but I checked /var/log/messages. I see segfaults from git,
that I missed somehow (don't remember seeing anything awry on the
terminal):
Feb 14 10:05:07 lisa kernel: git[21648]: segfault at 0000000000000000 rip 0000003f5eb709d0 rsp 0000007fbfffc158 error 4
Feb 14 10:05:43 lisa kernel: git[21710]: segfault at 0000000000000000 rip 0000003f5eb709d0 rsp 0000007fbfffc158 error 4
Feb 14 10:06:28 lisa kernel: git[21858]: segfault at 0000000000000000 rip 0000003f5eb709d0 rsp 0000007fbfffc158 error 4
10:05 is just before I posted my first note of this to the git list, and
the first instance of a segfault that I see.
Ok, this is almost certainly what's up. For some strange reason your git
binary segfaults on the clone. The scary thing is, it left your cloned
repo in a bad state without even telling you. That's not good. Normally
we should always die() and give a _reason_ for a failure.
If you have that particular git binary, doing a
gdb git
and then at the gdb prompt doing
x/5i 0x0000003f5eb709d0
will at least tell where the SIGSEGV happened, but it doesn't give a
backtrace so unless it's obvious, it can be a bit hard to debug remotely..
Linus