fatal: serious inflate inconsistency

Hello,
and here is another one I guess: ;-(

        (tomcat-05) [/usr/src] git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
        Initialized empty Git repository in /usr/src/linux-2.6/.git/
        remote: Generating pack...
        remote: Done counting 496911 objects.
        remote: Deltifying 496911 objects...
        remote:  100% (496911/496911) done
        Indexing 496911 objects...
        remote: Total 496911 (delta 401539), reused 493712 (delta 398653)
        100% (496911/496911) done
        Resolving 401539 deltas...
        fatal: serious inflate inconsistency
        fatal: index-pack died with error code 128
        fetch-pack from
        'git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git' failed.
        (tomcat-05) [/usr/src] git version
        git version 1.5.2.1

Greetings,
        Thomas

On Fri, 8 Jun 2007, Thomas Glanzmann wrote:

Hello,
and here is another one I guess: ;-(

        (tomcat-05) [/usr/src] git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
        Initialized empty Git repository in /usr/src/linux-2.6/.git/
        remote: Generating pack...
        remote: Done counting 496911 objects.
        remote: Deltifying 496911 objects...
        remote:  100% (496911/496911) done
        Indexing 496911 objects...
        remote: Total 496911 (delta 401539), reused 493712 (delta 398653)
        100% (496911/496911) done
        Resolving 401539 deltas...
        fatal: serious inflate inconsistency
        fatal: index-pack died with error code 128
        fetch-pack from
        'git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git' failed.
        (tomcat-05) [/usr/src] git version
        git version 1.5.2.1

This is bad.

index-pack works like this:

1) it receives data over the network:
   1a) it writes a copy of that data to disk to create a pack
   1b) it deflates that data in parallel to compute its SHA1

2) when done, it resolves deltas and to do so:
   2a) it deflates the base object data from 1a
   2b) it applies the delta over the base object data then compute its 
       SHA1

In your case it seems that 1B and 2A don't produce the same deflated 
data somehow.

Is it reproducible?
Are you using Linux?  What kernel version?
Are you using NFS?

If it isn't reproducible then you might possibly have been the victim of 
faulty hardware.


Nicolas

Hello,

Is it reproducible?

I can try. But that produces 150 Mbyte Traffic on Kernel.org

Are you using Linux?  What kernel version?

Yes, I do. Debian etch; 2.6.20

Are you using NFS?

Nope.

If it isn't reproducible then you might possibly have been the victim
of faulty hardware.

I give it a try and come back to you. I am off the net for 1 hour.

        Thomas

Hello,

I give it a try and come back to you. I am off the net for 1 hour.

        (tomcat-05) [/usr/src] git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
        Initialized empty Git repository in /usr/src/linux-2.6/.git/
        remote: Generating pack...
        remote: Done counting 496911 objects.
        remote: Deltifying 496911 objects...
        remote:  100% (496911/496911) done
        Indexing 496911 objects...
        remote: Total 496911 (delta 401539), reused 493712 (delta 398653)
        100% (496911/496911) done
        Resolving 401539 deltas...
        fatal: serious inflate inconsistency
        fatal: index-pack died with error code 128
        fetch-pack from 'git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git' failed.

yes, it is reproducable. Again I am off the net for an hour.

        Thomas

Hello again,

yes, it is reproducable. Again I am off the net for an hour.

I used the debian packages that built a friend of mine:

        deb http://rmdir.de/~michael/git/ ./

If you want to reproduce it by yourself.

        Thomas

On Fri, 8 Jun 2007, Thomas Glanzmann wrote:

Hello,

quoted

I give it a try and come back to you. I am off the net for 1 hour.

        (tomcat-05) [/usr/src] git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
        Initialized empty Git repository in /usr/src/linux-2.6/.git/
        remote: Generating pack...
        remote: Done counting 496911 objects.
        remote: Deltifying 496911 objects...
        remote:  100% (496911/496911) done
        Indexing 496911 objects...
        remote: Total 496911 (delta 401539), reused 493712 (delta 398653)
        100% (496911/496911) done
        Resolving 401539 deltas...
        fatal: serious inflate inconsistency
        fatal: index-pack died with error code 128
        fetch-pack from 'git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git' failed.

yes, it is reproducable. Again I am off the net for an hour.

I can't reproduce that.

$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
Initialized empty Git repository in /home/nico/git/tst/linux-2.6/.git/
remote: Generating pack...
remote: Done counting 496911 objects.
remote: Deltifying 496911 objects...
remote:  100% (496911/496911) done
Indexing 496911 objects...
remote: Total 496911 (delta 401539), reused 493712 (delta 398653)
 100% (496911/496911) done
Resolving 401539 deltas...
 100% (401539/401539) done
Checking 22413 files out...
 100% (22413/22413) done
$ git version
git version 1.5.2.1

I used the debian packages that built a friend of mine:

        deb http://rmdir.de/~michael/git/ ./

If you want to reproduce it by yourself.

I don't use Debian, sorry.


Nicolas

On Fri, 8 Jun 2007, Thomas Glanzmann wrote:

yes, it is reproducable. Again I am off the net for an hour.

Could you compile your own git? If so, please try this patch on top of 
current git, to make the error message a bit more informative.

It does sound like a corrupt stream, but maybe debian has some strange 
zlib version that git doesn't like.

		Linus
---
 index-pack.c |    8 ++++++--
 1 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/index-pack.c b/index-pack.c
index 82c8da3..542687b 100644
--- a/index-pack.c
+++ b/index-pack.c

@@ -277,9 +277,13 @@ static void *get_data_from_pack(struct object_entry *obj)
 	stream.avail_in = len;
 	inflateInit(&stream);
 	while ((st = inflate(&stream, Z_FINISH)) == Z_OK);
+	if (st != Z_STREAM_END)
+		die("serious inflate inconsistency: %d (%s)",
+			st, stream.msg ? stream.msg : "no message");
+	if (stream.total_out != obj->size)
+		die("serious inflate inconsistency: got %lu bytes, expected %lu",
+			stream.total_out, obj->size);
 	inflateEnd(&stream);
-	if (st != Z_STREAM_END || stream.total_out != obj->size)
-		die("serious inflate inconsistency");
 	free(src);
 	return data;
 }

Hello Linus,

Could you compile your own git? If so, please try this patch on top of 
current git, to make the error message a bit more informative.

        (tomcat-05) [/usr/src] scp 131.188.30.102:/var/tmp/sithglan/git-core-1.5.2.1/../git-core_1.5.2.1-1_i386.deb .
        Warning: Permanently added '131.188.30.102' (RSA) to the list of known hosts.
        git-core_1.5.2.1-1_i386.deb                                                    100% 3511KB   1.7MB/s   00:02
        (tomcat-05) [/usr/src] / dpkg -i git-core_1.5.2.1-1_i386.deb
        (Reading database ... 13273 files and directories currently installed.)
        Preparing to replace git-core 1:1.5.2.1-1 (using git-core_1.5.2.1-1_i386.deb) ...
        Unpacking replacement git-core ...
        Setting up git-core (1.5.2.1-1) ...
        (tomcat-05) [/usr/src] git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
        Initialized empty Git repository in /usr/src/linux-2.6/.git/
        remote: Generating pack...
        remote: Done counting 496911 objects.
        remote: Deltifying 496911 objects...
        remote:  100% (496911/496911) done
        Indexing 496911 objects...
        remote: Total 496911 (delta 401539), reused 493712 (delta 398653)
        100% (496911/496911) done
        Resolving 401539 deltas...
        fatal: serious inflate inconsistency: -3 (invalid distance too far back)
        fatal: index-pack died with error code 128
        fetch-pack from 'git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git' failed.

I talked to Michael and he is certain that the machine has a memory
corruption the next thing I am going to do is compiling a kernel in an
endless loop. Again if somone needs shell access just send me your ssh
key.

The complete build log is on (well not the complete log the first line is
missing because my screen backlog has _only_ 10000 lines (a memory issue)):

The first line is:
wget http://ftp.debian.org/debian/pool/main/g/git-core/git-core_1.5.2.1-1.dsc

The rest is here:

http://wwwcip.informatik.uni-erlangen.de/~sithglan/screen-exchange.bz2 (32k)

Thanks,
        Thomas

On Fri, 8 Jun 2007, Thomas Glanzmann wrote:

        fatal: serious inflate inconsistency: -3 (invalid distance too far back)

Whee. That's Z_DATA_ERROR, and yeah, I think the only thing that causes it 
is a corrupt zlib input stream.

I talked to Michael and he is certain that the machine has a memory
corruption the next thing I am going to do is compiling a kernel in an
endless loop. Again if somone needs shell access just send me your ssh
key.

I'd not expect memory corruption to be that *repeatable*. But the message 
certainly implies zlib data stream corruption _somewhere_, although the 
fact that it seems to be so repeatable does make me suspect program error.

Of course, it could be a hard bit-error in memory, but even then it would 
kind of have to hit the same page allocation history each time to be 
repeatable. And the page cache is the only thing that is that sticky under 
Linux, so it would have to be something like the zlib static data or code 
that had hit the memory corruption.

Sounds strange. I wonder if we might have finally hit a case of different 
versions of zlib acting differently?

But it _could_ just be a git bug too.

I don't think it's on the kernel.org side (or we would have had more 
reports of this), but maybe there is some wild pointer thing inside git, 
and nobody else noticed because it depends on some specific memory 
allocation pattern (which in turn depends on things like kernel memory 
layout choices, and on libc 'malloc()' implementation etc).

I'll build with ElectricFence, and see if it says anything for me. It 
passes all the tests, but fetching the kernel is almost certainly going to 
do things that we don't test for..

			Linus

On Fri, 8 Jun 2007, Linus Torvalds wrote:

I'll build with ElectricFence, and see if it says anything for me. It 
passes all the tests, but fetching the kernel is almost certainly going to 
do things that we don't test for..

It passes smaller clones fine with ElectricFence (and bigger ones fail 
because EF uses *so* much memory for its checking that you cannot actually 
complete them), so I don't think this is a git bug. Sounds like either the 
hw problems are real, or there is some zlib (or other library) issue..

		Linus

Hello Linus,

It passes smaller clones fine with ElectricFence (and bigger ones fail 
because EF uses *so* much memory for its checking that you cannot actually 
complete them), so I don't think this is a git bug. Sounds like either the 
hw problems are real, or there is some zlib (or other library) issue..

        (mailrelay) [/var/tmp] git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
        Initialized empty Git repository in /var/tmp/linux-2.6/.git/
        remote: Generating pack...
        remote: Done counting 496911 objects.
        remote: Deltifying 496911 objects...
        remote:  100% (496911/496911) done
        Indexing 496911 objects...
        remote: Total 496911 (delta 401539), reused 493712 (delta 398653)
        100% (496911/496911) done
        Resolving 401539 deltas...
        100% (401539/401539) done
        Checking 22413 files out...
        100% (22413/22413) done

I have a very similar machine. For this one it works. It has 32 bit kernel and
userland. Debian etch again. v2.6.20.

Does someone know a command for debian that checks the checksums for every
installed packages and the file that are included in that package?
I know there is one for rpm so there must be one for debian, too.

        Thomas

On Friday June 8th 2007 at 21:33 Thomas Glanzmann wrote:

Does someone know a command for debian that checks the checksums for every
installed packages and the file that are included in that package?

Try debsums. Without arguments it checks all installed packages and
their files, but you can also check individual packages.
-- 
Marco Roeland

Hello,
the OS is not corrupt. So I am going to compile a few kernels and wait
for gcc to segfault.

        Thomas

Hello again,
the machine that had the issues just got down because of a broken power
supply and took half the rack with it. So it was definitive a hardware
fault. Sorry for all the trouble. And the bottom line is: It was _not_
git, it was broken hardware!

        Thomas

On Sat, 9 Jun 2007, Thomas Glanzmann wrote:

the machine that had the issues just got down because of a broken power
supply and took half the rack with it. So it was definitive a hardware
fault. Sorry for all the trouble. And the bottom line is: It was _not_
git, it was broken hardware!

Good. That was one of the design goals for git (ie the "you know you can 
trust the data" thing relies on very strong protection at all levels, 
even in the presense of disk/memory/cpu corruption).

I think we have a pretty good record on this: the only real git-induced 
corruption I can remember was the embarrassing bug with use-after-free of 
the CRLF auto-translation. But let's not get complacent.

		Linus

On 6/9/07, Thomas Glanzmann [off-list ref] wrote:

I have a very similar machine. For this one it works. It has 32 bit kernel and
userland. Debian etch again. v2.6.20.

Wanted to mention -- we've been using git on debian sarge/etch on
several arches a lot and never seen any lib compatiblity problem.

Does someone know a command for debian that checks the checksums

apt-get install debsums
man debsums # sorry, I don't use it often enough to remember ;-)

cheers,


m

Hello,
the machine came back with a new power supply. And guess what:

(tomcat-05) [/usr/src] git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
Initialized empty Git repository in /usr/src/linux-2.6/.git/
remote: Generating pack...
remote: Done counting 499634 objects.
remote: Deltifying 499634 objects...
remote:  100% (499634/499634) done
Indexing 499634 objects...
remote: Total 499634 (delta 403845), reused 495761 (delta 400337)
100% (499634/499634) done
Resolving 403845 deltas...
100% (403845/403845) done
Checking 22411 files out...
100% (22411/22411) done

        Thomas