git fsck not identifying corrupted packs

Hi,

I have a pack that contains a corrupted object.
It is an old corrupted repo that I have conserved.

As expected, git gc cries out loud about it.
It indicates an inflate error (data stream error with incorrect data check),
and then the impossibility to read an object from a certain offset in the pack.

However, git fsck does not complain at all about the repo.
I guess that for speed reasons, git fsck does not try to inflate the objects.

Is there a means to have fsck to a truly full check on the sanity of a repo?

This both on git 1.6.5.1 and 1.6.4.2.

Thanks

Sergio Callegari

Sergio Callegari schrieb:

Is there a means to have fsck to a truly full check on the sanity of a repo?

git fsck --full

RTFM, please.

-- Hannes

Hi,

On Mon, 19 Oct 2009, Johannes Sixt wrote:

Sergio Callegari schrieb:

quoted

Is there a means to have fsck to a truly full check on the sanity of a 
repo?

git fsck --full

RTFM, please.

Now, now.

If you were to test a new filesystem, say, wonderfulfs, and wanted to 
check its integrity, would you not just run "fsck-wonderfulfs" if that 
exists, rather than reading the fantamagastic manual?  Would you not 
expect that it Does The Right Thing?  Would you not expect that it 
follows the Law Of Minimal Surprise?

So FWIW I can see where Sergio is coming from.

Ciao,
Dscho

Johannes Sixt wrote:

Sergio Callegari schrieb:
  

quoted

Is there a means to have fsck to a truly full check on the sanity of a repo?
    

git fsck --full

RTFM, please.

  

Right... sorry for the noise, I mismatched --strict for --full in a script.

BTW, the short help for fsck at --full only says "consider objects in 
alternate repositories".

My apologize.

Sergio

On Mon, Oct 19, 2009 at 11:11:33AM +0200, Johannes Sixt wrote:

quoted

Is there a means to have fsck to a truly full check on the sanity of a repo?

git fsck --full

RTFM, please.

That still does not catch everything. About a week ago I wanted to check
out a branch in a local repo and I got an error that it was corrupt. But
"git fsck --full" only complained about some dangling objects, it did
not notice the corruption. I used git version 1.6.4.3 (Debian unstable
at that time). It would be nice to have a
"git fsck --i-really-want-to-check-everything".

Gabor

-- 
     ---------------------------------------------------------
     MTA SZTAKI Computer and Automation Research Institute
                Hungarian Academy of Sciences
     ---------------------------------------------------------

Johannes Schindelin [off-list ref] writes:

On Mon, 19 Oct 2009, Johannes Sixt wrote:

quoted

Sergio Callegari schrieb:

quoted

Is there a means to have fsck to a truly full check on the sanity of a 
repo?

git fsck --full

RTFM, please.

Now, now.

If you were to test a new filesystem, say, wonderfulfs, and wanted to 
check its integrity, would you not just run "fsck-wonderfulfs" if that 
exists, rather than reading the fantamagastic manual?  Would you not 
expect that it Does The Right Thing?  Would you not expect that it 
follows the Law Of Minimal Surprise?

So FWIW I can see where Sergio is coming from.

Linus and other git developers from the early days trained their fingers
to type the command, every once in a while even without thinking, to check
the consistency of the repository back when the lower core part of the git
was still being developed.  Developers who wanted to make sure that git
correctly dealt with packfiles could deliberately trigger their creation
and checked them after they were created carefully, but loose objects are
the ones that are written by various commands from random codepaths.  It
made some technical sense to have a mode that checked only loose objects
from the debugging point of view for that reason.

    Side note.  I think the help description of --full option is wrong (or
    at least stale).  We always look at alternate object store these days
    since e15ef66 (fsck: check loose objects from alternate object stores
    by default, 2009-01-30).  It probably should read "check packed
    objects fully" or something.

The above paragraph is merely a historical background, and in this case
the "history" refers to early-to-mid 2005.  Even for git developers there
no longer is any reason to type "git fsck" in fear of some newly created
objects might be corrupt due to recent change to git these days.

The reason we did not make "--full" the default is probably we trust our
filesystems a bit too much.  At least, we trusted filesystems more than we
trusted the lower core part of git that was under development ;-)

Once a packfile is created and we always use it read-only, there didn't
seem to be much point in suspecting that the underlying filesystems or
disks may corrupt them in such a way that is not caught by the SHA-1
checksum over the entire packfile and per object checksum.  That trust in
the filesystems might have been a good tradeoff between fsck performance
and reliability on platforms git was initially developed on and for, but
it might not be true anymore as we run on more platforms these days.

It probably makes sense to ship 1.7.0 with a version of "fsck" in which
"--full" is the default; it would still accept "--full" but it would be a
no-op.  This would be a backward incompatible change, but the difference
is primarily about performance ("it takes a lot longer than before!"), and
not correctness, so we probably can live with it.  As I already said,
there is not much reason to run "fsck" every five minutes anymore to begin
with (unless your filesystem is so unreliable that it might eat one file
every five minutes, that is).

It probably is also a good idea to add a "--loose" option that does what
"fsck" currently does without "--full".  It is a good name because (1) to
people who do not know the internal of git, it means "check only loosely",
which would discourage them from running "fack" with that option to begin
with, and (2) to others, it exactly tells what the option makes the
command check.

On Monday 19 October 2009 04:56:07 Sergio Callegari wrote:

Johannes Sixt wrote:

quoted

Sergio Callegari schrieb:

quoted

Is there a means to have fsck to a truly full check on the sanity of a
repo?

git fsck --full

RTFM, please.

Right... sorry for the noise, I mismatched --strict for --full in a
script.

BTW, the short help for fsck at --full only says "consider objects in
alternate repositories".

Until I read this thread, I didn't realize you needed --full to check
objects in packs.

Since just every git repository I ever use has 99%+ of it's objects in
packs, this means every time I've run "git fsck" it's essentially been
a no-op and I didn't know it. I imagine this is a common confusion.

Also, having --full mean both "check alternate object pools", and "check
objects in packs" seems to be rolling up two orthogonal issues.

But anyway, here is a patch that at least fixes the short option help to
match the manual and the current behavior:

--- 8< ---

From 8fc3cd68d496bf00faad4f0a7b6ae4fee9437e68 Mon Sep 17 00:00:00 2001
From: Wesley J. Landaker <redacted>
Date: Mon, 19 Oct 2009 12:48:07 -0600
Subject: [PATCH] Update git fsck --full short description to mention packs

The '--full' option to git fsck does two things:

  1) Check objects in packs
  2) Check alternate objects

This is documented in the git fsck manual; this patch reflects that in
the short git fsck option help message as well.

Signed-off-by: Wesley J. Landaker <redacted>
---
 builtin-fsck.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/builtin-fsck.c b/builtin-fsck.c
index c58b0e3..63212ea 100644
--- a/builtin-fsck.c
+++ b/builtin-fsck.c

@@ -576,7 +576,7 @@ static struct option fsck_opts[] = {
 	OPT_BOOLEAN(0, "root", &show_root, "report root nodes"),
 	OPT_BOOLEAN(0, "cache", &keep_cache_objects, "make index objects head nodes"),
 	OPT_BOOLEAN(0, "reflogs", &include_reflogs, "make reflogs head nodes (default)"),
-	OPT_BOOLEAN(0, "full", &check_full, "also consider alternate objects"),
+	OPT_BOOLEAN(0, "full", &check_full, "also consider packs and alternate objects"),
 	OPT_BOOLEAN(0, "strict", &check_strict, "enable more strict checking"),
 	OPT_BOOLEAN(0, "lost-found", &write_lost_and_found,
 				"write dangling objects in .git/lost-found"),

-- 
1.6.5

(Not CCing everyone, since this is mostly curiosa in the "using git as it 
was never intended" section):

On Monday 19 October 2009 13:03:42 Junio C Hamano wrote:

Once a packfile is created and we always use it read-only, there didn't
seem to be much point in suspecting that the underlying filesystems or
disks may corrupt them in such a way that is not caught by the SHA-1
checksum over the entire packfile and per object checksum.  That trust in
the filesystems might have been a good tradeoff between fsck performance
and reliability on platforms git was initially developed on and for, but
it might not be true anymore as we run on more platforms these days.

Filesystems are mostly reliable, but only until your crazy users do strange 
and terrible things. I have a real, non-toy environment where I use this 
stack as a [horrible] workaround for some issues beyond my control:

git -> ext4 -> lvm -> dmcrypt -> loop -> sshfs -> cygwin sshd -> SMB share

Amazingly, this works pretty reliably with many gigabytes of data in a git 
repository, even with the occasional crash because of flakiness with the 
"sshfs -> cygwin sshd" piece of the puzzle. But a good "git fsck" sure 
doesn't hurt in this environment! =)

måndag 19 oktober 2009 21:27:48 skrev  Wesley J. Landaker:

(Not CCing everyone, since this is mostly curiosa in the "using git as it
was never intended" section):

On Monday 19 October 2009 13:03:42 Junio C Hamano wrote:

quoted

Once a packfile is created and we always use it read-only, there didn't
seem to be much point in suspecting that the underlying filesystems or
disks may corrupt them in such a way that is not caught by the SHA-1
checksum over the entire packfile and per object checksum.  That trust in
the filesystems might have been a good tradeoff between fsck performance
and reliability on platforms git was initially developed on and for, but
it might not be true anymore as we run on more platforms these days.

Filesystems are mostly reliable, but only until your crazy users do strange
and terrible things. I have a real, non-toy environment where I use this
stack as a [horrible] workaround for some issues beyond my control:

git -> ext4 -> lvm -> dmcrypt -> loop -> sshfs -> cygwin sshd -> SMB share

The obvious follow up question here is: Why?

-- robin

On Tuesday 20 October 2009 09:41:50 Robin Rosenberg wrote:

måndag 19 oktober 2009 21:27:48 skrev  Wesley J. Landaker:

quoted

(Not CCing everyone, since this is mostly curiosa in the "using git as
it was never intended" section):

[...]

quoted

Filesystems are mostly reliable, but only until your crazy users do
strange and terrible things. I have a real, non-toy environment where I
use this stack as a [horrible] workaround for some issues beyond my
control:

git -> ext4 -> lvm -> dmcrypt -> loop -> sshfs -> cygwin sshd -> SMB
share

My main point was to illustrate that having "git fsck" do a REALLY GOOD 
CHECK is still desirable, as we still haven't reached the days of file-
system utopia where nothing ever gets corrupted (even with a smaller, 
simpler stack).

The actual application where I use this stack is because of odd requirements 
and circumstances like data must be physically stored on a particular 
Windows server on the network that uses a weird authentication method that 
samba doesn't support, and it has to go over the network encrypted anyway, 
there are lots of holes in the data, so I want ext4 for the extent support, 
file-size limitations on the target, etc.

It's a really an exotic love-hate mix between an off-by-one-please-no-never-
again kind of situation coupled with a bit of "because I can".

The obvious follow up question here is: Why?

If you are both nerdy and morbidly curious enough to care, send me a "but, 
no ... really, WHY?!" with the git list CC dropped and we can talk about 
details and/or other crazy stuff. (I don't want to get wildly off-topic on 
this list.)