Re: My git repo is broken, how to fix it ?

[ Sorry, I got sidetracked, have been looking at kernel bugs and git 
  optimizations. I added back the git mailing list, since there is no 
  private file data here any more, and I'd like others to follow this
  saga too ]

On Wed, 21 Mar 2007, Alexander Litvinov wrote:

quoted

Oh, btw, before I do that - do you by any chance have the *uncorrupt*
version of the file that should be this object? In other words, do you
have object 03312463e194d68d0d677b51e09b47cb29ca926a in another
repository? It should be a version of your file.

It is pity, but I don't have that version. It was broken at my last commit and 
I will redo it again. I remeber the changes I have done.

The new file has different sha1 sum but I attach it to show you the real file 
content.

Sadly, I actually would need to compare it to the exact object it *should* 
have generated, and that means that I'm not actually all that interested 
in the "real file content" per se, I really would need to see the exact 
blob, so that I can generate the object it should have been, and then 
compare that against the corrupt one... 

It's the binary data I'd like to compare, so that I can tell (for example) 
if there is just a chunk missing in the middle, or something like that. 
But since even slight differences in the source data will lead to 
different binary data, and since the compressed and corrupt data I have 
from you earlier doesn't make sense on its own, I do care about the exact 
object that got corrupted.

By the way, I now I am using git (3ba7a10) taken from next with tree your 
patches:
1. disables the hardlinking code for CYGWIN, and it also checks for errors 
from "close()"
2. Don't ever return corrupt objects from "parse_object()"
3. Be more careful about zlib return values.

Ok, apart from #1, those should be in current -git now, along with better 
validation checks (by Nico) when packing. So hopefully at least when there 
is corruption in a loose object, we will now always notice when we do a 
"git repack", and will never generate a broken pack-file. Knock wood.

Of course, I actually wonder if the bug might be in your version of zlib 
(miscompiled or some other thing), in which case *any* amount of 
pre-validation won't really help, because it will become corrupted when we 
deflate it prior to writing. For example, if "deflateBound()" sometimes 
doesn't give a valid upper bound and we allocate too little space..

Yesterday brakage was made by git with only first patch.

I see that you seem to be able to reproduce it again - I'll answer that 
email separately just so that the git list sees that message too. But it 
boils down to: if it's reproducible, I'd *really* like to see the 
corrupted object and the exact file that it should have been generated 
from..

		Linus

On Thu, 22 Mar 2007, Linus Torvalds wrote:

Ok, apart from #1, those should be in current -git now, along with better 
validation checks (by Nico) when packing. So hopefully at least when there 
is corruption in a loose object, we will now always notice when we do a 
"git repack", and will never generate a broken pack-file. Knock wood.

Not yet actually.  What I did do is to make index-pack perform more 
validation and ensure it never accept SHA1 collisions.

For the repack case... I think there should be a better way.  Either we 
revalidate the full SHA1 which would be expensive as we'd basically lose 
most advantages of direct pack data copy.

What I'm pondering is some sort of lightweight checksum like adler32 for 
object data in the pack but stored in the index.  Since index-pack 
already perform the full SHA1 already, it could as well provide a 
checksum for the raw pack object data for the repack case.  Currently we 
try to validate reused pack data by attempting an inflate pass on the 
object payload, but that doesn't validate the object type nor the 
reference SHA1 to delta base objects which could get corrupted and 
copied without noticing into another pack.

Of course, I actually wonder if the bug might be in your version of zlib 
(miscompiled or some other thing), in which case *any* amount of 
pre-validation won't really help, because it will become corrupted when we 
deflate it prior to writing. For example, if "deflateBound()" sometimes 
doesn't give a valid upper bound and we allocate too little space..

Well, since we provide the size of the allocated output buffer to zlib 
it would be seriously broken if it overflowed it.  Also zlib perform a 
checksum verification of the deflated data if I remember correctly.  So 
it seems to me that zlib should be quite self validating already.


Nicolas