Debugging strange "corrupt pack" errors on SuSE 9

A colleage at work is using git to manage code updates to a heavily
firewalled machine at a client site. In a nutshell, we "push" the
interesting code to a repo on usb-stick, and we pull from it on the
client's machine.

We do some coding and testing on that machine, so ocassionally we
bring some patches back.

Now the working repo on the client machine has started to die with
"corrupt pack" errors. I am trying to get my hands on the literal
error messages, and exact software versions installed. Right now all I
know is that it is SuSE 9 x86, git 1.4.x, cogito .17.x . The error
appears on git-diff, git-fsck-objects --full

We did bring a copy of the working copy with the "corrupt pack" to our
office, and here git (v1.5.1 and 1.5.2) thinks it's perfectly well.

So I am a bit puzzled - while we try to get 1.5.x on the client
machine and see what happens, is there anything that could be causing
this? Any additional tests that we should run?

cheers,


martin

On Wed, 20 Jun 2007, Martin Langhoff wrote:

A colleage at work is using git to manage code updates to a heavily
firewalled machine at a client site. In a nutshell, we "push" the
interesting code to a repo on usb-stick, and we pull from it on the
client's machine.

We do some coding and testing on that machine, so ocassionally we
bring some patches back.

Now the working repo on the client machine has started to die with
"corrupt pack" errors. I am trying to get my hands on the literal
error messages, and exact software versions installed. Right now all I
know is that it is SuSE 9 x86, git 1.4.x, cogito .17.x . The error
appears on git-diff, git-fsck-objects --full

The full exact error message would be highly useful indeed.

We did bring a copy of the working copy with the "corrupt pack" to our
office, and here git (v1.5.1 and 1.5.2) thinks it's perfectly well.

So I am a bit puzzled - while we try to get 1.5.x on the client
machine and see what happens, is there anything that could be causing
this? Any additional tests that we should run?

Maybe the client machine runs git version < 1.4.2.2, in which case it is 
possible that your push created a pack containing delta objects with 
offset to base which git versions prior 1.4.2.2 do not understand.

If this is the problem you are facing (the error message should confirm 
this) then the easiest solution is to upgrade git on the client.

A quick fix for the client is to set repack.usedeltabaseoffset to 
false on the machine where you have git 1.5 installed, then run "git 
repack -a -d", and finally copy the pack over to the client repository.  
But because you push to a local repository (a mounted USB stick is 
considered a local repo) then you don't get to negociate the pack 
capabilities of the final destination, and therefore more "bad" delta 
objects might sneak in again.


Nicolas

On 6/20/07, Nicolas Pitre [off-list ref] wrote:

The full exact error message would be highly useful indeed.

Yes. I haven't seen it first hand either, and the machine is closely
guarded :-/ but I think you are on to something: it's a v1.4.0.

# git --version

git version 1.4.0

# cat /etc/issue
Welcome to SUSE LINUX Enterprise Server 9 (i586) - Kernel \r (\l).
# uname -a
Linux lhostname 2.6.5-7.244-bigsmp-100hz #3 SMP Sun Aug 13 21:50:35 NZST
2006 i686 i686 i386 GNU/Linux

The errors look like

# git-pull /home/jun/moodle-r2.git
ird-mk2:fromusb
fatal: git-unpack-objects exec failed
fatal: git-unpack-objects died with error code 128
Fetch failure: /home/jun/moodle-r2.git

# cg-update fromusb
Using hard links
Fetching head...
Fetching objects...
Getting pack 445d79a6fe09a3d03489449f63faef0d9e9e2668
 which contains 4f4773fb3403f3ec4097ab7c7b1fdec23b9aa924
fatal: corrupted pack file
.git/objects/pack/pack-445d79a6fe09a3d03489449f63faef0d9e9e2668.pack
progress: 2 objects, 0 bytes
cg-fetch: objects fetch failed

Maybe the client machine runs git version < 1.4.2.2, in which case it is
possible that your push created a pack containing delta objects with
offset to base which git versions prior 1.4.2.2 do not understand.

Ouch. We weren't supposed to have non-backwards compatible changes...

If this is the problem you are facing (the error message should confirm
this) then the easiest solution is to upgrade git on the client.

Ha ha. Not particularly easy, unfortunately.

A quick fix for the client is to set repack.usedeltabaseoffset to
false on the machine where you have git 1.5 installed, then run "git
repack -a -d", and finally copy the pack over to the client repository.

That'll be a bit easier -- it's a fix we can do on the transfer repo ourselves.

Thanks! I do wonder though -- isn't a backwards-incompatible change
like this worthy of don't we bump core.repositoryformatversion?

But because you push to a local repository (a mounted USB stick is
considered a local repo) then you don't get to negociate the pack
capabilities of the final destination, and therefore more "bad" delta
objects might sneak in again.

How does that work? So any repo we push _from_ can override (and muck
up) the destination repo, ignoring its config?

That sounds a bit broken - the pack being built for a local
destination should respect the settings of the destination repo.


m

On 6/20/07, Martin Langhoff [off-list ref] wrote:

quoted

But because you push to a local repository (a mounted USB stick is
considered a local repo) then you don't get to negociate the pack
capabilities of the final destination, and therefore more "bad" delta
objects might sneak in again.

How does that work? So any repo we push _from_ can override (and muck
up) the destination repo, ignoring its config?

That sounds a bit broken - the pack being built for a local
destination should respect the settings of the destination repo.

OTOH, as a workaround, it _should_ work if I force a repack on the
usb-repo after each push, right? It'll wear the USB disk out, waste
human+cpu time and kill some kittens along the way, but it'll do for
the time being.

cheers,


m

On Wed, 20 Jun 2007, Martin Langhoff wrote:

On 6/20/07, Nicolas Pitre [off-list ref] wrote:

quoted

Maybe the client machine runs git version < 1.4.2.2, in which case it is
possible that your push created a pack containing delta objects with
offset to base which git versions prior 1.4.2.2 do not understand.

Ouch. We weren't supposed to have non-backwards compatible changes...

Well... to be fair, we should say that your setup is a bit non 
conventional.

quoted

If this is the problem you are facing (the error message should confirm
this) then the easiest solution is to upgrade git on the client.

Ha ha. Not particularly easy, unfortunately.

quoted

A quick fix for the client is to set repack.usedeltabaseoffset to
false on the machine where you have git 1.5 installed, then run "git
repack -a -d", and finally copy the pack over to the client repository.

That'll be a bit easier -- it's a fix we can do on the transfer repo
ourselves.

Thanks! I do wonder though -- isn't a backwards-incompatible change
like this worthy of don't we bump core.repositoryformatversion?

The repository hasn't really changed, and even such a version bump 
wouldn't help you anyway.

I also note that you're using cogito.  And from what I can deduce, it 
seems that cogito is simply copying the pack(s) over without further 
sanity checks.  This only serves to muddy things even further.

If your client machine is so important then you should consider 
upgrading to a later git version, and stop using cogito.  It will solve 
this pack problem, and you'll get added pack sanity checks as a bonus.


Nicolas

On 6/20/07, Nicolas Pitre [off-list ref] wrote:

Well... to be fair, we should say that your setup is a bit non
conventional.

You mean this doesn't affect things between 1.4.0 and 1.5.0 on either
end of the git protocol?

I also note that you're using cogito.  And from what I can deduce, it
seems that cogito is simply copying the pack(s) over without further
sanity checks.  This only serves to muddy things even further.

Yep - we are moving away from cogito; the machine is important, but
it's very awkward to install/upgrade sw on it.

cheers,


m

On Wed, 20 Jun 2007, Martin Langhoff wrote:

On 6/20/07, Nicolas Pitre [off-list ref] wrote:

quoted

But because you push to a local repository (a mounted USB stick is
considered a local repo) then you don't get to negociate the pack
capabilities of the final destination, and therefore more "bad" delta
objects might sneak in again.

How does that work? So any repo we push _from_ can override (and muck
up) the destination repo, ignoring its config?

That sounds a bit broken - the pack being built for a local
destination should respect the settings of the destination repo.

But it does!  The problem is that _you_ are cheating it by removing the 
USB stick and mounting it somewhere else.

Then you're using cogito which bypasses things.

If you use pure git on the client machine, and actually use git-pull or 
git-fetch _without_ the -l switch then validation of the updated data 
will occur and there would be no way to muck up the destination repo.


Nicolas

On Wed, 20 Jun 2007, Martin Langhoff wrote:

OTOH, as a workaround, it _should_ work if I force a repack on the
usb-repo after each push, right? It'll wear the USB disk out, waste
human+cpu time and kill some kittens along the way, but it'll do for
the time being.

Right.  And you don't need to use -f with git-repack for it to work.


Nicolas

On Wed, 20 Jun 2007, Martin Langhoff wrote:

On 6/20/07, Nicolas Pitre [off-list ref] wrote:

quoted

Well... to be fair, we should say that your setup is a bit non
conventional.

You mean this doesn't affect things between 1.4.0 and 1.5.0 on either
end of the git protocol?

Exact.


Nicolas

Nicolas Pitre wrote:

But because you push to a local repository (a mounted USB stick is 
considered a local repo) then you don't get to negociate the pack 
capabilities of the final destination, and therefore more "bad" delta 
objects might sneak in again.

So if he pushes over ssh (ssh localhost) he would get capabilities
negotiation, at the cost of being slower?

-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git