Hi Johannes,

[CC'ing linux-fsdevel and tytso]

Am 21.06.2015 um 14:28 schrieb Johannes Schindelin:

Hi Richard,

On 2015-06-20 21:40, Richard Weinberger wrote:

quoted

Yesterday our git server faced a power cut and a git repository broke.
The server is running a ext4 filesystem on top of Linux 3.16 (stable
from openSUSE) and git 2.1.4.
We had a backup, so no data was lost but I really would like to figure out
what happened.

This is the output of git fsck:
Checking object directories: 100% (256/256), done.
error: object file objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d is empty
error: object file objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d is empty
fatal: loose object cef7627fc160ad7294b1f728db0c1ddb65a38b1d (stored
in objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d) is corrupt

To me it seems like git was creating a new object and got interrupted
before fsync/fdatasync'ing it.
As the object was referenced before syncing the data to disk the repo broke.
Could this have happened?
Also, is git designed to survive power cuts? Then referencing an
object before synching it do disk would make no sense.

I had similar issues with ext4 in the past, even with local repositories when using Git without pushing. My then-current laptop would not report battery power correctly, so I ran into out-of-power situations that would result in a loose object file that was simply empty, i.e. its length was zero. As far as my analysis back then went, this was not Git's fault, because its `write_loose_object()` function would write to a temporary file first and only move that file into place once it was written fully.

I was then shocked to learn that ext4 apparently has a default setting that allows it to truncate files upon power failure (something about a full journal vs a fast journal or some such) when I had expected the default to be a true journaled file system with proper atomicity regarding writes and moves. I remember that back then, I angrily fixed that setting to make my file system fully journaled.

You mean the ext4 delayed block allocation feature/issue?
IIRC Ted added some hacks to ext4 to detect misbehaving applications (Gnome and KDE).
But to my knowledge such an file corruption must not happen if the application behaves well. And it can happen on all file systems.
Ted, maybe you can help us? BTW: I'm using ext4's default mount options from openSUSE, data=ordered.

Maybe this leads you into the direction of a work-around in your setup?

I'm still not sure who to blame. ;-)

Thanks,
//richard
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in