From: Eric Sandeen <hidden> Date: 2012-07-20 23:13:52
I'm looking at a situation where a root filesystem encountered an error and shut down, and therefore the error was stored in the journal.
But for the root fs, it seems that nothing can clear it.
If we do e2fsck -fy on a readonly mounted filesystem, then remount,rw the error persists:
[25124.319387] EXT4-fs warning (device loop3): ext4_clear_journal_err:4281: Filesystem error recorded from previous mount: IO failure
[25124.331140] EXT4-fs warning (device loop3): ext4_clear_journal_err:4282: Marking fs in need of filesystem check.
ad infinitum.
It may be my fever-addled brain this week but I'm having a hard time following how this error is supposed to get set & cleared, especially if e2fsck has modified it while mounted ro.
As soon as I mount rw again, load_superblock() sees the journal superblock has an error set, and copies it back into the journal->j_errno.
After Ted's "e2fsck: correctly propagate error from journal to superblock" in e2fsprogs, at least an unmounted fs gets cleaned up, but I'm not sure what to do to fix this when it's mounted.
Am I missing something obvious?
Thanks,
-Eric
On Fri, Jul 20, 2012 at 06:13:50PM -0500, Eric Sandeen wrote:
I'm looking at a situation where a root filesystem encountered an error and shut down, and therefore the error was stored in the journal.
But for the root fs, it seems that nothing can clear it.
If we do e2fsck -fy on a readonly mounted filesystem, then remount,rw the error persists:
[25124.319387] EXT4-fs warning (device loop3): ext4_clear_journal_err:4281: Filesystem error recorded from previous mount: IO failure
[25124.331140] EXT4-fs warning (device loop3): ext4_clear_journal_err:4282: Marking fs in need of filesystem check.
ad infinitum.
It may be my fever-addled brain this week but I'm having a hard time following how this error is supposed to get set & cleared, especially if e2fsck has modified it while mounted ro.
As soon as I mount rw again, load_superblock() sees the journal superblock has an error set, and copies it back into the journal->j_errno.
After Ted's "e2fsck: correctly propagate error from journal to superblock" in e2fsprogs, at least an unmounted fs gets cleaned up, but I'm not sure what to do to fix this when it's mounted.
The problem you describe should have been fixed by commit 6d75685e:
e2fsck: handle an already recovered journal with a non-zero s_error
field, which is in e2fsprogs 1.42.4.
- Ted
From: Eric Sandeen <hidden> Date: 2012-07-23 16:29:03
On 7/20/12 7:39 PM, Theodore Ts'o wrote:
On Fri, Jul 20, 2012 at 06:13:50PM -0500, Eric Sandeen wrote:
quoted
I'm looking at a situation where a root filesystem encountered an error and shut down, and therefore the error was stored in the journal.
But for the root fs, it seems that nothing can clear it.
If we do e2fsck -fy on a readonly mounted filesystem, then remount,rw the error persists:
[25124.319387] EXT4-fs warning (device loop3): ext4_clear_journal_err:4281: Filesystem error recorded from previous mount: IO failure
[25124.331140] EXT4-fs warning (device loop3): ext4_clear_journal_err:4282: Marking fs in need of filesystem check.
ad infinitum.
It may be my fever-addled brain this week but I'm having a hard time following how this error is supposed to get set & cleared, especially if e2fsck has modified it while mounted ro.
As soon as I mount rw again, load_superblock() sees the journal superblock has an error set, and copies it back into the journal->j_errno.
After Ted's "e2fsck: correctly propagate error from journal to superblock" in e2fsprogs, at least an unmounted fs gets cleaned up, but I'm not sure what to do to fix this when it's mounted.
The problem you describe should have been fixed by commit 6d75685e:
e2fsck: handle an already recovered journal with a non-zero s_error
field, which is in e2fsprogs 1.42.4.
Well, I tested e2fsprogs from git and saw the same trouble. Let me look more closely at that commit.
-Eric
nWell, I tested e2fsprogs from git and saw the same trouble. Let me
look more closely at that commit.
Hmm, would it be possible to send me /tmp/loop3.qcow.bz2 after running:
e2image -Q /dev/loop3 /tmp/loop3.qcow
bzip2 /tmp/loop3.qcow
Regardless of what the problem ends up being, this looks like
something for our regression test suite....
- Ted
From: Eric Sandeen <hidden> Date: 2012-07-23 20:21:49
On 7/23/12 2:14 PM, Theodore Ts'o wrote:
quoted
nWell, I tested e2fsprogs from git and saw the same trouble. Let me
look more closely at that commit.
Hmm, would it be possible to send me /tmp/loop3.qcow.bz2 after running:
e2image -Q /dev/loop3 /tmp/loop3.qcow
bzip2 /tmp/loop3.qcow
Regardless of what the problem ends up being, this looks like
something for our regression test suite....
- Ted
Sure I can do that, will do it offline.
FWIW, the commit you mentioned changes e2fsck_check_ext3_journal, and
we only get there from main() like this:
if ((ctx->mount_flags & EXT2_MF_MOUNTED) &&
!(sb->s_feature_incompat & EXT3_FEATURE_INCOMPAT_RECOVER))
goto skip_journal;
retval = e2fsck_check_ext3_journal(ctx);
In my case I am mounted ro and recovery is done, so
EXT3_FEATURE_INCOMPAT_RECOVER is not, set, and so we skip over it with
the goto.
or else from here:
if (sb->s_feature_incompat & EXT3_FEATURE_INCOMPAT_RECOVER) {
if (ctx->options & E2F_OPT_READONLY) {
...
} else {
...
retval = e2fsck_run_ext3_journal(ctx);
And again, I've already done recovery and am mounted RO so we won't go that way.
If make the first test above a little later:
On Mon, Jul 23, 2012 at 03:21:45PM -0500, Eric Sandeen wrote:
If make the first test above a little later:
...
it clears it up for me but TBH I'm not totally clear on what all is going
on here.
Your patch effectively reverts commit 47c1b8e1666: e2fsck: Skip journal
checks if the fs is mounted and doesn't need recovery
The reason why this patch was added was that it slightly speeded up
the HDD boots. I've looked at this closely, and it's really not that
big of a speed up. It only saves a single seek and read (to read the
journal superblock). I've measured what it would be on a 1TB laptop
drive, and the difference is 60ms (70ms versus 10ms).
So I'm going to revert that commit, which should fix the problem that
we're seeing. But it also turns out that the kernel is not quite
doing the right thing. It should have never left the journal
superblock with a non-zero s_error indicator after it transfered the
error indicator to the superblock. The problem was that we weren't
actually flushing the cleared errno field out to the disk for the
journal superblock.
- Ted
After we transfer set the EXT4_ERROR_FS bit in the file system
superblock, it's not enough to call jbd2_journal_clear_err() to clear
the error indication from journal superblock --- we need to call
jbd2_journal_update_sb_errno() as well. Otherwise, when the root file
system is mounted read-only, the journal is replayed, and the error
indicator is transferred to the superblock --- but the s_errno field
in the jbd2 superblock is left set (since although we cleared it in
memory, we never flushed it out to disk).
This can end up confusing e2fsck. We should make e2fsck more robust
in this case, but the kernel shouldn't be leaving things in this
confused state, either.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Cc: stable@kernel.org
---
fs/ext4/super.c | 1 +
fs/jbd2/journal.c | 3 ++-
include/linux/jbd2.h | 1 +
3 files changed, 4 insertions(+), 1 deletion(-)
This reverts commit 47c1b8e16668daa6e74cee3c7b8bdf237ffefe70.
The original reason for this commit was to speed up boots for hard
drives. However, I've measured the time difference on a 1TB laptop
drive, and it's not significant: 70ms vs 10ms when running e2fsck on a
clean file system.
The problem with this optimization is that we don't notice if the
journal superblock has a non-zero s_errno field. If we don't transfer
the error indicator from the journal superblock to the file system
superblock, then the kernel will transfer it when the file system is
remounted read-write, causing scary messages to appear in the syslog.
(And since there was a bug in the kernel code which didn't clear the
error indicator in the journal superblock, it would never get
cleared.)
Reported-by: Eric Sandeen <redacted>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
---
e2fsck/unix.c | 5 -----
1 file changed, 5 deletions(-)
Previously e2fsck would only allow a mounted file system to be checked
if it was the root file system and it was mounted read-only. Now
allow any file system mounted read-only if the -f option is specified.
This makes it easier to test how e2fsck handles checking file systems
which are mounted without having to test on the root file system.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
---
e2fsck/unix.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)