From: Christoph Hellwig <hch@infradead.org> Date: 2011-09-07 06:40:07
On Wed, Sep 07, 2011 at 12:29:30PM +0900, Masayoshi MIZUMA wrote:
Hi,
When I checked the freeze feature for ext3 filesystem using fsfreeze
command at 3.1.0-rc4, I think the following deadlock problem happened.
How to reproduce:
# mkfs -t ext3 /dev/sdd1
# mount /dev/sdd1 /MNT
# ./fsstress -d /MNT/tmp -n 10 -p 1000 > /dev/null 2>&1 &
# fsfreeze -f /MNT
# fsfreeze -u /MNT
On Wed, Sep 07, 2011 at 12:29:30PM +0900, Masayoshi MIZUMA wrote:
quoted
Hi,
When I checked the freeze feature for ext3 filesystem using fsfreeze
command at 3.1.0-rc4, I think the following deadlock problem happened.
How to reproduce:
# mkfs -t ext3 /dev/sdd1
# mount /dev/sdd1 /MNT
# ./fsstress -d /MNT/tmp -n 10 -p 1000 > /dev/null 2>&1 &
# fsfreeze -f /MNT
# fsfreeze -u /MNT
Can you add this testcase to xfstests?
I don't know how to add this testcase to xfstests much, but I will
try to create it.
Please wait for a while...
Thanks,
Masayoshi
On Wed, Sep 7, 2011 at 2:40 AM, Christoph Hellwig [off-list ref] wrote:
On Wed, Sep 07, 2011 at 12:29:30PM +0900, Masayoshi MIZUMA wrote:
quoted
Hi,
When I checked the freeze feature for ext3 filesystem using fsfreeze
command at 3.1.0-rc4, I think the following deadlock problem happened.
How to reproduce:
# mkfs -t ext3 /dev/sdd1
# mount /dev/sdd1 /MNT
# ./fsstress -d /MNT/tmp -n 10 -p 1000 > /dev/null 2>&1 &
# fsfreeze -f /MNT
# fsfreeze -u /MNT
Can you add this testcase to xfstests?
Christoph,
Isn't that just a matter of extending test 068 to ext4
====
@@ -51,7 +51,7 @@ . ./common.filter # real QA test starts here-_supported_fs xfs+_supported_fs xfs ext3 ext4 _supported_os Linux IRIX _require_scratch
====
That's a totally untested patch if someone wants to try it.
Greg
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Eric Sandeen <hidden> Date: 2011-09-07 17:10:53
On 9/7/11 11:50 AM, Christoph Hellwig wrote:
On Wed, Sep 07, 2011 at 12:45:34PM -0400, Greg Freemyer wrote:
quoted
quoted
Can you add this testcase to xfstests?
Christoph,
Isn't that just a matter of extending test 068 to ext4
If you have recent enough xfsprogs that allow the freeze command
for foreign filesystems that might work.
where "recent enough" is since Tue Feb 10 14:41:51 2009 -0600
I say go for it :)
Could always add a quick helper to make sure the xfs_io command
doesn't fail on freeze, and _notrun if it does.
-Eric
From: Christoph Hellwig <hch@infradead.org> Date: 2011-09-07 17:17:28
On Wed, Sep 07, 2011 at 12:10:07PM -0500, Eric Sandeen wrote:
quoted
If you have recent enough xfsprogs that allow the freeze command
for foreign filesystems that might work.
where "recent enough" is since Tue Feb 10 14:41:51 2009 -0600
I say go for it :)
Could always add a quick helper to make sure the xfs_io command
doesn't fail on freeze, and _notrun if it does.
With the _notrun trick it could probably even claim generic fs
support.
From: Jan Kara <jack@suse.cz> Date: 2011-09-07 17:34:54
Hello,
Thanks for report!
On Wed 07-09-11 12:29:30, Masayoshi MIZUMA wrote:
When I checked the freeze feature for ext3 filesystem using fsfreeze
command at 3.1.0-rc4, I think the following deadlock problem happened.
How to reproduce:
# mkfs -t ext3 /dev/sdd1
# mount /dev/sdd1 /MNT
# ./fsstress -d /MNT/tmp -n 10 -p 1000 > /dev/null 2>&1 &
# fsfreeze -f /MNT
# fsfreeze -u /MNT
If this deadlock is reproduced, "fsfreeze -u /MNT" does not return.
The detail of deadlock:
o [flush-8:16:1523]
wb_do_writeback
wb_writeback
...
ext3_journalled_writepage
journal_start
start_this_handle
# waiting until journal->j_barrier_count turns 0...
# j_barrier_count was incremented by journal_lock_updates()
# via ext3_freeze().
o [fsstress:2673]
sys_sync
sync_filesystems
iterate_supers
down_read(sb->s_umount)
sync_one_sb
__sync_filesystem
writeback_inodes_sb
writeback_inodes_sb_nr
wait_for_completion
wait_for_common
# waiting for completion of [flush-8:16:1523]...
o [fsfreeze:2749]
sys_ioctl
do_vfs_ioctl
thaw_super
# waiting for down_write(sb->s_umount)...
# [fsfreeze:2673] did down_read(sb->s_umount).
Yes, this is a classical deadlock that can happen for any filesystem. The
problem is flusher thread holds s_umount semaphore (either directly, or as
in your case, indirectly via blocked sync) and tries to do some IO which
blocks on frozen filesystem. It's particularly easy to hit for ext3 because
it doesn't do vfs_check_frozen() checks but all other filesystems have the
race window as well. Val Henson is working on fixing the problem - she even
has some first version of patches I believe.
Honza
--
Jan Kara [off-list ref]
SUSE Labs, CR
On Wed, Sep 7, 2011 at 1:34 PM, Jan Kara [off-list ref] wrote:
Hello,
Thanks for report!
On Wed 07-09-11 12:29:30, Masayoshi MIZUMA wrote:
quoted
When I checked the freeze feature for ext3 filesystem using fsfreeze
command at 3.1.0-rc4, I think the following deadlock problem happened.
How to reproduce:
# mkfs -t ext3 /dev/sdd1
# mount /dev/sdd1 /MNT
# ./fsstress -d /MNT/tmp -n 10 -p 1000 > /dev/null 2>&1 &
# fsfreeze -f /MNT
# fsfreeze -u /MNT
If this deadlock is reproduced, "fsfreeze -u /MNT" does not return.
The detail of deadlock:
o [flush-8:16:1523]
wb_do_writeback
wb_writeback
...
ext3_journalled_writepage
journal_start
start_this_handle
# waiting until journal->j_barrier_count turns 0...
# j_barrier_count was incremented by journal_lock_updates()
# via ext3_freeze().
o [fsstress:2673]
sys_sync
sync_filesystems
iterate_supers
down_read(sb->s_umount)
sync_one_sb
__sync_filesystem
writeback_inodes_sb
writeback_inodes_sb_nr
wait_for_completion
wait_for_common
# waiting for completion of [flush-8:16:1523]...
o [fsfreeze:2749]
sys_ioctl
do_vfs_ioctl
thaw_super
# waiting for down_write(sb->s_umount)...
# [fsfreeze:2673] did down_read(sb->s_umount).
Yes, this is a classical deadlock that can happen for any filesystem. The
problem is flusher thread holds s_umount semaphore (either directly, or as
in your case, indirectly via blocked sync) and tries to do some IO which
blocks on frozen filesystem. It's particularly easy to hit for ext3 because
it doesn't do vfs_check_frozen() checks but all other filesystems have the
race window as well. Val Henson is working on fixing the problem - she even
has some first version of patches I believe.
Honza
xfstests test 068 has been around since kernel 2.4 days and should
have caught it if xfs is impacted.
I know I ran the 2002 version many times to prove to myself that
fsfreeze for xfs was stable when teamed with LVM. (It wasn't when I
first wrote 068 way back then).
068 has been greatly simplified since 2002, but it still looks like it
should do a good job.
Is there a problem with 068? Does it need extra test coverage even for xfs?
Greg
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Jan Kara <jack@suse.cz> Date: 2011-09-07 22:32:08
On Wed 07-09-11 13:56:08, Greg Freemyer wrote:
On Wed, Sep 7, 2011 at 1:34 PM, Jan Kara [off-list ref] wrote:
quoted
Hello,
Thanks for report!
On Wed 07-09-11 12:29:30, Masayoshi MIZUMA wrote:
quoted
When I checked the freeze feature for ext3 filesystem using fsfreeze
command at 3.1.0-rc4, I think the following deadlock problem happened.
How to reproduce:
# mkfs -t ext3 /dev/sdd1
# mount /dev/sdd1 /MNT
# ./fsstress -d /MNT/tmp -n 10 -p 1000 > /dev/null 2>&1 &
# fsfreeze -f /MNT
# fsfreeze -u /MNT
If this deadlock is reproduced, "fsfreeze -u /MNT" does not return.
The detail of deadlock:
o [flush-8:16:1523]
wb_do_writeback
wb_writeback
...
ext3_journalled_writepage
journal_start
start_this_handle
# waiting until journal->j_barrier_count turns 0...
# j_barrier_count was incremented by journal_lock_updates()
# via ext3_freeze().
o [fsstress:2673]
sys_sync
sync_filesystems
iterate_supers
down_read(sb->s_umount)
sync_one_sb
__sync_filesystem
writeback_inodes_sb
writeback_inodes_sb_nr
wait_for_completion
wait_for_common
# waiting for completion of [flush-8:16:1523]...
o [fsfreeze:2749]
sys_ioctl
do_vfs_ioctl
thaw_super
# waiting for down_write(sb->s_umount)...
# [fsfreeze:2673] did down_read(sb->s_umount).
Yes, this is a classical deadlock that can happen for any filesystem. The
problem is flusher thread holds s_umount semaphore (either directly, or as
in your case, indirectly via blocked sync) and tries to do some IO which
blocks on frozen filesystem. It's particularly easy to hit for ext3 because
it doesn't do vfs_check_frozen() checks but all other filesystems have the
race window as well. Val Henson is working on fixing the problem - she even
has some first version of patches I believe.
Honza
xfstests test 068 has been around since kernel 2.4 days and should
have caught it if xfs is impacted.
I know I ran the 2002 version many times to prove to myself that
fsfreeze for xfs was stable when teamed with LVM. (It wasn't when I
first wrote 068 way back then).
068 has been greatly simplified since 2002, but it still looks like it
should do a good job.
Is there a problem with 068? Does it need extra test coverage even for xfs?
I believe at least mmapped writes can trigger the deadlock even for xfs
and fsstress (slightly surprisingly) does not test that. It's a narrow race
window but it is there and it has been triggered in practice (for ext4 but
it's a race in VFS code used by both XFS and ext4). So maybe extending
fsstress would be a way to go?
Honza
--
Jan Kara [off-list ref]
SUSE Labs, CR
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Sep 7, 2011 at 6:32 PM, Jan Kara [off-list ref] wrote:
On Wed 07-09-11 13:56:08, Greg Freemyer wrote:
quoted
On Wed, Sep 7, 2011 at 1:34 PM, Jan Kara [off-list ref] wrote:
quoted
Hello,
Thanks for report!
On Wed 07-09-11 12:29:30, Masayoshi MIZUMA wrote:
quoted
When I checked the freeze feature for ext3 filesystem using fsfreeze
command at 3.1.0-rc4, I think the following deadlock problem happened.
How to reproduce:
# mkfs -t ext3 /dev/sdd1
# mount /dev/sdd1 /MNT
# ./fsstress -d /MNT/tmp -n 10 -p 1000 > /dev/null 2>&1 &
# fsfreeze -f /MNT
# fsfreeze -u /MNT
If this deadlock is reproduced, "fsfreeze -u /MNT" does not return.
The detail of deadlock:
o [flush-8:16:1523]
wb_do_writeback
wb_writeback
...
ext3_journalled_writepage
journal_start
start_this_handle
# waiting until journal->j_barrier_count turns 0...
# j_barrier_count was incremented by journal_lock_updates()
# via ext3_freeze().
o [fsstress:2673]
sys_sync
sync_filesystems
iterate_supers
down_read(sb->s_umount)
sync_one_sb
__sync_filesystem
writeback_inodes_sb
writeback_inodes_sb_nr
wait_for_completion
wait_for_common
# waiting for completion of [flush-8:16:1523]...
o [fsfreeze:2749]
sys_ioctl
do_vfs_ioctl
thaw_super
# waiting for down_write(sb->s_umount)...
# [fsfreeze:2673] did down_read(sb->s_umount).
Yes, this is a classical deadlock that can happen for any filesystem. The
problem is flusher thread holds s_umount semaphore (either directly, or as
in your case, indirectly via blocked sync) and tries to do some IO which
blocks on frozen filesystem. It's particularly easy to hit for ext3 because
it doesn't do vfs_check_frozen() checks but all other filesystems have the
race window as well. Val Henson is working on fixing the problem - she even
has some first version of patches I believe.
Honza
xfstests test 068 has been around since kernel 2.4 days and should
have caught it if xfs is impacted.
I know I ran the 2002 version many times to prove to myself that
fsfreeze for xfs was stable when teamed with LVM. (It wasn't when I
first wrote 068 way back then).
068 has been greatly simplified since 2002, but it still looks like it
should do a good job.
Is there a problem with 068? Does it need extra test coverage even for xfs?
I believe at least mmapped writes can trigger the deadlock even for xfs
and fsstress (slightly surprisingly) does not test that. It's a narrow race
window but it is there and it has been triggered in practice (for ext4 but
it's a race in VFS code used by both XFS and ext4). So maybe extending
fsstress would be a way to go?
Honza
That's a surprisingly large hole in xfstests.
That sounds like a pretty core and significant change. I'll have to
leave that to one of the main developers.
Greg
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Sep 7, 2011 at 10:34 AM, Jan Kara [off-list ref] wrote:
Hello,
Thanks for report!
On Wed 07-09-11 12:29:30, Masayoshi MIZUMA wrote:
quoted
When I checked the freeze feature for ext3 filesystem using fsfreeze
command at 3.1.0-rc4, I think the following deadlock problem happened.
How to reproduce:
# mkfs -t ext3 /dev/sdd1
# mount /dev/sdd1 /MNT
# ./fsstress -d /MNT/tmp -n 10 -p 1000 > /dev/null 2>&1 &
# fsfreeze -f /MNT
# fsfreeze -u /MNT
If this deadlock is reproduced, "fsfreeze -u /MNT" does not return.
The detail of deadlock:
o [flush-8:16:1523]
wb_do_writeback
wb_writeback
...
ext3_journalled_writepage
journal_start
start_this_handle
# waiting until journal->j_barrier_count turns 0...
# j_barrier_count was incremented by journal_lock_updates()
# via ext3_freeze().
o [fsstress:2673]
sys_sync
sync_filesystems
iterate_supers
down_read(sb->s_umount)
sync_one_sb
__sync_filesystem
writeback_inodes_sb
writeback_inodes_sb_nr
wait_for_completion
wait_for_common
# waiting for completion of [flush-8:16:1523]...
o [fsfreeze:2749]
sys_ioctl
do_vfs_ioctl
thaw_super
# waiting for down_write(sb->s_umount)...
# [fsfreeze:2673] did down_read(sb->s_umount).
Yes, this is a classical deadlock that can happen for any filesystem. The
problem is flusher thread holds s_umount semaphore (either directly, or as
in your case, indirectly via blocked sync) and tries to do some IO which
blocks on frozen filesystem. It's particularly easy to hit for ext3 because
it doesn't do vfs_check_frozen() checks but all other filesystems have the
race window as well. Val Henson is working on fixing the problem - she even
has some first version of patches I believe.
Yes, if the bug reporter could test the patches I just sent out, that
would be great. I'm happy to resend privately. Thanks!
-VAL
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Sep 7, 2011 at 10:34 AM, Jan Kara [off-list ref] wrote:
quoted
Hello,
Thanks for report!
On Wed 07-09-11 12:29:30, Masayoshi MIZUMA wrote:
quoted
When I checked the freeze feature for ext3 filesystem using fsfreeze
command at 3.1.0-rc4, I think the following deadlock problem happened.
How to reproduce:
# mkfs -t ext3 /dev/sdd1
# mount /dev/sdd1 /MNT
# ./fsstress -d /MNT/tmp -n 10 -p 1000 > /dev/null 2>&1 &
# fsfreeze -f /MNT
# fsfreeze -u /MNT
If this deadlock is reproduced, "fsfreeze -u /MNT" does not return.
The detail of deadlock:
o [flush-8:16:1523]
wb_do_writeback
wb_writeback
...
ext3_journalled_writepage
journal_start
start_this_handle
# waiting until journal->j_barrier_count turns 0...
# j_barrier_count was incremented by journal_lock_updates()
# via ext3_freeze().
o [fsstress:2673]
sys_sync
sync_filesystems
iterate_supers
down_read(sb->s_umount)
sync_one_sb
__sync_filesystem
writeback_inodes_sb
writeback_inodes_sb_nr
wait_for_completion
wait_for_common
# waiting for completion of [flush-8:16:1523]...
o [fsfreeze:2749]
sys_ioctl
do_vfs_ioctl
thaw_super
# waiting for down_write(sb->s_umount)...
# [fsfreeze:2673] did down_read(sb->s_umount).
Yes, this is a classical deadlock that can happen for any filesystem. The
problem is flusher thread holds s_umount semaphore (either directly, or as
in your case, indirectly via blocked sync) and tries to do some IO which
blocks on frozen filesystem. It's particularly easy to hit for ext3 because
it doesn't do vfs_check_frozen() checks but all other filesystems have the
race window as well. Val Henson is working on fixing the problem - she even
has some first version of patches I believe.
Yes, if the bug reporter could test the patches I just sent out, that
would be great. I'm happy to resend privately. Thanks!
I put your patches to 3.1.0-rc4 and tested it. Then, the deadlock was
not reproduced, so your patches work fine, thank you!
Masayoshi