On Wed 07-09-11 13:56:08, Greg Freemyer wrote:

On Wed, Sep 7, 2011 at 1:34 PM, Jan Kara [off-list ref] wrote:

quoted

 Hello,

 Thanks for report!

On Wed 07-09-11 12:29:30, Masayoshi MIZUMA wrote:

quoted

When I checked the freeze feature for ext3 filesystem using fsfreeze
command at 3.1.0-rc4, I think the following deadlock problem happened.

How to reproduce:
 # mkfs -t ext3 /dev/sdd1
 # mount /dev/sdd1 /MNT
 # ./fsstress -d /MNT/tmp -n 10 -p 1000 > /dev/null 2>&1 &
 # fsfreeze -f /MNT
 # fsfreeze -u /MNT

 If this deadlock is reproduced, "fsfreeze -u /MNT" does not return.

The detail of deadlock:
o [flush-8:16:1523]
  wb_do_writeback
   wb_writeback
   ...
     ext3_journalled_writepage
      journal_start
       start_this_handle
       # waiting until journal->j_barrier_count turns 0...
       # j_barrier_count was incremented by journal_lock_updates()
       # via ext3_freeze().

o [fsstress:2673]
  sys_sync
   sync_filesystems
    iterate_supers
     down_read(sb->s_umount)
     sync_one_sb
      __sync_filesystem
       writeback_inodes_sb
        writeback_inodes_sb_nr
         wait_for_completion
          wait_for_common
          # waiting for completion of [flush-8:16:1523]...

o [fsfreeze:2749]
  sys_ioctl
   do_vfs_ioctl
    thaw_super
    # waiting for down_write(sb->s_umount)...
    # [fsfreeze:2673] did down_read(sb->s_umount).

 Yes, this is a classical deadlock that can happen for any filesystem. The
problem is flusher thread holds s_umount semaphore (either directly, or as
in your case, indirectly via blocked sync) and tries to do some IO which
blocks on frozen filesystem. It's particularly easy to hit for ext3 because
it doesn't do vfs_check_frozen() checks but all other filesystems have the
race window as well. Val Henson is working on fixing the problem - she even
has some first version of patches I believe.

                                                               Honza

xfstests test 068 has been around since kernel 2.4 days and should
have caught it if xfs is impacted.

I know I ran the 2002 version many times to prove to myself that
fsfreeze for xfs was stable when teamed with LVM.  (It wasn't when I
first wrote 068 way back then).

068 has been greatly simplified since 2002, but it still looks like it
should do a good job.

Is there a problem with 068?  Does it need extra test coverage even for xfs?

  I believe at least mmapped writes can trigger the deadlock even for xfs
and fsstress (slightly surprisingly) does not test that. It's a narrow race
window but it is there and it has been triggered in practice (for ext4 but
it's a race in VFS code used by both XFS and ext4). So maybe extending
fsstress would be a way to go?

								Honza
-- 
Jan Kara [off-list ref]
SUSE Labs, CR
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html