On Tue, Aug 07, 2012 at 02:54:48PM +0900, Guk-Bong, Kwon wrote:

HI all

I tested xfs over nfs using bonnie++

xfs and nfs hang when xfs filesystem filled

What's the problem?

It appears to be blocked in writeback, getting ENOSPC errors when
they shouldn't occur.

see below
--------------------------------

1. nfs server

    a. uname -a
        - Linux nfs_server 2.6.32.58 #1 SMP Thu Mar 22 13:33:34 KST 2012 x86_64
        Intel(R) Xeon(R) CPU E5606 @ 2.13GHz GenuineIntel GNU/Linux

Old kernel. Upgrade.

================================================================================
/test   0.0.0.0/0.0.0.0(rw,async,wdelay,hide,nocrossmnt,insecure,no_root_squash,no_all_squash,no_subtree_check,secure_locks,acl,fsid=1342087477,anonuid=65534,anongid=65534)
================================================================================

You're using the async export option, which means the server/client
write throttling mechanisms built into the NFs protocol are not
active. That leads to clients swamping the server with dirty data
and not backing off when the server is overloaded, and leads to
-data loss- when the server fails.

IOWs, you're massively overcomitting allocation from lots of
threads which means you are probably depleting the free space pool,
and that leads to -data loss- and potentially deadlocks. If this is
what your production systems do, then a) increase the reserve pool,
and b) fix your producton systems not to do this.

Aug  2 18:17:58 anystor1 kernel: Call Trace:
Aug  2 18:17:58 anystor1 kernel: [<ffffffff811738ce>] ?  xfs_btree_is_lastrec+0x4e/0x60
Aug  2 18:17:58 anystor1 kernel: [<ffffffff8135edad>] ?  schedule_timeout+0x1ed/0x250
Aug  2 18:17:58 anystor1 kernel: [<ffffffff8135fcd1>] ? __down+0x61/0xa0
Aug  2 18:17:58 anystor1 kernel: [<ffffffff810572d6>] ? down+0x46/0x50
Aug  2 18:17:58 anystor1 kernel: [<ffffffff811af6a4>] ?  _xfs_buf_find+0x134/0x220
Aug  2 18:17:58 anystor1 kernel: [<ffffffff811af7fe>] ?  xfs_buf_get_flags+0x6e/0x190
Aug  2 18:17:58 anystor1 kernel: [<ffffffff811a525e>] ?  xfs_trans_get_buf+0x10e/0x160
Aug  2 18:17:58 anystor1 kernel: [<ffffffff81161954>] ?  xfs_alloc_fix_freelist+0x144/0x450
Aug  2 18:17:58 anystor1 kernel: [<ffffffff8119e597>] ?  xfs_icsb_disable_counter+0x17/0x160
Aug  2 18:17:58 anystor1 kernel: [<ffffffff8116d2f2>] ?  xfs_bmap_add_extent_delay_real+0x8d2/0x11a0
Aug  2 18:17:58 anystor1 kernel: [<ffffffff811a4b83>] ?  xfs_trans_log_buf+0x63/0xa0
Aug  2 18:17:58 anystor1 kernel: [<ffffffff8119e731>] ?  xfs_icsb_balance_counter_locked+0x31/0xf0
Aug  2 18:17:58 anystor1 kernel: [<ffffffff81161ed1>] ?  xfs_alloc_vextent+0x1b1/0x4c0
Aug  2 18:17:58 anystor1 kernel: [<ffffffff8116e946>] ?  xfs_bmap_btalloc+0x596/0xa70
Aug  2 18:17:58 anystor1 kernel: [<ffffffff8117125a>] ? xfs_bmapi+0x9fa/0x1230
Aug  2 18:17:58 anystor1 kernel: [<ffffffff811965f6>] ?  xlog_state_release_iclog+0x56/0xe0
Aug  2 18:17:58 anystor1 kernel: [<ffffffff811a3a0f>] ?  xfs_trans_reserve+0x9f/0x210
Aug  2 18:17:58 anystor1 kernel: [<ffffffff81192d0e>] ?  xfs_iomap_write_allocate+0x24e/0x3d0
Aug  2 18:17:58 anystor1 kernel: [<ffffffff811c29c0>] ? elv_insert+0xf0/0x260
Aug  2 18:17:58 anystor1 kernel: [<ffffffff8119396b>] ? xfs_iomap+0x2cb/0x300
Aug  2 18:17:58 anystor1 kernel: [<ffffffff811aba05>] ? xfs_map_blocks+0x25/0x30
Aug  2 18:17:58 anystor1 kernel: [<ffffffff811acb64>] ?  xfs_page_state_convert+0x414/0x6d0
Aug  2 18:17:58 anystor1 kernel: [<ffffffff811ad137>] ?  xfs_vm_writepage+0x77/0x130
Aug  2 18:17:58 anystor1 kernel: [<ffffffff8107c8ca>] ? __writepage+0xa/0x40
Aug  2 18:17:58 anystor1 kernel: [<ffffffff8107d0af>] ?  write_cache_pages+0x1df/0x3d0
Aug  2 18:17:58 anystor1 kernel: [<ffffffff8107c8c0>] ? __writepage+0x0/0x40
Aug  2 18:17:58 anystor1 kernel: [<ffffffff81076f4c>] ?  __filemap_fdatawrite_range+0x4c/0x60
Aug  2 18:17:58 anystor1 kernel: [<ffffffff811da3a1>] ?  radix_tree_gang_lookup+0x71/0xf0
Aug  2 18:17:58 anystor1 kernel: [<ffffffff811b029d>] ?  xfs_flush_pages+0xad/0xc0
Aug  2 18:17:58 anystor1 kernel: [<ffffffff811b795a>] ?  xfs_sync_inode_data+0xca/0xf0
Aug  2 18:17:58 anystor1 kernel: [<ffffffff811b7aa0>] ?  xfs_inode_ag_walk+0x80/0x140
Aug  2 18:17:58 anystor1 kernel: [<ffffffff811b7890>] ?  xfs_sync_inode_data+0x0/0xf0
Aug  2 18:17:58 anystor1 kernel: [<ffffffff811b7be8>] ?  xfs_inode_ag_iterator+0x88/0xd0
Aug  2 18:17:58 anystor1 kernel: [<ffffffff811b7890>] ?  xfs_sync_inode_data+0x0/0xf0
Aug  2 18:17:58 anystor1 kernel: [<ffffffff8135ed1d>] ?  schedule_timeout+0x15d/0x250
Aug  2 18:17:58 anystor1 kernel: [<ffffffff811b7f40>] ? xfs_sync_data+0x30/0x60
Aug  2 18:17:58 anystor1 kernel: [<ffffffff811b7f8e>] ?  xfs_flush_inodes_work+0x1e/0x50
Aug  2 18:17:58 anystor1 kernel: [<ffffffff811b726c>] ? xfssyncd+0x13c/0x1d0
Aug  2 18:17:58 anystor1 kernel: [<ffffffff811b7130>] ? xfssyncd+0x0/0x1d0
Aug  2 18:17:58 anystor1 kernel: [<ffffffff810529d6>] ? kthread+0x96/0xb0

There's your problem - writeback of data is blocked waiting on a
metadata buffer, and everything else is blocked behind it. Upgrade
your kernel.

In summary, you are doing something silly on a very old kernel and
you broke it. As a prize, you get to keep all the broken pieces.....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs