Hi Zheng,

Interesting analysis. 

* On Sun, Jan 15, 2012 at 03:17:12PM -0500, Zheng Da [off-list ref] wrote:

Thanks. I was reading the code of kernel 3.0. XFS starts to support
concurrent direct IO since kernel 3.1.5.
But concurrent direct IO write still doesn't work well in kernel 3.2. 

 From what I have heard it has supported it from sometime back. I 
think you may need to ask in xfs general ML about this.

I wrote a test program that accesses a 4G file randomly (read and write), and
I ran it with 8 threads and the machine has 8 cores. It turns out that only
1 core is running. I'm pretty sure xfs_rw_ilock is locked
with XFS_IOLOCK_SHARED in xfs_file_dio_aio_write.

lockstat shows me that there is a lot of wait time in ip->i_lock. It seems
the lock is locked exclusively.
              &(&ip->i_lock)->mr_lock-W:         31568          36170
     0.24       20048.25     7589157.99         130154        3146848
     0.00         217.70     1238310.72
              &(&ip->i_lock)->mr_lock-R:         11251          11886
     0.24       20043.01     2895595.18          46671         526309
     0.00          63.80      264097.96
              -------------------------
                &(&ip->i_lock)->mr_lock          36170
[<ffffffffa03be122>] xfs_ilock+0xb2/0x110 [xfs]
                &(&ip->i_lock)->mr_lock          11886
[<ffffffffa03be15a>] xfs_ilock+0xea/0x110 [xfs]
              -------------------------
                &(&ip->i_lock)->mr_lock          38555
[<ffffffffa03be122>] xfs_ilock+0xb2/0x110 [xfs]
                &(&ip->i_lock)->mr_lock           9501
[<ffffffffa03be15a>] xfs_ilock+0xea/0x110 [xfs]

Then I used systemtap to instrument xfs_ilock and find there are at least 3
functions that lock ip->i_lock exclusively during write.

 From what I saw in xfs_file_dio_aio_write code, it uses EXCL only 
if there is unaligned IO   or there are cached pages to be 
invalidated after shared lock is obtained *but* it demotes that 
lock to SHARED just before generic_file_direct_write. 

Is there any popular FS that supports concurrent direct IO well?

Thanks,
Da

On Sat, Jan 14, 2012 at 6:45 AM, Raghavendra D Prabhu <
raghu.prabhu13 at gmail.com> wrote:

quoted

Hi Zheng,

quoted

* On Fri, Jan 13, 2012 at 04:41:16PM -0500, Zheng Da <
zhengda1936 at gmail.com> wrote:

quoted

quoted

Hello,

quoted

quoted

I'm looking for a FS in Linux that supports concurrent direct IO write.
ext4 supports concurrent direct IO read if we mount it with
dioread_nolock,
but doesn't support concurrent writes. XFS doesn't support concurrent
direct IO at all. It locks the inode exclusive if it's direct IO. I tried
btrfs, and it seems it doesn't support concurrent direct IO either though
I
haven't looked into its code.
Is there a local FS that support concurrent direct IO write? It seems NFS
supports it (
http://kevinclosson.wordpress.**com/2011/08/12/file-systems-**
for-a-database-choose-one-**that-couples-direct-io-and-**
concurrent-io-whats-this-have-**to-do-with-nfs-harken-back-5-**
2-years-to-find-out/<http://kevinclosson.wordpress.com/2011/08/12/file-systems-for-a-database-choose-one-that-couples-direct-io-and-concurrent-io-whats-this-have-to-do-with-nfs-harken-back-5-2-years-to-find-out/>
),
but I'm looking for local FS.

quoted

quoted

Thanks,
Da

quoted

 ______________________________**_________________

quoted

Kernelnewbies mailing list
Kernelnewbies at kernelnewbies.**org [off-list ref]
http://lists.kernelnewbies.**org/mailman/listinfo/**kernelnewbies<http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies>

quoted

XFS locks inode exclusive only if it is an unaligned Direct IO, which is
apparently done to prevent race conditions -- refer to this
http://oss.sgi.com/archives/**xfs/2011-01/msg00157.html<http://oss.sgi.com/archives/xfs/2011-01/msg00157.html>Also the behavior of Ext4 under dioread_nolock is supported by XFS by
default and in a much better way. Also Ext4 is the only one which uses
DIO_LOCKING while doing direct io.

quoted

Regards,
--
Raghavendra Prabhu
GPG Id : 0xD72BE977
Fingerprint: B93F EBCB 8E05 7039 CD3C A4B8 A616 DCA1 D72B E977
www: wnohang.net

Regards,
-- 
Raghavendra Prabhu
GPG Id : 0xD72BE977
Fingerprint: B93F EBCB 8E05 7039 CD3C A4B8 A616 DCA1 D72B E977
www: wnohang.net
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 490 bytes
Desc: not available
Url : http://lists.kernelnewbies.org/pipermail/kernelnewbies/attachments/20120116/80d92106/attachment.bin