On Wed, 2011-03-30 at 13:17 +1100, Dave Chinner wrote:

On Tue, Mar 29, 2011 at 05:33:30PM -0700, Mingming Cao wrote:

quoted

Ric,

May I propose some discussion about concurrent direct IO support for
ext4?

Just look at the way XFS does it and copy that?  i.e. it has a
filesytem level IO lock and an inode lock both with shared/exclusive
semantics. These lie below the i_mutex (i.e. locking order is
i_mutex, i_iolock, i_ilock), and effectively result in the i_mutex
only being used for VFS level synchronisation and as such is rarely
used inside XFS itself.

Inode attribute operations are protected by the inode lock, while IO
operations and truncation synchronisation is provided by the IO
lock.

Right, inode attribute operations should be covered by the i_lock. in
ext4 the i_mutex is used to protect IO and truncation synch, along with
the i_datasem to pretect concurrent access.modification to file's
allocation. 

So for buffered IO, the IO lock is used in shared mode for reads
and exclusive mode for writes. This gives normal POSIX buffered IO
semantics and holding the IO lock exclusive allows sycnhronisation
against new IO of any kind for truncate.

For direct IO, the IO lock is always taken in shared mode, so we can
have concurrent read and write operations taking place at once
regardless of the offset into the file.

thanks for reminding me,in xfs concurrent direct IO write to the same
offset is allowed.

quoted

I am looking for some discussion about removing the i_mutex lock in the
direct IO write code path for ext4, when multiple threads are
direct write to different offset of the same file. This would require
some way to track the in-fly DIO IO range, either done at ext4 level or
above th vfs layer. 

Direct IO semantics have always been that the application is allowed
to overlap IO to the same range if it wants to. The result is
undefined (just like issuing overlapping reads and writes to a disk
at the same time) so it's the application's responsibility to avoid
overlapping IO if it is a problem.

I was thinking along the line to provide finer granularity lock to allow
concurrent direct IO to different offset/range, but to same offset, they
have to be serialized. If it's undefined behavior, i.e. overlapping is
allowed, then concurrent dio implementation is much easier. But not sure
if any apps currently using DIO aware of the ordering has to be done at
the application level. 

Cheers,

Dave.