This series adds nonblocking feature to asynchronous I/O writes.
io_submit() can be delayed because of a number of reason:
- Block allocation for files
- Data writebacks for direct I/O
- Sleeping because of waiting to acquire i_rwsem
- Congested block device
The goal of the patch series is to return -EAGAIN/-EWOULDBLOCK if
any of these conditions are met. This way userspace can push most
of the write()s to the kernel to the best of its ability to complete
and if it returns -EAGAIN, can defer it to another thread.
In order to enable this, IOCB_RW_FLAG_NOWAIT is introduced in
uapi/linux/aio_abi.h. If set for aio_rw_flags, it translates to
IOCB_NOWAIT for struct iocb, REQ_NOWAIT for bio.bi_opf and IOMAP_NOWAIT for
iomap. aio_rw_flags is a new flag replacing aio_reserved1. We could
not use aio_flags because it is not currently checked for invalidity
in the kernel.
This feature is provided for direct I/O of asynchronous I/O only. I have
tested it against xfs, ext4, and btrfs while I intend to add more filesystems.
The nowait feature is for request based devices. In the future, I intend to
add support to stacked devices such as md.
Applications will have to check supportability
by sending a async direct write and any other error besides -EAGAIN
would mean it is not supported.
First two patches are prep patches into nowait I/O.
Changes since v1:
+ changed name from _NONBLOCKING to *_NOWAIT
+ filemap_range_has_page call moved to closer to (just before) calling filemap_write_and_wait_range().
+ BIO_NOWAIT limited to get_request()
+ XFS fixes
- included reflink
- use of xfs_ilock_nowait() instead of a XFS_IOLOCK_NONBLOCKING flag
- Translate the flag through IOMAP_NOWAIT (iomap) to check for
block allocation for the file.
+ ext4 coding style
Changes since v2:
+ Using aio_reserved1 as aio_rw_flags instead of aio_flags
+ blk-mq support
+ xfs uptodate with kernel and reflink changes
Changes since v3:
+ Added FS_NOWAIT, which is set if the filesystem supports NOWAIT feature.
+ Checks in generic_make_request() to make sure BIO_NOWAIT comes in
for async direct writes only.
+ Added QUEUE_FLAG_NOWAIT, which is set if the device supports BIO_NOWAIT.
This is added (rather not set) to block devices such as dm/md currently.
Changes since v4:
+ Ported AIO code to use RWF_* flags. Check for RWF_* flags in
generic_file_write_iter().
+ Changed IOCB_RW_FLAGS_NOWAIT to RWF_NOWAIT.
Changes since v5:
+ BIO_NOWAIT to REQ_NOWAIT
+ Common helper for RWF flags.
Changes since v6:
+ REQ_NOWAIT will be ignored for request based devices since they
cannot block. So, removed QUEUE_FLAG_NOWAIT since it is not
required in the current implementation. It will be resurrected
when we program for stacked devices.
+ changed kiocb_rw_flags() to kiocb_set_rw_flags() in order to accomodate
for errors. Moved checks in the function.
Changes since v7:
+ split patches into prep so the main patches are smaller and easier
to understand
+ All patches are reviewed or acked!
Changes since v8:
+ Err out AIO reads with -EINVAL flagged as RWF_NOWAIT
Changes since v9:
+ Retract - Err out AIO reads with -EINVAL flagged as RWF_NOWAIT
+ XFS returns EAGAIN if extent list is not in memory
+ Man page updates to io_submit with iocb description and nowait features.
Changes since v10:
+ Corrected comment and subject in "return on congested block device"
--
Goldwyn
From: Goldwyn Rodrigues <redacted>
filemap_range_has_page() return true if the file's mapping has
a page within the range mentioned. This function will be used
to check if a write() call will cause a writeback of previous
writes.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Jan Kara <jack@suse.cz>
Signed-off-by: Goldwyn Rodrigues <redacted>
---
include/linux/fs.h | 2 ++
mm/filemap.c | 33 +++++++++++++++++++++++++++++++++
2 files changed, 35 insertions(+)
From: Goldwyn Rodrigues <redacted>
aio_rw_flags is introduced in struct iocb (using aio_reserved1) which will
carry the RWF_* flags. We cannot use aio_flags because they are not
checked for validity which may break existing applications.
Note, the only place RWF_HIPRI comes in effect is dio_await_one().
All the rest of the locations, aio code return -EIOCBQUEUED before the
checks for RWF_HIPRI.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Jan Kara <jack@suse.cz>
Signed-off-by: Goldwyn Rodrigues <redacted>
---
fs/aio.c | 8 +++++++-
include/uapi/linux/aio_abi.h | 2 +-
2 files changed, 8 insertions(+), 2 deletions(-)
@@ -79,7 +79,7 @@ struct io_event {structiocb{/* these are internal to the kernel/libc. */__u64aio_data;/* data to be returned in event's data */-__u32PADDED(aio_key,aio_reserved1);+__u32PADDED(aio_key,aio_rw_flags);/* the kernel sets aio_key to the req # *//* common fields */
From: Goldwyn Rodrigues <redacted>
Find out if the write will trigger a wait due to writeback. If yes,
return -EAGAIN.
Return -EINVAL for buffered AIO: there are multiple causes of
delay such as page locks, dirty throttling logic, page loading
from disk etc. which cannot be taken care of.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Jan Kara <jack@suse.cz>
Signed-off-by: Goldwyn Rodrigues <redacted>
---
mm/filemap.c | 17 ++++++++++++++---
1 file changed, 14 insertions(+), 3 deletions(-)
@@ -2743,9 +2746,17 @@ generic_file_direct_write(struct kiocb *iocb, struct iov_iter *from)write_len=iov_iter_count(from);end=(pos+write_len-1)>>PAGE_SHIFT;-written=filemap_write_and_wait_range(mapping,pos,pos+write_len-1);-if(written)-gotoout;+if(iocb->ki_flags&IOCB_NOWAIT){+/* If there are pages to writeback, return */+if(filemap_range_has_page(inode->i_mapping,pos,+pos+iov_iter_count(from)))+return-EAGAIN;+}else{+written=filemap_write_and_wait_range(mapping,pos,+pos+write_len-1);+if(written)+gotoout;+}/**Afterawritewewantbufferedreadstobesuretogotodisktoget
From: Goldwyn Rodrigues <redacted>
RWF_NOWAIT informs kernel to bail out if an AIO request will block
for reasons such as file allocations, or a writeback triggered,
or would block while allocating requests while performing
direct I/O.
RWF_NOWAIT is translated to IOCB_NOWAIT for iocb->ki_flags.
The check for -EOPNOTSUPP is placed in generic_file_write_iter(). This
is called by most filesystems, either through fsops.write_iter() or through
the function defined by write_iter(). If not, we perform the check defined
by .write_iter() which is called for direct IO specifically.
Filesystems xfs, btrfs and ext4 would be supported in the following patches.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Jan Kara <jack@suse.cz>
Signed-off-by: Goldwyn Rodrigues <redacted>
---
fs/9p/vfs_file.c | 3 +++
fs/aio.c | 6 ++++++
fs/ceph/file.c | 3 +++
fs/cifs/file.c | 3 +++
fs/fuse/file.c | 3 +++
fs/nfs/direct.c | 3 +++
fs/ocfs2/file.c | 3 +++
include/linux/fs.h | 5 ++++-
include/uapi/linux/fs.h | 1 +
mm/filemap.c | 3 +++
10 files changed, 32 insertions(+), 1 deletion(-)
From: Goldwyn Rodrigues <redacted>
Return EAGAIN if any of the following checks fail for direct I/O:
+ i_rwsem is lockable
+ Writing beyond end of file (will trigger allocation)
+ Blocks are not allocated at the write location
Signed-off-by: Goldwyn Rodrigues <redacted>
Reviewed-by: Jan Kara <jack@suse.cz>
---
fs/ext4/file.c | 20 ++++++++++++++++----
1 file changed, 16 insertions(+), 4 deletions(-)
@@ -235,9 +241,15 @@ ext4_file_write_iter(struct kiocb *iocb, struct iov_iter *from)iocb->private=&overwrite;/* Check whether we do a DIO overwrite or not */-if(o_direct&&ext4_should_dioread_nolock(inode)&&!unaligned_aio&&-ext4_overwrite_io(inode,iocb->ki_pos,iov_iter_count(from)))-overwrite=1;+if(o_direct&&!unaligned_aio){+if(ext4_overwrite_io(inode,iocb->ki_pos,iov_iter_count(from))){+if(ext4_should_dioread_nolock(inode))+overwrite=1;+}elseif(iocb->ki_flags&IOCB_NOWAIT){+ret=-EAGAIN;+gotoout;+}+}ret=__generic_file_write_iter(iocb,from);inode_unlock(inode);
From: Goldwyn Rodrigues <redacted>
A new bio operation flag REQ_NOWAIT is introduced to identify bio's
orignating from iocb with IOCB_NOWAIT. This flag indicates
to return immediately if a request cannot be made instead
of retrying.
Stacked devices such as md (the ones with make_request_fn hooks)
currently are not supported because it may block for housekeeping.
For example, an md can have a part of the device suspended.
For this reason, only request based devices are supported.
In the future, this feature will be expanded to stacked devices
by teaching them how to handle the REQ_NOWAIT flags.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Goldwyn Rodrigues <redacted>
---
block/blk-core.c | 23 +++++++++++++++++++++--
block/blk-mq-sched.c | 3 +++
block/blk-mq.c | 2 ++
fs/direct-io.c | 10 ++++++++--
include/linux/bio.h | 6 ++++++
include/linux/blk_types.h | 2 ++
6 files changed, 42 insertions(+), 4 deletions(-)
@@ -1900,6 +1905,16 @@ generic_make_request_checks(struct bio *bio)gotoend_io;}+/*+*ForaREQ_NOWAITbasedrequest,return-EOPNOTSUPP+*ifqueueisnotarequestbasedqueue.+*/++if((bio->bi_opf&REQ_NOWAIT)&&!queue_is_rq_based(q)){+err=-EOPNOTSUPP;+gotoend_io;+}+part=bio->bi_bdev->bd_part;if(should_fail_request(part,bio->bi_iter.bi_size)||should_fail_request(&part_to_disk(part)->part0,
@@ -2057,7 +2072,7 @@ blk_qc_t generic_make_request(struct bio *bio)do{structrequest_queue*q=bdev_get_queue(bio->bi_bdev);-if(likely(blk_queue_enter(q,false)==0)){+if(likely(blk_queue_enter(q,bio->bi_opf&REQ_NOWAIT)==0)){structbio_listlower,same;/* Create a fresh bio_list for all subordinate requests */
@@ -2082,7 +2097,11 @@ blk_qc_t generic_make_request(struct bio *bio)bio_list_merge(&bio_list_on_stack[0],&same);bio_list_merge(&bio_list_on_stack[0],&bio_list_on_stack[1]);}else{-bio_io_error(bio);+if(unlikely(!blk_queue_dying(q)&&+(bio->bi_opf&REQ_NOWAIT)))+bio_wouldblock_error(bio);+else+bio_io_error(bio);}bio=bio_list_pop(&bio_list_on_stack[0]);}while(bio);
@@ -480,8 +480,12 @@ static int dio_bio_complete(struct dio *dio, struct bio *bio)unsignedi;interr;-if(bio->bi_error)-dio->io_error=-EIO;+if(bio->bi_error){+if(bio->bi_error==-EAGAIN&&(bio->bi_opf&REQ_NOWAIT))+dio->io_error=-EAGAIN;+else+dio->io_error=-EIO;+}if(dio->is_async&&dio->op==REQ_OP_READ&&dio->should_dirty){err=bio->bi_error;
@@ -205,6 +205,7 @@ enum req_flag_bits {/* command specific flags for REQ_OP_WRITE_ZEROES: */__REQ_NOUNMAP,/* do not free blocks when zeroing */+__REQ_NOWAIT,/* Don't wait if request will block */__REQ_NR_BITS,/* stops here */};
From: Goldwyn Rodrigues <redacted>
Return EAGAIN if any of the following checks fail
+ i_rwsem is not lockable
+ NODATACOW or PREALLOC is not set
+ Cannot nocow at the desired location
+ Writing beyond end of file which is not allocated
Acked-by: David Sterba <dsterba@suse.com>
Signed-off-by: Goldwyn Rodrigues <redacted>
---
fs/btrfs/file.c | 25 ++++++++++++++++++++-----
fs/btrfs/inode.c | 3 +++
2 files changed, 23 insertions(+), 5 deletions(-)
From: Goldwyn Rodrigues <redacted>
IOCB_NOWAIT translates to IOMAP_NOWAIT for iomaps.
This is used by XFS in the XFS patch.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Jan Kara <jack@suse.cz>
Signed-off-by: Goldwyn Rodrigues <redacted>
---
fs/iomap.c | 2 ++
include/linux/iomap.h | 1 +
2 files changed, 3 insertions(+)
From: Goldwyn Rodrigues <redacted>
If IOCB_NOWAIT is set, bail if the i_rwsem is not lockable
immediately.
IF IOMAP_NOWAIT is set, return EAGAIN in xfs_file_iomap_begin
if it needs allocation either due to file extension, writing to a hole,
or COW or waiting for other DIOs to finish.
Return -EAGAIN if we don't have extent list in memory.
Signed-off-by: Goldwyn Rodrigues <redacted>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Darrick J. Wong <redacted>
---
fs/xfs/xfs_file.c | 19 ++++++++++++++-----
fs/xfs/xfs_iomap.c | 22 ++++++++++++++++++++++
2 files changed, 36 insertions(+), 5 deletions(-)
@@ -553,9 +556,15 @@ xfs_file_dio_aio_write(*otherwisedemotethelockifwehadtotaketheexclusivelock*forotherreasonsinxfs_file_aio_write_checks.*/-if(unaligned_io)-inode_dio_wait(inode);-elseif(iolock==XFS_IOLOCK_EXCL){+if(unaligned_io){+/* If we are going to wait for other DIO to finish, bail */+if(iocb->ki_flags&IOCB_NOWAIT){+if(atomic_read(&inode->i_dio_count))+return-EAGAIN;+}else{+inode_dio_wait(inode);+}+}elseif(iolock==XFS_IOLOCK_EXCL){xfs_ilock_demote(ip,XFS_IOLOCK_EXCL);iolock=XFS_IOLOCK_SHARED;}
@@ -1016,6 +1021,15 @@ xfs_file_iomap_begin(if((flags&(IOMAP_WRITE|IOMAP_ZERO))&&xfs_is_reflink_inode(ip)){if(flags&IOMAP_DIRECT){+/*+*AreflinkedinodewillresultinCoWalloc.+*FIXME:Itcouldstilloverwriteonunsharedextents+*andnotneedallocation.+*/+if(flags&IOMAP_NOWAIT){+error=-EAGAIN;+gotoout_unlock;+}/* may drop and re-acquire the ilock */error=xfs_reflink_allocate_cow(ip,&imap,&shared,&lockmode);
From: Christoph Hellwig <hch@infradead.org> Date: 2017-06-08 07:39:14
As already indicated this whole series looks fine to me.
Al: are you going to pick this up? Or Andrew?
On Tue, Jun 06, 2017 at 06:19:29AM -0500, Goldwyn Rodrigues wrote:
This series adds nonblocking feature to asynchronous I/O writes.
io_submit() can be delayed because of a number of reason:
- Block allocation for files
- Data writebacks for direct I/O
- Sleeping because of waiting to acquire i_rwsem
- Congested block device
The goal of the patch series is to return -EAGAIN/-EWOULDBLOCK if
any of these conditions are met. This way userspace can push most
of the write()s to the kernel to the best of its ability to complete
and if it returns -EAGAIN, can defer it to another thread.
In order to enable this, IOCB_RW_FLAG_NOWAIT is introduced in
uapi/linux/aio_abi.h. If set for aio_rw_flags, it translates to
IOCB_NOWAIT for struct iocb, REQ_NOWAIT for bio.bi_opf and IOMAP_NOWAIT for
iomap. aio_rw_flags is a new flag replacing aio_reserved1. We could
not use aio_flags because it is not currently checked for invalidity
in the kernel.
This feature is provided for direct I/O of asynchronous I/O only. I have
tested it against xfs, ext4, and btrfs while I intend to add more filesystems.
The nowait feature is for request based devices. In the future, I intend to
add support to stacked devices such as md.
Applications will have to check supportability
by sending a async direct write and any other error besides -EAGAIN
would mean it is not supported.
First two patches are prep patches into nowait I/O.
Changes since v1:
+ changed name from _NONBLOCKING to *_NOWAIT
+ filemap_range_has_page call moved to closer to (just before) calling filemap_write_and_wait_range().
+ BIO_NOWAIT limited to get_request()
+ XFS fixes
- included reflink
- use of xfs_ilock_nowait() instead of a XFS_IOLOCK_NONBLOCKING flag
- Translate the flag through IOMAP_NOWAIT (iomap) to check for
block allocation for the file.
+ ext4 coding style
Changes since v2:
+ Using aio_reserved1 as aio_rw_flags instead of aio_flags
+ blk-mq support
+ xfs uptodate with kernel and reflink changes
Changes since v3:
+ Added FS_NOWAIT, which is set if the filesystem supports NOWAIT feature.
+ Checks in generic_make_request() to make sure BIO_NOWAIT comes in
for async direct writes only.
+ Added QUEUE_FLAG_NOWAIT, which is set if the device supports BIO_NOWAIT.
This is added (rather not set) to block devices such as dm/md currently.
Changes since v4:
+ Ported AIO code to use RWF_* flags. Check for RWF_* flags in
generic_file_write_iter().
+ Changed IOCB_RW_FLAGS_NOWAIT to RWF_NOWAIT.
Changes since v5:
+ BIO_NOWAIT to REQ_NOWAIT
+ Common helper for RWF flags.
Changes since v6:
+ REQ_NOWAIT will be ignored for request based devices since they
cannot block. So, removed QUEUE_FLAG_NOWAIT since it is not
required in the current implementation. It will be resurrected
when we program for stacked devices.
+ changed kiocb_rw_flags() to kiocb_set_rw_flags() in order to accomodate
for errors. Moved checks in the function.
Changes since v7:
+ split patches into prep so the main patches are smaller and easier
to understand
+ All patches are reviewed or acked!
Changes since v8:
+ Err out AIO reads with -EINVAL flagged as RWF_NOWAIT
Changes since v9:
+ Retract - Err out AIO reads with -EINVAL flagged as RWF_NOWAIT
+ XFS returns EAGAIN if extent list is not in memory
+ Man page updates to io_submit with iocb description and nowait features.
Changes since v10:
+ Corrected comment and subject in "return on congested block device"
--
Goldwyn
From: Al Viro <viro@ZenIV.linux.org.uk> Date: 2017-06-10 05:27:08
On Tue, Jun 06, 2017 at 06:19:33AM -0500, Goldwyn Rodrigues wrote:
From: Goldwyn Rodrigues <rgoldwyn-IBi9RG/b67k@public.gmane.org>
RWF_NOWAIT informs kernel to bail out if an AIO request will block
for reasons such as file allocations, or a writeback triggered,
or would block while allocating requests while performing
direct I/O.
RWF_NOWAIT is translated to IOCB_NOWAIT for iocb->ki_flags.
The check for -EOPNOTSUPP is placed in generic_file_write_iter(). This
is called by most filesystems, either through fsops.write_iter() or through
the function defined by write_iter(). If not, we perform the check defined
by .write_iter() which is called for direct IO specifically.
Filesystems xfs, btrfs and ext4 would be supported in the following patches.
Umm... What about ->write_iter() instances outside of fs/*? Even in fs/*,
consider e.g.
int cifs_get_writer(struct cifsInodeInfo *cinode)
{
int rc;
start:
rc = wait_on_bit(&cinode->flags, CIFS_INODE_PENDING_OPLOCK_BREAK,
TASK_KILLABLE);
and cifs_file_write_iter() calling it before going to generic_file_write_iter().
Ditto for cifs_struct_writev()... coda_file_write_iter() does inode_lock()
before calling vfs_iter_write(). ext2_dax_write_iter(): inode_lock().
f2fs_file_write_iter(): ditto. fuse_file_write_iter(): ditto in case when
->writeback_cache is false. gfs2 is O_APPEND case: almost certainly blocks.
ncp_file_write_iter(): blocks (mutex_lock(&NCP_FINFO(inode)->open_mutex)
in ncp_make_open(), not to mention anything else). ntfs_file_write_iter():
inode_lock(). orangefs_file_write_iter(): ditto. ubifs_write_iter():
may block in update_mctime(). udf_file_write_iter(): inode_lock().
Lustre sure as hell does block before it gets anywhere near mm/filemap.c.
And that - just from looking at regular files. Then we have sockets and
pipes, not to mention weird stuff like fs/fuse/cuse.c, etc.
From: Al Viro <viro@ZenIV.linux.org.uk> Date: 2017-06-10 05:34:45
On Thu, Jun 08, 2017 at 12:39:10AM -0700, Christoph Hellwig wrote:
As already indicated this whole series looks fine to me.
Al: are you going to pick this up? Or Andrew?
The main issue here is "let's bail out from ->write_iter() instances"
patch. It very obviously has holes in coverage.
Could we have FMODE_AIO_NOWAIT and make those who claim to support it
set that in ->open()? And make aio check that and bail out if asked
for nowait on a file without that flag...
On Thu, Jun 08, 2017 at 12:39:10AM -0700, Christoph Hellwig wrote:
quoted
As already indicated this whole series looks fine to me.
Al: are you going to pick this up? Or Andrew?
The main issue here is "let's bail out from ->write_iter() instances"
patch. It very obviously has holes in coverage.
Could we have FMODE_AIO_NOWAIT and make those who claim to support it
set that in ->open()? And make aio check that and bail out if asked
for nowait on a file without that flag...
Yes, I would agree.
We had FS_NOWAIT in filesystem type flags (in v3), but retracted it
later in v4.
Another option could be to keep the feature against FS_REQUIRES_DEV to
rule out filesystems which are not local, but it again has the problem
of holes in coverage.
I will work on adding FMODE_AIO_NOWAIT in the meantime.
Thanks,
--
Goldwyn
From: Christoph Hellwig <hch@infradead.org> Date: 2017-06-13 06:14:36
On Mon, Jun 12, 2017 at 05:38:13PM -0500, Goldwyn Rodrigues wrote:
We had FS_NOWAIT in filesystem type flags (in v3), but retracted it
later in v4.
A per-fs flag is wrong as file_operation may have different
capabilities.
I will work on adding FMODE_AIO_NOWAIT in the meantime.
If Al prefers that let's go with it for now. One other thing I'd
love is a supported flags field in struct file_operations that we
can compare the flags against. Al, does that sound good as well?
From: Al Viro <viro@ZenIV.linux.org.uk> Date: 2017-06-15 11:12:03
On Mon, Jun 12, 2017 at 11:14:31PM -0700, Christoph Hellwig wrote:
On Mon, Jun 12, 2017 at 05:38:13PM -0500, Goldwyn Rodrigues wrote:
quoted
We had FS_NOWAIT in filesystem type flags (in v3), but retracted it
later in v4.
A per-fs flag is wrong as file_operation may have different
capabilities.
quoted
I will work on adding FMODE_AIO_NOWAIT in the meantime.
If Al prefers that let's go with it for now. One other thing I'd
love is a supported flags field in struct file_operations that we
can compare the flags against. Al, does that sound good as well?
Which flags are you talking about? aio ones? AFAICS, it's the same
kind of thing as "can we lseek?" or "can we read/pread?", etc.
What would that field look like? Note that some of those might depend
upon the flags passed to open(), so shoving them into file_operations
might mean splitting file_operations instances... Details, please.
From: Christoph Hellwig <hch@infradead.org> Date: 2017-06-15 14:55:44
On Thu, Jun 15, 2017 at 12:11:58PM +0100, Al Viro wrote:
Which flags are you talking about? aio ones? AFAICS, it's the same
kind of thing as "can we lseek?" or "can we read/pread?", etc.
What would that field look like? Note that some of those might depend
upon the flags passed to open(), so shoving them into file_operations
might mean splitting file_operations instances... Details, please.