Re: aio poll, io_pgetevents and a new in-kernel poll API V3

Jeff Moyer [off-list ref] writes:

FYI, this kernel has issues.  It will boot up, but I don't have
networking, and even rebooting doesn't succeed.  I'm looking into it.

A bisect lands on: eventfd: switch to ->poll_mask.  That's not super
helpful, though.  I did run the ltp eventfd2 tests, and they all pass.

The actual issue I get on boot is that several services don't start:

[FAILED] Failed to start Modem Manager.
See 'systemctl status ModemManager.service' for details.
[FAILED] Failed to start Authorization Manager.
See 'systemctl status polkit.service' for details.
[DEPEND] Dependency failed for Dynamic System Tuning Daemon.
[FAILED] Failed to start Network Manager.
See 'systemctl status NetworkManager.service' for details.
[DEPEND] Dependency failed for Network Manager Wait Online.
...

Christoph, are you able to reproduce this?

-Jeff

Christoph Hellwig [off-list ref] writes:

quoted

Hi all,

this series adds support for the IOCB_CMD_POLL operation to poll for the
readyness of file descriptors using the aio subsystem.  The API is based
on patches that existed in RHAS2.1 and RHEL3, which means it already is
supported by libaio.  To implement the poll support efficiently new
methods to poll are introduced in struct file_operations:  get_poll_head
and poll_mask.  The first one returns a wait_queue_head to wait on
(lifetime is bound by the file), and the second does a non-blocking
check for the POLL* events.  This allows aio poll to work without
any additional context switches, unlike epoll.

To make the interface fully useful a new io_pgetevents system call is
added, which atomically saves and restores the signal mask over the
io_pgetevents system call.  It it the logical equivalent to pselect and
ppoll for io_pgetevents.

The corresponding libaio changes for io_pgetevents support and
documentation, as well as a test case will be posted in a separate
series.

The changes were sponsored by Scylladb, and improve performance
of the seastar framework up to 10%, while also removing the need
for a privileged SCHED_FIFO epoll listener thread.

The patches are on top of Als __poll_t annoations, so I've also
prepared a git branch on top of those here:

    git://git.infradead.org/users/hch/vfs.git aio-poll.3

Gitweb:

    http://git.infradead.org/users/hch/vfs.git/shortlog/refs/heads/aio-poll.3

Libaio changes:

    https://pagure.io/libaio.git io-poll

Seastar changes (not updated for the new io_pgetevens ABI yet):

    https://github.com/avikivity/seastar/commits/aio

Changes since V2:
 - removed a double initialization
 - new vfs_get_poll_head helper
 - document that ->get_poll_head can return NULL
 - call ->poll_mask before sleeping
 - various ACKs
 - add conversion of random to ->poll_mask
 - add conversion of af_alg to ->poll_mask
 - lacking ->poll_mask support now returns -EINVAL for IOCB_CMD_POLL
 - reshuffled the series so that prep patches and everything not
   requiring the new in-kernel poll API is in the beginning

Changes since V1:
 - handle the NULL ->poll case in vfs_poll
 - dropped the file argument to the ->poll_mask socket operation
 - replace the ->pre_poll socket operation with ->get_poll_head as
   in the file operations

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>

On Thu, Jan 18, 2018 at 11:44:03AM -0500, Jeff Moyer wrote:

Jeff Moyer [off-list ref] writes:

quoted

FYI, this kernel has issues.  It will boot up, but I don't have
networking, and even rebooting doesn't succeed.  I'm looking into it.

A bisect lands on: eventfd: switch to ->poll_mask.  That's not super
helpful, though.  I did run the ltp eventfd2 tests, and they all pass.

The actual issue I get on boot is that several services don't start:

[...]

Christoph, are you able to reproduce this?

No, I can't reproduce any of that.  But I don't have a Fedora system
either, so this might be a new systemd version doing funky things.

The major change in this version was to call ->poll_mask before setting up
the wait queue as well.  This does the right thing for poll and aio poll,
but the more I dig into the epoll code the less sure I am it does the right
thing for it, or in fact that epoll does the right thing in general..

Do you still see it with the patch below applied?

diff --git a/fs/select.c b/fs/select.c
index 707abe79536b..1784c1a29253 100644
--- a/fs/select.c
+++ b/fs/select.c

@@ -53,9 +53,9 @@ __poll_t vfs_poll(struct file *file, struct poll_table_struct *pt)
 		head = vfs_get_poll_head(file, events);
 		if (!head)
 			return DEFAULT_POLLMASK;
-		mask = file->f_op->poll_mask(file, events);
-		if (mask)
-			return mask;
+//		mask = file->f_op->poll_mask(file, events);
+//		if (mask)
+//			return mask;
 
 		pt->_qproc(file, head, pt);
 	}

--

To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>

On Thu, Jan 18, 2018, at 11:44 AM, Jeff Moyer wrote:

Jeff Moyer [off-list ref] writes:

quoted

FYI, this kernel has issues.  It will boot up, but I don't have
networking, and even rebooting doesn't succeed.  I'm looking into it.

A bisect lands on: eventfd: switch to ->poll_mask.  That's not super
helpful, though.  I did run the ltp eventfd2 tests, and they all pass.

FWIW: https://git.gnome.org/browse/glib/commit/?id=3904c8761a60dbadbdfaf98fe23ff19cbdcc4a9a

Since that a lot of userspace (including NetworkManager) uses eventfd. I haven't
tried this patchset myself but I'd look at what the GLib mainloop is doing.

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to majordomo@kvack.org.  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"aart@kvack.org">aart@kvack.org</a>

On Thu, Jan 18, 2018 at 12:55:47PM -0500, Colin Walters wrote:

FWIW: https://git.gnome.org/browse/glib/commit/?id=3904c8761a60dbadbdfaf98fe23ff19cbdcc4a9a

Since that a lot of userspace (including NetworkManager) uses eventfd. I haven't
tried this patchset myself but I'd look at what the GLib mainloop is doing.

eventfd really isn't the interesting piece, the interesting piece
is epoll, especially if used in level triggered mode.