Re: [PATCH] epoll: add exclusive wakeups flag

Hi,

On 01/28/2016 02:16 AM, Michael Kerrisk (man-pages) wrote:

Hi Jason,

On 12/08/2015 04:23 AM, Jason Baron wrote:

quoted

Hi,

Re-post of an old series addressing thundering herd issues when sharing
an event source fd amongst multiple epoll fds. Last posting was here
for reference: https://lkml.org/lkml/2015/2/25/56
 
The patch herein drops the core scheduler 'rotate' changes I had previously
proposed as this patch seems performant without those.

I was prompted to re-post this because Madars Vitolins reported some good
speedups with this patch using Enduro/X application. His writeup is here:
https://mvitolin.wordpress.com/2015/12/05/endurox-testing-epollexclusive-flag/

Thanks,

-Jason

Sample epoll_clt text:

Thanks for the proposed text. I have some questions about points
that are not quite clear to me.

quoted

EPOLLEXCLUSIVE
        Sets an exclusive wakeup mode for the epfd file descriptor that is
	being attached to the target file descriptor, fd. Thus, when an
	event occurs and multiple epfd file descriptors are attached to the
	same target file using EPOLLEXCLUSIVE, one or more epfds will receive
	an event with epoll_wait(2). The default in this scenario (when
	EPOLLEXCLUSIVE is not set) is for all epfds to receive an event.
	EPOLLEXLUSVIE may only be specified with the op EPOLL_CTL_ADD.

So, assuming an FD is present in the interest list of multiple (say 6)
epoll FDs, and some (say 3) of those attachments were done using
EPOLLEXCLUSVE. Which of the following statements are correct:

(a) It's guaranteed that *none* of the epoll FDs that did NOT specify
    EPOLLEXCLUSIVE will receive an event.

(b) It's guaranteed that *all* of the epoll FDs that did NOT specify
    EPOLLEXCLUSIVE will receive an event.

(c) From 1 to 3 of the epoll FDs that did specify EPOLLEXCLUSIVE
    will receive an event.

(d) Exactly one epoll FD that did specify EPOLLEXCLUSIVE will get
    an event, and it is indeterminate which one.

So b and c. All the non-exclusive adds will get it and at least 1 of the
exclusive adds will as well.

I suppose one point I'm trying to uncover in the above is: what is
the scope of EPOLLEXCLUSIVE? Is it just applicable for one process's
FD, or is it setting an attribute in the epoll "interest list" record
for that FD that affects notification behavior across all processes?

Right - so 'EPOLLEXCLUSIVE' will affect other epoll sets that are also
using 'EPOLLEXCLUSIVE' against the the same fd, but will have no affect
on epoll sets connected to fd that do not specify it.

And then:

(1) What are the semantics of EPOLLEXCLUSIVE if the added FD becomes
    disabled via EPOLLONESHOT (or explicitly via EPOLL_CTL_MOD with
    the 'events' field set to 0)?

In the case of EPOLLEXCLUSIVE and EPOLLONESHOT, one would have to re-arm
at least 1 of threads that was woken up by doing EPOLL_CTL_MOD to
guarantee further wakeups.

And like-wise with an EPOLL_CTL_MOD with 'events' all set to 0, one
would need to either re-arm the thread that set the 'events' field to 0
(by setting back to non-zero), or re-arm in at least one other thread
via EPOLL_CTL_MOD (or delete and add).

(2) The source code contains a comment "we do not currently supported 
    nested exclusive wakeups". Could you elaborate on this point? It
    sounds like something that should be documented.

So I was just trying to say that we return -EINVAL if you try to do and
EPOLL_CTL_ADD with EPOLLEXCLUSIVE and the 'fd' argument is a epoll fd
returned via epoll_create().

Thanks,

-Jason