Re: [RFC PATCH 4/4 v0.3] sched/umcg: RFC: implement UMCG syscalls

On Mon, Jul 19, 2021 at 11:13 AM Thierry Delisle [off-list ref] wrote:

 > Latency/efficiency: on worker wakeup an idle server can be picked from
 > the list and context-switched into synchronously, on the same CPU.
 > Using FDs and select/poll/epoll will add extra layers of abstractions;
 > synchronous context-switches (not yet fully implemented in UMCG) will
 > most likely be impossible. This patchset seems much more efficient and
 > lightweight than whatever can be built on top of FDs.

I can believe that.

Are you planning to support separate scheduling instances within a
single user
space? That is having multiple sets of server threads and workers can
only run
within a specific set.

Yes, this is naturally supported in the current patchset on the kernel
side, and is supported in libumcg (to be posted, later when the kernel
side is settled); internally at Google, some applications use
different "groups" of workers/servers per NUMA node.

I believe the problem with the idle_servers_ptr as specified is that it
is not
possible to reclaim used nodes safely. I don't see any indication of which
nodes the kernel can concurrently access and on which some memory
reclamation
scheme could be based.

Please see the attached atomic_stack.h file - I use it in my tests,
things seem to be working. Specifically, atomic_stack_gc does the
cleanup. For the kernel side of things, see the third patch in this
patchset.

What is the benefit of having users maintain themselves a list of idle
servers
rather than each servers marking themselves as 'out of work' and having the
kernel maintain the list?

To keep the kernel side light and simple. To also protect the kernel
from spinning if userspace misbehaves. Basically, the overall approach
is to delegate most of the work to the userspace, and keep the bare
minimum in the kernel.