From: Chris Metcalf <hidden> Date: 2015-05-08 17:58:56
The existing nohz_full mode does a nice job of suppressing extraneous
kernel interrupts for cores that desire it. However, there is a need
for a more deterministic mode that rigorously disallows kernel
interrupts, even at a higher cost in user/kernel transition time:
for example, high-speed networking applications running userspace
drivers that will drop packets if they are ever interrupted.
These changes attempt to provide an initial draft of such a framework;
the changes do not add any overhead to the usual non-nohz_full mode,
and only very small overhead to the typical nohz_full mode. A prctl()
option (PR_SET_DATAPLANE) is added to control whether processes have
requested this stricter semantics, and within that prctl() option we
provide a number of different bits for more precise control.
Additionally, we add a new command-line boot argument to facilitate
debugging where unexpected interrupts are being delivered from.
Code that is conceptually similar has been in use in Tilera's
Multicore Development Environment since 2008, known as Zero-Overhead
Linux, and has seen wide adoption by a range of customers. This patch
series represents the first serious attempt to upstream that
functionality. Although the current state of the kernel isn't quite
ready to run with absolutely no kernel interrupts (for example,
workqueues on dataplane cores still remain to be dealt with), this
patch series provides a way to make dynamic tradeoffs between avoiding
kernel interrupts on the one hand, and making voluntary calls in and
out of the kernel more expensive, for tasks that want it.
The series (based currently on my arch/tile master tree for 4.2,
in turn based on 4.1-rc1) is available at:
git://git.kernel.org/pub/scm/linux/kernel/git/cmetcalf/linux-tile.git dataplane
Chris Metcalf (6):
nohz_full: add support for "dataplane" mode
nohz: dataplane: allow tick to be fully disabled for dataplane
dataplane nohz: run softirqs synchronously on user entry
nohz: support PR_DATAPLANE_QUIESCE
nohz: support PR_DATAPLANE_STRICT mode
nohz: add dataplane_debug boot flag
Documentation/kernel-parameters.txt | 6 ++
arch/tile/mm/homecache.c | 5 +-
include/linux/sched.h | 3 +
include/linux/tick.h | 12 ++++
include/uapi/linux/prctl.h | 8 +++
kernel/context_tracking.c | 3 +
kernel/irq_work.c | 4 +-
kernel/sched/core.c | 18 ++++++
kernel/signal.c | 5 ++
kernel/smp.c | 4 ++
kernel/softirq.c | 15 ++++-
kernel/sys.c | 8 +++
kernel/time/tick-sched.c | 112 +++++++++++++++++++++++++++++++++++-
13 files changed, 198 insertions(+), 5 deletions(-)
--
2.1.2
From: Chris Metcalf <hidden> Date: 2015-05-08 17:59:05
The existing nohz_full mode makes tradeoffs to minimize userspace
interruptions while still attempting to avoid overheads in the
kernel entry/exit path, to provide 100% kernel semantics, etc.
However, some applications require a stronger commitment from the
kernel to avoid interruptions, in particular userspace device
driver style applications, such as high-speed networking code.
This change introduces a framework to allow applications to elect
to have the stronger semantics as needed, specifying
prctl(PR_SET_DATAPLANE, PR_DATAPLANE_ENABLE) to do so.
Subsequent commits will add additional flags and additional
semantics.
The dataplane state is indicated by setting a new task struct
field, dataplane_flags, to the value passed by prctl(). When the
_ENABLE bit is set for a task, and it is returning to userspace
on a nohz_full core, it calls the new tick_nohz_dataplane_enter()
routine to take additional actions to help the task avoid being
interrupted in the future.
For this first patch, the only action taken is to call
lru_add_drain() to prevent being interrupted by a subsequent
lru_add_drain_all() call on another core.
Signed-off-by: Chris Metcalf <redacted>
---
include/linux/sched.h | 3 +++
include/linux/tick.h | 10 ++++++++++
include/uapi/linux/prctl.h | 5 +++++
kernel/context_tracking.c | 3 +++
kernel/sys.c | 8 ++++++++
kernel/time/tick-sched.c | 13 +++++++++++++
6 files changed, 42 insertions(+)
From: Chris Metcalf <hidden> Date: 2015-05-08 17:59:19
With QUIESCE mode, the task is in principle guaranteed not to be
interrupted by the kernel, but only if it behaves. In particular,
if it enters the kernel via system call, page fault, or any of
a number of other synchronous traps, it may be unexpectedly
exposed to long latencies. Add a simple flag that puts the process
into a state where any such kernel entry is fatal.
To allow the state to be entered and exited, we add an internal
bit to current->dataplane_flags that is set when prctl() sets the
flags. That way, when we are exiting the kernel after calling
prctl() to forbid future kernel exits, we don't get immediately
killed.
Signed-off-by: Chris Metcalf <redacted>
---
include/uapi/linux/prctl.h | 2 ++
kernel/sys.c | 2 +-
kernel/time/tick-sched.c | 17 +++++++++++++++++
3 files changed, 20 insertions(+), 1 deletion(-)
@@ -464,6 +478,9 @@ void tick_nohz_dataplane_enter(void)if((current->dataplane_flags&PR_DATAPLANE_QUIESCE)!=0)dataplane_quiesce();+/* Clear the bit set by prctl() when it updates the flags. */+current->dataplane_flags&=~PR_DATAPLANE_PRCTL;+/**Disableinterruptsagainsinceothercoderunninginthis*functionmayhaveenabledthem,andthecallerexpects
From: Andy Lutomirski <luto@amacapital.net> Date: 2015-05-09 07:29:10
On May 8, 2015 11:44 PM, "Chris Metcalf" [off-list ref] wrote:
With QUIESCE mode, the task is in principle guaranteed not to be
interrupted by the kernel, but only if it behaves. In particular,
if it enters the kernel via system call, page fault, or any of
a number of other synchronous traps, it may be unexpectedly
exposed to long latencies. Add a simple flag that puts the process
into a state where any such kernel entry is fatal.
To allow the state to be entered and exited, we add an internal
bit to current->dataplane_flags that is set when prctl() sets the
flags. That way, when we are exiting the kernel after calling
prctl() to forbid future kernel exits, we don't get immediately
killed.
Is there any reason this can't already be addressed in userspace using
/proc/interrupts or perf_events? ISTM the real goal here is to detect
when we screw up and fail to avoid an interrupt, and killing the task
seems like overkill to me.
Also, can we please stop further torturing the exit paths? We have a
disaster of assembly code that calls into syscall_trace_leave and
do_notify_resume. Those functions, in turn, *both* call user_enter
(WTF?), and on very brief inspection user_enter makes it into the nohz
code through multiple levels of indirection, which, with these
patches, has yet another conditionally enabled helper, which does this
new stuff. It's getting to be impossible to tell what happens when we
exit to user space any more.
Also, I think your code is buggy. There's no particular guarantee
that user_enter is only called once between sys_prctl and the final
exit to user mode (see the above WTF), so you might spuriously kill
the process.
Also, I think that most users will be quite surprised if "strict
dataplane" code causes any machine check on the system to kill your
dataplane task. Similarly, a user accidentally running perf record -a
probably should have some reasonable semantics. /proc/interrupts gets
that right as is. Sure, MCEs will hurt your RT performance, but Intel
screwed up the way that MCEs work, so we should make do.
--Andy
From: Gilad Ben Yossef <hidden> Date: 2015-05-09 10:53:36
From: Andy Lutomirski [mailto:luto@amacapital.net]
Sent: Saturday, May 09, 2015 10:29 AM
To: Chris Metcalf
Cc: Srivatsa S. Bhat; Paul E. McKenney; Frederic Weisbecker; Ingo Molnar;
Rik van Riel; linux-doc@vger.kernel.org; Andrew Morton; linux-
kernel@vger.kernel.org; Thomas Gleixner; Tejun Heo; Peter Zijlstra; Steven
Rostedt; Christoph Lameter; Gilad Ben Yossef; Linux API
Subject: Re: [PATCH 5/6] nohz: support PR_DATAPLANE_STRICT mode
On May 8, 2015 11:44 PM, "Chris Metcalf" [off-list ref] wrote:
quoted
With QUIESCE mode, the task is in principle guaranteed not to be
interrupted by the kernel, but only if it behaves. In particular,
if it enters the kernel via system call, page fault, or any of
a number of other synchronous traps, it may be unexpectedly
exposed to long latencies. Add a simple flag that puts the process
into a state where any such kernel entry is fatal.
To allow the state to be entered and exited, we add an internal
bit to current->dataplane_flags that is set when prctl() sets the
flags. That way, when we are exiting the kernel after calling
prctl() to forbid future kernel exits, we don't get immediately
killed.
Is there any reason this can't already be addressed in userspace using
/proc/interrupts or perf_events? ISTM the real goal here is to detect
when we screw up and fail to avoid an interrupt, and killing the task
seems like overkill to me.
Also, can we please stop further torturing the exit paths?
So, I don't know if it is a practical suggestion or not, but would it better/easier to mark a pending signal on kernel entry for this case?
The upsides I see is that the user gets her notification (killing the task or just logging the event in a signal handler) and hopefully since return to userspace with a pending signal is already handled we don't need new code in the exit path?
Gilad
From: Chris Metcalf <hidden> Date: 2015-05-11 19:13:58
On 05/09/2015 03:28 AM, Andy Lutomirski wrote:
On May 8, 2015 11:44 PM, "Chris Metcalf" [off-list ref] wrote:
quoted
With QUIESCE mode, the task is in principle guaranteed not to be
interrupted by the kernel, but only if it behaves. In particular,
if it enters the kernel via system call, page fault, or any of
a number of other synchronous traps, it may be unexpectedly
exposed to long latencies. Add a simple flag that puts the process
into a state where any such kernel entry is fatal.
To allow the state to be entered and exited, we add an internal
bit to current->dataplane_flags that is set when prctl() sets the
flags. That way, when we are exiting the kernel after calling
prctl() to forbid future kernel exits, we don't get immediately
killed.
Is there any reason this can't already be addressed in userspace using
/proc/interrupts or perf_events? ISTM the real goal here is to detect
when we screw up and fail to avoid an interrupt, and killing the task
seems like overkill to me.
Patch 6/6 proposes a mechanism to track down times when the
kernel screws up and delivers an IRQ to a userspace-only task.
Here, we're just trying to identify the times when an application
screws itself up out of cluelessness, and provide a mechanism
that allows the developer to easily figure out why and fix it.
In particular, /proc/interrupts won't show syscalls or page faults,
which are two easy ways applications can screw themselves
when they think they're in userspace-only mode. Also, they don't
provide sufficient precision to make it clear what part of the
application caused the undesired kernel entry.
In this case, killing the task is appropriate, since that's exactly
the semantics that have been asked for - it's like on architectures
that don't natively support unaligned accesses, but fake it relatively
slowly in the kernel, and in development you just say "give me a
SIGBUS when that happens" and in production you might say
"fix it up and let's try to keep going".
You can argue that this is something that can be done by ftrace,
but certainly you'd want to have a way to programmatically
turn on ftrace at the moment when you're entering userspace-only
mode, so we'd want some API around that anyway. And honestly,
it's so easy to test a task state bit in a couple of places and
generate the failurel on the spot, vs. the relative complexity
of setting up and understanding ftrace, that I think it merits
inclusion on that basis alone.
Also, can we please stop further torturing the exit paths? We have a
disaster of assembly code that calls into syscall_trace_leave and
do_notify_resume. Those functions, in turn, *both* call user_enter
(WTF?), and on very brief inspection user_enter makes it into the nohz
code through multiple levels of indirection, which, with these
patches, has yet another conditionally enabled helper, which does this
new stuff. It's getting to be impossible to tell what happens when we
exit to user space any more.
Also, I think your code is buggy. There's no particular guarantee
that user_enter is only called once between sys_prctl and the final
exit to user mode (see the above WTF), so you might spuriously kill
the process.
This is a good point; I also find the x86 kernel entry and exit
paths confusing, although I've reviewed them a bunch of times.
The tile architecture paths are a little easier to understand.
That said, I think the answer here is avoid non-idempotent
actions in the dataplane code, such as clearing a syscall bit.
A better implementation, I think, is to put the tests for "you
screwed up and synchronously entered the kernel" in
the syscall_trace_enter() code, which TIF_NOHZ already
gets us into; there, we can test if the dataplane "strict" bit is
set and the syscall is not prctl(), then we generate the error.
(We'd exclude exit and exit_group here too, since we don't
need to shoot down a task that's just trying to kill itself.)
This needs a bit of platform-specific code for each platform,
but that doesn't seem like too big a problem.
Likewise we can test in exception_enter() since that's only
called for all the synchronous user entries like page faults.
Also, I think that most users will be quite surprised if "strict
dataplane" code causes any machine check on the system to kill your
dataplane task.
Fair point, and avoided by testing as described above instead.
(Though presumably in development it's not such a big deal,
and as I said you'd likely turn it off in production.)
Similarly, a user accidentally running perf record -a
probably should have some reasonable semantics.
Yes, also avoided by doing this as above, though I'd argue we
could also just say that running perf disables this mode.
But it's not as clean as the above suggestion.
On 05/09/2015 06:37 AM, Gilad Ben Yossef wrote:
So, I don't know if it is a practical suggestion or not, but would it better/easier to mark a pending signal on kernel entry for this case?
The upsides I see is that the user gets her notification (killing the task or just logging the event in a signal handler) and hopefully since return to userspace with a pending signal is already handled we don't need new code in the exit path?
We could certainly do this now that I'm planning to do the
test at kernel entry rather than super-late in kernel exit.
Rather than just do_group_exit(SIGKILL), we should raise
a proper SIGKILL signal via send_sig(SIGKILL, current, 1),
and then we could catch it in the debugger; the pc should
help identify if it was a syscall, page fault, or other trap.
I'm not sure there's an argument to be made for the user
process being able to catch the signal itself; presumably in
production you don't turn this mode on anyway, and in
development, assuming a debugger is probably fine.
But if you want to argue for another signal (SIGILL?) please
do; I'm curious to hear if you think it would make more sense.
--
Chris Metcalf, EZChip Semiconductor
http://www.ezchip.com
From: Andy Lutomirski <luto@amacapital.net> Date: 2015-05-11 22:29:03
[add peterz due to perf stuff]
On Mon, May 11, 2015 at 12:13 PM, Chris Metcalf [off-list ref] wrote:
On 05/09/2015 03:28 AM, Andy Lutomirski wrote:
quoted
On May 8, 2015 11:44 PM, "Chris Metcalf" [off-list ref] wrote:
quoted
With QUIESCE mode, the task is in principle guaranteed not to be
interrupted by the kernel, but only if it behaves. In particular,
if it enters the kernel via system call, page fault, or any of
a number of other synchronous traps, it may be unexpectedly
exposed to long latencies. Add a simple flag that puts the process
into a state where any such kernel entry is fatal.
To allow the state to be entered and exited, we add an internal
bit to current->dataplane_flags that is set when prctl() sets the
flags. That way, when we are exiting the kernel after calling
prctl() to forbid future kernel exits, we don't get immediately
killed.
Is there any reason this can't already be addressed in userspace using
/proc/interrupts or perf_events? ISTM the real goal here is to detect
when we screw up and fail to avoid an interrupt, and killing the task
seems like overkill to me.
Patch 6/6 proposes a mechanism to track down times when the
kernel screws up and delivers an IRQ to a userspace-only task.
Here, we're just trying to identify the times when an application
screws itself up out of cluelessness, and provide a mechanism
that allows the developer to easily figure out why and fix it.
In particular, /proc/interrupts won't show syscalls or page faults,
which are two easy ways applications can screw themselves
when they think they're in userspace-only mode. Also, they don't
provide sufficient precision to make it clear what part of the
application caused the undesired kernel entry.
Perf does, though, complete with context.
In this case, killing the task is appropriate, since that's exactly
the semantics that have been asked for - it's like on architectures
that don't natively support unaligned accesses, but fake it relatively
slowly in the kernel, and in development you just say "give me a
SIGBUS when that happens" and in production you might say
"fix it up and let's try to keep going".
I think more control is needed. I also think that, if we go this
route, we should distinguish syscalls, synchronous non-syscall
entries, and asynchronous non-syscall entries. They're quite
different.
You can argue that this is something that can be done by ftrace,
but certainly you'd want to have a way to programmatically
turn on ftrace at the moment when you're entering userspace-only
mode, so we'd want some API around that anyway. And honestly,
it's so easy to test a task state bit in a couple of places and
generate the failurel on the spot, vs. the relative complexity
of setting up and understanding ftrace, that I think it merits
inclusion on that basis alone.
perf_event, not ftrace.
quoted
Also, can we please stop further torturing the exit paths? We have a
disaster of assembly code that calls into syscall_trace_leave and
do_notify_resume. Those functions, in turn, *both* call user_enter
(WTF?), and on very brief inspection user_enter makes it into the nohz
code through multiple levels of indirection, which, with these
patches, has yet another conditionally enabled helper, which does this
new stuff. It's getting to be impossible to tell what happens when we
exit to user space any more.
Also, I think your code is buggy. There's no particular guarantee
that user_enter is only called once between sys_prctl and the final
exit to user mode (see the above WTF), so you might spuriously kill
the process.
This is a good point; I also find the x86 kernel entry and exit
paths confusing, although I've reviewed them a bunch of times.
The tile architecture paths are a little easier to understand.
That said, I think the answer here is avoid non-idempotent
actions in the dataplane code, such as clearing a syscall bit.
A better implementation, I think, is to put the tests for "you
screwed up and synchronously entered the kernel" in
the syscall_trace_enter() code, which TIF_NOHZ already
gets us into;
No, not unless you're planning on using that to distinguish syscalls
from other stuff *and* people think that's justified.
It's far to easy to just make a tiny change to the entry code. Add a
tiny trivial change here, a few lines of asm (that's you, audit!)
there, some weird written-in-asm scheduling code over here, and you
end up with the truly awful mess that we currently have.
If it really makes sense for this stuff to go with context tracking,
then fine, but we should *fix* the context tracking first rather than
kludging around it. I already have a prototype patch for the relevant
part of that.
there, we can test if the dataplane "strict" bit is
set and the syscall is not prctl(), then we generate the error.
(We'd exclude exit and exit_group here too, since we don't
need to shoot down a task that's just trying to kill itself.)
This needs a bit of platform-specific code for each platform,
but that doesn't seem like too big a problem.
I'd rather avoid that, too. This feature isn't really arch-specific,
so let's avoid the arch stuff if at all possible.
Likewise we can test in exception_enter() since that's only
called for all the synchronous user entries like page faults.
Let's try to generalize a bit. There's also irq_entry and ist_enter,
and some of the exception_enter cases are for synchronous entries
while (IIRC -- could be wrong) others aren't always like that.
quoted
Also, I think that most users will be quite surprised if "strict
dataplane" code causes any machine check on the system to kill your
dataplane task.
Fair point, and avoided by testing as described above instead.
(Though presumably in development it's not such a big deal,
and as I said you'd likely turn it off in production.)
Until you forget to turn it off in production because it worked so
nicely in development.
What if we added a mode to perf where delivery of a sample
synchronously (or semi-synchronously by catching it on the next exit
to userspace) freezes the delivering task? It would be like debugger
support via perf.
peterz, do you think this would be a sensible thing to add to perf?
It would only make sense for some types of events (tracepoints and
hw_breakpoints mostly, I think).
quoted
So, I don't know if it is a practical suggestion or not, but would it
better/easier to mark a pending signal on kernel entry for this case?
The upsides I see is that the user gets her notification (killing the task
or just logging the event in a signal handler) and hopefully since return to
userspace with a pending signal is already handled we don't need new code in
the exit path?
We could certainly do this now that I'm planning to do the
test at kernel entry rather than super-late in kernel exit.
Rather than just do_group_exit(SIGKILL), we should raise
a proper SIGKILL signal via send_sig(SIGKILL, current, 1),
and then we could catch it in the debugger; the pc should
help identify if it was a syscall, page fault, or other trap.
I'm not sure there's an argument to be made for the user
process being able to catch the signal itself; presumably in
production you don't turn this mode on anyway, and in
development, assuming a debugger is probably fine.
But if you want to argue for another signal (SIGILL?) please
do; I'm curious to hear if you think it would make more sense.
From: Chris Metcalf <hidden> Date: 2015-05-12 21:06:17
On 05/11/2015 06:28 PM, Andy Lutomirski wrote:
[add peterz due to perf stuff]
On Mon, May 11, 2015 at 12:13 PM, Chris Metcalf [off-list ref] wrote:
quoted
Patch 6/6 proposes a mechanism to track down times when the
kernel screws up and delivers an IRQ to a userspace-only task.
Here, we're just trying to identify the times when an application
screws itself up out of cluelessness, and provide a mechanism
that allows the developer to easily figure out why and fix it.
In particular, /proc/interrupts won't show syscalls or page faults,
which are two easy ways applications can screw themselves
when they think they're in userspace-only mode. Also, they don't
provide sufficient precision to make it clear what part of the
application caused the undesired kernel entry.
Perf does, though, complete with context.
The perf_event suggestions are interesting, but I think it's plausible
for this to be an alternate way to debug the issues that STRICT
addresses.
quoted
In this case, killing the task is appropriate, since that's exactly
the semantics that have been asked for - it's like on architectures
that don't natively support unaligned accesses, but fake it relatively
slowly in the kernel, and in development you just say "give me a
SIGBUS when that happens" and in production you might say
"fix it up and let's try to keep going".
I think more control is needed. I also think that, if we go this
route, we should distinguish syscalls, synchronous non-syscall
entries, and asynchronous non-syscall entries. They're quite
different.
I don't think it's necessary to distinguish the types. As long as we
have a PC pointing to the instruction that triggered the problem,
we can see if it's a system call instruction, a memory write that
caused a page fault, a trap instruction, etc. We certainly could
add infrastructure to capture syscall numbers, fault/signal numbers,
etc etc, but I think it's overkill if it adds kernel overhead on
entry/exit.
quoted
A better implementation, I think, is to put the tests for "you
screwed up and synchronously entered the kernel" in
the syscall_trace_enter() code, which TIF_NOHZ already
gets us into;
No, not unless you're planning on using that to distinguish syscalls
from other stuff *and* people think that's justified.
So, the question is how we separate synchronous entries
from IRQs? At a high level, IRQs are kernel bugs (for cpu-isolated
tasks), and synchronous entries are application bugs. We'd
like to deliver a signal for the latter, and do some kind of
kernel diagnostics for the former. So we can't just add the
test in the context tracking code, which doesn't actually know
why we're entering or exiting.
That's why I was thinking that the syscall_trace_entry and
exception_enter paths were the best choices. I'm fairly sure
that exception_enter is only done for synchronous traps,
page faults, etc.
Certainly on the tile architecture we include the trap number
in the pt_regs, so it's possible to just examine the pt_regs and
know why you entered or are exiting the kernel, but I don't
think we can rely on that for all architectures.
It's far to easy to just make a tiny change to the entry code. Add a
tiny trivial change here, a few lines of asm (that's you, audit!)
there, some weird written-in-asm scheduling code over here, and you
end up with the truly awful mess that we currently have.
If it really makes sense for this stuff to go with context tracking,
then fine, but we should *fix* the context tracking first rather than
kludging around it. I already have a prototype patch for the relevant
part of that.
quoted
there, we can test if the dataplane "strict" bit is
set and the syscall is not prctl(), then we generate the error.
(We'd exclude exit and exit_group here too, since we don't
need to shoot down a task that's just trying to kill itself.)
This needs a bit of platform-specific code for each platform,
but that doesn't seem like too big a problem.
I'd rather avoid that, too. This feature isn't really arch-specific,
so let's avoid the arch stuff if at all possible.
I'll put out a v2 of my patch that does both the things you
advise against :-) just so we can have a strawman to think
about how to do it better - unless you have a suggestion
offhand as to how we can better differentiate sync and async
entries into the kernel in a platform-independent way.
I could imagine modifying user_exit() and exception_enter()
to pass an identifier into the context system saying why they
were changing contexts, so we could have syscalls, trap
numbers, fault numbers, etc., and some way to query as
to whether they were synchronous or asynchronous, and
build this scheme on top of that, but I'm not sure the extra
infrastructure is worthwhile.
quoted
Likewise we can test in exception_enter() since that's only
called for all the synchronous user entries like page faults.
Let's try to generalize a bit. There's also irq_entry and ist_enter,
and some of the exception_enter cases are for synchronous entries
while (IIRC -- could be wrong) others aren't always like that.
I don't think we need to generalize this piece. irq_entry()
shouldn't be reported by the STRICT mechanism but by
kernel bug reporting. For ist_enter(), it looks like if you're
coming from userspace it's just handled with exception_enter().
I'm more familiar with the tile architecture mechanisms than
with x86, though, to be honest.
quoted
quoted
Also, I think that most users will be quite surprised if "strict
dataplane" code causes any machine check on the system to kill your
dataplane task.
Fair point, and avoided by testing as described above instead.
(Though presumably in development it's not such a big deal,
and as I said you'd likely turn it off in production.)
Until you forget to turn it off in production because it worked so
nicely in development.
I guess that's an argument for using a non-fatal signal with a
handler from the get-go, since then even in production you'll
just end up with a slightly heavier-weight kernel overhead
(whatever stupid thing your application did, plus the time
spent in the signal handler), but then after that you can get
back to processing packets or whatever the app is doing.
You had mentioned some alternatives to a catchable signal
(a signal to some other process, or queuing to an fd); I think
it still seems reasonable to just deliver a signal to the process,
configurably by the prctl, and not do anything more complex.
Does this seem reasonable to you at this point?
What if we added a mode to perf where delivery of a sample
synchronously (or semi-synchronously by catching it on the next exit
to userspace) freezes the delivering task? It would be like debugger
support via perf.
peterz, do you think this would be a sensible thing to add to perf?
It would only make sense for some types of events (tracepoints and
hw_breakpoints mostly, I think).
I suspect it's reasonable to consider this orthogonal, particularly
if there is some skid between the actual violation by the
application, and the freeze happening.
You pushed back somewhat on prctl() in favor of a quiesce()
syscall in your email, but it seemed like at the end of your
email you were adopting the prctl() perspective. Is that true?
I admit the prctl() still seems cleaner from my perspective.
--
Chris Metcalf, EZChip Semiconductor
http://www.ezchip.com
From: Andy Lutomirski <luto@amacapital.net> Date: 2015-05-12 22:23:27
On May 13, 2015 6:06 AM, "Chris Metcalf" [off-list ref] wrote:
On 05/11/2015 06:28 PM, Andy Lutomirski wrote:
quoted
[add peterz due to perf stuff]
On Mon, May 11, 2015 at 12:13 PM, Chris Metcalf [off-list ref] wrote:
quoted
Patch 6/6 proposes a mechanism to track down times when the
kernel screws up and delivers an IRQ to a userspace-only task.
Here, we're just trying to identify the times when an application
screws itself up out of cluelessness, and provide a mechanism
that allows the developer to easily figure out why and fix it.
In particular, /proc/interrupts won't show syscalls or page faults,
which are two easy ways applications can screw themselves
when they think they're in userspace-only mode. Also, they don't
provide sufficient precision to make it clear what part of the
application caused the undesired kernel entry.
Perf does, though, complete with context.
The perf_event suggestions are interesting, but I think it's plausible
for this to be an alternate way to debug the issues that STRICT
addresses.
quoted
quoted
In this case, killing the task is appropriate, since that's exactly
the semantics that have been asked for - it's like on architectures
that don't natively support unaligned accesses, but fake it relatively
slowly in the kernel, and in development you just say "give me a
SIGBUS when that happens" and in production you might say
"fix it up and let's try to keep going".
I think more control is needed. I also think that, if we go this
route, we should distinguish syscalls, synchronous non-syscall
entries, and asynchronous non-syscall entries. They're quite
different.
I don't think it's necessary to distinguish the types. As long as we
have a PC pointing to the instruction that triggered the problem,
we can see if it's a system call instruction, a memory write that
caused a page fault, a trap instruction, etc.
Not true. PC right after a syscall insn could be any type of kernel
entry, and you can't even reliably tell whether the syscall insn was
executed or, on x86, whether it was a syscall at all. (x86 insns
can't be reliably decided backwards.)
PC pointing at a load could be a page fault or an IPI.
We certainly could
add infrastructure to capture syscall numbers, fault/signal numbers,
etc etc, but I think it's overkill if it adds kernel overhead on
entry/exit.
None of these should add overhead.
quoted
quoted
A better implementation, I think, is to put the tests for "you
screwed up and synchronously entered the kernel" in
the syscall_trace_enter() code, which TIF_NOHZ already
gets us into;
No, not unless you're planning on using that to distinguish syscalls
from other stuff *and* people think that's justified.
So, the question is how we separate synchronous entries
from IRQs? At a high level, IRQs are kernel bugs (for cpu-isolated
tasks), and synchronous entries are application bugs. We'd
like to deliver a signal for the latter, and do some kind of
kernel diagnostics for the former. So we can't just add the
test in the context tracking code, which doesn't actually know
why we're entering or exiting.
Synchronous entries could be VM bugs, too.
That's why I was thinking that the syscall_trace_entry and
exception_enter paths were the best choices. I'm fairly sure
that exception_enter is only done for synchronous traps,
page faults, etc.
Maybe. Doing it through the actual entry/exit slow paths would be
overhead-free, although I'm not sure that IRQs have real slow paths
for entry.
Certainly on the tile architecture we include the trap number
in the pt_regs, so it's possible to just examine the pt_regs and
know why you entered or are exiting the kernel, but I don't
think we can rely on that for all architectures.
x86 can't do this.
I'll put out a v2 of my patch that does both the things you
advise against :-) just so we can have a strawman to think
about how to do it better - unless you have a suggestion
offhand as to how we can better differentiate sync and async
entries into the kernel in a platform-independent way.
I could imagine modifying user_exit() and exception_enter()
to pass an identifier into the context system saying why they
were changing contexts, so we could have syscalls, trap
numbers, fault numbers, etc., and some way to query as
to whether they were synchronous or asynchronous, and
build this scheme on top of that, but I'm not sure the extra
infrastructure is worthwhile.
I'll take a look.
Again, though, I think we really do need to distinguish at least MCE
and NMI (on x86) from the others.
quoted
What if we added a mode to perf where delivery of a sample
synchronously (or semi-synchronously by catching it on the next exit
to userspace) freezes the delivering task? It would be like debugger
support via perf.
peterz, do you think this would be a sensible thing to add to perf?
It would only make sense for some types of events (tracepoints and
hw_breakpoints mostly, I think).
I suspect it's reasonable to consider this orthogonal, particularly
if there is some skid between the actual violation by the
application, and the freeze happening.
I think it could be done without skid, except for async entries, but
for asynx entries we don't care about exact user state anyway.
You pushed back somewhat on prctl() in favor of a quiesce()
syscall in your email, but it seemed like at the end of your
email you were adopting the prctl() perspective. Is that true?
I admit the prctl() still seems cleaner from my perspective.
Prctl for the strict thing seems much more reasonable to me than prctl
for quiescing. Also, the scheduler people seem to thing that
quiescing should be automatic.
Anyway, I'll happily look at code and maybe even write more coherent
emails when I'm back in town in a week. Since you're thinking that
async entries should give kernel diagnostics instead of signals, maybe
the right thing to do is to separate them out completely and try to
address the individual entry types separately and as needed.
--Andy
From: Chris Metcalf <hidden> Date: 2015-05-15 21:25:31
On 05/12/2015 06:23 PM, Andy Lutomirski wrote:
On May 13, 2015 6:06 AM, "Chris Metcalf" [off-list ref] wrote:
quoted
On 05/11/2015 06:28 PM, Andy Lutomirski wrote:
quoted
On Mon, May 11, 2015 at 12:13 PM, Chris Metcalf [off-list ref] wrote:
quoted
In this case, killing the task is appropriate, since that's exactly
the semantics that have been asked for - it's like on architectures
that don't natively support unaligned accesses, but fake it relatively
slowly in the kernel, and in development you just say "give me a
SIGBUS when that happens" and in production you might say
"fix it up and let's try to keep going".
I think more control is needed. I also think that, if we go this
route, we should distinguish syscalls, synchronous non-syscall
entries, and asynchronous non-syscall entries. They're quite
different.
I don't think it's necessary to distinguish the types. As long as we
have a PC pointing to the instruction that triggered the problem,
we can see if it's a system call instruction, a memory write that
caused a page fault, a trap instruction, etc.
Not true. PC right after a syscall insn could be any type of kernel
entry, and you can't even reliably tell whether the syscall insn was
executed or, on x86, whether it was a syscall at all. (x86 insns
can't be reliably decided backwards.)
PC pointing at a load could be a page fault or an IPI.
All that we are trying to do with this API, though, is distinguish
synchronous faults. So IPIs, etc., should not be happening
(they would be bugs), and hopefully we are mostly just
distinguishing different types of synchronous program entries.
That said, I did a si_info flag to differentiate syscalls from other
synchronous entries, and I'm open to looking at more such if
it seems useful.
Again, though, I think we really do need to distinguish at least MCE
and NMI (on x86) from the others.
Yes, those are both interesting cases, and I'm not entirely
sure what the right way to handle them is - for example,
likely disable STRICT if you are running with perf enabled.
I look forward to hearing more when you're back next week!
--
Chris Metcalf, EZChip Semiconductor
http://www.ezchip.com
So while I'm all for hard fails like this, can we not provide a wee bit
more information in the siginfo ? And maybe use a slightly less fatal
signal, such that userspace can actually catch it and dump state in
debug modes?
So while I'm all for hard fails like this, can we not provide a wee bit
more information in the siginfo ? And maybe use a slightly less fatal
signal, such that userspace can actually catch it and dump state in
debug modes?
Agreed, a bit more debug state would be helpful.
Thanx, Paul
From: Chris Metcalf <hidden> Date: 2015-05-08 18:00:04
This prctl() flag for PR_SET_DATAPLANE sets a mode that requires the
kernel to quiesce any pending timer interrupts prior to returning
to userspace. When running with this mode set, sys calls (and page
faults, etc.) can be inordinately slow. However, user applications
that want to guarantee that no unexpected interrupts will occur
(even if they call into the kernel) can set this flag to guarantee
that semantics.
Signed-off-by: Chris Metcalf <redacted>
---
include/uapi/linux/prctl.h | 1 +
kernel/time/tick-sched.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 55 insertions(+)
@@ -392,6 +392,53 @@ void __init tick_nohz_init(void)}/*+*Wenormallyreturnimmediatelytouserspace.+*+*ThePR_DATAPLANE_QUIESCEflagcausesustowaituntilnomore+*interruptsarepending.Otherwisewenapwithinterruptsenabled+*andwaitforthenextinterrupttofire,thenloopbackandretry.+*+*Notethatifyouscheduletwoprocessesonthesamecoreandboth+*specifyPR_DATAPLANE_QUIESCE,neitherwilleverleavethekernel,+*andonewillhavetobekilledmanually.Otherwiseinsituations+*whereanotherprocessisintherunqueueonthiscpu,thistask+*willjustwaitforthatothertasktogoidlebeforereturningto+*userspace.+*/+staticvoiddataplane_quiesce(void)+{+structclock_event_device*dev=+__this_cpu_read(tick_cpu_device.evtdev);+structtask_struct*task=current;+unsignedlongstart=jiffies;+boolwarned=false;++while(ACCESS_ONCE(dev->next_event.tv64)!=KTIME_MAX){+if(!warned&&(jiffies-start)>=(5*HZ)){+pr_warn("%s/%d: cpu %d: dataplane task blocked for %ld jiffies\n",+task->comm,task->pid,smp_processor_id(),+(jiffies-start));+warned=true;+}+if(should_resched())+schedule();+if(test_thread_flag(TIF_SIGPENDING))+break;++/* Idle with interrupts enabled and wait for the tick. */+set_current_state(TASK_INTERRUPTIBLE);+arch_cpu_idle();+set_current_state(TASK_RUNNING);+}+if(warned){+pr_warn("%s/%d: cpu %d: dataplane task unblocked after %ld jiffies\n",+task->comm,task->pid,smp_processor_id(),+(jiffies-start));+dump_stack();+}+}++/**Whenreturningtouserspaceonanohz_fullcoreafterdoing*prctl(PR_DATAPLANE_SET,1),wecomehereandtrymoreaggressively*topreventthiscorefrombeinginterruptedlater.
From: Peter Zijlstra <peterz@infradead.org> Date: 2015-05-12 09:34:18
On Fri, May 08, 2015 at 01:58:45PM -0400, Chris Metcalf wrote:
This prctl() flag for PR_SET_DATAPLANE sets a mode that requires the
kernel to quiesce any pending timer interrupts prior to returning
to userspace. When running with this mode set, sys calls (and page
faults, etc.) can be inordinately slow. However, user applications
that want to guarantee that no unexpected interrupts will occur
(even if they call into the kernel) can set this flag to guarantee
that semantics.
Currently people hot-unplug and hot-plug the CPU to do this. Obviously
that's a wee bit horrible :-)
Not sure if a prctl like this is any better though. This is a CPU
properly not a process one.
ISTR people talking about 'quiesce' sysfs file, along side the hotplug
stuff, I can't quite remember.
On Fri, May 08, 2015 at 01:58:45PM -0400, Chris Metcalf wrote:
quoted
This prctl() flag for PR_SET_DATAPLANE sets a mode that requires the
kernel to quiesce any pending timer interrupts prior to returning
to userspace. When running with this mode set, sys calls (and page
faults, etc.) can be inordinately slow. However, user applications
that want to guarantee that no unexpected interrupts will occur
(even if they call into the kernel) can set this flag to guarantee
that semantics.
Currently people hot-unplug and hot-plug the CPU to do this.
Obviously that's a wee bit horrible :-)
Not sure if a prctl like this is any better though. This is a CPU
properly not a process one.
So if then a prctl() (or other system call) could be a shortcut to:
- move the task to an isolated CPU
- make sure there _is_ such an isolated domain available
I.e. have some programmatic, kernel provided way for an application to
be sure it's running in the right environment. Relying on random
administration flags here and there won't cut it.
Thanks,
Ingo
From: Peter Zijlstra <peterz@infradead.org> Date: 2015-05-12 10:38:25
On Tue, May 12, 2015 at 11:50:30AM +0200, Ingo Molnar wrote:
* Peter Zijlstra [off-list ref] wrote:
quoted
On Fri, May 08, 2015 at 01:58:45PM -0400, Chris Metcalf wrote:
quoted
This prctl() flag for PR_SET_DATAPLANE sets a mode that requires the
kernel to quiesce any pending timer interrupts prior to returning
to userspace. When running with this mode set, sys calls (and page
faults, etc.) can be inordinately slow. However, user applications
that want to guarantee that no unexpected interrupts will occur
(even if they call into the kernel) can set this flag to guarantee
that semantics.
Currently people hot-unplug and hot-plug the CPU to do this.
Obviously that's a wee bit horrible :-)
Not sure if a prctl like this is any better though. This is a CPU
properly not a process one.
So if then a prctl() (or other system call) could be a shortcut to:
- move the task to an isolated CPU
- make sure there _is_ such an isolated domain available
I.e. have some programmatic, kernel provided way for an application to
be sure it's running in the right environment. Relying on random
administration flags here and there won't cut it.
No, we already have sched_setaffinity() and we should not duplicate its
ability to move tasks about.
What this is about is 'clearing' CPU state, its nothing to do with
tasks.
Ideally we'd never have to clear the state because it should be
impossible to get into this predicament in the first place.
The typical example here is a periodic timer that found its way onto the
cpu and stays there. We're actually working on allowing such self arming
timers to migrate, so once we have that sorted this could be fixed
proper I think.
Not sure if there's more pollution that people worry about.
The hotplug hack worked because unplug force migrates the timers away.
So if then a prctl() (or other system call) could be a shortcut
to:
- move the task to an isolated CPU
- make sure there _is_ such an isolated domain available
I.e. have some programmatic, kernel provided way for an
application to be sure it's running in the right environment.
Relying on random administration flags here and there won't cut
it.
No, we already have sched_setaffinity() and we should not duplicate
its ability to move tasks about.
But sched_setaffinity() does not guarantee isolation - it's just a
syscall to move a task to a set of CPUs, which might be isolated or
not.
What I suggested is that it might make sense to offer a system call,
for example a sched_setparam() variant, that makes such guarantees.
Say if user-space does:
ret = sched_setscheduler(0, BIND_ISOLATED, &isolation_params);
... then we would get the task moved to an isolated domain and get a 0
return code if the kernel is able to do all that and if the current
uid/namespace/etc. has the required permissions and such.
( BIND_ISOLATED will not replace the current p->policy value, so it's
still possible to use the regular policies as well on top of this. )
I.e. make it programatic instead of relying on a fragile, kernel
version dependent combination of sysctl, sysfs, kernel config and boot
parameter details to get us this result.
I.e. provide a central hub to offer this feature in a more structured,
easier to use fashion.
We might still require the admin (or distro) to separately set up the
domain of isolated CPUs, and it would still be possible to simply
'move' tasks there using existing syscalls - but I say that it's not a
bad idea at all to offer a single central syscall interface for apps
to request such treatment.
What this is about is 'clearing' CPU state, its nothing to do with
tasks.
Ideally we'd never have to clear the state because it should be
impossible to get into this predicament in the first place.
That I absolutely agree about, that bit is nonsense.
We might offer debugging facilities to debug such bugs, but we won't
work or hack it around.
Thanks,
Ingo
From: Andy Lutomirski <luto@amacapital.net> Date: 2015-05-13 04:35:52
On Tue, May 12, 2015 at 5:52 AM, Ingo Molnar [off-list ref] wrote:
* Peter Zijlstra [off-list ref] wrote:
quoted
quoted
So if then a prctl() (or other system call) could be a shortcut
to:
- move the task to an isolated CPU
- make sure there _is_ such an isolated domain available
I.e. have some programmatic, kernel provided way for an
application to be sure it's running in the right environment.
Relying on random administration flags here and there won't cut
it.
No, we already have sched_setaffinity() and we should not duplicate
its ability to move tasks about.
But sched_setaffinity() does not guarantee isolation - it's just a
syscall to move a task to a set of CPUs, which might be isolated or
not.
What I suggested is that it might make sense to offer a system call,
for example a sched_setparam() variant, that makes such guarantees.
Say if user-space does:
ret = sched_setscheduler(0, BIND_ISOLATED, &isolation_params);
... then we would get the task moved to an isolated domain and get a 0
return code if the kernel is able to do all that and if the current
uid/namespace/etc. has the required permissions and such.
( BIND_ISOLATED will not replace the current p->policy value, so it's
still possible to use the regular policies as well on top of this. )
I think we shouldn't have magic selection of an isolated domain.
Anyone using this has already configured some isolated CPUs and
probably wants to choose the CPU and, especially, NUMA node
themselves. Also, maybe it should be a special type of realtime
class/priority -- doing this should require RT permission IMO.
--Andy
From: Paul E. McKenney <hidden> Date: 2015-05-13 21:00:27
On Tue, May 12, 2015 at 09:35:25PM -0700, Andy Lutomirski wrote:
On Tue, May 12, 2015 at 5:52 AM, Ingo Molnar [off-list ref] wrote:
quoted
* Peter Zijlstra [off-list ref] wrote:
quoted
quoted
So if then a prctl() (or other system call) could be a shortcut
to:
- move the task to an isolated CPU
- make sure there _is_ such an isolated domain available
I.e. have some programmatic, kernel provided way for an
application to be sure it's running in the right environment.
Relying on random administration flags here and there won't cut
it.
No, we already have sched_setaffinity() and we should not duplicate
its ability to move tasks about.
But sched_setaffinity() does not guarantee isolation - it's just a
syscall to move a task to a set of CPUs, which might be isolated or
not.
What I suggested is that it might make sense to offer a system call,
for example a sched_setparam() variant, that makes such guarantees.
Say if user-space does:
ret = sched_setscheduler(0, BIND_ISOLATED, &isolation_params);
... then we would get the task moved to an isolated domain and get a 0
return code if the kernel is able to do all that and if the current
uid/namespace/etc. has the required permissions and such.
( BIND_ISOLATED will not replace the current p->policy value, so it's
still possible to use the regular policies as well on top of this. )
I think we shouldn't have magic selection of an isolated domain.
Anyone using this has already configured some isolated CPUs and
probably wants to choose the CPU and, especially, NUMA node
themselves. Also, maybe it should be a special type of realtime
class/priority -- doing this should require RT permission IMO.
I have no real argument against special permissions, but this feature
is totally orthogonal to realtime classes/priorities. It is perfectly
legitimate for a given CPU's single runnable task to be SCHED_OTHER,
for example.
Thanx, Paul
From: Chris Metcalf <hidden> Date: 2015-05-14 20:55:24
On 05/12/2015 08:52 AM, Ingo Molnar wrote:
What I suggested is that it might make sense to offer a system call,
for example a sched_setparam() variant, that makes such guarantees.
Say if user-space does:
ret = sched_setscheduler(0, BIND_ISOLATED, &isolation_params);
... then we would get the task moved to an isolated domain and get a 0
return code if the kernel is able to do all that and if the current
uid/namespace/etc. has the required permissions and such.
Unfortunately I don't know nearly as much about the scheduler
and scheduler policies as I might, since I mostly focused on
make the scheduler stay out of the way. :-) This does seem like
another way to set a policy bit on a process. I assume you
could only validly issue this call on a nohz_full core, and that
you're not assuming it migrates the cpu to such a core?
You suggested that BIND_ISOLATED would not replace the usual
scheduler policies, but perhaps SCHED_ISOLATED as a full
replacement would make sense - it would make it an error
to have any other schedulable task on that core. I guess that
brings it around to whether the "cpu_isolated" task just loses when
another task is scheduled on the core with it (the current
approach I'm proposing) or if it ends up truly owning the core
and other processes can be denied the right to run there:
which in that case clearly does get us into the area of requiring
privileges to set up, as Andy pointed out later.
This would leave the notion of "strict" as proposed elsewhere
as a separate thing, but presumably it could still be a prctl()
as originally proposed.
I admit I don't know enough to say whether this sounds like
a better approach than just using a prctl() to set the
cpu_isolated state. My instinct is that it's cleanest to avoid
requiring permissions to do this, and to simply enable the
quiescing semantics the process requested when it happens
to be alone on a core. If so, it's somewhat orthogonal to the
actual scheduler policy in force, so best not to conflate it with
the notion of scheduler code at all via sched_setscheduler()?
--
Chris Metcalf, EZChip Semiconductor
http://www.ezchip.com
From: Chris Metcalf <hidden> Date: 2015-05-14 20:54:57
On 05/12/2015 05:33 AM, Peter Zijlstra wrote:
On Fri, May 08, 2015 at 01:58:45PM -0400, Chris Metcalf wrote:
quoted
This prctl() flag for PR_SET_DATAPLANE sets a mode that requires the
kernel to quiesce any pending timer interrupts prior to returning
to userspace. When running with this mode set, sys calls (and page
faults, etc.) can be inordinately slow. However, user applications
that want to guarantee that no unexpected interrupts will occur
(even if they call into the kernel) can set this flag to guarantee
that semantics.
Currently people hot-unplug and hot-plug the CPU to do this. Obviously
that's a wee bit horrible :-)
Not sure if a prctl like this is any better though. This is a CPU
properly not a process one.
The CPU property aspects, I think, should be largely handled by
fixing kernel bugs that let work end up running on nohz_full cores
without having been explicitly requested to run there.
As you said in a follow-up email:
On 05/12/2015 06:38 AM, Peter Zijlstra wrote:
Ideally we'd never have to clear the state because it should be
impossible to get into this predicament in the first place.
What my prctl() proposal does is quiesce things that end up
happening specifically because the user process called on purpose
into the kernel. For example, perhaps RCU was invoked in the
kernel, and the core has to wait a timer tick to quiesce RCU.
Whatever causes it, the intent is that you're not allowed back into
userspace until everything has settled down from your call into
the kernel; the presumption is that it's all due to the kernel entry
that was just made, and not from other stray work.
In that sense, it's very appropriate for it to be a process property.
ISTR people talking about 'quiesce' sysfs file, along side the hotplug
stuff, I can't quite remember.
It seems somewhat similar (adding Viresh to the cc's) but does
seem like it might have been more intended to address the
CPU properties rather than process properties:
https://lkml.org/lkml/2014/4/4/99
One thing the original Tilera dataplane code did was to require
setting dataplane flags to succeed only on dataplane cores,
and only when the task had been affinitized to that single core.
This did not protect the task from later being re-affinitized in
a way that broke those assumptions, but I suppose you could
also imagine make sched_setaffinity() fail for such a process.
Somewhat unrelated, but it occurred to me in the context of this
reply, so what do you think? I can certainly add this to the
patch series if it seems like it makes setting the prctl() flags
more conservative.
--
Chris Metcalf, EZChip Semiconductor
http://www.ezchip.com
From: Steven Rostedt <rostedt@goodmis.org> Date: 2015-05-08 21:22:16
On Fri, 8 May 2015 14:18:24 -0700
Andrew Morton [off-list ref] wrote:
On Fri, 8 May 2015 13:58:41 -0400 Chris Metcalf [off-list ref] wrote:
quoted
A prctl() option (PR_SET_DATAPLANE) is added
Dumb question: what does the term "dataplane" mean in this context? I
can't see the relationship between those words and what this patch
does.
I was thinking the same thing. I haven't gotten around to searching
DATAPLANE yet.
I would assume we want a name that is more meaningful for what is
happening.
-- Steve
From: Chris Metcalf <hidden> Date: 2015-05-08 23:11:28
On 5/8/2015 5:22 PM, Steven Rostedt wrote:
On Fri, 8 May 2015 14:18:24 -0700
Andrew Morton [off-list ref] wrote:
quoted
On Fri, 8 May 2015 13:58:41 -0400 Chris Metcalf [off-list ref] wrote:
quoted
A prctl() option (PR_SET_DATAPLANE) is added
Dumb question: what does the term "dataplane" mean in this context? I
can't see the relationship between those words and what this patch
does.
I was thinking the same thing. I haven't gotten around to searching
DATAPLANE yet.
I would assume we want a name that is more meaningful for what is
happening.
The text in the commit message and the 0/6 cover letter do try to explain
the concept. The terminology comes, I think, from networking line cards,
where the "dataplane" is the part of the application that handles all the
fast path processing of network packets, and the "control plane" is the part
that handles routing updates, etc., generally slow-path stuff. I've probably
just been using the terms so long they seem normal to me.
That said, what would be clearer? NO_HZ_STRICT as a superset of
NO_HZ_FULL? Or move away from the NO_HZ terminology a bit; after all,
we're talking about no interrupts of any kind, and maybe NO_HZ is too
limited in scope? So, NO_INTERRUPTS? USERSPACE_ONLY? Or look
to vendors who ship bare-metal runtimes and call it BARE_METAL?
Borrow the Tilera marketing name and call it ZERO_OVERHEAD?
Maybe BARE_METAL seems most plausible -- after DATAPLANE, to me,
of course :-)
--
Chris Metcalf, EZChip Semiconductor
http://www.ezchip.com
From: Andrew Morton <akpm@linux-foundation.org> Date: 2015-05-08 23:19:15
On Fri, 8 May 2015 19:11:10 -0400 Chris Metcalf [off-list ref] wrote:
On 5/8/2015 5:22 PM, Steven Rostedt wrote:
quoted
On Fri, 8 May 2015 14:18:24 -0700
Andrew Morton [off-list ref] wrote:
quoted
On Fri, 8 May 2015 13:58:41 -0400 Chris Metcalf [off-list ref] wrote:
quoted
A prctl() option (PR_SET_DATAPLANE) is added
Dumb question: what does the term "dataplane" mean in this context? I
can't see the relationship between those words and what this patch
does.
I was thinking the same thing. I haven't gotten around to searching
DATAPLANE yet.
I would assume we want a name that is more meaningful for what is
happening.
The text in the commit message and the 0/6 cover letter do try to explain
the concept. The terminology comes, I think, from networking line cards,
where the "dataplane" is the part of the application that handles all the
fast path processing of network packets, and the "control plane" is the part
that handles routing updates, etc., generally slow-path stuff. I've probably
just been using the terms so long they seem normal to me.
That said, what would be clearer? NO_HZ_STRICT as a superset of
NO_HZ_FULL? Or move away from the NO_HZ terminology a bit; after all,
we're talking about no interrupts of any kind, and maybe NO_HZ is too
limited in scope? So, NO_INTERRUPTS? USERSPACE_ONLY? Or look
to vendors who ship bare-metal runtimes and call it BARE_METAL?
Borrow the Tilera marketing name and call it ZERO_OVERHEAD?
Maybe BARE_METAL seems most plausible -- after DATAPLANE, to me,
of course :-)
On Fri, 8 May 2015 19:11:10 -0400 Chris Metcalf [off-list ref] wrote:
quoted
On 5/8/2015 5:22 PM, Steven Rostedt wrote:
quoted
On Fri, 8 May 2015 14:18:24 -0700
Andrew Morton [off-list ref] wrote:
quoted
On Fri, 8 May 2015 13:58:41 -0400 Chris Metcalf [off-list ref] wrote:
quoted
A prctl() option (PR_SET_DATAPLANE) is added
Dumb question: what does the term "dataplane" mean in this context? I
can't see the relationship between those words and what this patch
does.
I was thinking the same thing. I haven't gotten around to searching
DATAPLANE yet.
I would assume we want a name that is more meaningful for what is
happening.
The text in the commit message and the 0/6 cover letter do try to explain
the concept. The terminology comes, I think, from networking line cards,
where the "dataplane" is the part of the application that handles all the
fast path processing of network packets, and the "control plane" is the part
that handles routing updates, etc., generally slow-path stuff. I've probably
just been using the terms so long they seem normal to me.
That said, what would be clearer? NO_HZ_STRICT as a superset of
NO_HZ_FULL? Or move away from the NO_HZ terminology a bit; after all,
we're talking about no interrupts of any kind, and maybe NO_HZ is too
limited in scope? So, NO_INTERRUPTS? USERSPACE_ONLY? Or look
to vendors who ship bare-metal runtimes and call it BARE_METAL?
Borrow the Tilera marketing name and call it ZERO_OVERHEAD?
Maybe BARE_METAL seems most plausible -- after DATAPLANE, to me,
of course :-)
'baremetal' has uses in virtualization speak, so I think that would be
confusing.
I like NO_INTERRUPTS. Simple, direct.
NO_HZ_PURE?
That's what it's really about: user-space wants to run exclusively, in
pure user-mode, without any interrupts.
So I don't like 'NO_HZ_NO_INTERRUPTS' for a couple of reasons:
- It is similar to a term we use in perf: PERF_PMU_CAP_NO_INTERRUPT.
- Another reason is that 'NO_INTERRUPTS', in most existing uses in
the kernel generally relates to some sort of hardware weakness,
limitation, a negative property: that we try to limp along without
having a hardware interrupt and have to poll. In other driver code
that uses variants of NO_INTERRUPT it appears to be similar. So I
think there's some confusion potential here.
- Here the fact that we don't disturb user-space is an absolutely
positive property, not a limitation, a kernel feature we work hard
to achieve. NO_HZ_PURE would convey that while NO_HZ_NO_INTERRUPTS
wouldn't.
- NO_HZ_NO_INTERRUPTS has a double negation, and it's also too long,
compared to NO_HZ_FULL or NO_HZ_PURE ;-) The term 'no HZ' already
expresses that we don't have periodic interruptions. We just
duplicate that information with NO_HZ_NO_INTERRUPTS, while
NO_HZ_FULL or NO_HZ_PURE qualifies it, makes it a stronger
property - which is what we want I think.
So I think we should either rename NO_HZ_FULL to NO_HZ_PURE, or keep
it at NO_HZ_FULL: because the intention of NO_HZ_FULL was always to be
such a 'zero overhead' mode of operation, where if user-space runs, it
won't get interrupted in any way.
There's no need to add yet another Kconfig variant - lets just enhance
the current stuff and maybe rename it to NO_HZ_PURE to better express
its intent.
Thanks,
Ingo
From: Andy Lutomirski <luto@amacapital.net> Date: 2015-05-09 07:19:31
On Sat, May 9, 2015 at 12:05 AM, Ingo Molnar [off-list ref] wrote:
* Andrew Morton [off-list ref] wrote:
quoted
On Fri, 8 May 2015 19:11:10 -0400 Chris Metcalf [off-list ref] wrote:
quoted
On 5/8/2015 5:22 PM, Steven Rostedt wrote:
quoted
On Fri, 8 May 2015 14:18:24 -0700
Andrew Morton [off-list ref] wrote:
quoted
On Fri, 8 May 2015 13:58:41 -0400 Chris Metcalf [off-list ref] wrote:
quoted
A prctl() option (PR_SET_DATAPLANE) is added
Dumb question: what does the term "dataplane" mean in this context? I
can't see the relationship between those words and what this patch
does.
I was thinking the same thing. I haven't gotten around to searching
DATAPLANE yet.
I would assume we want a name that is more meaningful for what is
happening.
The text in the commit message and the 0/6 cover letter do try to explain
the concept. The terminology comes, I think, from networking line cards,
where the "dataplane" is the part of the application that handles all the
fast path processing of network packets, and the "control plane" is the part
that handles routing updates, etc., generally slow-path stuff. I've probably
just been using the terms so long they seem normal to me.
That said, what would be clearer? NO_HZ_STRICT as a superset of
NO_HZ_FULL? Or move away from the NO_HZ terminology a bit; after all,
we're talking about no interrupts of any kind, and maybe NO_HZ is too
limited in scope? So, NO_INTERRUPTS? USERSPACE_ONLY? Or look
to vendors who ship bare-metal runtimes and call it BARE_METAL?
Borrow the Tilera marketing name and call it ZERO_OVERHEAD?
Maybe BARE_METAL seems most plausible -- after DATAPLANE, to me,
of course :-)
'baremetal' has uses in virtualization speak, so I think that would be
confusing.
quoted
I like NO_INTERRUPTS. Simple, direct.
NO_HZ_PURE?
Naming aside, I don't think this should be a per-task flag at all. We
already have way too much overhead per syscall in nohz mode, and it
would be nice to get the per-syscall overhead as low as possible. We
should strive, for all tasks, to keep syscall overhead down *and*
avoid as many interrupts as possible.
That being said, I do see a legitimate use for a way to tell the
kernel "I'm going to run in userspace for a long time; stay away".
But shouldn't that be a single operation, not an ongoing flag? IOW, I
think that we should have a new syscall quiesce() or something rather
than a prctl.
--Andy
From: Chris Metcalf <hidden> Date: 2015-05-11 19:54:51
(Oops, resending and forcing html off.)
On 05/09/2015 03:19 AM, Andy Lutomirski wrote:
Naming aside, I don't think this should be a per-task flag at all. We
already have way too much overhead per syscall in nohz mode, and it
would be nice to get the per-syscall overhead as low as possible. We
should strive, for all tasks, to keep syscall overhead down*and*
avoid as many interrupts as possible.
That being said, I do see a legitimate use for a way to tell the
kernel "I'm going to run in userspace for a long time; stay away".
But shouldn't that be a single operation, not an ongoing flag? IOW, I
think that we should have a new syscall quiesce() or something rather
than a prctl.
Yes, if all you are concerned about is quiescing the tick, we could
probably do it as a new syscall.
I do note that you'd want to try to actually do the quiesce as late as
possible - in particular, if you just did it in the usual syscall, you
might miss out on a timer that is set by softirq, or even something
that happened when you called schedule() on the syscall exit path.
Doing it as late as we are doing helps to ensure that that doesn't
happen. We could still arrange for this semantics by having a new
quiesce() syscall set a temporary task bit that was cleared on
return to userspace, but as you pointed out in a different email,
that gets tricky if you end up doing multiple user_exit() calls on
your way back to userspace.
More to the point, I think it's actually important to know when an
application believes it's in userspace-only mode as an actual state
bit, rather than just during its transitional moment. If an
application calls the kernel at an unexpected time (third-party code
is the usual culprit for our customers, whether it's syscalls, page
faults, or other things) we would prefer to have the "quiesce"
semantics stay in force and cause the third-party code to be
visibly very slow, rather than cause a totally unexpected and
hard-to-diagnose interrupt show up later as we are still going
around the loop that we thought was safely userspace-only.
And, for debugging the kernel, it's crazy helpful to have that state
bit in place: see patch 6/6 in the series for how we can diagnose
things like "a different core just queued an IPI that will hit a
dataplane core unexpectedly". Having that state bit makes this sort
of thing a trivial check in the kernel and relatively easy to debug.
Finally, I proposed a "strict" mode in patch 5/6 where we kill the
process if it voluntarily enters the kernel by mistake after saying it
wasn't going to any more. To do this requires a state bit, so
carrying another state bit for "quiesce on user entry" seems pretty
reasonable.
--
Chris Metcalf, EZChip Semiconductor
http://www.ezchip.com
From: Andy Lutomirski <luto@amacapital.net> Date: 2015-05-11 22:15:46
On May 12, 2015 4:54 AM, "Chris Metcalf" [off-list ref] wrote:
(Oops, resending and forcing html off.)
On 05/09/2015 03:19 AM, Andy Lutomirski wrote:
quoted
Naming aside, I don't think this should be a per-task flag at all. We
already have way too much overhead per syscall in nohz mode, and it
would be nice to get the per-syscall overhead as low as possible. We
should strive, for all tasks, to keep syscall overhead down*and*
avoid as many interrupts as possible.
That being said, I do see a legitimate use for a way to tell the
kernel "I'm going to run in userspace for a long time; stay away".
But shouldn't that be a single operation, not an ongoing flag? IOW, I
think that we should have a new syscall quiesce() or something rather
than a prctl.
Yes, if all you are concerned about is quiescing the tick, we could
probably do it as a new syscall.
I do note that you'd want to try to actually do the quiesce as late as
possible - in particular, if you just did it in the usual syscall, you
might miss out on a timer that is set by softirq, or even something
that happened when you called schedule() on the syscall exit path.
Doing it as late as we are doing helps to ensure that that doesn't
happen. We could still arrange for this semantics by having a new
quiesce() syscall set a temporary task bit that was cleared on
return to userspace, but as you pointed out in a different email,
that gets tricky if you end up doing multiple user_exit() calls on
your way back to userspace.
We should fix that, then. A quiesce() syscall can certainly arrange
to clean up on final exit.
More to the point, I think it's actually important to know when an
application believes it's in userspace-only mode as an actual state
bit, rather than just during its transitional moment.
We can do that, too, with a new flag that's cleared on the next entry.
If an
application calls the kernel at an unexpected time (third-party code
is the usual culprit for our customers, whether it's syscalls, page
faults, or other things) we would prefer to have the "quiesce"
semantics stay in force and cause the third-party code to be
visibly very slow, rather than cause a totally unexpected and
hard-to-diagnose interrupt show up later as we are still going
around the loop that we thought was safely userspace-only.
I'm not really convinced that we should design this feature around
ease of debugging userspace screwups. There are already plenty of
ways to do that part. Userspace getting an interrupt because
userspace accidentally did a syscall is very different from userspace
getting interrupted due to an IPI.
And, for debugging the kernel, it's crazy helpful to have that state
bit in place: see patch 6/6 in the series for how we can diagnose
things like "a different core just queued an IPI that will hit a
dataplane core unexpectedly". Having that state bit makes this sort
of thing a trivial check in the kernel and relatively easy to debug.
As above, this can be done with a one-time operation, too.
Finally, I proposed a "strict" mode in patch 5/6 where we kill the
process if it voluntarily enters the kernel by mistake after saying it
wasn't going to any more. To do this requires a state bit, so
carrying another state bit for "quiesce on user entry" seems pretty
reasonable.
I still dislike that in the form you chose. It's too deadly to be
useful for anyone but the hardest RT users.
I think I'd be okay with variants, though: let a suitably privileged
process ask for a signal on inadvertent kernel entry or rig up an fd
to be notified when one of these bad entries happens. Queueing
something to a pollable fd would work, too.
See that thread for more comments.
--Andy
From: Mike Galbraith <hidden> Date: 2015-05-09 07:19:51
On Sat, 2015-05-09 at 09:05 +0200, Ingo Molnar wrote:
* Andrew Morton [off-list ref] wrote:
quoted
On Fri, 8 May 2015 19:11:10 -0400 Chris Metcalf [off-list ref] wrote:
quoted
On 5/8/2015 5:22 PM, Steven Rostedt wrote:
quoted
On Fri, 8 May 2015 14:18:24 -0700
Andrew Morton [off-list ref] wrote:
quoted
On Fri, 8 May 2015 13:58:41 -0400 Chris Metcalf [off-list ref] wrote:
quoted
A prctl() option (PR_SET_DATAPLANE) is added
Dumb question: what does the term "dataplane" mean in this context? I
can't see the relationship between those words and what this patch
does.
I was thinking the same thing. I haven't gotten around to searching
DATAPLANE yet.
I would assume we want a name that is more meaningful for what is
happening.
The text in the commit message and the 0/6 cover letter do try to explain
the concept. The terminology comes, I think, from networking line cards,
where the "dataplane" is the part of the application that handles all the
fast path processing of network packets, and the "control plane" is the part
that handles routing updates, etc., generally slow-path stuff. I've probably
just been using the terms so long they seem normal to me.
That said, what would be clearer? NO_HZ_STRICT as a superset of
NO_HZ_FULL? Or move away from the NO_HZ terminology a bit; after all,
we're talking about no interrupts of any kind, and maybe NO_HZ is too
limited in scope? So, NO_INTERRUPTS? USERSPACE_ONLY? Or look
to vendors who ship bare-metal runtimes and call it BARE_METAL?
Borrow the Tilera marketing name and call it ZERO_OVERHEAD?
Maybe BARE_METAL seems most plausible -- after DATAPLANE, to me,
of course :-)
'baremetal' has uses in virtualization speak, so I think that would be
confusing.
quoted
I like NO_INTERRUPTS. Simple, direct.
NO_HZ_PURE?
Hm, coke light, coke zero... OS_LIGHT and OS_ZERO?
-Mike
From: Gilad Ben Yossef <hidden> Date: 2015-05-09 10:51:31
From: Mike Galbraith [mailto:umgwanakikbuti@gmail.com]
Sent: Saturday, May 09, 2015 10:20 AM
To: Ingo Molnar
Cc: Andrew Morton; Chris Metcalf; Steven Rostedt; Gilad Ben Yossef; Ingo
Molnar; Peter Zijlstra; Rik van Riel; Tejun Heo; Frederic Weisbecker;
Thomas Gleixner; Paul E. McKenney; Christoph Lameter; Srivatsa S. Bhat;
linux-doc@vger.kernel.org; linux-api@vger.kernel.org; linux-
kernel@vger.kernel.org
Subject: Re: [PATCH 0/6] support "dataplane" mode for nohz_full
On Sat, 2015-05-09 at 09:05 +0200, Ingo Molnar wrote:
quoted
* Andrew Morton [off-list ref] wrote:
quoted
On Fri, 8 May 2015 19:11:10 -0400 Chris Metcalf [off-list ref]
wrote:
quoted
quoted
quoted
On 5/8/2015 5:22 PM, Steven Rostedt wrote:
quoted
On Fri, 8 May 2015 14:18:24 -0700
Andrew Morton [off-list ref] wrote:
quoted
On Fri, 8 May 2015 13:58:41 -0400 Chris Metcalf
[off-list ref] wrote:
quoted
quoted
quoted
quoted
quoted
quoted
A prctl() option (PR_SET_DATAPLANE) is added
Dumb question: what does the term "dataplane" mean in this
context? I
quoted
quoted
quoted
quoted
quoted
can't see the relationship between those words and what this
patch
quoted
quoted
quoted
quoted
quoted
does.
I was thinking the same thing. I haven't gotten around to
searching
quoted
quoted
quoted
quoted
DATAPLANE yet.
I would assume we want a name that is more meaningful for what is
happening.
The text in the commit message and the 0/6 cover letter do try to
explain
quoted
quoted
quoted
the concept. The terminology comes, I think, from networking line
cards,
quoted
quoted
quoted
where the "dataplane" is the part of the application that handles
all the
quoted
quoted
quoted
fast path processing of network packets, and the "control plane" is
the part
quoted
quoted
quoted
that handles routing updates, etc., generally slow-path stuff. I've
probably
quoted
quoted
quoted
just been using the terms so long they seem normal to me.
That said, what would be clearer? NO_HZ_STRICT as a superset of
NO_HZ_FULL? Or move away from the NO_HZ terminology a bit; after
all,
quoted
quoted
quoted
we're talking about no interrupts of any kind, and maybe NO_HZ is
too
quoted
quoted
quoted
limited in scope? So, NO_INTERRUPTS? USERSPACE_ONLY? Or look
to vendors who ship bare-metal runtimes and call it BARE_METAL?
Borrow the Tilera marketing name and call it ZERO_OVERHEAD?
Maybe BARE_METAL seems most plausible -- after DATAPLANE, to me,
of course :-)
'baremetal' has uses in virtualization speak, so I think that would be
confusing.
quoted
I like NO_INTERRUPTS. Simple, direct.
NO_HZ_PURE?
Hm, coke light, coke zero... OS_LIGHT and OS_ZERO?
LOL... you forgot OS_CLASSIC for backwards compatibility :-)
How about TASK_SOLO?
Yes, you are trying to achieve the least amount of interference but the bigger context is about monopolizing a single CPU for yourself.
Anyway it is worth pointing out that while NO_HZ_FULL is very useful in conjunction with this turning the tick off is useful also if you have multiple tasks runnable (e.g. if you know you only need to context switch in 100 ms, why keep a periodic interrupt running?) even though we don't support it *right now*. It might be a good idea not to entangle these concepts too much.
Gilad
Gilad Ben-Yossef
Chief Software Architect
EZchip Technologies Ltd.
37 Israel Pollak Ave, Kiryat Gat 82025 ,Israel
Tel: +972-4-959-6666 ext. 576, Fax: +972-8-681-1483
Mobile: +972-52-826-0388, US Mobile: +1-973-826-0388
Email: giladb@ezchip.com, Web: http://www.ezchip.com
From: Steven Rostedt <rostedt@goodmis.org> Date: 2015-05-11 12:58:07
NO_HZ_LEAVE_ME_THE_FSCK_ALONE!
On Sat, 9 May 2015 09:05:38 +0200
Ingo Molnar [off-list ref] wrote:
So I think we should either rename NO_HZ_FULL to NO_HZ_PURE, or keep
it at NO_HZ_FULL: because the intention of NO_HZ_FULL was always to be
such a 'zero overhead' mode of operation, where if user-space runs, it
won't get interrupted in any way.
All kidding aside, I think this is the real answer. We don't need a new
NO_HZ, we need to make NO_HZ_FULL work. Right now it doesn't do exactly
what it was created to do. That should be fixed.
Please lets get NO_HZ_FULL up to par. That should be the main focus.
-- Steve
On Mon, May 11, 2015 at 08:57:59AM -0400, Steven Rostedt wrote:
NO_HZ_LEAVE_ME_THE_FSCK_ALONE!
On Sat, 9 May 2015 09:05:38 +0200
Ingo Molnar [off-list ref] wrote:
quoted
So I think we should either rename NO_HZ_FULL to NO_HZ_PURE, or keep
it at NO_HZ_FULL: because the intention of NO_HZ_FULL was always to be
such a 'zero overhead' mode of operation, where if user-space runs, it
won't get interrupted in any way.
All kidding aside, I think this is the real answer. We don't need a new
NO_HZ, we need to make NO_HZ_FULL work. Right now it doesn't do exactly
what it was created to do. That should be fixed.
Please lets get NO_HZ_FULL up to par. That should be the main focus.
Now if we can achieve to make NO_HZ_FULL behave in a specific way
that fits everyone's usecase, I'll be happy.
But some people may expect hard isolation requirement (Real Time, deterministic
latency) and others softer isolation (HPC, only interested in performance, can
live with one rare random tick, so no need to loop before returning to userspace
until we have the no-noise guarantee).
I expect some Real Time users may want this kind of dataplane mode where a syscall
or whatever sleeps until the system is ready to provide the guarantee that no
disturbance is going to happen for a given time. I'm not sure HPC users are interested
in that.
In fact it goes along the fact that NO_HZ_FULL was really only supposed to be about
the tick and now people are introducing more and more kernel default presetting that
assume NO_HZ_FULL implies ISOLATION which is about all kind of noise (tick, tasks, irqs,
...). Which is true but what kind of ISOLATION?
Probably NO_HZ_FULL should really only be about stopping the tick then some sort
of CONFIG_ISOLATION would drive the kind of isolation we are interested in
and hereby the behaviour of NO_HZ_FULL, workqueues, timers, tasks affinity, irqs
affinity, dataplane mode, ...
From: Mike Galbraith <hidden> Date: 2015-05-11 19:19:48
On Mon, 2015-05-11 at 17:36 +0200, Frederic Weisbecker wrote:
I expect some Real Time users may want this kind of dataplane mode where a syscall
or whatever sleeps until the system is ready to provide the guarantee that no
disturbance is going to happen for a given time. I'm not sure HPC users are interested
in that.
I bet they are. RT is just a different way to spell HPC, and reverse.
In fact it goes along the fact that NO_HZ_FULL was really only supposed to be about
the tick and now people are introducing more and more kernel default presetting that
assume NO_HZ_FULL implies ISOLATION which is about all kind of noise (tick, tasks, irqs,
...). Which is true but what kind of ISOLATION?
True, nohz mode and various isolation measures are distinct properties.
NO_HZ_FULL is kinda pointless without isolation measures to go with it,
but you're right.
I really shouldn't have acked nohz_full -> isolcpus. Beside the fact
that old static isolcpus was _supposed_ to crawl off and die, I know
beyond doubt that having isolated a cpu as well as you can definitely
does NOT imply that said cpu should become tickless. I routinely run a
load model that wants all the isolation it can get. It's not single
task compute though, rt executive coordinating rt workers, and of course
wants every cycle it can get, so nohz_full is less than helpful.
-Mike
From: Chris Metcalf <hidden> Date: 2015-05-11 19:25:38
On 05/11/2015 03:19 PM, Mike Galbraith wrote:
I really shouldn't have acked nohz_full -> isolcpus. Beside the fact
that old static isolcpus was_supposed_ to crawl off and die, I know
beyond doubt that having isolated a cpu as well as you can definitely
does NOT imply that said cpu should become tickless.
True, at a high level, I agree that it would be better to have a
top-level concept like Frederic's proposed ISOLATION that includes
isolcpus and nohz_cpu (and other stuff as needed).
That said, what you wrote above is wrong; even with the patch you
acked, setting isolcpus does not automatically turn on nohz_full for
a given cpu. The patch made it true the other way around: when
you say nohz_full, you automatically get isolcpus on that cpu too.
That does, at least, make sense for the semantics of nohz_full.
--
Chris Metcalf, EZChip Semiconductor
http://www.ezchip.com
From: Mike Galbraith <hidden> Date: 2015-05-12 01:48:05
On Mon, 2015-05-11 at 15:25 -0400, Chris Metcalf wrote:
On 05/11/2015 03:19 PM, Mike Galbraith wrote:
quoted
I really shouldn't have acked nohz_full -> isolcpus. Beside the fact
that old static isolcpus was_supposed_ to crawl off and die, I know
beyond doubt that having isolated a cpu as well as you can definitely
does NOT imply that said cpu should become tickless.
True, at a high level, I agree that it would be better to have a
top-level concept like Frederic's proposed ISOLATION that includes
isolcpus and nohz_cpu (and other stuff as needed).
That said, what you wrote above is wrong; even with the patch you
acked, setting isolcpus does not automatically turn on nohz_full for
a given cpu. The patch made it true the other way around: when
you say nohz_full, you automatically get isolcpus on that cpu too.
That does, at least, make sense for the semantics of nohz_full.
I didn't write that, I wrote nohz_full implies (spelled '->') isolcpus.
Yes, with nohz_full currently being static, the old allegedly dying but
also static isolcpus scheduler off switch is a convenient thing to wire
the nohz_full CPU SET (<- hint;) property to.
-Mike
From: Mike Galbraith <hidden> Date: 2015-05-12 04:35:42
On Tue, 2015-05-12 at 03:47 +0200, Mike Galbraith wrote:
On Mon, 2015-05-11 at 15:25 -0400, Chris Metcalf wrote:
quoted
On 05/11/2015 03:19 PM, Mike Galbraith wrote:
quoted
I really shouldn't have acked nohz_full -> isolcpus. Beside the fact
that old static isolcpus was_supposed_ to crawl off and die, I know
beyond doubt that having isolated a cpu as well as you can definitely
does NOT imply that said cpu should become tickless.
True, at a high level, I agree that it would be better to have a
top-level concept like Frederic's proposed ISOLATION that includes
isolcpus and nohz_cpu (and other stuff as needed).
That said, what you wrote above is wrong; even with the patch you
acked, setting isolcpus does not automatically turn on nohz_full for
a given cpu. The patch made it true the other way around: when
you say nohz_full, you automatically get isolcpus on that cpu too.
That does, at least, make sense for the semantics of nohz_full.
I didn't write that, I wrote nohz_full implies (spelled '->') isolcpus.
Yes, with nohz_full currently being static, the old allegedly dying but
also static isolcpus scheduler off switch is a convenient thing to wire
the nohz_full CPU SET (<- hint;) property to.
BTW, another facet of this: Rik wants to make isolcpus immune to
cpusets, which makes some sense, user did say isolcpus=, but that also
makes isolcpus truly static. If the user now says nohz_full=, they lose
the ability to deactivate CPU isolation, making the set fairly useless
for anything other than HPC. Currently, the user can flip the isolation
switch as he sees fit. He takes a size extra large performance hit for
having said nohz_full=, but he doesn't lose generic utility.
-Mike
From: Paul E. McKenney <hidden> Date: 2015-05-11 17:19:25
On Mon, May 11, 2015 at 08:57:59AM -0400, Steven Rostedt wrote:
NO_HZ_LEAVE_ME_THE_FSCK_ALONE!
NO_HZ_OVERFLOWING?
Kconfig naming controversy aside, I believe this patchset is addressing
a real need. Might need additional adjustment, but something useful.
Thanx, Paul
On Sat, 9 May 2015 09:05:38 +0200
Ingo Molnar [off-list ref] wrote:
quoted
So I think we should either rename NO_HZ_FULL to NO_HZ_PURE, or keep
it at NO_HZ_FULL: because the intention of NO_HZ_FULL was always to be
such a 'zero overhead' mode of operation, where if user-space runs, it
won't get interrupted in any way.
All kidding aside, I think this is the real answer. We don't need a new
NO_HZ, we need to make NO_HZ_FULL work. Right now it doesn't do exactly
what it was created to do. That should be fixed.
Please lets get NO_HZ_FULL up to par. That should be the main focus.
-- Steve
From: Andrew Morton <akpm@linux-foundation.org> Date: 2015-05-11 17:27:48
On Mon, 11 May 2015 10:19:16 -0700 "Paul E. McKenney" [off-list ref] wrote:
On Mon, May 11, 2015 at 08:57:59AM -0400, Steven Rostedt wrote:
quoted
NO_HZ_LEAVE_ME_THE_FSCK_ALONE!
NO_HZ_OVERFLOWING?
Actually, "NO_HZ" shouldn't appear in the name at all. The objective
is to permit userspace to execute without interruption. NO_HZ is a
part of that, as is NO_INTERRUPTS. The "NO_HZ" thing is a historical
artifact from an early partial implementation.
On Mon, May 11, 2015 at 10:27:44AM -0700, Andrew Morton wrote:
On Mon, 11 May 2015 10:19:16 -0700 "Paul E. McKenney" [off-list ref] wrote:
quoted
On Mon, May 11, 2015 at 08:57:59AM -0400, Steven Rostedt wrote:
quoted
NO_HZ_LEAVE_ME_THE_FSCK_ALONE!
NO_HZ_OVERFLOWING?
Actually, "NO_HZ" shouldn't appear in the name at all. The objective
is to permit userspace to execute without interruption. NO_HZ is a
part of that, as is NO_INTERRUPTS. The "NO_HZ" thing is a historical
artifact from an early partial implementation.
Agreed! Which is why I'd rather advocate in favour of CONFIG_ISOLATION.
From: Steven Rostedt <rostedt@goodmis.org> Date: 2015-05-11 18:00:21
On Mon, 11 May 2015 19:33:06 +0200
Frederic Weisbecker [off-list ref] wrote:
On Mon, May 11, 2015 at 10:27:44AM -0700, Andrew Morton wrote:
quoted
On Mon, 11 May 2015 10:19:16 -0700 "Paul E. McKenney" [off-list ref] wrote:
quoted
On Mon, May 11, 2015 at 08:57:59AM -0400, Steven Rostedt wrote:
quoted
NO_HZ_LEAVE_ME_THE_FSCK_ALONE!
NO_HZ_OVERFLOWING?
Actually, "NO_HZ" shouldn't appear in the name at all. The objective
is to permit userspace to execute without interruption. NO_HZ is a
part of that, as is NO_INTERRUPTS. The "NO_HZ" thing is a historical
artifact from an early partial implementation.
Agreed! Which is why I'd rather advocate in favour of CONFIG_ISOLATION.
Then we should have CONFIG_LEAVE_ME_THE_FSCK_ALONE. Hmm, I guess that's
just an synonym for CONFIG_ISOLATION.
-- Steve
From: Chris Metcalf <hidden> Date: 2015-05-11 18:10:25
A bunch of issues have been raised by various folks (thanks!) and
I'll try to break them down and respond to them in a few different
emails. This email is just about the issue of naming and whether the
proposed patch series should even have its own "name" or just be part
of NO_HZ_FULL.
First, Ingo and Steven both suggested that this new "dataplane" mode
(or whatever we want to call it; see below) should just be rolled into
the existing NO_HZ_FULL and that we should focus on making that work
better.
Steven writes:
All kidding aside, I think this is the real answer. We don't need a new
NO_HZ, we need to make NO_HZ_FULL work. Right now it doesn't do exactly
what it was created to do. That should be fixed.
The claim I'm making is that it's worthwhile to differentiate the two
semantics. Plain NO_HZ_FULL just says "kernel makes a best effort to
avoid periodic interrupts without incurring any serious overhead". My
patch series allows an app to request "kernel makes an absolute
commitment to avoid all interrupts regardless of cost when leaving
kernel space". These are different enough ideas, and serve different
enough application needs, that I think they should be kept distinct.
Frederic actually summed this up very nicely in his recent email when
he wrote "some people may expect hard isolation requirement (Real
Time, deterministic latency) and others softer isolation (HPC, only
interested in performance, can live with one rare random tick, so no
need to loop before returning to userspace until we have the no-noise
guarantee)."
So we need a way for apps to ask for the "harder" mode and let
the softer mode be the default.
What about naming? We may or may not want to have a Kconfig flag
for this, and we may or may not have a separate mode for it, but
we still will need some kind of name to talk about it with. (In
particular there's the prctl name, if we take that approach, and
potential boot command-line flags to consider naming for.)
I'll quickly cover the suggestions that have been raised:
- DATAPLANE. My suggestion, seemingly broadly disliked by folks
who felt it wasn't apparent what it meant. Probably a fair point.
- NO_INTERRUPTS (Andrew). Captures some of the sense, but was
criticized pretty fairly by Ingo as being too negative, confusing
with perf nomenclature, and too long :-)
- PURE (Ingo). Proposed as an alternative to NO_HZ_FULL, but we could
use it as a name for this new mode. However, I think it's not clear
enough how FULL and PURE can/should relate to each other from the
names alone.
- BARE_METAL (me). Ingo observes it's confusing with respect to
virtualization.
- TASK_SOLO (Gilad). Not sure this conveys enough of the semantics.
- OS_LIGHT/OS_ZERO and NO_HZ_LEAVE_ME_THE_FSCK_ALONE. Excellent
ideas :-)
- ISOLATION (Frederic). I like this but it conflicts with other uses
of "isolation" in the kernel: cgroup isolation, lru page isolation,
iommu isolation, scheduler isolation (at least it's a superset of
that one), etc. Also, we're not exactly isolating a task - often
a "dataplane" app consists of a bunch of interacting threads in
userspace, so not exactly isolated. So perhaps it's too confusing.
- OVERFLOWING (Steven) - not sure I understood this one, honestly.
I suggested earlier a few other candidates that I don't love, but no
one commented on: NO_HZ_STRICT, USERSPACE_ONLY, and ZERO_OVERHEAD.
One thing I'm leaning towards is to remove the intermediate state of
DATAPLANE_ENABLE and say that there is really only one primary state,
DATAPLANE_QUIESCE (or whatever we call it). The "dataplane but no
quiesce" state probably isn't that useful, since it doesn't offer the
hard guarantee that is the entire point of this patch series. So that
opens the idea of using the name NO_HZ_QUIESCE or just QUIESCE as the
word that describes the mode; of course this sort of conflicts with
RCU quiesce (though it is a superset of that so maybe that's OK).
One new idea I had is to use NO_HZ_HARD to reflect what Frederic was
suggesting about "soft" and "hard" requirements for NO_HZ. So
enabling NO_HZ_HARD would enable my suggested QUIESCE mode.
One way to focus this discussion is on the user API naming. I had
prctl(PR_SET_DATAPLANE), which was attractive in being a "positive"
noun. A lot of the other suggestions fail this test in various way.
Reasonable candidates seem to be:
PR_SET_OS_ZERO
PR_SET_TASK_SOLO
PR_SET_ISOLATION
Another possibility:
PR_SET_NONSTOP
Or take Andrew's NO_INTERRUPTS and have:
PR_SET_UNINTERRUPTED
I slightly favor ISOLATION at this point despite the overlap with
other kernel concepts.
Let the bike-shedding continue! :-)
--
Chris Metcalf, EZChip Semiconductor
http://www.ezchip.com
From: Steven Rostedt <rostedt@goodmis.org> Date: 2015-05-11 18:36:50
On Mon, 11 May 2015 14:09:59 -0400
Chris Metcalf [off-list ref] wrote:
Steven writes:
quoted
All kidding aside, I think this is the real answer. We don't need a new
NO_HZ, we need to make NO_HZ_FULL work. Right now it doesn't do exactly
what it was created to do. That should be fixed.
The claim I'm making is that it's worthwhile to differentiate the two
semantics. Plain NO_HZ_FULL just says "kernel makes a best effort to
avoid periodic interrupts without incurring any serious overhead". My
patch series allows an app to request "kernel makes an absolute
commitment to avoid all interrupts regardless of cost when leaving
kernel space". These are different enough ideas, and serve different
enough application needs, that I think they should be kept distinct.
Frederic actually summed this up very nicely in his recent email when
he wrote "some people may expect hard isolation requirement (Real
Time, deterministic latency) and others softer isolation (HPC, only
interested in performance, can live with one rare random tick, so no
need to loop before returning to userspace until we have the no-noise
guarantee)."
So we need a way for apps to ask for the "harder" mode and let
the softer mode be the default.
Fair enough. But I would hope that this would improve on NO_HZ_FULL as
well.
What about naming? We may or may not want to have a Kconfig flag
for this, and we may or may not have a separate mode for it, but
we still will need some kind of name to talk about it with. (In
particular there's the prctl name, if we take that approach, and
potential boot command-line flags to consider naming for.)
I'll quickly cover the suggestions that have been raised:
- DATAPLANE. My suggestion, seemingly broadly disliked by folks
who felt it wasn't apparent what it meant. Probably a fair point.
- NO_INTERRUPTS (Andrew). Captures some of the sense, but was
criticized pretty fairly by Ingo as being too negative, confusing
with perf nomenclature, and too long :-)
What about NO_INTERRUPTIONS
- PURE (Ingo). Proposed as an alternative to NO_HZ_FULL, but we could
use it as a name for this new mode. However, I think it's not clear
enough how FULL and PURE can/should relate to each other from the
names alone.
I would find the two confusing as well.
- BARE_METAL (me). Ingo observes it's confusing with respect to
virtualization.
This is also confusing.
- TASK_SOLO (Gilad). Not sure this conveys enough of the semantics.
Agreed.
- OS_LIGHT/OS_ZERO and NO_HZ_LEAVE_ME_THE_FSCK_ALONE. Excellent
ideas :-)
At least the LEAVE_ME_ALONE conveys the semantics ;-)
- ISOLATION (Frederic). I like this but it conflicts with other uses
of "isolation" in the kernel: cgroup isolation, lru page isolation,
iommu isolation, scheduler isolation (at least it's a superset of
that one), etc. Also, we're not exactly isolating a task - often
a "dataplane" app consists of a bunch of interacting threads in
userspace, so not exactly isolated. So perhaps it's too confusing.
- OVERFLOWING (Steven) - not sure I understood this one, honestly.
Actually, that was suggested by Paul McKenney.
I suggested earlier a few other candidates that I don't love, but no
one commented on: NO_HZ_STRICT, USERSPACE_ONLY, and ZERO_OVERHEAD.
One thing I'm leaning towards is to remove the intermediate state of
DATAPLANE_ENABLE and say that there is really only one primary state,
DATAPLANE_QUIESCE (or whatever we call it). The "dataplane but no
quiesce" state probably isn't that useful, since it doesn't offer the
hard guarantee that is the entire point of this patch series. So that
opens the idea of using the name NO_HZ_QUIESCE or just QUIESCE as the
word that describes the mode; of course this sort of conflicts with
RCU quiesce (though it is a superset of that so maybe that's OK).
One new idea I had is to use NO_HZ_HARD to reflect what Frederic was
suggesting about "soft" and "hard" requirements for NO_HZ. So
enabling NO_HZ_HARD would enable my suggested QUIESCE mode.
One way to focus this discussion is on the user API naming. I had
prctl(PR_SET_DATAPLANE), which was attractive in being a "positive"
noun. A lot of the other suggestions fail this test in various way.
Reasonable candidates seem to be:
PR_SET_OS_ZERO
PR_SET_TASK_SOLO
PR_SET_ISOLATION
Another possibility:
PR_SET_NONSTOP
Or take Andrew's NO_INTERRUPTS and have:
PR_SET_UNINTERRUPTED
For another possible answer, what about
SET_TRANQUILITY
A state with no disturbances.
-- Steve
I slightly favor ISOLATION at this point despite the overlap with
other kernel concepts.
Let the bike-shedding continue! :-)
- ISOLATION (Frederic). I like this but it conflicts with other uses
of "isolation" in the kernel: cgroup isolation, lru page isolation,
iommu isolation, scheduler isolation (at least it's a superset of
that one), etc. Also, we're not exactly isolating a task - often
a "dataplane" app consists of a bunch of interacting threads in
userspace, so not exactly isolated. So perhaps it's too confusing.
So I'd vote for Frederic's CONFIG_ISOLATION=y, mostly because this is
a high level kernel feature, so it won't conflict with isolation
concepts in lower level subsystems such as IOMMU isolation - and other
higher level features like scheduler isolation are basically another
partial implementation we want to merge with all this...
nohz, RCU tricks, watchdog defaults, isolcpus and various other
measures to keep these CPUs and workloads as isolated as possible
are (or should become) components of this high level concept.
Ideally CONFIG_ISOLATION=y would be a kernel feature that has almost
zero overhead on normal workloads and on non-isolated CPUs, so that
Linux distributions can enable it.
Enabling CONFIG_ISOLATION=y should be the only 'kernel config' step
needed: just like cpusets, the configuration of isolated CPUs should
be a completely boot option free excercise that can be dynamically
done and undone by the administrator via an intuitive interface.
Thanks,
Ingo
From: Peter Zijlstra <peterz@infradead.org> Date: 2015-05-12 11:48:29
On Tue, May 12, 2015 at 11:10:32AM +0200, Ingo Molnar wrote:
So I'd vote for Frederic's CONFIG_ISOLATION=y, mostly because this is
a high level kernel feature, so it won't conflict with isolation
concepts in lower level subsystems such as IOMMU isolation - and other
higher level features like scheduler isolation are basically another
partial implementation we want to merge with all this...
But why do we need a CONFIG flag for something that has no content?
That is, I do not see anything much; except the 'I want to stay in
userspace and kill me otherwise' flag, and I'm not sure that warrants a
CONFIG flag like this.
Other than that, its all a combination of NOHZ_FULL and cpusets/isolcpus
and whatnot.
On Tue, May 12, 2015 at 11:10:32AM +0200, Ingo Molnar wrote:
quoted
So I'd vote for Frederic's CONFIG_ISOLATION=y, mostly because this
is a high level kernel feature, so it won't conflict with
isolation concepts in lower level subsystems such as IOMMU
isolation - and other higher level features like scheduler
isolation are basically another partial implementation we want to
merge with all this...
But why do we need a CONFIG flag for something that has no content?
That is, I do not see anything much; except the 'I want to stay in
userspace and kill me otherwise' flag, and I'm not sure that
warrants a CONFIG flag like this.
Other than that, its all a combination of NOHZ_FULL and
cpusets/isolcpus and whatnot.
Yes, that's what I meant: CONFIG_ISOLATION would trigger what is
NO_HZ_FULL today - we could possibly even remove CONFIG_NO_HZ_FULL as
an individual Kconfig option?
CONFIG_ISOLATION=y would express the guarantee from the kernel that
it's possible for user-space to configure itself to run undisturbed -
instead of the current inconsistent set of options and facilities.
A bit like CONFIG_PREEMPT_RT is more than just preemptable spinlocks,
it also tries to offer various facilities and tune the defaults to
turn the kernel hard-rt.
Does that make sense to you?
Thanks,
Ingo
From: Peter Zijlstra <peterz@infradead.org> Date: 2015-05-12 12:39:34
On Tue, May 12, 2015 at 02:34:40PM +0200, Ingo Molnar wrote:
Yes, that's what I meant: CONFIG_ISOLATION would trigger what is
NO_HZ_FULL today - we could possibly even remove CONFIG_NO_HZ_FULL as
an individual Kconfig option?
Ah, as a rename of nohz_full, sure that might work.
On Tue, May 12, 2015 at 02:34:40PM +0200, Ingo Molnar wrote:
quoted
Yes, that's what I meant: CONFIG_ISOLATION would trigger what is
NO_HZ_FULL today - we could possibly even remove CONFIG_NO_HZ_FULL
as an individual Kconfig option?
Ah, as a rename of nohz_full, sure that might work.
It could also be named CONFIG_CPU_ISOLATION=y, to make it more
explicit what it's about.
Thanks,
Ingo
On Tue, May 12, 2015 at 02:34:40PM +0200, Ingo Molnar wrote:
* Peter Zijlstra [off-list ref] wrote:
quoted
On Tue, May 12, 2015 at 11:10:32AM +0200, Ingo Molnar wrote:
quoted
So I'd vote for Frederic's CONFIG_ISOLATION=y, mostly because this
is a high level kernel feature, so it won't conflict with
isolation concepts in lower level subsystems such as IOMMU
isolation - and other higher level features like scheduler
isolation are basically another partial implementation we want to
merge with all this...
But why do we need a CONFIG flag for something that has no content?
That is, I do not see anything much; except the 'I want to stay in
userspace and kill me otherwise' flag, and I'm not sure that
warrants a CONFIG flag like this.
Other than that, its all a combination of NOHZ_FULL and
cpusets/isolcpus and whatnot.
Yes, that's what I meant: CONFIG_ISOLATION would trigger what is
NO_HZ_FULL today - we could possibly even remove CONFIG_NO_HZ_FULL as
an individual Kconfig option?
Right, we could return to what we had previously: CONFIG_NO_HZ. A config
that enables dynticks-idle by default and allows full dynticks if nohz_full=
boot option is passed (or something driven by higher level isolation interface).
Because eventually, distros enable NO_HZ_FULL so that their 0.0001% users
can use it. Well at least Red Hat does.
CONFIG_ISOLATION=y would express the guarantee from the kernel that
it's possible for user-space to configure itself to run undisturbed -
instead of the current inconsistent set of options and facilities.
A bit like CONFIG_PREEMPT_RT is more than just preemptable spinlocks,
it also tries to offer various facilities and tune the defaults to
turn the kernel hard-rt.
Does that make sense to you?
Right although distros tend to want features to be enabled dynamically
so that they have a single kernel to maintain. Things like PREEMPT_RT
really need to be a different kernel because fundamental primitives like
spinlocks must be implemented statically.
But isolation can be a boot-enabled, or even runtime-enabled, as it's only
about timer,irq,task affinity. Full Nohz is more complicated but it can
be runtime toggled in the future.
So we can bring CONFIG_CPU_ISOLATION, at least for distros that are really
not interested in that so they can disable it. CONFIG_CPU_ISOLATION=y would
bring an ability which is default-disabled and driven dynamically through whatever
interface.
From: Chris Metcalf <hidden> Date: 2015-05-12 21:05:59
On 05/12/2015 05:10 AM, Ingo Molnar wrote:
* Chris Metcalf [off-list ref] wrote:
quoted
- ISOLATION (Frederic). I like this but it conflicts with other uses
of "isolation" in the kernel: cgroup isolation, lru page isolation,
iommu isolation, scheduler isolation (at least it's a superset of
that one), etc. Also, we're not exactly isolating a task - often
a "dataplane" app consists of a bunch of interacting threads in
userspace, so not exactly isolated. So perhaps it's too confusing.
So I'd vote for Frederic's CONFIG_ISOLATION=y, mostly because this is
a high level kernel feature, so it won't conflict with isolation
concepts in lower level subsystems such as IOMMU isolation - and other
higher level features like scheduler isolation are basically another
partial implementation we want to merge with all this...
nohz, RCU tricks, watchdog defaults, isolcpus and various other
measures to keep these CPUs and workloads as isolated as possible
are (or should become) components of this high level concept.
Ideally CONFIG_ISOLATION=y would be a kernel feature that has almost
zero overhead on normal workloads and on non-isolated CPUs, so that
Linux distributions can enable it.
Using CONFIG_CPU_ISOLATION to capture all this stuff instead of
making CONFIG_NO_HZ_FULL do it seems plausible for naming.
However, this feels like just bombing the current naming to this
new name, right? I'd like to argue that this is orthogonal to adding
new isolation functionality into no_hz_full, as my patch series has
been doing. Perhaps we can defer this to a follow-up patch series?
I'm happy to do the work but I'm not sure we want to bundle all
that churn into the current patch series under consideration.
I can use cpu_isolation_xxx for naming in the current patch series
so we don't have to come back and bomb that later.
Enabling CONFIG_ISOLATION=y should be the only 'kernel config' step
needed: just like cpusets, the configuration of isolated CPUs should
be a completely boot option free excercise that can be dynamically
done and undone by the administrator via an intuitive interface.
Eventually isolation can be runtime-enabled, but for now I think
it makes sense to be boot-enabled. As Frederic suggested, we
can arrange full nohz to be runtime toggled in the future.
I agree that it should be reasonable to compile it in by default.
On 05/12/2015 07:48 AM, Peter Zijlstra wrote:
But why do we need a CONFIG flag for something that has no content?
That is, I do not see anything much; except the 'I want to stay in
userspace and kill me otherwise' flag, and I'm not sure that warrants a
CONFIG flag like this.
Other than that, its all a combination of NOHZ_FULL and cpusets/isolcpus
and whatnot.
There are three major pieces here - one is the STRICT piece
that you allude to, but there is also the piece where we quiesce
tasks in the kernel until no timer interrupts are pending, and the
piece that allows easy debugging of stray IRQs etc to isolated cpus.
--
Chris Metcalf, EZChip Semiconductor
http://www.ezchip.com