From: GeHao Kang <hidden> Date: 2016-08-17 06:38:46
Hi Frederic and Chris,
When the lmbench runs on the tickless isolated CPU, the context switch
latency on
this CPU is higher than the one on other CPU. The test platform is
Linux 4.4.12 with NO_HZ_FULL on I.MX6Q sabresd. The following is the
lmbench results about context switch:
lmbench runs on nonspecific CPU:
Context switching - times in microseconds - smaller is better
-------------------------------------------------------------------------
Host OS 2p/0K 2p/16K 2p/64K 8p/16K 8p/64K 16p/16K 16p/64K
ctxsw ctxsw ctxsw ctxsw ctxsw ctxsw ctxsw
--------- ------------- ------ ------ ------ ------ ------ ------- -------
imx6qsabr Linux 4.4.12- 12.6 12.8 16.1 26.6 42.1 36.5 70.0
lmbench runs on the isolated CPU:
Context switching - times in microseconds - smaller is better
-------------------------------------------------------------------------
Host OS 2p/0K 2p/16K 2p/64K 8p/16K 8p/64K 16p/16K 16p/64K
ctxsw ctxsw ctxsw ctxsw ctxsw ctxsw ctxsw
--------- ------------- ------ ------ ------ ------ ------ ------- -------
imx6qsabr Linux 4.4.12- 17.7 21.9 27.6 42.0 40.3 44.0 77.1
From the results, only the test case with 8p/64K on the isolated CPU
has lower latency.
To investigate the cause, I use the kernel event tracer to find out
the events, user_enter and user_exit, of context_tracking would happen
on tickless isolated CPU. These two events means that this CPU enters
and exits the RCU extended quiescent state. Besides, the execution
time of these two events are 3us and 2us,
which are measured by ktime. Is this the reason why the context switch
has higher
latency on the tickless isolated CPU?
Thanks,
Regards,
- Kang
From: Chris Metcalf <hidden> Date: 2016-08-17 14:53:03
On 8/17/2016 2:26 AM, GeHao Kang wrote:
To investigate the cause, I use the kernel event tracer to find out
the events, user_enter and user_exit, of context_tracking would happen
on tickless isolated CPU. These two events means that this CPU enters
and exits the RCU extended quiescent state. Besides, the execution
time of these two events are 3us and 2us,
which are measured by ktime. Is this the reason why the context switch
has higher
latency on the tickless isolated CPU?
The increased context switch time is likely from the increased
time to return from the kernel to userspace, due to ensuring
that various things in the kernel are quiesced.
Of course I'm sure it goes without saying that context switch
time is probably near the absolute bottom of things that
we care about as a metric for task isolation, since when you
are using it as designed, you never actually context switch.
But that said, it's always good to quantify what the overheads
are, so thanks.
--
Chris Metcalf, Mellanox Technologies
http://www.mellanox.com
From: GeHao Kang <hidden> Date: 2016-08-18 03:25:32
Hi Chris,
Thanks for your reply.
Is the increased time fixed in each context switch? Because this increased time
will be the latency of the real time application, we hope to confirm it.
Thanks.
Regards,
- Kang
On Wed, Aug 17, 2016 at 8:18 PM, Chris Metcalf [off-list ref] wrote:
On 8/17/2016 2:26 AM, GeHao Kang wrote:
quoted
To investigate the cause, I use the kernel event tracer to find out
the events, user_enter and user_exit, of context_tracking would happen
on tickless isolated CPU. These two events means that this CPU enters
and exits the RCU extended quiescent state. Besides, the execution
time of these two events are 3us and 2us,
which are measured by ktime. Is this the reason why the context switch
has higher
latency on the tickless isolated CPU?
The increased context switch time is likely from the increased
time to return from the kernel to userspace, due to ensuring
that various things in the kernel are quiesced.
Of course I'm sure it goes without saying that context switch
time is probably near the absolute bottom of things that
we care about as a metric for task isolation, since when you
are using it as designed, you never actually context switch.
But that said, it's always good to quantify what the overheads
are, so thanks.
--
Chris Metcalf, Mellanox Technologies
http://www.mellanox.com
From: Peter Zijlstra <peterz@infradead.org> Date: 2016-08-19 12:34:47
On Thu, Aug 18, 2016 at 11:25:00AM +0800, GeHao Kang wrote:
Is the increased time fixed in each context switch? Because this increased time
will be the latency of the real time application, we hope to confirm it.
Thanks.
Why are you wanting to use nohz_full if you do syscalls?
From: GeHao Kang <hidden> Date: 2016-08-21 11:26:07
On Fri, Aug 19, 2016 at 8:34 PM, Peter Zijlstra [off-list ref] wrote:
Why are you wanting to use nohz_full if you do syscalls?
We hope to reduce the overhead of the tick while the real time
applications run,
and these applications might do some syscalls to operate the I/O devices like
EtherCAT.
From: Paul E. McKenney <hidden> Date: 2016-08-21 14:53:00
On Sun, Aug 21, 2016 at 07:26:04PM +0800, GeHao Kang wrote:
On Fri, Aug 19, 2016 at 8:34 PM, Peter Zijlstra [off-list ref] wrote:
quoted
Why are you wanting to use nohz_full if you do syscalls?
We hope to reduce the overhead of the tick while the real time
applications run,
and these applications might do some syscalls to operate the I/O devices like
EtherCAT.
If latency is all you care about, one approach is to map the device
registers into userspace and do the I/O without assistance from the
kernel.
Alternatively, use in-memory mailbox/queuing techniques to hand the
I/O off to some other thread.
Thanx, Paul
From: GeHao Kang <hidden> Date: 2016-08-22 09:40:07
On Sun, Aug 21, 2016 at 10:53 PM, Paul E. McKenney
[off-list ref] wrote:
If latency is all you care about, one approach is to map the device
registers into userspace and do the I/O without assistance from the
kernel.
In addition to the context switch latency, local interrupts are also
closed during
user_enter and user_exit of the context tracking. Therefore, the interrupt
latency might be also increased on the isolated tickless CPU. That
will degrade the
real time performance. Are these two events determined?
Thanks,
Kang
From: Paul E. McKenney <hidden> Date: 2016-08-22 14:47:58
On Mon, Aug 22, 2016 at 05:40:03PM +0800, GeHao Kang wrote:
On Sun, Aug 21, 2016 at 10:53 PM, Paul E. McKenney
[off-list ref] wrote:
quoted
If latency is all you care about, one approach is to map the device
registers into userspace and do the I/O without assistance from the
kernel.
In addition to the context switch latency, local interrupts are also
closed during
user_enter and user_exit of the context tracking. Therefore, the interrupt
latency might be also increased on the isolated tickless CPU. That
will degrade the
real time performance. Are these two events determined?
Hmmm... Why would you be taking interrupts on your isolated tickless
CPUs? Doesn't that defeat the purpose of designating them as isolated
and tickless?
The key point being that effective use of NO_HZ_FULL requires
careful configuration and complete understanding of your workload.
And it is quite possible that you instead need to use something
other than NO_HZ_FULL.
If your question is instead "why must interrupts be disabled during
context tracking", I must defer to people who understand the x86
entry/exit code paths better than I do.
Thanx, Paul
From: Mark Hounschell <hidden> Date: 2016-08-22 15:12:56
On 08/22/2016 10:48 AM, Paul E. McKenney wrote:
On Mon, Aug 22, 2016 at 05:40:03PM +0800, GeHao Kang wrote:
quoted
On Sun, Aug 21, 2016 at 10:53 PM, Paul E. McKenney
[off-list ref] wrote:
quoted
If latency is all you care about, one approach is to map the device
registers into userspace and do the I/O without assistance from the
kernel.
In addition to the context switch latency, local interrupts are also
closed during
user_enter and user_exit of the context tracking. Therefore, the interrupt
latency might be also increased on the isolated tickless CPU. That
will degrade the
real time performance. Are these two events determined?
Hmmm... Why would you be taking interrupts on your isolated tickless
CPUs? Doesn't that defeat the purpose of designating them as isolated
and tickless?
Don't mean to butt in here but think about a "special" PCI card that
does nothing but take an external interrupt or external interrupts from
an outside source where the latency between the time it occurs on the
outside and the time an isolated processor can act on that event. The
IRQ of that card also being pinned/isolated to that processor. This is a
very common thing in the RT world.
Mark
The key point being that effective use of NO_HZ_FULL requires
careful configuration and complete understanding of your workload.
And it is quite possible that you instead need to use something
other than NO_HZ_FULL.
If your question is instead "why must interrupts be disabled during
context tracking", I must defer to people who understand the x86
entry/exit code paths better than I do.
Thanx, Paul
From: Paul E. McKenney <hidden> Date: 2016-08-22 15:37:27
On Mon, Aug 22, 2016 at 11:12:45AM -0400, Mark Hounschell wrote:
On 08/22/2016 10:48 AM, Paul E. McKenney wrote:
quoted
On Mon, Aug 22, 2016 at 05:40:03PM +0800, GeHao Kang wrote:
quoted
On Sun, Aug 21, 2016 at 10:53 PM, Paul E. McKenney
[off-list ref] wrote:
quoted
If latency is all you care about, one approach is to map the device
registers into userspace and do the I/O without assistance from the
kernel.
In addition to the context switch latency, local interrupts are also
closed during
user_enter and user_exit of the context tracking. Therefore, the interrupt
latency might be also increased on the isolated tickless CPU. That
will degrade the
real time performance. Are these two events determined?
Hmmm... Why would you be taking interrupts on your isolated tickless
CPUs? Doesn't that defeat the purpose of designating them as isolated
and tickless?
Don't mean to butt in here but think about a "special" PCI card that
does nothing but take an external interrupt or external interrupts
from an outside source where the latency between the time it occurs
on the outside and the time an isolated processor can act on that
event. The IRQ of that card also being pinned/isolated to that
processor. This is a very common thing in the RT world.
In this case, the host OS would see an event-driven real-time workload
from the PCI card, which would lead me to suggest -not- using NO_HZ_FULL
on the host OS.
Of course, if you are instead building an OS to run on the PCI card
itself, then the choice of configuration would depend on how the PCI
card was set up. If it polled hardware, then NO_HZ_FULL on the PCI card
might work quite well. But then you wouldn't have interrupts (on the
PCI card), so I am guessing that you mean the scenario covered in the
first paragraph.
Or am I missing your point?
Thanx, Paul
Mark
quoted
The key point being that effective use of NO_HZ_FULL requires
careful configuration and complete understanding of your workload.
And it is quite possible that you instead need to use something
other than NO_HZ_FULL.
If your question is instead "why must interrupts be disabled during
context tracking", I must defer to people who understand the x86
entry/exit code paths better than I do.
Thanx, Paul
From: Mark Hounschell <hidden> Date: 2016-08-22 16:41:40
On 08/22/2016 11:37 AM, Paul E. McKenney wrote:
On Mon, Aug 22, 2016 at 11:12:45AM -0400, Mark Hounschell wrote:
quoted
On 08/22/2016 10:48 AM, Paul E. McKenney wrote:
quoted
On Mon, Aug 22, 2016 at 05:40:03PM +0800, GeHao Kang wrote:
quoted
On Sun, Aug 21, 2016 at 10:53 PM, Paul E. McKenney
[off-list ref] wrote:
quoted
If latency is all you care about, one approach is to map the device
registers into userspace and do the I/O without assistance from the
kernel.
In addition to the context switch latency, local interrupts are also
closed during
user_enter and user_exit of the context tracking. Therefore, the interrupt
latency might be also increased on the isolated tickless CPU. That
will degrade the
real time performance. Are these two events determined?
Hmmm... Why would you be taking interrupts on your isolated tickless
CPUs? Doesn't that defeat the purpose of designating them as isolated
and tickless?
Don't mean to butt in here but think about a "special" PCI card that
does nothing but take an external interrupt or external interrupts
from an outside source where the latency between the time it occurs
on the outside and the time an isolated processor can act on that
event. The IRQ of that card also being pinned/isolated to that
processor. This is a very common thing in the RT world.
In this case, the host OS would see an event-driven real-time workload
from the PCI card, which would lead me to suggest -not- using NO_HZ_FULL
on the host OS.
Of course, if you are instead building an OS to run on the PCI card
itself, then the choice of configuration would depend on how the PCI
card was set up. If it polled hardware, then NO_HZ_FULL on the PCI card
might work quite well. But then you wouldn't have interrupts (on the
PCI card), so I am guessing that you mean the scenario covered in the
first paragraph.
Or am I missing your point?
Thanx, Paul
First paragraph scenario is the one I was referring.
Thanks
Mark
quoted
Mark
quoted
The key point being that effective use of NO_HZ_FULL requires
careful configuration and complete understanding of your workload.
And it is quite possible that you instead need to use something
other than NO_HZ_FULL.
If your question is instead "why must interrupts be disabled during
context tracking", I must defer to people who understand the x86
entry/exit code paths better than I do.
Thanx, Paul