From: Steven Rostedt <rostedt@goodmis.org> Date: 2016-08-11 16:37:00
On Thu, 11 Aug 2016 19:15:40 +0300
Grygorii Strashko [off-list ref] wrote:
Mark CPSW Rx/Tx IRQs as IRQF_NO_THREAD and avoid double scheduling on -RT
where this IRQs are forced threaded:
rx-irq
|- schedule threaded rx-irq handler
...
|- threaded rx-irq handler -> cpsw_rx_interrupt()
|- napi_schedule()
|- __raise_softirq_irqoff()
|- wakeup_proper_softirq()
...
napi
after:
rx-irq
|- cpsw_rx_interrupt()
|- napi_schedule()
|- irq_exit()
|- invoke_softirq()
|- wakeup_softirqd()
...
napi
And, as result, get benefits from the following improvements (tested
on am57xx-evm):
1) "[ 78.348599] NOHZ: local_softirq_pending 80" message will not be
seen any more. Now these warnings can be seen once iperf is started.
# iperf -c $IPERFHOST -w 128K -d -t 60
2) latency reduction when cyclictest is run in parallel with network load
where net_perf.sh is:
iperf -c $IPERFHOST -w 8K -d -t 60
iperf -c $IPERFHOST -w 16K -d -t 60
iperf -c $IPERFHOST -w 32K -d -t 60
iperf -c $IPERFHOST -w 64K -d -t 60
iperf -c $IPERFHOST -w 128K -d -t 60
before:
T: 0 ( 1326) P:98 I:1000 C: 240000 Min: 8 Act: 13 Avg: 18 Max: 70
T: 1 ( 1327) P:98 I:1500 C: 159981 Min: 9 Act: 15 Avg: 16 Max: 43
after:
T: 0 ( 1331) P:98 I:1000 C: 240000 Min: 8 Act: 15 Avg: 14 Max: 51
T: 1 ( 1332) P:98 I:1500 C: 159953 Min: 8 Act: 16 Avg: 15 Max: 33
3) network performance increase
win, K Mbits/s
before after %
8K 354 350.3 0.0
16K 412 551 33.7
32K 423 659.5 55.9
64K 436 728.3 67.0
128K 537 845 57.4
This change does not affect on non-RT.
This looks fine to me, but it should go into the development branch,
which is currently 4.6-rt. And I can then pull it from there.
-- Steve
quoted hunk
Signed-off-by: Grygorii Strashko <grygorii.strashko@ti.com>
---
Hi All,
I'll be appreciated on any feedback or tested-by.
In case of positive feedback I'll resend it for upstream.
drivers/net/ethernet/ti/cpsw.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
From: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Date: 2016-08-12 08:32:06
On 2016-08-11 19:15:40 [+0300], Grygorii Strashko wrote:
Mark CPSW Rx/Tx IRQs as IRQF_NO_THREAD and avoid double scheduling on -RT
where this IRQs are forced threaded:
rx-irq
|- schedule threaded rx-irq handler
...
|- threaded rx-irq handler -> cpsw_rx_interrupt()
|- napi_schedule()
|- __raise_softirq_irqoff()
|- wakeup_proper_softirq()
...
napi
This should not be the default path. The default should be napi running
in the context of the threaded rx-irq handler once the handler is done.
The wakeup_proper_softirq() part is only done if napi thinks that the
callback functions runs for too long. So in *that* case you continue
NAPI in the softirq-thread which runs at SCHED_OTHER.
Since you schedule the softirq from an IRQ-off region / without a
process context you force the softirq to run in the thread at
SCHED_OTHER priority.
And, as result, get benefits from the following improvements (tested
on am57xx-evm):
1) "[ 78.348599] NOHZ: local_softirq_pending 80" message will not be
seen any more. Now these warnings can be seen once iperf is started.
# iperf -c $IPERFHOST -w 128K -d -t 60
Do you also see "sched: RT throttling activated"? Because I don't see
otherwise why this should pop up.
-d 0 to have I: set to the same value.
What does -i 250 say?
And without network load we are where we were at "after" values?
What happens if s/__raise_softirq_irqoff_ksoft/__raise_softirq_irqoff/
in net/core/dev.c and chrt the priority of you network interrupt
handlers to SCHED_OTHER priority?
3) network performance increase
win, K Mbits/s
before after %
8K 354 350.3 0.0
16K 412 551 33.7
32K 423 659.5 55.9
64K 436 728.3 67.0
128K 537 845 57.4
Hi Sebastian,
On 08/12/2016 11:32 AM, Sebastian Andrzej Siewior wrote:
On 2016-08-11 19:15:40 [+0300], Grygorii Strashko wrote:
quoted
Mark CPSW Rx/Tx IRQs as IRQF_NO_THREAD and avoid double scheduling on -RT
where this IRQs are forced threaded:
rx-irq
|- schedule threaded rx-irq handler
...
|- threaded rx-irq handler -> cpsw_rx_interrupt()
|- napi_schedule()
|- __raise_softirq_irqoff()
|- wakeup_proper_softirq()
...
napi
This should not be the default path. The default should be napi running
in the context of the threaded rx-irq handler once the handler is done.
The wakeup_proper_softirq() part is only done if napi thinks that the
callback functions runs for too long. So in *that* case you continue
NAPI in the softirq-thread which runs at SCHED_OTHER.
Thankds for comment. You're right:
irq_thread()->irq_forced_thread_fn()->local_bh_enable()
but wouldn't here two wake_up_process() calls any way,
plus preempt_check_resched_rt() in napi_schedule().
Since you schedule the softirq from an IRQ-off region / without a
process context you force the softirq to run in the thread at
SCHED_OTHER priority.
Thanks a lot for comments.
quoted
And, as result, get benefits from the following improvements (tested
on am57xx-evm):
1) "[ 78.348599] NOHZ: local_softirq_pending 80" message will not be
seen any more. Now these warnings can be seen once iperf is started.
# iperf -c $IPERFHOST -w 128K -d -t 60
Do you also see "sched: RT throttling activated"? Because I don't see
otherwise why this should pop up.
I've reverted my patch an did requested experiments (some additional info below).
I do not see "sched: RT throttling activated" :(
root@am57xx-evm:~# ./net_perf.sh & cyclictest -m -Sp98 -q -D4m
[1] 1301
# /dev/cpu_dma_latency set to 0us
Linux am57xx-evm 4.4.16-rt23-00321-ga195e6a-dirty #92 SMP PREEMPT RT Fri Aug 12 14:03:59 EEST 2016 armv7l GNU/Linux
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 16.0 KByte (WARNING: requested 8.00 KByte)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 172.22.39.17, TCP port 5001
TCP window size: 16.0 KByte (WARNING: requested 8.00 KByte)
------------------------------------------------------------
[ 3] local 172.22.39.21 port 47500 connected with 172.22.39.17 port 5001
[ 5] local 172.22.39.21 port 5001 connected with 172.22.39.17 port 59955
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-60.0 sec 1.31 GBytes 188 Mbits/sec
[ 5] 0.0-60.0 sec 868 MBytes 121 Mbits/sec
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 32.0 KByte (WARNING: requested 16.0 KByte)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 172.22.39.17, TCP port 5001
TCP window size: 32.0 KByte (WARNING: requested 16.0 KByte)
------------------------------------------------------------
[ 5] local 172.22.39.21 port 47502 connected with 172.22.39.17 port 5001
[ 4] local 172.22.39.21 port 5001 connected with 172.22.39.17 port 59957
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-60.0 sec 1.92 GBytes 275 Mbits/sec
[ 4] 0.0-60.0 sec 1.14 GBytes 163 Mbits/sec
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (WARNING: requested 32.0 KByte)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 172.22.39.17, TCP port 5001
TCP window size: 64.0 KByte (WARNING: requested 32.0 KByte)
------------------------------------------------------------
[ 5] local 172.22.39.21 port 47504 connected with 172.22.39.17 port 5001
[ 4] local 172.22.39.21 port 5001 connected with 172.22.39.17 port 59965
[ 2285.928516] NOHZ: local_softirq_pending 80
[ 2285.928549] NOHZ: local_softirq_pending 80
[ 2285.928594] NOHZ: local_softirq_pending 80
[ 2285.928689] NOHZ: local_softirq_pending 80
[ 2286.348572] NOHZ: local_softirq_pending 80
[ 2290.358623] NOHZ: local_softirq_pending 80
[ 2291.588624] NOHZ: local_softirq_pending 80
[ 2291.698579] NOHZ: local_softirq_pending 80
[ 2292.608594] NOHZ: local_softirq_pending 80
[ 2293.178627] NOHZ: local_softirq_pending 80
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-60.0 sec 1.43 GBytes 205 Mbits/sec
[ 4] 0.0-60.0 sec 1.58 GBytes 226 Mbits/sec
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 128 KByte (WARNING: requested 64.0 KByte)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 172.22.39.17, TCP port 5001
TCP window size: 128 KByte (WARNING: requested 64.0 KByte)
------------------------------------------------------------
[ 5] local 172.22.39.21 port 47506 connected with 172.22.39.17 port 5001
[ 4] local 172.22.39.21 port 5001 connected with 172.22.39.17 port 59980
T: 0 ( 1309) P:98 I:1000 C: 240000 Min: 8 Act: 13 Avg: 18 Max: 60
T: 1 ( 1310) P:98 I:1500 C: 159968 Min: 8 Act: 14 Avg: 16 Max: 58
root@am57xx-evm:~# [ ID] Interval Transfer Bandwidth
[ 5] 0.0-60.0 sec 1.84 GBytes 264 Mbits/sec
[ 4] 0.0-60.0 sec 1.47 GBytes 210 Mbits/sec
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 256 KByte (WARNING: requested 128 KByte)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 172.22.39.17, TCP port 5001
TCP window size: 256 KByte (WARNING: requested 128 KByte)
------------------------------------------------------------
[ 5] local 172.22.39.21 port 47508 connected with 172.22.39.17 port 5001
[ 4] local 172.22.39.21 port 5001 connected with 172.22.39.17 port 59982
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-60.0 sec 974 MBytes 136 Mbits/sec
[ 4] 0.0-60.0 sec 2.84 GBytes 406 Mbits/sec
[1]+ Done ./net_perf.sh
What happens if s/__raise_softirq_irqoff_ksoft/__raise_softirq_irqoff/
in net/core/dev.c and chrt the priority of you network interrupt
handlers to SCHED_OTHER priority?
===== without this patch + __raise_softirq_irqoff + netIRQs->SCHED_OTHER
with net load:
cyclictest -m -Sp98 -q -D4m -i250 -d0
T: 0 ( 1325) P:98 I:1000 C: 240000 Min: 8 Act: 22 Avg: 17 Max: 51
T: 1 ( 1326) P:98 I:1500 C: 159981 Min: 8 Act: 15 Avg: 15 Max: 39
cyclictest -m -Sp98 -q -D4m -i250 -d0
T: 0 ( 1307) P:98 I:250 C: 960000 Min: 7 Act: 13 Avg: 16 Max: 50
T: 1 ( 1308) P:98 I:250 C: 959819 Min: 8 Act: 12 Avg: 14 Max: 37
and net parformance is better:
root@am57xx-evm:~# ps -A | grep 4848
82 ? 00:00:00 irq/354-4848400
83 ? 00:00:00 irq/355-4848400
root@am57xx-evm:~# chrt -o -p 0 82
root@am57xx-evm:~# chrt -o -p 0 83
./net_perf.sh & cyclictest -m -Sp98 -q -D4m -i250 -d0
[1] 1298
# /dev/cpu_dma_latency set to 0us
Linux am57xx-evm 4.4.16-rt23-00321-ga195e6a-dirty #95 SMP PREEMPT RT Fri Aug 12 16:20:42 EEST 2016 armv7l GNU/Linux
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 16.0 KByte (WARNING: requested 8.00 KByte)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 172.22.39.17, TCP port 5001
TCP window size: 16.0 KByte (WARNING: requested 8.00 KByte)
------------------------------------------------------------
[ 3] local 172.22.39.21 port 58672 connected with 172.22.39.17 port 5001
[ 5] local 172.22.39.21 port 5001 connected with 172.22.39.17 port 60672
[ 78.718828] NOHZ: local_softirq_pending 80
[ 78.908849] NOHZ: local_softirq_pending 80
[ 78.908927] NOHZ: local_softirq_pending 80
[ 78.918827] NOHZ: local_softirq_pending 80
[ 79.058826] NOHZ: local_softirq_pending 80
[ 81.488918] NOHZ: local_softirq_pending 80
[ 81.489162] NOHZ: local_softirq_pending 80
[ 81.489398] NOHZ: local_softirq_pending 80
[ 81.489646] NOHZ: local_softirq_pending 80
[ 81.489895] NOHZ: local_softirq_pending 80
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-60.0 sec 1.30 GBytes 187 Mbits/sec
[ 5] 0.0-60.0 sec 637 MBytes 89.0 Mbits/sec
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 32.0 KByte (WARNING: requested 16.0 KByte)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 172.22.39.17, TCP port 5001
TCP window size: 32.0 KByte (WARNING: requested 16.0 KByte)
------------------------------------------------------------
[ 5] local 172.22.39.21 port 58674 connected with 172.22.39.17 port 5001
[ 4] local 172.22.39.21 port 5001 connected with 172.22.39.17 port 60680
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-60.0 sec 1.78 GBytes 255 Mbits/sec
[ 4] 0.0-60.0 sec 1.47 GBytes 210 Mbits/sec
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (WARNING: requested 32.0 KByte)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 172.22.39.17, TCP port 5001
TCP window size: 64.0 KByte (WARNING: requested 32.0 KByte)
------------------------------------------------------------
[ 5] local 172.22.39.21 port 58676 connected with 172.22.39.17 port 5001
[ 4] local 172.22.39.21 port 5001 connected with 172.22.39.17 port 60693
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-60.0 sec 1.62 GBytes 232 Mbits/sec
[ 4] 0.0-60.0 sec 2.07 GBytes 296 Mbits/sec
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 128 KByte (WARNING: requested 64.0 KByte)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 172.22.39.17, TCP port 5001
TCP window size: 128 KByte (WARNING: requested 64.0 KByte)
------------------------------------------------------------
[ 5] local 172.22.39.21 port 58678 connected with 172.22.39.17 port 5001
[ 4] local 172.22.39.21 port 5001 connected with 172.22.39.17 port 60702
T: 0 ( 1307) P:98 I:250 C: 960000 Min: 7 Act: 13 Avg: 16 Max: 50
T: 1 ( 1308) P:98 I:250 C: 959819 Min: 8 Act: 12 Avg: 14 Max: 37
root@am57xx-evm:~# [ ID] Interval Transfer Bandwidth
[ 5] 0.0-60.0 sec 1.77 GBytes 254 Mbits/sec
[ 4] 0.0-60.0 sec 1.77 GBytes 253 Mbits/sec
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 256 KByte (WARNING: requested 128 KByte)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 172.22.39.17, TCP port 5001
TCP window size: 256 KByte (WARNING: requested 128 KByte)
------------------------------------------------------------
[ 5] local 172.22.39.21 port 58680 connected with 172.22.39.17 port 5001
[ 4] local 172.22.39.21 port 5001 connected with 172.22.39.17 port 60711
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-60.0 sec 2.18 GBytes 312 Mbits/sec
[ 4] 0.0-60.0 sec 2.53 GBytes 362 Mbits/sec
quoted
3) network performance increase
win, K Mbits/s
before after %
8K 354 350.3 0.0
16K 412 551 33.7
32K 423 659.5 55.9
64K 436 728.3 67.0
128K 537 845 57.4
On 08/12/2016 11:32 AM, Sebastian Andrzej Siewior wrote:
On 2016-08-11 19:15:40 [+0300], Grygorii Strashko wrote:
quoted
Mark CPSW Rx/Tx IRQs as IRQF_NO_THREAD and avoid double scheduling on -RT
where this IRQs are forced threaded:
rx-irq
|- schedule threaded rx-irq handler
...
|- threaded rx-irq handler -> cpsw_rx_interrupt()
|- napi_schedule()
|- __raise_softirq_irqoff()
|- wakeup_proper_softirq()
...
napi
This should not be the default path. The default should be napi running
in the context of the threaded rx-irq handler once the handler is done.
The wakeup_proper_softirq() part is only done if napi thinks that the
callback functions runs for too long. So in *that* case you continue
NAPI in the softirq-thread which runs at SCHED_OTHER.
Since you schedule the softirq from an IRQ-off region / without a
process context you force the softirq to run in the thread at
SCHED_OTHER priority.
quoted
And, as result, get benefits from the following improvements (tested
on am57xx-evm):
1) "[ 78.348599] NOHZ: local_softirq_pending 80" message will not be
seen any more. Now these warnings can be seen once iperf is started.
# iperf -c $IPERFHOST -w 128K -d -t 60
Do you also see "sched: RT throttling activated"? Because I don't see
otherwise why this should pop up.
From: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Date: 2016-09-08 14:28:21
On 2016-08-12 18:58:21 [+0300], Grygorii Strashko wrote:
Hi Sebastian,
Hi Grygorii,
Thankds for comment. You're right:
irq_thread()->irq_forced_thread_fn()->local_bh_enable()
but wouldn't here two wake_up_process() calls any way,
plus preempt_check_resched_rt() in napi_schedule().
Usually you prefer BH handling in the IRQ-thread because it runs at
higher priority and is not interrupted by a SCHED_OTHER process. And you
can assign it a higher priority if it should be preferred over an other
interrupt. However, if the processing of the interrupt is taking too
much time (like that ping flood, a lot of network traffic) then we push
it to the softirq thread. If you do this now unconditionally in the
SCHED_OTHER softirq thread then you take away all the `good' things we
had (like processing important packets at higher priority as long as
nobody floods us). Plus you share this thread with everything else that
runs in there.
quoted
quoted
And, as result, get benefits from the following improvements (tested
on am57xx-evm):
1) "[ 78.348599] NOHZ: local_softirq_pending 80" message will not be
seen any more. Now these warnings can be seen once iperf is started.
# iperf -c $IPERFHOST -w 128K -d -t 60
Do you also see "sched: RT throttling activated"? Because I don't see
otherwise why this should pop up.
I've reverted my patch an did requested experiments (some additional info below).
I do not see "sched: RT throttling activated" :(
That is okay. However if aim for throughput you might want to switch
away from NO_HZ (and deactivate the software watchdog wich runs at
prio 99 if enabled).
root@am57xx-evm:~# ./net_perf.sh & cyclictest -m -Sp98 -q -D4m
[1] 1301
# /dev/cpu_dma_latency set to 0us
Linux am57xx-evm 4.4.16-rt23-00321-ga195e6a-dirty #92 SMP PREEMPT RT Fri Aug 12 14:03:59 EEST 2016 armv7l GNU/Linux
…
[1]+ Done ./net_perf.sh
I can't parse this. But that local_softirq_pending() warning might
contribute to lower numbers.
So the max value dropped by ~50% with your patch. Interesting. What I
remember from testing is that once you had, say, one hour of hackbench
running then after that, the extra network traffic didn't contribute
much (if at all) to the max value.
That said it is hard to believe that one extra context switch
contributes about 40us to the max value on CPU0.
quoted
What happens if s/__raise_softirq_irqoff_ksoft/__raise_softirq_irqoff/
in net/core/dev.c and chrt the priority of you network interrupt
handlers to SCHED_OTHER priority?
We don't serve TIMER & SCHED because those two are pushed to the
ksoftirq thread(s). So we keep mostly doing NET_RX and now we switch to
the next best thing which is RCU.
As you see ksoftirqd left the CPU with a D so I would assume it is
blocked on a lock and waits.
NET_RX is in progress but scheduled out due to RCUC which is also
scheduled out.
I assume we got to softirq because nothing else can run. It will see
that NET_RX is pending and tries it but blocks on the lock
(lock_softirq()). It schedules out. Nothing left -> idle.
The idle code checks to see if a softirq is pending and in fact there is
SCHED on the list and ksoftirq was about to handle it but due to
ordering complication (NET_RX before SCHED) it can't. And we have the
warning.
This
@@ -132,6 +133,15 @@ void softirq_check_pending_idle(void)}}+if(warnpending&&ksoft_tsk){+raw_spin_lock(&ksoft_tsk->pi_lock);+if(ksoft_tsk->pi_blocked_on||ksoft_tsk->state==TASK_RUNNING){+/* Clear all bits pending in that task */+warnpending&=~(ksoft_tsk->softirqs_raised);+}+raw_spin_unlock(&ksoft_tsk->pi_lock);+}+if(warnpending){printk(KERN_ERR"NOHZ: local_softirq_pending %02x\n",warnpending);
should avoid the warning if the softirq thread is blocked.
However:
- RCU boosting with RT prio might help here or there.
- having the hard-IRQ and IRQ-thread on the same CPU might help, too. It
is not strictly required but saves a few cycles if you don't have to
perform cross CPU wake ups and migrate task forth and back. The latter
happens at prio 99.
- I am not sure NAPI works as expected. I would assume so. There is IRQ
354 and 355 which fire after each other. One would be enough I guess.
And they seem to be short living / fire often. If NAPI works then it
should put an end to it and push it to the softirq thread.
If you have IRQ-pacing support I suggest to use something like 10ms or
so. That means your ping response will go from <= 1ms to 10ms in the
worst case but since you process more packets at a time your
throughput should increase.
If I count this correct, it too you alsmost 4ms from "raise SCHED" to
"try process SCHED" and most of the time was spent in 35[45] hard irq,
raise NET_RX or cross wakeup the IRQ thread.
Sebastian
On 09/08/2016 05:28 PM, Sebastian Andrzej Siewior wrote:
On 2016-08-12 18:58:21 [+0300], Grygorii Strashko wrote:
quoted
Hi Sebastian,
Hi Grygorii,
quoted
Thankds for comment. You're right:
irq_thread()->irq_forced_thread_fn()->local_bh_enable()
but wouldn't here two wake_up_process() calls any way,
plus preempt_check_resched_rt() in napi_schedule().
Usually you prefer BH handling in the IRQ-thread because it runs at
higher priority and is not interrupted by a SCHED_OTHER process. And you
can assign it a higher priority if it should be preferred over an other
interrupt. However, if the processing of the interrupt is taking too
much time (like that ping flood, a lot of network traffic) then we push
it to the softirq thread. If you do this now unconditionally in the
SCHED_OTHER softirq thread then you take away all the `good' things we
had (like processing important packets at higher priority as long as
nobody floods us). Plus you share this thread with everything else that
runs in there.
That's i understand, but effect from this patch on network throughput
is pretty amazing :)
quoted
quoted
quoted
And, as result, get benefits from the following improvements (tested
on am57xx-evm):
1) "[ 78.348599] NOHZ: local_softirq_pending 80" message will not be
seen any more. Now these warnings can be seen once iperf is started.
# iperf -c $IPERFHOST -w 128K -d -t 60
Do you also see "sched: RT throttling activated"? Because I don't see
otherwise why this should pop up.
I've reverted my patch an did requested experiments (some additional info below).
I do not see "sched: RT throttling activated" :(
That is okay. However if aim for throughput you might want to switch
away from NO_HZ (and deactivate the software watchdog wich runs at
prio 99 if enabled).
quoted
root@am57xx-evm:~# ./net_perf.sh & cyclictest -m -Sp98 -q -D4m
[1] 1301
# /dev/cpu_dma_latency set to 0us
Linux am57xx-evm 4.4.16-rt23-00321-ga195e6a-dirty #92 SMP PREEMPT RT Fri Aug 12 14:03:59 EEST 2016 armv7l GNU/Linux
…
quoted
[1]+ Done ./net_perf.sh
I can't parse this. But that local_softirq_pending() warning might
contribute to lower numbers.
So the max value dropped by ~50% with your patch. Interesting. What I
remember from testing is that once you had, say, one hour of hackbench
running then after that, the extra network traffic didn't contribute
much (if at all) to the max value.
That said it is hard to believe that one extra context switch
contributes about 40us to the max value on CPU0.
Yup. but short time testing provides very stable results.
This patch is going to be tested more intensively shortly.
quoted
quoted
What happens if s/__raise_softirq_irqoff_ksoft/__raise_softirq_irqoff/
in net/core/dev.c and chrt the priority of you network interrupt
handlers to SCHED_OTHER priority?
Yah, This improvement, in general.
But the fact that so significant net performance drop observed out of the box
(without any tunning) and on idle system - triggers a lot of questions ;(
I'm worry if observed original behavior can depend on usage NAPI polling for both RX/TX
in CPSW driver. CPSW request two IRQs RX and TX and both handler just do napi_schedule()[NET_RX].
--
regards,
-grygorii
We don't serve TIMER & SCHED because those two are pushed to the
ksoftirq thread(s). So we keep mostly doing NET_RX and now we switch to
the next best thing which is RCU.
It looks like scheduler playing ping-pong between CPUs with threaded irqs irq/354-355.
And seems this might be the case - if I pin both threaded IRQ handlers to CPU0
I can see better latency and netperf improvement
cyclictest -m -Sp98 -q -D4m
T: 0 ( 1318) P:98 I:1000 C: 240000 Min: 9 Act: 14 Avg: 15 Max: 42
T: 1 ( 1319) P:98 I:1500 C: 159909 Min: 9 Act: 14 Avg: 16 Max: 39
if I arrange hwirqs and pin pin both threaded IRQ handlers on CPU1
I can observe more less similar results as with this patch.
As you see ksoftirqd left the CPU with a D so I would assume it is
blocked on a lock and waits.
NET_RX is in progress but scheduled out due to RCUC which is also
scheduled out.
I assume we got to softirq because nothing else can run. It will see
that NET_RX is pending and tries it but blocks on the lock
(lock_softirq()). It schedules out. Nothing left -> idle.
The idle code checks to see if a softirq is pending and in fact there is
SCHED on the list and ksoftirq was about to handle it but due to
ordering complication (NET_RX before SCHED) it can't. And we have the
warning.
This
@@ -132,6 +133,15 @@ void softirq_check_pending_idle(void)}}+if(warnpending&&ksoft_tsk){+raw_spin_lock(&ksoft_tsk->pi_lock);+if(ksoft_tsk->pi_blocked_on||ksoft_tsk->state==TASK_RUNNING){+/* Clear all bits pending in that task */+warnpending&=~(ksoft_tsk->softirqs_raised);+}+raw_spin_unlock(&ksoft_tsk->pi_lock);+}+if(warnpending){printk(KERN_ERR"NOHZ: local_softirq_pending %02x\n",warnpending);
should avoid the warning if the softirq thread is blocked.
with this change i do not see "NOHZ: local_softirq_pending 80" any more
Tested-by: Grygorii Strashko <grygorii.strashko@ti.com>
However:
- RCU boosting with RT prio might help here or there.
That I'll try. current settings
CONFIG_RCU_BOOST=y
CONFIG_RCU_KTHREAD_PRIO=1
CONFIG_RCU_BOOST_DELAY=500
- having the hard-IRQ and IRQ-thread on the same CPU might help, too. It
is not strictly required but saves a few cycles if you don't have to
perform cross CPU wake ups and migrate task forth and back. The latter
happens at prio 99.
- I am not sure NAPI works as expected. I would assume so. There is IRQ
354 and 355 which fire after each other. One would be enough I guess.
And they seem to be short living / fire often. If NAPI works then it
should put an end to it and push it to the softirq thread.
If you have IRQ-pacing support I suggest to use something like 10ms or
so. That means your ping response will go from <= 1ms to 10ms in the
worst case but since you process more packets at a time your
throughput should increase.
If I count this correct, it too you alsmost 4ms from "raise SCHED" to
"try process SCHED" and most of the time was spent in 35[45] hard irq,
raise NET_RX or cross wakeup the IRQ thread.
The question I have to dial with is why switching to RT cause so significant
netperf drop (without additional tunning) comparing to vanilla - ~120% for 256K and ~200% for 128K windows?
It's of course expected to see netperf drop, but I assume not so significant :(
And I can't find any reports or statistic related to this. Does the same happen on x86?
Thanks a lot for your comments.
--
regards,
-grygorii
From: Sebastian Andrzej Siewior <bigeasy@linutronix.de> Date: 2016-09-15 14:39:33
On 2016-09-09 15:46:44 [+0300], Grygorii Strashko wrote:
It looks like scheduler playing ping-pong between CPUs with threaded irqs irq/354-355.
And seems this might be the case - if I pin both threaded IRQ handlers to CPU0
I can see better latency and netperf improvement
cyclictest -m -Sp98 -q -D4m
T: 0 ( 1318) P:98 I:1000 C: 240000 Min: 9 Act: 14 Avg: 15 Max: 42
T: 1 ( 1319) P:98 I:1500 C: 159909 Min: 9 Act: 14 Avg: 16 Max: 39
if I arrange hwirqs and pin pin both threaded IRQ handlers on CPU1
I can observe more less similar results as with this patch.
so no patch then.
with this change i do not see "NOHZ: local_softirq_pending 80" any more
Tested-by: Grygorii Strashko <grygorii.strashko@ti.com>
okay. So I need to think what I do about this. Either this or trying to
run the "higher" softirq first but this could break things.
Thanks for the confirmation.
quoted
- having the hard-IRQ and IRQ-thread on the same CPU might help, too. It
is not strictly required but saves a few cycles if you don't have to
perform cross CPU wake ups and migrate task forth and back. The latter
happens at prio 99.
There is irq_thread() => irq_thread_check_affinity(). It might not work
as expected on ARM but it makes sense to follow the affinity mask HW irq
for the thread.
quoted
- I am not sure NAPI works as expected. I would assume so. There is IRQ
354 and 355 which fire after each other. One would be enough I guess.
And they seem to be short living / fire often. If NAPI works then it
should put an end to it and push it to the softirq thread.
If you have IRQ-pacing support I suggest to use something like 10ms or
so. That means your ping response will go from <= 1ms to 10ms in the
worst case but since you process more packets at a time your
throughput should increase.
If I count this correct, it too you alsmost 4ms from "raise SCHED" to
"try process SCHED" and most of the time was spent in 35[45] hard irq,
raise NET_RX or cross wakeup the IRQ thread.
The question I have to dial with is why switching to RT cause so significant
netperf drop (without additional tunning) comparing to vanilla - ~120% for 256K and ~200% for 128K windows?
You have a sched / thread ping/pong. That is one thing. !RT with
threaded irqs should show similar problems. The higher latency is caused
by the migration thread.
It's of course expected to see netperf drop, but I assume not so significant :(
And I can't find any reports or statistic related to this. Does the same happen on x86?
It should. Maybe at a lower level if it handles migration more
effective. There is this watchdog thread (for instance) which tries to
detect lockups and runs at P99. It causes "worse" cyclictest numbers on
x86 and on ARM but on ARM this is more visible than on x86.
Sebastian
We don't serve TIMER & SCHED because those two are pushed to the
ksoftirq thread(s). So we keep mostly doing NET_RX and now we switch to
the next best thing which is RCU.
It looks like scheduler playing ping-pong between CPUs with threaded irqs irq/354-355.
And seems this might be the case - if I pin both threaded IRQ handlers to CPU0
I can see better latency and netperf improvement
cyclictest -m -Sp98 -q -D4m
T: 0 ( 1318) P:98 I:1000 C: 240000 Min: 9 Act: 14 Avg: 15 Max: 42
T: 1 ( 1319) P:98 I:1500 C: 159909 Min: 9 Act: 14 Avg: 16 Max: 39
if I arrange hwirqs and pin pin both threaded IRQ handlers on CPU1
I can observe more less similar results as with this patch.
As you see ksoftirqd left the CPU with a D so I would assume it is
blocked on a lock and waits.
NET_RX is in progress but scheduled out due to RCUC which is also
scheduled out.
I assume we got to softirq because nothing else can run. It will see
that NET_RX is pending and tries it but blocks on the lock
(lock_softirq()). It schedules out. Nothing left -> idle.
The idle code checks to see if a softirq is pending and in fact there is
SCHED on the list and ksoftirq was about to handle it but due to
ordering complication (NET_RX before SCHED) it can't. And we have the
warning.
This
@@ -132,6 +133,15 @@ void softirq_check_pending_idle(void)}}+if(warnpending&&ksoft_tsk){+raw_spin_lock(&ksoft_tsk->pi_lock);+if(ksoft_tsk->pi_blocked_on||ksoft_tsk->state==TASK_RUNNING){+/* Clear all bits pending in that task */+warnpending&=~(ksoft_tsk->softirqs_raised);+}+raw_spin_unlock(&ksoft_tsk->pi_lock);+}+if(warnpending){printk(KERN_ERR"NOHZ: local_softirq_pending %02x\n",warnpending);
should avoid the warning if the softirq thread is blocked.
with this change i do not see "NOHZ: local_softirq_pending 80" any more
Tested-by: Grygorii Strashko <grygorii.strashko@ti.com>
Above change is not present in K4.9 and I can still see NOHZ messages if
i run iperf on am57xx-evm, I'd like to know if you have any plans regarding this?
Copy pasting here your comments from another e-mail
--
okay. So I need to think what I do about this. Either this or trying to
run the "higher" softirq first but this could break things.
Thanks for the confirmation.
((c) Sebastian Andrzej Siewior)
--
root@am57xx-evm:~# iperf -c 192.168.1.1 -w128K -d -i5 -t120 & cyclictest -n -m -Sp97 -q -D2m
[1] 1078
# /dev/cpu_dma_latency set to 0us
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 256 KByte (WARNING: requested 128 KByte)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.1.1, TCP port 5001
TCP window size: 256 KByte (WARNING: requested 128 KByte)
------------------------------------------------------------
[ 5] local 192.168.1.2 port 39346 connected with 192.168.1.1 port 5001
[ 4] local 192.168.1.2 port 5001 connected with 192.168.1.1 port 50594
[ 90.829952] NOHZ: local_softirq_pending 80
[ 90.830001] NOHZ: local_softirq_pending 80
[ 91.529926] NOHZ: local_softirq_pending 80
[ 93.299956] NOHZ: local_softirq_pending 80
[ 93.680078] NOHZ: local_softirq_pending 80
[ 93.680128] NOHZ: local_softirq_pending 80
[ ID] Interval Transfer Bandwidth
[ 5] 0.0- 5.0 sec 73.2 MBytes 123 Mbits/sec
[ 4] 0.0- 5.0 sec 47.5 MBytes 79.7 Mbits/sec
[ 94.719951] NOHZ: local_softirq_pending 80
[ 96.439923] NOHZ: local_softirq_pending 80
[ 96.569922] NOHZ: local_softirq_pending 80
[ 96.569951] NOHZ: local_softirq_pending 80
[ 5] 5.0-10.0 sec 73.5 MBytes 123 Mbits/sec
PS: if i set net irqs to run on cpu0 (or CPU1) only it improve net throughput and
eliminates this NOHZ messages.
--
regards,
-grygorii