From: Eric Dumazet <hidden> Date: 2013-01-03 12:28:59
From: Eric Dumazet <edumazet@google.com>
In various network workloads, __do_softirq() latencies can be up
to 20 ms if HZ=1000, and 200 ms if HZ=100.
This is because we iterate 10 times in the softirq dispatcher,
and some actions can consume a lot of cycles.
This patch changes the fallback to ksoftirqd condition to :
- A time limit of 2 ms.
- need_resched() being set on current task
When one of this condition is met, we wakeup ksoftirqd for further
softirq processing if we still have pending softirqs.
I ran several benchmarks and got no significant difference in
throughput, but a very significant reduction of maximum latencies.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: David Miller <davem@davemloft.net>
Cc: Tom Herbert <redacted>
---
kernel/softirq.c | 17 +++++++++--------
1 file changed, 9 insertions(+), 8 deletions(-)
From: Andrew Morton <akpm@linux-foundation.org> Date: 2013-01-03 20:46:11
On Thu, 03 Jan 2013 04:28:52 -0800
Eric Dumazet [off-list ref] wrote:
From: Eric Dumazet <edumazet@google.com>
In various network workloads, __do_softirq() latencies can be up
to 20 ms if HZ=1000, and 200 ms if HZ=100.
This is because we iterate 10 times in the softirq dispatcher,
and some actions can consume a lot of cycles.
hm, where did that "20 ms" come from? What caused it? Is it simply
the case that you happened to have actions which consume 2ms if HZ=1000
and 20ms if HZ=100?
This patch changes the fallback to ksoftirqd condition to :
- A time limit of 2 ms.
- need_resched() being set on current task
When one of this condition is met, we wakeup ksoftirqd for further
softirq processing if we still have pending softirqs.
Do we need both tests? The need_resched() test alone might be
sufficient?
With this change, there is a possibility that a rapidly-rescheduling
task will cause softirq starvation?
Can this change cause worsened latencies in some situations? Say there
are a large number of short-running actions queued. Presently we'll
dispatch ten of them and return. With this change we'll dispatch many
more of them - however many consume 2ms. So worst-case latency
increases from "10 * not-much" to "2 ms".
From: Eric Dumazet <hidden> Date: 2013-01-03 22:41:21
On Thu, 2013-01-03 at 12:46 -0800, Andrew Morton wrote:
On Thu, 03 Jan 2013 04:28:52 -0800
Eric Dumazet [off-list ref] wrote:
quoted
From: Eric Dumazet <edumazet@google.com>
In various network workloads, __do_softirq() latencies can be up
to 20 ms if HZ=1000, and 200 ms if HZ=100.
This is because we iterate 10 times in the softirq dispatcher,
and some actions can consume a lot of cycles.
hm, where did that "20 ms" come from? What caused it? Is it simply
the case that you happened to have actions which consume 2ms if HZ=1000
and 20ms if HZ=100?
net_rx_action() has such behavior yes.
In the worst/busy case, we spend 2 ticks per call, and even more
in some cases you dont want to know about (like triggering a IPv6 route
garbage collect)
quoted
This patch changes the fallback to ksoftirqd condition to :
- A time limit of 2 ms.
- need_resched() being set on current task
When one of this condition is met, we wakeup ksoftirqd for further
softirq processing if we still have pending softirqs.
Do we need both tests? The need_resched() test alone might be
sufficient?
I tried a need_resched() only, but could trigger watchdog faults and
reboots, in case a cpu was dedicated to softirq and all other tasks run
on other cpus.
In other cases, the following RCU splat was triggered :
(need_resched() doesnt know the current cpu is blocking RCU )
Jan 2 21:33:40 lpq83 kernel: [ 311.678050] INFO: rcu_sched self-detected stall on CPU { 2} (t=21000 jiffies g=11416 c=11415 q=2665)
Jan 2 21:33:40 lpq83 kernel: [ 311.687314] Pid: 1460, comm: simple-watchdog Not tainted 3.8.0-smp-DEV #63
Jan 2 21:33:40 lpq83 kernel: [ 311.687316] Call Trace:
Jan 2 21:33:40 lpq83 kernel: [ 311.687317] <IRQ> [<ffffffff81100e92>] rcu_check_callbacks+0x212/0x7a0
Jan 2 21:33:40 lpq83 kernel: [ 311.687326] [<ffffffff81097018>] update_process_times+0x48/0x90
Jan 2 21:33:40 lpq83 kernel: [ 311.687329] [<ffffffff810cfe31>] tick_sched_timer+0x81/0xd0
Jan 2 21:33:40 lpq83 kernel: [ 311.687332] [<ffffffff810ad69d>] __run_hrtimer+0x7d/0x220
Jan 2 21:33:40 lpq83 kernel: [ 311.687333] [<ffffffff810cfdb0>] ? tick_nohz_handler+0x100/0x100
Jan 2 21:33:40 lpq83 kernel: [ 311.687337] [<ffffffff810ca02c>] ? ktime_get_update_offsets+0x4c/0xd0
Jan 2 21:33:40 lpq83 kernel: [ 311.687339] [<ffffffff810adfa7>] hrtimer_interrupt+0xf7/0x230
Jan 2 21:33:40 lpq83 kernel: [ 311.687343] [<ffffffff815b7089>] smp_apic_timer_interrupt+0x69/0x99
Jan 2 21:33:40 lpq83 kernel: [ 311.687345] [<ffffffff815b630a>] apic_timer_interrupt+0x6a/0x70
Jan 2 21:33:40 lpq83 kernel: [ 311.687348] [<ffffffffa01a1b66>] ? ipt_do_table+0x106/0x5b0 [ip_tables]
Jan 2 21:33:40 lpq83 kernel: [ 311.687352] [<ffffffff810fb357>] ? handle_edge_irq+0x77/0x130
Jan 2 21:33:40 lpq83 kernel: [ 311.687354] [<ffffffff8108e9c9>] ? irq_exit+0x79/0xb0
Jan 2 21:33:40 lpq83 kernel: [ 311.687356] [<ffffffff815b6fa3>] ? do_IRQ+0x63/0xe0
Jan 2 21:33:40 lpq83 kernel: [ 311.687359] [<ffffffffa004a0d3>] iptable_filter_hook+0x33/0x64 [iptable_filter]
Jan 2 21:33:40 lpq83 kernel: [ 311.687362] [<ffffffff81523dff>] nf_iterate+0x8f/0xd0
Jan 2 21:33:40 lpq83 kernel: [ 311.687364] [<ffffffff81529fc0>] ? ip_rcv_finish+0x360/0x360
Jan 2 21:33:40 lpq83 kernel: [ 311.687366] [<ffffffff81523ebd>] nf_hook_slow+0x7d/0x150
Jan 2 21:33:40 lpq83 kernel: [ 311.687368] [<ffffffff81529fc0>] ? ip_rcv_finish+0x360/0x360
Jan 2 21:33:40 lpq83 kernel: [ 311.687370] [<ffffffff8152a39e>] ip_local_deliver+0x5e/0xa0
Jan 2 21:33:40 lpq83 kernel: [ 311.687372] [<ffffffff81529d79>] ip_rcv_finish+0x119/0x360
Jan 2 21:33:40 lpq83 kernel: [ 311.687374] [<ffffffff8152a631>] ip_rcv+0x251/0x300
Jan 2 21:33:40 lpq83 kernel: [ 311.687377] [<ffffffff814f4c72>] __netif_receive_skb+0x582/0x820
Jan 2 21:33:40 lpq83 kernel: [ 311.687379] [<ffffffff81560697>] ? inet_gro_receive+0x197/0x200
Jan 2 21:33:40 lpq83 kernel: [ 311.687381] [<ffffffff814f50ad>] netif_receive_skb+0x2d/0x90
Jan 2 21:33:40 lpq83 kernel: [ 311.687383] [<ffffffff814f5943>] napi_gro_frags+0xf3/0x2a0
Jan 2 21:33:40 lpq83 kernel: [ 311.687387] [<ffffffffa01aa87c>] mlx4_en_process_rx_cq+0x6cc/0x7b0 [mlx4_en]
Jan 2 21:33:40 lpq83 kernel: [ 311.687390] [<ffffffffa01aa9ff>] mlx4_en_poll_rx_cq+0x3f/0x80 [mlx4_en]
Jan 2 21:33:40 lpq83 kernel: [ 311.687392] [<ffffffff814f53c1>] net_rx_action+0x111/0x210
Jan 2 21:33:40 lpq83 kernel: [ 311.687393] [<ffffffff814f3a51>] ? net_tx_action+0x81/0x1d0
With this change, there is a possibility that a rapidly-rescheduling
task will cause softirq starvation?
Only if this task has higher priority than ksoftirqd, but then, people
wanting/playing with high priority tasks know what they do ;)
Can this change cause worsened latencies in some situations? Say there
are a large number of short-running actions queued. Presently we'll
dispatch ten of them and return. With this change we'll dispatch many
more of them - however many consume 2ms. So worst-case latency
increases from "10 * not-much" to "2 ms".
I tried to reproduce such workload but couldnt. 2 ms (or more exactly 1
to 2 ms given the jiffies/HZ granularity) is about the time needed to
process 1000 frames on current hardware.
Certainly, this patch will increase number of scheduler calls in some
situations. But with the increase of cores, it seems a bit odd to allow
softirq to be a bad guy. The current logic was more suited for the !SMP
age.
From: Ben Hutchings <hidden> Date: 2013-01-03 22:08:06
On Thu, 2013-01-03 at 04:28 -0800, Eric Dumazet wrote:
From: Eric Dumazet <edumazet@google.com>
In various network workloads, __do_softirq() latencies can be up
to 20 ms if HZ=1000, and 200 ms if HZ=100.
This is because we iterate 10 times in the softirq dispatcher,
and some actions can consume a lot of cycles.
This patch changes the fallback to ksoftirqd condition to :
- A time limit of 2 ms.
- need_resched() being set on current task
[...]
Really? Never iterate if HZ < 500?
Ben.
--
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
From: Eric Dumazet <hidden> Date: 2013-01-03 22:40:26
On Thu, 2013-01-03 at 22:08 +0000, Ben Hutchings wrote:
On Thu, 2013-01-03 at 04:28 -0800, Eric Dumazet wrote:
quoted
From: Eric Dumazet <edumazet@google.com>
In various network workloads, __do_softirq() latencies can be up
to 20 ms if HZ=1000, and 200 ms if HZ=100.
This is because we iterate 10 times in the softirq dispatcher,
and some actions can consume a lot of cycles.
This patch changes the fallback to ksoftirqd condition to :
- A time limit of 2 ms.
- need_resched() being set on current task
From: Eric Dumazet <hidden> Date: 2013-01-04 07:49:47
From: Eric Dumazet <edumazet@google.com>
In various network workloads, __do_softirq() latencies can be up
to 20 ms if HZ=1000, and 200 ms if HZ=100.
This is because we iterate 10 times in the softirq dispatcher,
and some actions can consume a lot of cycles.
This patch changes the fallback to ksoftirqd condition to :
- A time limit of 2 ms.
- need_resched() being set on current task
When one of this condition is met, we wakeup ksoftirqd for further
softirq processing if we still have pending softirqs.
Using need_resched() as the only condition can trigger RCU stalls,
as we can keep BH disabled for too long.
I ran several benchmarks and got no significant difference in
throughput, but a very significant reduction of latencies (one order
of magnitude) :
In following bench, 200 antagonist "netperf -t TCP_RR" are started in
background, using all available cpus.
Then we start one "netperf -t TCP_RR", bound to the cpu handling the NIC
IRQ (hard+soft)
Before patch :
# netperf -H 7.7.7.84 -t TCP_RR -T2,2 -- -k
RT_LATENCY,MIN_LATENCY,MAX_LATENCY,P50_LATENCY,P90_LATENCY,P99_LATENCY,MEAN_LATENCY,STDDEV_LATENCY
MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET
to 7.7.7.84 () port 0 AF_INET : first burst 0 : cpu bind
RT_LATENCY=550110.424
MIN_LATENCY=146858
MAX_LATENCY=997109
P50_LATENCY=305000
P90_LATENCY=550000
P99_LATENCY=710000
MEAN_LATENCY=376989.12
STDDEV_LATENCY=184046.92
After patch :
# netperf -H 7.7.7.84 -t TCP_RR -T2,2 -- -k
RT_LATENCY,MIN_LATENCY,MAX_LATENCY,P50_LATENCY,P90_LATENCY,P99_LATENCY,MEAN_LATENCY,STDDEV_LATENCY
MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET
to 7.7.7.84 () port 0 AF_INET : first burst 0 : cpu bind
RT_LATENCY=40545.492
MIN_LATENCY=9834
MAX_LATENCY=78366
P50_LATENCY=33583
P90_LATENCY=59000
P99_LATENCY=69000
MEAN_LATENCY=38364.67
STDDEV_LATENCY=12865.26
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: David Miller <davem@davemloft.net>
Cc: Tom Herbert <redacted>
Cc: Ben Hutchings <redacted>
---
v2: min(1, (2*HZ/1000)) -> max(1, (2*HZ/1000)), as spotted by Ben
kernel/softirq.c | 17 +++++++++--------
1 file changed, 9 insertions(+), 8 deletions(-)
From: Joe Perches <joe@perches.com> Date: 2013-01-04 08:15:54
On Thu, 2013-01-03 at 23:49 -0800, Eric Dumazet wrote:
In various network workloads, __do_softirq() latencies can be up
to 20 ms if HZ=1000, and 200 ms if HZ=100.
This patch changes the fallback to ksoftirqd condition to :
- A time limit of 2 ms.
[]
quoted hunk
diff --git a/kernel/softirq.c b/kernel/softirq.c
[]
+#define MAX_SOFTIRQ_TIME max(1, (2*HZ/1000))
And if HZ is 10000?
asmlinkage void __do_softirq(void)
{
[]
+ unsigned long end = jiffies + MAX_SOFTIRQ_TIME;
Perhaps MAX_SOFTIRQ_TIME should be
#define MAX_SOFTIRQ_TIME msecs_to_jiffies(2)
though it would be nicer if it were a compile time constant.
From: Eric Dumazet <hidden> Date: 2013-01-04 08:23:12
On Fri, 2013-01-04 at 00:15 -0800, Joe Perches wrote:
On Thu, 2013-01-03 at 23:49 -0800, Eric Dumazet wrote:
quoted
In various network workloads, __do_softirq() latencies can be up
to 20 ms if HZ=1000, and 200 ms if HZ=100.
This patch changes the fallback to ksoftirqd condition to :
- A time limit of 2 ms.
[]
quoted
diff --git a/kernel/softirq.c b/kernel/softirq.c
[]
quoted
+#define MAX_SOFTIRQ_TIME max(1, (2*HZ/1000))
And if HZ is 10000?
Then its OK. 2*10000/1000 -> 20 ticks -> 2 ms
quoted
asmlinkage void __do_softirq(void)
{
[]
quoted
+ unsigned long end = jiffies + MAX_SOFTIRQ_TIME;
Perhaps MAX_SOFTIRQ_TIME should be
#define MAX_SOFTIRQ_TIME msecs_to_jiffies(2)
though it would be nicer if it were a compile time constant.
If you send a patch to convert msecs_to_jiffies() to an inline function
when HZ = 1000, I will gladly use it instead of (2*HZ/1000)
Right now, max(1, msecs_to_jiffies(2)) uses way too many instructions,
while it should be the constant 2, known at compile time.
From: Joe Perches <joe@perches.com> Date: 2013-01-04 09:12:08
On Fri, 2013-01-04 at 00:23 -0800, Eric Dumazet wrote:
On Fri, 2013-01-04 at 00:15 -0800, Joe Perches wrote:
quoted
Perhaps MAX_SOFTIRQ_TIME should be
#define MAX_SOFTIRQ_TIME msecs_to_jiffies(2)
though it would be nicer if it were a compile time constant.
If you send a patch to convert msecs_to_jiffies() to an inline function
when HZ = 1000, I will gladly use it instead of (2*HZ/1000)
Right now, max(1, msecs_to_jiffies(2)) uses way too many instructions,
while it should be the constant 2, known at compile time.
Something like this might work.
This is incomplete, it just does msecs_to_jiffies,
and it should convert usecs_to_jiffies and the
jiffies_to_<foo> types too.
Maybe it's worthwhile.
It does reduce object size by 16 bytes per call site
(x86-32) when the argument is a constant. There are
about 800 of these jiffies conversions in kernel sources.
What do you think?
include/linux/jiffies.h | 49 ++++++++++++++++++++++++++++++++++++++++++++++++-
kernel/time.c | 42 +++---------------------------------------
2 files changed, 51 insertions(+), 40 deletions(-)
From: Eric Dumazet <hidden> Date: 2013-01-04 17:00:34
On Fri, 2013-01-04 at 01:12 -0800, Joe Perches wrote:
On Fri, 2013-01-04 at 00:23 -0800, Eric Dumazet wrote:
quoted
On Fri, 2013-01-04 at 00:15 -0800, Joe Perches wrote:
quoted
Perhaps MAX_SOFTIRQ_TIME should be
#define MAX_SOFTIRQ_TIME msecs_to_jiffies(2)
though it would be nicer if it were a compile time constant.
If you send a patch to convert msecs_to_jiffies() to an inline function
when HZ = 1000, I will gladly use it instead of (2*HZ/1000)
Right now, max(1, msecs_to_jiffies(2)) uses way too many instructions,
while it should be the constant 2, known at compile time.
Something like this might work.
This is incomplete, it just does msecs_to_jiffies,
and it should convert usecs_to_jiffies and the
jiffies_to_<foo> types too.
Maybe it's worthwhile.
It does reduce object size by 16 bytes per call site
(x86-32) when the argument is a constant. There are
about 800 of these jiffies conversions in kernel sources.
What do you think?
I think this is something to discuss in another thread, and definitely
worth to do, at least for msecs_to_jiffies()
We have many HZ references everywhere that could be cleaned up using
this.
From: Joe Perches <joe@perches.com> Date: 2013-01-04 21:15:48
Do the multiplications and divisions at compile time
instead of runtime when the converted value is a constant.
Make the calculation functions static __always_inline to jiffies.h.
Add #defines with __builtin_constant_p to test and use the
static inline or the runtime functions as appropriate.
Prefix the old exported symbols/functions with __
Signed-off-by: Joe Perches <joe@perches.com>
---
include/linux/jiffies.h | 136 ++++++++++++++++++++++++++++++++++++++++++++++--
kernel/time.c | 106 +++++--------------------------------
2 files changed, 144 insertions(+), 98 deletions(-)
From: David Miller <davem@davemloft.net> Date: 2013-01-04 21:49:14
From: Eric Dumazet <redacted>
Date: Thu, 03 Jan 2013 23:49:40 -0800
From: Eric Dumazet <edumazet@google.com>
In various network workloads, __do_softirq() latencies can be up
to 20 ms if HZ=1000, and 200 ms if HZ=100.
This is because we iterate 10 times in the softirq dispatcher,
and some actions can consume a lot of cycles.
This patch changes the fallback to ksoftirqd condition to :
- A time limit of 2 ms.
- need_resched() being set on current task
When one of this condition is met, we wakeup ksoftirqd for further
softirq processing if we still have pending softirqs.