Robert demonstrated to us sometime ago with a small
timestamping user program to show that it can get starved for
more than 6 seconds in his system. So userland starvation is an
issue.
softirqs are already capable of being offloaded to scheduler-friendy
kernel threads to avoid starvation, if this wasn't the case NAPI would
have no way to work in the first place and everything else would fall
apart too, not just the rcu-route-cache. I don't think high latencies
and starvation are the same thing, starvation means for "indefinite
time" and you can't hang userspace for indefinite time using softirqs.
For sure the irq based load, and in turn softirqs too, can take a large
amount of cpu (though not 100%, this is why it cannot be called
starvation).
the only real starvation you can claim is in presence of an _hard_irq
flood, not a softirq one.
There are no hardirqs in the case under investigation, remember?
What's about the problem it really splits to two ones:
1. The _new_ problem when bad latency of rcu hurts core
functionality. This is precise description:
to keep up with the softirq load. This has never been the case so far,
and serving softirq as fast as possible is normally a good thing for
server/firewalls, the small unfariness (note unfariness != starvation)
it generates has never been an issue, because so far the softirq never
required the scheduler to work in order to do their work, rcu changed
this in the routing cache specific case.
We had one full solution for this issue not changing anything
in scheduler/softirq relationship: to run rcu task for the things
sort of dst cache not from process context, but essentially as part
of do_softirq(). Simple, stupid and apparently solves new problems
which rcu created.
Another solution is just to increase memory consumption limits
to deal with current rcu latency. F.e. 300ms latency just requires
additional space for pps*300ms objects, which are handled by RCU.
The problem with this is that pps is the thing which increases
when cpu power grows and that 300ms is not a mathematically established
limit too.
So you're simply asking the ksoftirqd offloading to become more
aggressive,
It is the second challenge.
Andrea, it is experimenatl fact: this "small unfariness" stalls process
contexts for >=6 seconds and gives them microscopic slices. We could live
with this (provided RCU problem is solved in some way). Essentially,
the only trouble for me was that we could use existing rcu bits to make
offloading to ksoftirqd more smart (not aggressive, _smart_). The absense
of RCU quiescent states looks too close to absence of forward progress
in process contexts, it was anticipating similarity. The dumb throttling
do_softirq made not from ksoftirqd context when starvation is detected
which we tested the last summer is not only ugly, it really might hurt
router performance, you are right here too. It is the challenge:
or we proceed with this idea and invent something, or we just forget
about this concentrating on RCU.
Alexey
On Wed, Mar 31, 2004 at 12:05:05AM +0400, kuznet@ms2.inr.ac.ru wrote:
What's about the problem it really splits to two ones:
1. The _new_ problem when bad latency of rcu hurts core
functionality. This is precise description:
Which essentially happens due to userland starvation during a
softirq flood.
quoted
to keep up with the softirq load. This has never been the case so far,
and serving softirq as fast as possible is normally a good thing for
server/firewalls, the small unfariness (note unfariness != starvation)
it generates has never been an issue, because so far the softirq never
required the scheduler to work in order to do their work, rcu changed
this in the routing cache specific case.
We had one full solution for this issue not changing anything
in scheduler/softirq relationship: to run rcu task for the things
sort of dst cache not from process context, but essentially as part
of do_softirq(). Simple, stupid and apparently solves new problems
which rcu created.
Can you be a little bit more specific about this solution ? There
were a number of them discussed during our private emails and I
can't tell which one you are talking about.
Another solution is just to increase memory consumption limits
to deal with current rcu latency. F.e. 300ms latency just requires
additional space for pps*300ms objects, which are handled by RCU.
The problem with this is that pps is the thing which increases
when cpu power grows and that 300ms is not a mathematically established
limit too.
I don't think increasing memory consumption limit is a good idea.
Andrea, it is experimenatl fact: this "small unfariness" stalls process
contexts for >=6 seconds and gives them microscopic slices. We could live
with this (provided RCU problem is solved in some way). Essentially,
the only trouble for me was that we could use existing rcu bits to make
offloading to ksoftirqd more smart (not aggressive, _smart_). The absense
of RCU quiescent states looks too close to absence of forward progress
in process contexts, it was anticipating similarity. The dumb throttling
do_softirq made not from ksoftirqd context when starvation is detected
which we tested the last summer is not only ugly, it really might hurt
router performance, you are right here too. It is the challenge:
or we proceed with this idea and invent something, or we just forget
about this concentrating on RCU.
Exactly. And the throttling will likely not be enough by ksoftirqd
as indicated by my earlier experiments. We have potential fixes
for RCU through a call_rcu_bh() interface where completion of a
softirq handler is a quiescent state. I am working on forward porting
that old patch from our discussion last year and testing in my
environment. That should increase the number of quiescent state
points significantly and hopefully reduce the grace period significantly.
But this does nothing to help userland starvation.
Thanks
Dipankar
From: Andrea Arcangeli <hidden> Date: 2004-03-30 21:14:56
On Wed, Mar 31, 2004 at 12:05:05AM +0400, kuznet@ms2.inr.ac.ru wrote:
Hello!
quoted
quoted
Robert demonstrated to us sometime ago with a small
timestamping user program to show that it can get starved for
more than 6 seconds in his system. So userland starvation is an
issue.
softirqs are already capable of being offloaded to scheduler-friendy
kernel threads to avoid starvation, if this wasn't the case NAPI would
have no way to work in the first place and everything else would fall
apart too, not just the rcu-route-cache. I don't think high latencies
and starvation are the same thing, starvation means for "indefinite
time" and you can't hang userspace for indefinite time using softirqs.
For sure the irq based load, and in turn softirqs too, can take a large
amount of cpu (though not 100%, this is why it cannot be called
starvation).
the only real starvation you can claim is in presence of an _hard_irq
flood, not a softirq one.
There are no hardirqs in the case under investigation, remember?
no hardirqs? there must be tons of hardirqs if ksoftirqd never runs.
What's about the problem it really splits to two ones:
1. The _new_ problem when bad latency of rcu hurts core
functionality. This is precise description:
quoted
to keep up with the softirq load. This has never been the case so far,
and serving softirq as fast as possible is normally a good thing for
server/firewalls, the small unfariness (note unfariness != starvation)
it generates has never been an issue, because so far the softirq never
required the scheduler to work in order to do their work, rcu changed
this in the routing cache specific case.
We had one full solution for this issue not changing anything
in scheduler/softirq relationship: to run rcu task for the things
sort of dst cache not from process context, but essentially as part
of do_softirq(). Simple, stupid and apparently solves new problems
which rcu created.
I don't understand exactly the details of how this works but I trust you ;)
Another solution is just to increase memory consumption limits
to deal with current rcu latency. F.e. 300ms latency just requires
additional space for pps*300ms objects, which are handled by RCU.
The problem with this is that pps is the thing which increases
when cpu power grows and that 300ms is not a mathematically established
limit too.
I consider this more a workaround than a solution ;). It also depends
how much boost you get from the rcu scalability in smp to evaluate if
the above solution could be worthwhile, I don't know the exact numbers
myself.
quoted
So you're simply asking the ksoftirqd offloading to become more
aggressive,
It is the second challenge.
Andrea, it is experimenatl fact: this "small unfariness" stalls process
contexts for >=6 seconds and gives them microscopic slices. We could live
with this (provided RCU problem is solved in some way). Essentially,
yes, we could live with this if it wasn't for RCU not making any
progress and the irq load depending on the RCU to make progress.
the only trouble for me was that we could use existing rcu bits to make
offloading to ksoftirqd more smart (not aggressive, _smart_). The absense
of RCU quiescent states looks too close to absence of forward progress
in process contexts, it was anticipating similarity. The dumb throttling
do_softirq made not from ksoftirqd context when starvation is detected
which we tested the last summer is not only ugly, it really might hurt
router performance, you are right here too. It is the challenge:
or we proceed with this idea and invent something, or we just forget
about this concentrating on RCU.
So basically you're suggesting to use the rcu grace period "timeout" to
choose when to defer the softirq load to userspace, right? That should
work. I mean, it will need some instrumentation to define the "timeout",
but it's very similar to the patch I already posted, rather than
local_softirqd_running() we'll check for rcu_timed_out() and we'll
wakeup ksofitrqd in that case and we'll return immediatly from
do_softirq. That will still risk to lose some packet from the backlog
though, since we'll process softirqs at a slower peace (but that's
partly the feature).
That sounds a bit too rcu centric though, I mean, if we do this, maybe
we should make it indipendent from rcu, and we could detect a too long
time spent in softirq load, and we can offload softirqs in function of
that instead of the rcu_timed_out() thing.
From: David S. Miller <hidden> Date: 2004-03-30 21:30:54
On Tue, 30 Mar 2004 23:14:50 +0200
Andrea Arcangeli [off-list ref] wrote:
quoted
There are no hardirqs in the case under investigation, remember?
no hardirqs? there must be tons of hardirqs if ksoftirqd never runs.
NAPI should be kicking in for this workload, and I know for a fact it is
for Robert's case. There should only be a few thousand hard irqs per
second.
Until the RX ring is depleted the device's hardirqs will not be re-
enabled.
From: Andrea Arcangeli <hidden> Date: 2004-03-30 21:37:47
On Tue, Mar 30, 2004 at 01:30:00PM -0800, David S. Miller wrote:
On Tue, 30 Mar 2004 23:14:50 +0200
Andrea Arcangeli [off-list ref] wrote:
quoted
quoted
There are no hardirqs in the case under investigation, remember?
no hardirqs? there must be tons of hardirqs if ksoftirqd never runs.
NAPI should be kicking in for this workload, and I know for a fact it is
for Robert's case. There should only be a few thousand hard irqs per
second.
Until the RX ring is depleted the device's hardirqs will not be re-
enabled.
then Dipankar is reproducing with a workload that is completely
different. I've only seen the emails from Dipankar so I couldn't know it
was a NAPI load.
He posted these numbers:
softirq_count, ksoftirqd_count and other_softirq_count shows -
CPU 0 : 638240 554 637686
CPU 1 : 102316 1 102315
CPU 2 : 675696 557 675139
CPU 3 : 102305 0 102305
that means nothing runs in ksoftirqd for Dipankar, so he cannot be using
NAPI.
Either that or I'm misreading his numbers, or his stats results are wrong.
From: David S. Miller <hidden> Date: 2004-03-30 22:23:27
On Tue, 30 Mar 2004 23:37:42 +0200
Andrea Arcangeli [off-list ref] wrote:
that means nothing runs in ksoftirqd for Dipankar, so he cannot be using
NAPI.
Either that or I'm misreading his numbers, or his stats results are wrong.
If these numbers are with your "if (ksoftirqd_pending()) return;" thing
at the top of do_softirq() then I must agree with you.
Otherwise, keep in mind what I said, and also as Robert mentioned every
single local_bh_enable() is going to call do_softirq() if the count falls
to zero.
From: Andrea Arcangeli <hidden> Date: 2004-03-30 22:55:24
On Tue, Mar 30, 2004 at 02:22:10PM -0800, David S. Miller wrote:
Otherwise, keep in mind what I said, and also as Robert mentioned every
single local_bh_enable() is going to call do_softirq() if the count falls
to zero.
I was less concerned about the do_sofitrq in local_bh_enable, since that
runs in a scheduler-aware context, so at least the timeslice is
definitely accounted for and it'll schedule at some point (unlike with
an hardirq flood). Actually the length of the default timeslice matters
too here, lowering the max timeslice to 10msec would certainly reduce
the effect.
call_rcu_bh will fix the local_bh_enable too. The only problem with
call_rcu_bh is how to queue the tasklets in every cpu (an IPI sounds
overkill at high frequency, because effectively here we're running the rcu
callbacks in a potential fast path). OTOH if we've to add a spinlock to
queue the tasklet, then we might as well take a spinlock in the routing
cache in the first place (at least for this workload).
From: Robert Olsson <hidden> Date: 2004-03-30 22:58:39
Andrea Arcangeli writes:
> He posted these numbers:
>
> softirq_count, ksoftirqd_count and other_softirq_count shows -
>
> CPU 0 : 638240 554 637686
> CPU 1 : 102316 1 102315
> CPU 2 : 675696 557 675139
> CPU 3 : 102305 0 102305
>
> that means nothing runs in ksoftirqd for Dipankar, so he cannot be using
> NAPI.
>
> Either that or I'm misreading his numbers, or his stats results are wrong.
Well we have to ask Dipankar... But I'll buy a beer if it's not on. :)
Anyway w. NAPI enabled. 2 * 304 kpps DoS flows into eth0, eth2. Flows
are 2 * 10 Millions 64 byte pkts. 32 k buckets routehash. Full Internet
routing means ~130 k routes. Linux 2.6.4 2*2.66 MHz XEON.
26: 896 0 IO-APIC-level eth0
27: 25197 0 IO-APIC-level eth1
28: 8 579 IO-APIC-level eth2
29: 10 26112 IO-APIC-level eth3
T-put is seen on output dev. eth1, eth3. So about 16% of incoming load,
eth0 1500 0 1577468 9631270 9631270 8422828 237 0 0 0 BRU
eth1 1500 0 42 0 0 0 1573355 0 0 0 BRU
eth2 1500 0 1636154 9603432 9603432 8363849 41 0 0 0 BRU
eth3 1500 0 54 0 0 0 1632274 0 0 0 BRU
And lots of
.
.
printk: 1898 messages suppressed.
dst cache overflow
printk: 829 messages suppressed.
dst cache overflow
Cheers.
--ro
On Tue, Mar 30, 2004 at 11:37:42PM +0200, Andrea Arcangeli wrote:
On Tue, Mar 30, 2004 at 01:30:00PM -0800, David S. Miller wrote:
quoted
On Tue, 30 Mar 2004 23:14:50 +0200
Andrea Arcangeli [off-list ref] wrote:
quoted
quoted
There are no hardirqs in the case under investigation, remember?
no hardirqs? there must be tons of hardirqs if ksoftirqd never runs.
NAPI should be kicking in for this workload, and I know for a fact it is
for Robert's case. There should only be a few thousand hard irqs per
second.
Until the RX ring is depleted the device's hardirqs will not be re-
enabled.
then Dipankar is reproducing with a workload that is completely
different. I've only seen the emails from Dipankar so I couldn't know it
was a NAPI load.
He posted these numbers:
softirq_count, ksoftirqd_count and other_softirq_count shows -
CPU 0 : 638240 554 637686
CPU 1 : 102316 1 102315
CPU 2 : 675696 557 675139
CPU 3 : 102305 0 102305
that means nothing runs in ksoftirqd for Dipankar, so he cannot be using
NAPI.
And I am not. I am still on 2.6.0 and there seems to be no NAPI support
for the e100 there. Should I try 2.6.4 where e100 has NAPI support ?
Anyway, even without softirqs on the back of hardirqs, there are
other ways of softirq overload as seen in Robert's setup.
Thanks
Dipankar
From: Robert Olsson <hidden> Date: 2004-03-31 18:46:36
Dipankar Sarma writes:
> And I am not. I am still on 2.6.0 and there seems to be no NAPI support
> for the e100 there. Should I try 2.6.4 where e100 has NAPI support ?
>
> Anyway, even without softirqs on the back of hardirqs, there are
> other ways of softirq overload as seen in Robert's setup.
Well I see some hardirq's most from TX and timer interrups and HZ=1000 can
can change the way softirq's are run a bit.
I hacked do_softirq() sources so we can understand how things work a bit
better. (Use fastroute stats from /proc/net/softnet_stat).
ksoftird == softirq's sourced from ksofttirq
irqexit == softirq's sourced from interrupt exit
bh_enbl == softirq's sourced local_bh_enable
Before run
total droppped tsquz throttl bh_enbl ksoftird irqexit other
00000000 00000000 00000000 00000000 000000e8 0000017e 00030411 00000000
00000000 00000000 00000000 00000000 000000ae 00000277 00030349 00000000
After DoS (See description from previous mail)
total droppped tsquz throttl bh_enbl ksoftird irqexit other
00164c55 00000000 000021de 00000000 000000fc 0000229f 0003443c 00000000
001695e7 00000000 0000224d 00000000 00000162 0000236f 000342f7 00000000
So the major part of softirq's are run from irqexit and therefor out of
scheduler control. This even with RX polling (eth0, eth2) We still have
some TX interrupts plus timer interrupts now at 1000Hz. Which probably
reduces the number of softirq's that ksoftirqd runs.
CPU0 CPU1
0: 297156 0 IO-APIC-edge timer
1: 431 0 IO-APIC-edge i8042
2: 0 0 XT-PIC cascade
8: 0 0 IO-APIC-edge rtc
14: 34527 0 IO-APIC-edge ide0
26: 131 0 IO-APIC-level eth0
27: 22910 0 IO-APIC-level eth1
28: 8 124 IO-APIC-level eth2
29: 9 23197 IO-APIC-level eth3
NMI: 0 0
LOC: 297060 297059
On Wed, Mar 31, 2004 at 08:46:09PM +0200, Robert Olsson wrote:
Content-Description: message body text
Before run
total droppped tsquz throttl bh_enbl ksoftird irqexit other
00000000 00000000 00000000 00000000 000000e8 0000017e 00030411 00000000
00000000 00000000 00000000 00000000 000000ae 00000277 00030349 00000000
After DoS (See description from previous mail)
total droppped tsquz throttl bh_enbl ksoftird irqexit other
00164c55 00000000 000021de 00000000 000000fc 0000229f 0003443c 00000000
001695e7 00000000 0000224d 00000000 00000162 0000236f 000342f7 00000000
So the major part of softirq's are run from irqexit and therefor out of
scheduler control. This even with RX polling (eth0, eth2) We still have
some TX interrupts plus timer interrupts now at 1000Hz. Which probably
reduces the number of softirq's that ksoftirqd runs.
So, NAPI or not we get userland stalls due to packetflooding.
Looking at some of the old patches we discussed privately, it seems
this is what was done earlier -
1. Use rcu-softirq.patch which provides call_rcu_bh() for softirqs
only.
2. Limit non-ksoftirqd softirqs - get a measure of userland stall (using
an api rcu_grace_period(cpu)) and if it is too long, expire
the timeslice of the current process and start sending everything to
ksoftirqd.
By reducing the softirq time at the back of a hardirq or local_bh_enable(),
we should be able to bring a bit more fairness. I am working on the
patches, will test and publish later.
Thanks
Dipankar
On Wed, Mar 31, 2004 at 12:49:02AM +0200, Andrea Arcangeli wrote:
On Tue, Mar 30, 2004 at 02:22:10PM -0800, David S. Miller wrote:
quoted
Otherwise, keep in mind what I said, and also as Robert mentioned every
single local_bh_enable() is going to call do_softirq() if the count falls
to zero.
I was less concerned about the do_sofitrq in local_bh_enable, since that
runs in a scheduler-aware context, so at least the timeslice is
definitely accounted for and it'll schedule at some point (unlike with
an hardirq flood). Actually the length of the default timeslice matters
too here, lowering the max timeslice to 10msec would certainly reduce
the effect.
That is there in my list of things to test.
call_rcu_bh will fix the local_bh_enable too. The only problem with
call_rcu_bh is how to queue the tasklets in every cpu (an IPI sounds
overkill at high frequency, because effectively here we're running the rcu
callbacks in a potential fast path). OTOH if we've to add a spinlock to
queue the tasklet, then we might as well take a spinlock in the routing
cache in the first place (at least for this workload).
I don't do any of this. I just have a separate quiescent state counter
for softirq RCU. It is incremented for regular quiescent points
like cswitch, userland, idle loop as well as at the completion
of each softirq handler. call_rcu_bh() uses its own queues.
Everything else works like call_rcu().
Thanks
Dipankar
From: Andrea Arcangeli <hidden> Date: 2004-03-31 21:33:38
On Thu, Apr 01, 2004 at 02:07:50AM +0530, Dipankar Sarma wrote:
On Wed, Mar 31, 2004 at 08:46:09PM +0200, Robert Olsson wrote:
Content-Description: message body text
quoted
Before run
total droppped tsquz throttl bh_enbl ksoftird irqexit other
00000000 00000000 00000000 00000000 000000e8 0000017e 00030411 00000000
00000000 00000000 00000000 00000000 000000ae 00000277 00030349 00000000
After DoS (See description from previous mail)
total droppped tsquz throttl bh_enbl ksoftird irqexit other
00164c55 00000000 000021de 00000000 000000fc 0000229f 0003443c 00000000
001695e7 00000000 0000224d 00000000 00000162 0000236f 000342f7 00000000
So the major part of softirq's are run from irqexit and therefor out of
scheduler control. This even with RX polling (eth0, eth2) We still have
some TX interrupts plus timer interrupts now at 1000Hz. Which probably
reduces the number of softirq's that ksoftirqd runs.
So, NAPI or not we get userland stalls due to packetflooding.
indeed, the most of the softirq load happens within irqs even with NAPI
as we were talking about, so Alexey and DaveM were wrong about the
hardirq load being non significant.
Maybe the problem is simply that NAPI should be tuned more aggressively,
it may have to poll for a longer time before giving up.
Looking at some of the old patches we discussed privately, it seems
this is what was done earlier -
1. Use rcu-softirq.patch which provides call_rcu_bh() for softirqs
only.
this is the one I prefer if it performs.
2. Limit non-ksoftirqd softirqs - get a measure of userland stall (using
an api rcu_grace_period(cpu)) and if it is too long, expire
the timeslice of the current process and start sending everything to
ksoftirqd.
yep, this may be desiderable eventually just to be fair with tasks, but
I believe it's partly an orthogonal with the rcu grace period length.
By reducing the softirq time at the back of a hardirq or local_bh_enable(),
we should be able to bring a bit more fairness. I am working on the
patches, will test and publish later.
From: Andrea Arcangeli <hidden> Date: 2004-03-31 21:33:38
On Thu, Apr 01, 2004 at 02:16:11AM +0530, Dipankar Sarma wrote:
I don't do any of this. I just have a separate quiescent state counter
for softirq RCU. It is incremented for regular quiescent points
like cswitch, userland, idle loop as well as at the completion
of each softirq handler. call_rcu_bh() uses its own queues.
Everything else works like call_rcu().
the point is that you want this counter to increase in every cpu quick,
that's why I was thinking at posting the tasklet, if the counter doesn't
increase from softirq, you fallback in the grace period length of the
non-bh rcu.
maybe the softirq load is so high in all cpus that just the additional
counter will fix it w/o having to post any additional tasklet (I very
much hope so but especially with irq binding I don't see it happening,
however with irq binding you may have a call_rcu_bh_cpuset). You should
give it a try and see if it just works.
On Wed, Mar 31, 2004 at 11:28:17PM +0200, Andrea Arcangeli wrote:
On Thu, Apr 01, 2004 at 02:07:50AM +0530, Dipankar Sarma wrote:
quoted
So, NAPI or not we get userland stalls due to packetflooding.
indeed, the most of the softirq load happens within irqs even with NAPI
as we were talking about, so Alexey and DaveM were wrong about the
hardirq load being non significant.
Maybe the problem is simply that NAPI should be tuned more aggressively,
it may have to poll for a longer time before giving up.
Perhaps yes, but we still have softirqs from local_bh_enable()s to
deal with.
quoted
Looking at some of the old patches we discussed privately, it seems
this is what was done earlier -
1. Use rcu-softirq.patch which provides call_rcu_bh() for softirqs
only.
this is the one I prefer if it performs.
I just tried the attached patch (forward ported and some aggressive
things deleted) and there was no route cache overflow during
pktgen testing. So, this is a good approach, however this is
not going to solve userland stalls.
Robert, btw, this rcu-softirq patch is slightly different
from the earlier one in the sense that now every softirq
handler completion is a quiescent point. Earlier each iteration
of softirqs was a quiescent point. So this has more quiescent
points.
quoted
2. Limit non-ksoftirqd softirqs - get a measure of userland stall (using
an api rcu_grace_period(cpu)) and if it is too long, expire
the timeslice of the current process and start sending everything to
ksoftirqd.
yep, this may be desiderable eventually just to be fair with tasks, but
I believe it's partly an orthogonal with the rcu grace period length.
Yes, it is. Delaying softirqs will delay RCU grace periods. But it
will also affect i/o throughput. So this is a balancing act anyway.
quoted
By reducing the softirq time at the back of a hardirq or local_bh_enable(),
we should be able to bring a bit more fairness. I am working on the
patches, will test and publish later.
I consider this as the approch number 2 too.
Well, I don't really have #1 and #2. I think we need to understand
if there are the situations where such stalls are unacceptable
and if so fix softirqs for them. RCU OOMs we can probably work
around anyway.
Thanks
Dipankar
Provide a new call_rcu_bh() interface that can be used in softirq
only situations. Completion of a softirq handler is considered
a quiescent point apart from regular quiescent points. If there
is any read from process context, then it must be protected
by rcu_read_lock/unlock_bh().
include/linux/rcupdate.h | 40 ++++++---
kernel/rcupdate.c | 207 ++++++++++++++++++++++++++++-------------------
kernel/softirq.c | 12 ++
net/decnet/dn_route.c | 6 -
net/ipv4/route.c | 28 +++---
5 files changed, 183 insertions(+), 110 deletions(-)
diff -puN include/linux/rcupdate.h~rcu-softirq include/linux/rcupdate.h
On Wed, Mar 31, 2004 at 11:31:09PM +0200, Andrea Arcangeli wrote:
On Thu, Apr 01, 2004 at 02:16:11AM +0530, Dipankar Sarma wrote:
quoted
I don't do any of this. I just have a separate quiescent state counter
for softirq RCU. It is incremented for regular quiescent points
like cswitch, userland, idle loop as well as at the completion
of each softirq handler. call_rcu_bh() uses its own queues.
Everything else works like call_rcu().
the point is that you want this counter to increase in every cpu quick,
that's why I was thinking at posting the tasklet, if the counter doesn't
increase from softirq, you fallback in the grace period length of the
non-bh rcu.
maybe the softirq load is so high in all cpus that just the additional
counter will fix it w/o having to post any additional tasklet (I very
much hope so but especially with irq binding I don't see it happening,
however with irq binding you may have a call_rcu_bh_cpuset). You should
give it a try and see if it just works.
Ah, forcing CPUs for quiescent state is my last WMD if I have to use it ever :)
Thanks
Dipankar
From: Robert Olsson <hidden> Date: 2004-03-31 22:37:00
Andrea Arcangeli writes:
> Maybe the problem is simply that NAPI should be tuned more aggressively,
> it may have to poll for a longer time before giving up.
It cannot poll much more... 20 Million packets were injected in total
there were 250 RX irq's. Most from my ssh sessions. There are some TX
interrupts... it's another story
Packet flooding is just our way to generate load and kernel locking must
work with and without irq's. As far as I understand the real problem is
when do_softirq is run from irqexit etc.
Some thoughts...
If we tag the different do_softirq sources (look in my testpatch) we can
control the softirqs better. For example; do_softirq's from irqexit etc
could be given low a "max_restart" this to move processing to ksoftird
maybe even dynamic.
Cheers.
--ro
From: Andrea Arcangeli <hidden> Date: 2004-03-31 22:53:13
On Thu, Apr 01, 2004 at 12:36:00AM +0200, Robert Olsson wrote:
Andrea Arcangeli writes:
> Maybe the problem is simply that NAPI should be tuned more aggressively,
> it may have to poll for a longer time before giving up.
It cannot poll much more... 20 Million packets were injected in total
there were 250 RX irq's. Most from my ssh sessions. There are some TX
interrupts... it's another story
I didn't focus much on the irq count, but now that I look at it, it
looks like the biggest source of softirq in irq context is the timer
irq, not the network irq. That explains the problem and why NAPI
couldn't avoid the softirq load in irq context, NAPI avoids the network
irqs, but the softirqs keeps running in irq context.
So lowering HZ to 100 should mitigate the problem significantly.
But I feel like we should change the softirq code so that the irqexit
runs only the softirq setup by the current (or nested) irq handler. This
way the timer irq will stop executing the softirqs posted by the network
stack and it may very well fix the problem completely. This definitely
will help fairness too.
Packet flooding is just our way to generate load and kernel locking must
work with and without irq's. As far as I understand the real problem is
when do_softirq is run from irqexit etc.
yes, but it's run from the _timer_ irq as far as I can tell. Changing
the softirq code so that the irqexit only processes softirqs posted in
the irq handlers should fix it.
If we tag the different do_softirq sources (look in my testpatch) we can
control the softirqs better. For example; do_softirq's from irqexit etc
could be given low a "max_restart" this to move processing to ksoftird
maybe even dynamic.
max_restart is needed exactly to avoid irqexit load to be offloaded to
regular kernel context, so that's basically saying that we should
disable max_restart but that's not a good solution for some non-NAPI
workload.
From: Robert Olsson <hidden> Date: 2004-04-01 14:03:23
Andrea Arcangeli writes:
> But I feel like we should change the softirq code so that the irqexit
> runs only the softirq setup by the current (or nested) irq handler. This
> way the timer irq will stop executing the softirqs posted by the network
> stack and it may very well fix the problem completely. This definitely
> will help fairness too.
How would softirq's scheduled be softirq's be run?
For a definitive solution a think Alexey analysis make sense:
"All of them happening outside of ksoftirqd are equally bad, unaccountable,
uncontrollable and will show up in some situation."
FYI. I did some more experiments:
Simple sycall test: user app = ank-time-loop (does gettimeofday in a loop)
=========================================================================
Before
total droppped tsquz throttl bh_enbl ksoftird irqexit other
00000099 00000000 00000000 00000000 0000005d 00000005 000163a2 00000000
00000008 00000000 00000000 00000000 00000009 00000000 000162f6 00000000
ank-time-loop
After 10 sec.
000000c0 00000000 00000000 00000000 0000005d 00000005 0001c3d2 00000000
00000008 00000000 00000000 00000000 00000009 00000000 0001c2f3 00000000
So syscalls "in this context" seems to do_softirq() via irqexit.
Route DoS while BGP tables are loading. ank-time-loop is running.
================================================================
total droppped tsquz throttl bh_enbl ksoftird irqexit other
000000c2 00000000 00000000 00000000 00000042 00000005 0001aef8 00000000
00000008 00000000 00000000 00000000 00000017 00000001 0001adba 00000000
After run:
000b3a10 00000000 00000eda 00000000 000006f6 0000052e 00041c06 00000000
000b0500 00000000 00000ec1 00000000 0000075e 00000197 000419f2 00000000
Most softirq's are run from irqexit but we see local_bh_enable is running
softirq's too. And only a minority of softirq's is running under scheduler
control. As discussed we before with 100Hz timer we would expect lower
rates from irqexit.
Cheers.
--ro
On Mon, Apr 05, 2004 at 07:11:52PM +0200, Robert Olsson wrote:
Dipankar Sarma writes:
> Robert, btw, this rcu-softirq patch is slightly different
> from the earlier one in the sense that now every softirq
> handler completion is a quiescent point. Earlier each iteration
> of softirqs was a quiescent point. So this has more quiescent
> points.
Hello!
Yes it seems reduce RCU latency in our setup as well. It does not eliminate
overflows but reduces with ~50% and increases the throughput a bit. dst cache
overflow depends on RCU-delay + gc_min_interval and the number of entries
freed per sec so this means RCU has improved. Also the user app doing gettimeofday
seems to be better scheduled. The worst starvation improved from ~7.5 to ~4.4 sec.
Looks better atleast. Can you apply the following patch (rs-throttle-rcu)
on top of rcu-softirq.patch in your tree and see if helps a little bit more ?
Please make sure to set the kernel paramenters rcupdate.maxbatch to 4
and rcupdate.plugticks to 0. You can make sure of those parameters
by looking at dmesg (rcu prints them out during boot). I just merged
it, but have not tested this patch yet.
Thanks
Dipankar
Throttle rcu by forcing a limit on how many callbacks per softirq
and also implement a configurable plug. Applies on top of the
rcu-softirq (hence rs-) patch.
include/linux/list.h | 21 +++++++++++++++++++++
include/linux/rcupdate.h | 7 ++++++-
kernel/rcupdate.c | 33 +++++++++++++++++++++++++--------
kernel/sched.c | 2 ++
4 files changed, 54 insertions(+), 9 deletions(-)
diff -puN include/linux/list.h~rs-throttle-rcu include/linux/list.h
@@ -1486,6 +1486,8 @@ void scheduler_tick(int user_ticks, int if(rcu_pending(cpu))rcu_check_callbacks(cpu,user_ticks);+if(RCU_plugticks(cpu))+RCU_plugticks(cpu)--;/* note: this timer irq context must be accounted for as well */if(hardirq_count()-HARDIRQ_OFFSET){
From: Robert Olsson <hidden> Date: 2004-04-06 12:55:46
Dipankar Sarma writes:
> Looks better atleast. Can you apply the following patch (rs-throttle-rcu)
> on top of rcu-softirq.patch in your tree and see if helps a little bit more ?
> Please make sure to set the kernel paramenters rcupdate.maxbatch to 4
> and rcupdate.plugticks to 0. You can make sure of those parameters
> by looking at dmesg (rcu prints them out during boot). I just merged
> it, but have not tested this patch yet.
OK!
Well not tested yet but I don't think we will get rid overflow totally in my
setup. I've done a little experimental patch so *all* softirq's are run via
ksoftirqd.
total droppped tsquz throttl bh_enbl ksoftird irqexit other
009bee0c 00000000 00004aa7 00000000 00000000 0336a637 00000078 00000000
0054d381 00000000 00004ca0 00000000 00000000 032f8e48 00000000 00000000
I still see dst overflows. But the priority of the ksoftird's can now control
the user apps behavior even during softirq DoS.
On Tue, Apr 06, 2004 at 02:55:19PM +0200, Robert Olsson wrote:
Content-Description: message body text
Dipankar Sarma writes:
> Looks better atleast. Can you apply the following patch (rs-throttle-rcu)
> on top of rcu-softirq.patch in your tree and see if helps a little bit more ?
> Please make sure to set the kernel paramenters rcupdate.maxbatch to 4
> and rcupdate.plugticks to 0. You can make sure of those parameters
> by looking at dmesg (rcu prints them out during boot). I just merged
> it, but have not tested this patch yet.
OK!
Well not tested yet but I don't think we will get rid overflow totally in my
setup. I've done a little experimental patch so *all* softirq's are run via
ksoftirqd.
Robert, you should try out rs-throttle-rcu.patch. The idea is that
we don't run too many callbacks in a single rcu. In my setup,
at 100kpps, I see as many as 30000 rcu callbacks in a single
tasklet handler. That is likely hurting even the softirq-only
RCU grace periods. Setting rcupdate.maxbatch=4 will do only 4 per
tasklet thus providing more quiescent points to the system.
Thanks
Dipankar
On Wed, Apr 07, 2004 at 05:23:28PM +0200, Robert Olsson wrote:
Dipankar Sarma writes:
> Robert, you should try out rs-throttle-rcu.patch. The idea is that
> we don't run too many callbacks in a single rcu. In my setup,
> at 100kpps, I see as many as 30000 rcu callbacks in a single
> tasklet handler. That is likely hurting even the softirq-only
> RCU grace periods. Setting rcupdate.maxbatch=4 will do only 4 per
> tasklet thus providing more quiescent points to the system.
Hello!
No bad things happens, lots of overflows and drop in performance
and the userland app can stall for 32 sec. We seems to spin in
softirq to much and still don't get things done.
Argh!! Andrea, this means that throttling rcu callbacks with
back-to-back rcu tasklets for better scheduling latency is bad
for this kind of DoS situation. I think we will have to address
the softirq limiting question.
That said, Robert, one last experiment - if you are running UP,
can you try the following patchset (should apply on top of vanilla
2.6.x) ? This implements direct invocation of callbacks instead
of waiting for rcu grace periods in UP kernel. This would be a
good data point to understand what happens.
Thanks
Dipankar