ksoftirqd 100% after disabling IPv4 route cache on high pps.

Hi Eric,

After disabling the route cache, I found a forwarding performance
problem (inevitable ?). Basically the kernel couldn't keep up under
about 100k pps and ksoftirqd is dominating the CPU. This problem went
away right away if I do

echo  1 > rt_cache_rebuild_count

and comes back as soon as i do echo -1 > rt_cache_rebuild_count.

I then tried to use RPS/RFS to share the load on to mulitple cpus
(since ksoftirqd is only using 1 core, clearly). but that has little
effect.

Is there some tweaks/patches you recommend?

Thanks in advance.

On Tue, Nov 27, 2012 at 5:34 PM, 叶雨飞 [off-list ref] wrote:

Thanks!!! it works, after flushing cache it stays 0.

On Tue, Nov 27, 2012 at 5:01 PM, Eric Dumazet [off-list ref] wrote:

quoted

On Tue, 2012-11-27 at 15:15 -0800, 叶雨飞 wrote:

quoted

Hi,

I have a linux router running kernel 3.2  that receive public ingress
packets and route them through an GRE tunnel, return packets don't go
through it

I've recently faced a serious issue with the route cache,  when the
router received spoofed source , the route cache will quickly get
exhausted (depending on the size of it) and soon the ip dst cache
overflow will be printed and network subsystem will hang until
restarted.

So, my question is, how can I turn off the route cache without
recompile the kernel or adding the  patch for removal  in 3.7?  I
tried to set

echo 0 > /proc/sys/net/ipv4/route/max_size but that has no effect at all.

And if some one can share some insight on why when dst cache
overflows, the network subsystem hangs, it would be great.

echo -1 >/proc/sys/net/ipv4/rt_cache_rebuild_count

On Wed, 2012-12-05 at 13:22 -0800, 叶雨飞 wrote:

Hi Eric,

After disabling the route cache, I found a forwarding performance
problem (inevitable ?). Basically the kernel couldn't keep up under
about 100k pps and ksoftirqd is dominating the CPU. This problem went
away right away if I do

echo  1 > rt_cache_rebuild_count

and comes back as soon as i do echo -1 > rt_cache_rebuild_count.

I then tried to use RPS/RFS to share the load on to mulitple cpus
(since ksoftirqd is only using 1 core, clearly). but that has little
effect.

Is there some tweaks/patches you recommend?

Yes, upgrade your kernel to 3.6 or 3.7, this problem has been solved for
good. No more IP route cache and good performance.