From: "David S. Miller" <davem@davemloft.net> Date: 2004-12-06 21:50:13
On Mon, 6 Dec 2004 21:53:05 +0100
Karsten Desler [off-list ref] wrote:
20017 ipt_do_table 24.0589
It's spending nearly half of it's time in iptables.
Try to consolidate your rules if possible. This is the
part of netfilter that really doesn't scale well at all.
It's spending nearly half of it's time in iptables.
Try to consolidate your rules if possible. This is the
part of netfilter that really doesn't scale well at all.
Removing the iptables rules helps reducing the load a little, but the
majority of time is still spent somewhere else.
50kpps rx and 43kpps tx.
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 1802956 101032 104464 0 0 0 18 74 26 0 21 79 0
0 0 0 1802956 101032 104464 0 0 0 61 8810 28 0 25 75 0
0 0 0 1802956 101032 104464 0 0 0 233 8867 17 0 24 76 0
0 0 0 1802892 101032 104464 0 0 0 0 8865 16 0 21 79 0
0 0 0 1802892 101032 104464 0 0 0 0 8772 8 0 18 82 0 <- iptables -F
0 0 0 1802892 101032 104464 0 0 0 36 8863 18 0 19 81 0
0 0 0 1802892 101032 104464 0 0 0 80 8700 18 0 18 82 0
0 0 0 1802956 101032 104464 0 0 0 0 8779 7 0 17 83 0
0 0 0 1802948 101032 104464 0 0 0 223 8716 278 4 19 76 0
- Karsten
From: Con Kolivas <hidden> Date: 2004-12-06 23:58:28
Karsten Desler writes:
* David S. Miller wrote:
quoted
It's spending nearly half of it's time in iptables.
Try to consolidate your rules if possible. This is the
part of netfilter that really doesn't scale well at all.
Removing the iptables rules helps reducing the load a little, but the
majority of time is still spent somewhere else.
I had a similar scenario recently with a very low spec box and found it to
be the QoS. Disabling traffic shaping and removing the QoS modules made it
much faster. I don't know if you're using them but it's worth pointing out.
Cheers,
Con
I had a similar scenario recently with a very low spec box and found it to
be the QoS. Disabling traffic shaping and removing the QoS modules made it
much faster. I don't know if you're using them but it's worth pointing out.
No QoS loaded.
.config is attached.
Thanks,
Karsten
It's spending nearly half of it's time in iptables.
Try to consolidate your rules if possible. This is the
part of netfilter that really doesn't scale well at all.
Removing the iptables rules helps reducing the load a little, but the
majority of time is still spent somewhere else.
Well with NAPI it can be hard to tell CPU usage.
You may need to use something like cyclesoak to get
a true idea of CPU left.
Also have a look at http://www.hipac.org/ as netfilter
has silly scalability properties.
I also notice that a lot of time is spent allocating
and freeing the packet buffers (and possible hidden
time due to cache misses due to allocating on one
CPU and freeing on another?).
How many [RT]xDescriptors do you have configured by the way?
Anyway attached is a small patch that I used to make the e1000
"own" the packet buffers, and hence it does not alloc/free
per packet at all. Now this has only been tested in one
configuration where I was just sniffing the packets, so
definitely YMMV.
--
Pádraig Brady - http://www.pixelbeat.org
--
It's spending nearly half of it's time in iptables.
Try to consolidate your rules if possible. This is the
part of netfilter that really doesn't scale well at all.
Removing the iptables rules helps reducing the load a little, but the
majority of time is still spent somewhere else.
Well with NAPI it can be hard to tell CPU usage.
You may need to use something like cyclesoak to get
a true idea of CPU left.
Thanks, I'll look into it.
Also have a look at http://www.hipac.org/ as netfilter
has silly scalability properties.
I did before, but I read on Harald Weltes' weblog that 2.4 gives
slightly worse pps results than 2.6, and since the cpu usage is as high
as it is, I didn't want to take any more performance hits.
I'll try to see what performance impact the netfilter rules have during
peak load.
I also notice that a lot of time is spent allocating
and freeing the packet buffers (and possible hidden
time due to cache misses due to allocating on one
CPU and freeing on another?).
How many [RT]xDescriptors do you have configured by the way?
256. I increased them to 1024 shortly after the profiling run, but
didn't notice any change in the cpu usage (will try again with cyclesoak).
Anyway attached is a small patch that I used to make the e1000
"own" the packet buffers, and hence it does not alloc/free
per packet at all. Now this has only been tested in one
configuration where I was just sniffing the packets, so
definitely YMMV.
From: Robert Olsson <hidden> Date: 2004-12-07 12:39:12
Hello!
Well my experience is that it very hard not to say almost impossible to
extrapolate idle cpu into any network system capacity. I guess this is
what you are trying to do?
Rather load and overload the system with traffic having the characteristics
you expect as a bonus you will get some kind proof of robustness and
responsiveness a max load. There are tools for this type of tests.
Pádraig! Very funny... I started hacking 2 hours ago on idea I
had for long time, This to do a light version of skb recycling based
skb->users (the pktgen trick) with very minimal kernel change..
Anyway attached is a small patch that I used to make the e1000
"own" the packet buffers, and hence it does not alloc/free
per packet at all. Now this has only been tested in one
configuration where I was just sniffing the packets, so
definitely YMMV.
Well my experience is that it very hard not to say almost impossible to
extrapolate idle cpu into any network system capacity. I guess this is
what you are trying to do?
Kinda, yes.
I'm trying to evaluate if the behaviour I'm seeing is expected, which
would heavily influence my choice of hardware/software for future
projects (and of course to optimize the current setup).
Currently I'm having problems capturing packets with tcpdump (lots of
"packets dropped by kernel") which indicates to me that there's
genuinely not much (enough) idle time sitting around.
Rather load and overload the system with traffic having the characteristics
you expect as a bonus you will get some kind proof of robustness and
responsiveness a max load. There are tools for this type of tests.
Will do, that could take a couple of days though.
Cheers,
Karsten
On Tue, 2004-12-07 at 07:50, Karsten Desler wrote:
Currently I'm having problems capturing packets with tcpdump (lots of
"packets dropped by kernel") which indicates to me that there's
genuinely not much (enough) idle time sitting around.
Ah, more hints. So you are not trying to forward - rather just packet
capturing?
Are you using a tcpdump patched with mmaped packet socket?
The 230-240Kpps you are reporting as a capture dont seem as unreasonable
as i thought then. Neither would the CPU use.
cheers,
jamal
On Tue, 2004-12-07 at 07:50, Karsten Desler wrote:
quoted
Currently I'm having problems capturing packets with tcpdump (lots of
"packets dropped by kernel") which indicates to me that there's
genuinely not much (enough) idle time sitting around.
Ah, more hints. So you are not trying to forward - rather just packet
capturing?
forward/routing is the goal.
I was just trying to capture a tcpdump to analyze the traffic to
generate something that could emulate the trafficpattern for further
testing in a non-production environment.
Cheers,
Karsten
On Tue, 2004-12-07 at 07:50, Karsten Desler wrote:
quoted
Currently I'm having problems capturing packets with tcpdump (lots of
"packets dropped by kernel") which indicates to me that there's
genuinely not much (enough) idle time sitting around.
Ah, more hints. So you are not trying to forward - rather just packet
capturing?
Are you using a tcpdump patched with mmaped packet socket?
The 230-240Kpps you are reporting as a capture dont seem as unreasonable
as i thought then. Neither would the CPU use.
Yes this is vital Karsten, otherwise tcpdump
will do 2 syscalls per packet, which is the
bottleneck in my experience.
You may want to try a simpler capture program that
uses the kernel PACKET_MMAP feature directly:
http://www.scaramanga.co.uk/code-fu/lincap.c
--
Pádraig Brady - http://www.pixelbeat.org
--
Also have a look at http://www.hipac.org/ as netfilter
has silly scalability properties.
I did before, but I read on Harald Weltes' weblog that 2.4 gives
slightly worse pps results than 2.6, and since the cpu usage is as high
as it is, I didn't want to take any more performance hits.
I'll try to see what performance impact the netfilter rules have during
peak load.
using 2 CPUs
System load: 61.4% || Free: 51.0%(0) 26.3%(1)
System load: 59.6% || Free: 53.6%(0) 27.3%(1)
System load: 59.6% || Free: 53.6%(0) 27.3%(1)
System load: 59.7% || Free: 53.6%(0) 27.0%(1)
System load: 60.3% || Free: 53.0%(0) 26.4%(1)
System load: 51.9% || Free: 60.4%(0) 35.8%(1) <- iptables -F
System load: 50.1% || Free: 62.1%(0) 37.7%(1)
System load: 50.1% || Free: 62.0%(0) 37.8%(1)
System load: 50.6% || Free: 61.6%(0) 37.2%(1)
System load: 50.5% || Free: 61.7%(0) 37.3%(1)
quoted
I also notice that a lot of time is spent allocating
and freeing the packet buffers (and possible hidden
time due to cache misses due to allocating on one
CPU and freeing on another?).
How many [RT]xDescriptors do you have configured by the way?
256. I increased them to 1024 shortly after the profiling run, but
didn't notice any change in the cpu usage (will try again with cyclesoak).
Current packetload on eth0 (and reversed on eth1):
115kpps tx
135kpps rx
I totally forgot to mention: There are approximately 100k concurrent
flows.
From dmesg:
IP: routing cache hash table of 16384 buckets, 128Kbytes
Maybe there is some contention on the rt_hash_table spinlocks?
Is the attached patch enough to increase the size?
- Karsten
From: Robert Olsson <hidden> Date: 2004-12-07 22:40:38
Karsten Desler writes:
> I totally forgot to mention: There are approximately 100k concurrent
> flows.
> >From dmesg:
> IP: routing cache hash table of 16384 buckets, 128Kbytes
You can take a looks at stats w. rtstat. Hash spinning and how many new
entires create and how many warm you hit.
> Maybe there is some contention on the rt_hash_table spinlocks?
> Is the attached patch enough to increase the size?
There is boot option for this now
rhash_entries= [KNL,NET]
Set number of hash buckets for route cache
--ro
On Tue, 2004-12-07 at 16:10, Karsten Desler wrote:
Karsten Desler [off-list ref] wrote:
quoted
Current packetload on eth0 (and reversed on eth1):
115kpps tx
135kpps rx
I totally forgot to mention: There are approximately 100k concurrent
flows.
;-> Aha. That would make a huge difference. I know of noone
who has actually done this level of testing. I have tried upto about 50K
flows myself in early 2.6.x and was eventually compute bound.
Really try compiling out totaly iptables/netfilter - it will make a
difference.
You may also want to try something like LC trie algorithm that Robert
and co are playing with to see if it makes a difference with this many
flows.
cheers,
jamal
On Tue, Dec 07, 2004 at 12:21:39PM +0100, Karsten Desler wrote:
quoted
I also notice that a lot of time is spent allocating
and freeing the packet buffers (and possible hidden
time due to cache misses due to allocating on one
CPU and freeing on another?).
How many [RT]xDescriptors do you have configured by the way?
256. I increased them to 1024 shortly after the profiling run, but
didn't notice any change in the cpu usage (will try again with cyclesoak).
Have you checked the interrupts rate ? I had an e1000 eating many CPU cycles
because it would generate 50000 interrupts/s. Passing the module
InterruptThrottleRate=5000 definitely calmed it down, and more than doubled
the data rate.
Regards
Willy
On Tue, Dec 07, 2004 at 12:21:39PM +0100, Karsten Desler wrote:
quoted
quoted
I also notice that a lot of time is spent allocating
and freeing the packet buffers (and possible hidden
time due to cache misses due to allocating on one
CPU and freeing on another?).
How many [RT]xDescriptors do you have configured by the way?
256. I increased them to 1024 shortly after the profiling run, but
didn't notice any change in the cpu usage (will try again with cyclesoak).
Have you checked the interrupts rate ? I had an e1000 eating many CPU cycles
because it would generate 50000 interrupts/s. Passing the module
InterruptThrottleRate=5000 definitely calmed it down, and more than doubled
the data rate.
I was running mit ITR=3000, but as a test to see if NAPI works, I
disabled ITR on eth0 bringing the int/s rate up to 50k.
Is that normal? I always though NAPI was supposed to kick in way earlier.
Anyways, I'm going to try different ITR settings to see if they make any
difference.
Cheers,
Karsten
On Tue, 2004-12-07 at 16:10, Karsten Desler wrote:
quoted
I totally forgot to mention: There are approximately 100k concurrent
flows.
;-> Aha. That would make a huge difference. I know of noone
who has actually done this level of testing. I have tried upto about 50K
flows myself in early 2.6.x and was eventually compute bound.
Really try compiling out totaly iptables/netfilter - it will make a
difference.
Unfortunately I need netfilter (for now, I haven't had time to look into
replacing the rules with tc yet).
You may also want to try something like LC trie algorithm that Robert
and co are playing with to see if it makes a difference with this many
flows.
On a scale of one to ten, one being "will crash on the first packet",
five being "will allow moderate load, but is probably going to crash
under high load" and ten being "rock stable" how would the patch be rated?
The announcement doesn't look too promising:
| Locking.
| Not yet done.
Also when looking at the profiles, fib_* isn't showing up at (not even
near) the top. Testworthy none the less?
ip route|wc -l:
40
profile:
4 fib_validate_source 0.0064
39 fib_semantic_match 0.1875
[...]
76 ipt_route_hook 1.5833
219 __kmalloc 1.7109
985 e1000_clean_tx_irq 1.8107
157 kmem_cache_alloc 1.9625
405 skb_release_data 2.5312
633 eth_type_trans 2.6375
394 handle_IRQ_event 3.5179
1017 __kfree_skb 3.9727
989 alloc_skb 4.1208
1645 ip_route_input 4.6733
5128 ipt_do_table 6.1635
1678 e1000_intr 6.9917
289 _read_unlock_bh 18.0625
616 _read_lock_bh 19.2500
418 _spin_lock 26.1250
2076 e1000_irq_enable 43.2500
881 _spin_unlock_irqrestore 55.0625
96895 default_idle 1513.9844
Cheers,
Karsten
On Wed, 2004-12-08 at 08:08, Karsten Desler wrote:
I was running mit ITR=3000, but as a test to see if NAPI works, I
disabled ITR on eth0 bringing the int/s rate up to 50k.
Is that normal? I always though NAPI was supposed to kick in way earlier.
Anyways, I'm going to try different ITR settings to see if they make any
difference.
The one time you would need this ITR crap is when you are running low
traffic (relatively speaking: which could mean anything below 100Kpps
depending on h/ware). NAPI will consume a little more CPU
otherwise - given it does an extra IO (if it kicks in and out on every
packet or two). Granted you are doing some new types of tests, so
you may be seeing some things we havent experienced before.
Can you put a printk in ->open() of e1000 with
#if napi is defined
printk("%s: NAPI is on\n",dev->name);
#endif
This should print on ifconfig up and indicate whether NAPI is on or not.
cheers,
jamal
rhash_entries= [KNL,NET]
Set number of hash buckets for route cache
IP: routing cache hash table of 131072 buckets, 1024Kbytes
A 3 percent improvement over the average cpu usage in the last 8 hours
compared to the same time yesterday.
Still in the noise.
I'm going to try a day without netfilter next and then the realloc_skb
patch.
Cheers,
Karsten