From: Jamal Hadi <hidden> Date: 2003-06-10 11:23:28
On Tue, 10 Jun 2003, Simon Kirby wrote:
[some good stuff deleted]
Simon,
I havent looked at your data in details; i will. Someone like Robert
would be able to snuff it much faster than i do. I just wanna say thanks
for the effort, I will spend time catching up with you folks.
It is clear that our next hurudle is gc.
Do you have profiles for your data? Profiles would be nice to collect
as well.
In any case, setting gc_min_interval to 0 definitely helped, but I
suspect Dave's patches will make a bigger difference. Next up is
2.5.70-bk14 and 2.5.70-bk14+davem's stuff from yesterday.
Also since you are doing all that work post the kernels somewhere so
people like foo can grab them and test as well.
cheers,
jamal
From: Ralph Doncaster <hidden> Date: 2003-06-10 13:34:15
On Tue, 10 Jun 2003, Simon Kirby wrote:
Running with vanilla 2.4.21-rc7 (for now), the kernel manages to forward
an amazing 39,000 packets per second. Woohoo!
I hope that's sarcasm. I know if you posted to NANOG saying it took a
dual 1.4Ghz Opteron to route 39kpps under linux you'd be laughed off the
list. Maybe I should be bragging about my 3-minute lap times on the
Shannonville track in my M5!
-Ralph
From: Jamal Hadi <hidden> Date: 2003-06-10 13:39:38
On Tue, 10 Jun 2003, Ralph Doncaster wrote:
I hope that's sarcasm. I know if you posted to NANOG saying it took a
dual 1.4Ghz Opteron to route 39kpps under linux you'd be laughed off the
list. Maybe I should be bragging about my 3-minute lap times on the
Shannonville track in my M5!
Ralph,
Take a look at the sprint core routers i posted earlier. See how much
data they actually route ;->
This are damn expensive routers btw with OC48 interface.
On your other comment on being able to get rid of the route cache when you
dont need it, i am actually indifferent. You may have a point.
cheers,
jamal
I'd be happy to set up a repository ftp site or maybe even some cvs
servers so all of us can test
All these things and share data. We are an ISP so it wouldn't be too
hard to just pop up another server
To store all this :> Let me know
Paul xerox@foonet.net http://www.httpd.net
-----Original Message-----
From: Jamal Hadi [mailto:hadi@shell.cyberus.ca]
Sent: Tuesday, June 10, 2003 7:23 AM
To: Simon Kirby
Cc: ralph+d@istop.com; CIT/Paul; 'David S. Miller'; fw@deneb.enyo.de;
netdev@oss.sgi.com; linux-net@vger.kernel.org
Subject: Re: Route cache performance tests
On Tue, 10 Jun 2003, Simon Kirby wrote:
[some good stuff deleted]
Simon,
I havent looked at your data in details; i will. Someone like Robert
would be able to snuff it much faster than i do. I just wanna say thanks
for the effort, I will spend time catching up with you folks. It is
clear that our next hurudle is gc. Do you have profiles for your data?
Profiles would be nice to collect as well.
In any case, setting gc_min_interval to 0 definitely helped, but I
suspect Dave's patches will make a bigger difference. Next up is
2.5.70-bk14 and 2.5.70-bk14+davem's stuff from yesterday.
Also since you are doing all that work post the kernels somewhere so
people like foo can grab them and test as well.
cheers,
jamal
From: David S. Miller <hidden> Date: 2003-06-13 06:20:02
From: Simon Kirby [off-list ref]
Date: Tue, 10 Jun 2003 00:57:32 -0700
In any case, setting gc_min_interval to 0 definitely helped, but I
suspect Dave's patches will make a bigger difference. Next up is
2.5.70-bk14 and 2.5.70-bk14+davem's stuff from yesterday.
Did you get stuck in some mud? :-) It's been two days.
I even posted new patches for you to test, get on it :)))
From: Simon Kirby <hidden> Date: 2003-06-16 22:37:14
On Thu, Jun 12, 2003 at 11:20:02PM -0700, David S. Miller wrote:
In any case, setting gc_min_interval to 0 definitely helped, but I
suspect Dave's patches will make a bigger difference. Next up is
2.5.70-bk14 and 2.5.70-bk14+davem's stuff from yesterday.
Did you get stuck in some mud? :-) It's been two days.
I even posted new patches for you to test, get on it :)))
From: David S. Miller <hidden> Date: 2003-06-16 22:44:01
From: Simon Kirby [off-list ref]
Date: Mon, 16 Jun 2003 15:37:14 -0700
So, which kernels shall I try? When I set the thing up I was using
2.5.70-bk14, but I am compiling 2.5.71, and I will try with your patch
above and with Alexey's.
Thanks for your profiles.
I pushed all of our current work to Linus's tree.
But for your convenience here are the routing diffs
against plain 2.5.71
# This is a BitKeeper generated patch for the following project:
# Project Name: Linux kernel tree
# This patch format is intended for GNU patch command version 2.5 or higher.
# This patch includes the following deltas:
# ChangeSet 1.1318.1.15 -> 1.1318.1.16
# net/ipv4/route.c 1.63 -> 1.64
#
# The following is the BitKeeper ChangeSet Log
# --------------------------------------------
# 03/06/16 kuznet@ms2.inr.ac.ru 1.1318.1.16
# [IPV4]: More sane rtcache behavior.
# 1) More reasonable ip_rt_gc_min_interval default
# 2) Trim less valuable entries in hash chain during
# rt_intern_hash when such chains grow too long.
# --------------------------------------------
#
diff -Nru a/net/ipv4/route.c b/net/ipv4/route.c
--- a/net/ipv4/route.c Mon Jun 16 15:45:20 2003+++ b/net/ipv4/route.c Mon Jun 16 15:45:20 2003
@@ -456,6 +456,25 @@out:returnret;}+/* Bits of score are:+*31:veryvaluable+*30:notquiteuseless+*29..0:usagecounter+*/+staticinlineu32rt_score(structrtable*rt)+{+u32score=rt->u.dst.__use;++if(rt_valuable(rt))+score|=(1<<31);++if(!rt->fl.iif||+!(rt->rt_flags&(RTCF_BROADCAST|RTCF_MULTICAST|RTCF_LOCAL)))+score|=(1<<30);++returnscore;+}+/* This runs via a timer and thus is always in BH context. */staticvoidrt_check_expire(unsignedlongdummy){
@@ -755,7 +777,33 @@return0;}+if(!atomic_read(&rth->u.dst.__refcnt)){+u32score=rt_score(rth);++if(score<=min_score){+cand=rth;+candp=rthp;+min_score=score;+}+}++chain_length++;+rthp=&rth->u.rt_next;+}++if(cand){+/* ip_rt_gc_elasticity used to be average length of chain+*length,whenexceededgcbecomesreallyaggressive.+*+*Thesecondlimitislesscertain.Atthemomentitallows+*only2entriesperbucket.Wewillsee.+*/+if(chain_length>ip_rt_gc_elasticity||+(chain_length>1&&!(min_score&(1<<31)))){+*candp=cand->u.rt_next;+rt_free(cand);+}}/* Try to bind route to arp only if it is output
# This is a BitKeeper generated patch for the following project:
# Project Name: Linux kernel tree
# This patch format is intended for GNU patch command version 2.5 or higher.
# This patch includes the following deltas:
# ChangeSet 1.1320.1.1 -> 1.1320.1.2
# net/ipv4/route.c 1.64 -> 1.65
#
# The following is the BitKeeper ChangeSet Log
# --------------------------------------------
# 03/06/16 robert.olsson@data.slu.se 1.1320.1.2
# [IPV4]: In rt_intern_hash, reinit all state vars on branch to "restart".
# --------------------------------------------
#
diff -Nru a/net/ipv4/route.c b/net/ipv4/route.c
--- a/net/ipv4/route.c Mon Jun 16 15:46:05 2003+++ b/net/ipv4/route.c Mon Jun 16 15:46:05 2003
From: David S. Miller <hidden> Date: 2003-06-16 23:08:56
From: Simon Kirby [off-list ref]
Date: Mon, 16 Jun 2003 16:09:22 -0700
On Mon, Jun 16, 2003 at 03:44:01PM -0700, David S. Miller wrote:
> I pushed all of our current work to Linus's tree.
> But for your convenience here are the routing diffs
> against plain 2.5.71
Trying to apply against 2.5.71:
patching file net/ipv4/route.c
Hunk #2 succeeded at 454 (offset -2 lines).
Hunk #3 succeeded at 738 (offset -2 lines).
Hunk #4 succeeded at 775 (offset -2 lines).
patching file net/ipv4/route.c
Hunk #1 FAILED at 739.
1 out of 1 hunk FAILED -- saving rejects to file net/ipv4/route.c.rej
Trying to apply against 2.5.71-bk2:
patching file net/ipv4/route.c
patching file net/ipv4/route.c
Hunk #1 FAILED at 739.
1 out of 1 hunk FAILED -- saving rejects to file net/ipv4/route.c.rej
Missing something between?
Code from bk2:
static int rt_intern_hash(unsigned hash, struct rtable *rt, struct rtable **rp)
{
struct rtable *rth, **rthp;
unsigned long now = jiffies;
int attempts = !in_softirq();
Patch:
It depends upon the first patch that I enclosed.
What I gave you was a 2-part patch, the first one
did:
@@ -721,6 +740,9 @@ { struct rtable *rth, **rthp; unsigned long now = jiffies;+ struct rtable *cand = NULL, **candp = NULL;+ u32 min_score = ~(u32)0;+ int chain_length = 0; int attempts = !in_softirq(); restart:
The second one did:
@@ -739,13 +739,19 @@ static int rt_intern_hash(unsigned hash, struct rtable *rt, struct rtable **rp) { struct rtable *rth, **rthp;- unsigned long now = jiffies;- struct rtable *cand = NULL, **candp = NULL;- u32 min_score = ~(u32)0;- int chain_length = 0;+ unsigned long now;+ struct rtable *cand, **candp;+ u32 min_score;+ int chain_length; int attempts = !in_softirq();
...
I have no idea why it doesn't apply.
Nothing else has happened in these bits of code for a while.
From: Simon Kirby <hidden> Date: 2003-06-16 23:27:50
On Mon, Jun 16, 2003 at 04:08:56PM -0700, David S. Miller wrote:
It depends upon the first patch that I enclosed.
Never mind. :) Such patches don't work very well with patch --dry.
Simon-
[ Simon Kirby ][ Network Operations ]
[ sim@netnation.com ][ NetNation Communications Inc. ]
[ Opinions expressed are not necessarily those of my employer. ]
From: David S. Miller <hidden> Date: 2003-06-17 15:59:21
From: Simon Kirby [off-list ref]
Date: Mon, 16 Jun 2003 16:49:37 -0700
60.0073 seconds passed, avg forwarding rate: 157557.710 pps
...Looks like a tad worse than with your patch, but not by much.
Forwarding rate is still pretty crappy for an Opteron. Will fiddle
a bit more tonight to see what I can do.
To be honest, this isn't half-bad for pure DoS load.
This reminds me, maybe a good test would be PPS for "well behaved
flows" in the presence of DoS load. You'd probably need 4 systems
to carry out such a test accurately.
Because, really, who cares how fast we can forward the DoS traffic
as long as legitimate users still see good metrics.
From: Robert Olsson <hidden> Date: 2003-06-17 16:50:03
David S. Miller writes:
> 60.0073 seconds passed, avg forwarding rate: 157557.710 pps
> To be honest, this isn't half-bad for pure DoS load.
No thats pretty good and profiles looks as expected. It would interesting
to get the singeflow performance as a comparison.
Also think Simon used only /32 routes... I took "real" Internet-routing
and made a script so it can be used for experiments. I can make it available.
Cheers.
--ro
From: David S. Miller <hidden> Date: 2003-06-17 16:50:28
From: Robert Olsson [off-list ref]
Date: Tue, 17 Jun 2003 18:50:03 +0200
Also think Simon used only /32 routes... I took "real"
Internet-routing and made a script so it can be used for
experiments. I can make it available.
Please do, I'd like to play with such a list locally.
From: Robert Olsson <hidden> Date: 2003-06-17 17:29:25
David S. Miller writes:
> Internet-routing and made a script so it can be used for
> experiments. I can make it available.
>
> Please do, I'd like to play with such a list locally.
ftp://robur.slu.se/pub/Linux/net-development/inet_routes/
Just configure the script and run...
And Simon can you do a run with this routing table too? And even fibstat
output could be interesting to compare.
Cheers.
--ro
From: Mr. James W. Laferriere <hidden> Date: 2003-06-17 19:06:43
Hello Robert , First thank you for these tools . Now for the
questions .
Is 'fibstat' only for 2.5.x kernels ?
The reason for that ? is there isn't a /proc/net/fib_stat .
Under 2.4.21 , which has no mention of fib_stat anywhere in the
sources .
The next ? is packet-generator.c (appears) to require kernel
include files . Is there an updated user level net/ includes ?
Tia , JimL
On Tue, 17 Jun 2003, Robert Olsson wrote:
David S. Miller writes:
> Internet-routing and made a script so it can be used for
> experiments. I can make it available.
> Please do, I'd like to play with such a list locally.
ftp://robur.slu.se/pub/Linux/net-development/inet_routes/
Just configure the script and run...
And Simon can you do a run with this routing table too? And even fibstat
output could be interesting to compare.
--
+------------------------------------------------------------------+
| James W. Laferriere | System Techniques | Give me VMS |
| Network Engineer | P.O. Box 854 | Give me Linux |
| babydr@baby-dragons.com | Coudersport PA 16915 | only on AXP |
+------------------------------------------------------------------+
From: Simon Kirby <hidden> Date: 2003-06-17 20:07:21
On Tue, Jun 17, 2003 at 06:50:03PM +0200, Robert Olsson wrote:
David S. Miller writes:
> 60.0073 seconds passed, avg forwarding rate: 157557.710 pps
> To be honest, this isn't half-bad for pure DoS load.
No thats pretty good and profiles looks as expected. It would interesting
to get the singeflow performance as a comparison.
I changed Juno to send from a single IP, but it only spat out about
330000 pps, which the dual Tigon3 Opteron box forwarded completely.
In order to do a single flow forwarding test, I need to be able to create
more input traffic somehow. Seeing as you wrote pktgen.c, maybe you
could help in this department. :)
Also think Simon used only /32 routes... I took "real" Internet-routing
and made a script so it can be used for experiments. I can make it available.
Yes, I found that area less interesting since Dave M. fixed the hash
buckets. But yes, the prefix scanning will slow it down some.
Whoa. Uhm. A lot. I should compare with 2.4 again to see what's going
on here.
60.0042 seconds passed, avg forwarding rate: 50759.683 pps
60.0039 seconds passed, avg forwarding rate: 50311.258 pps
60.0046 seconds passed, avg forwarding rate: 50420.562 pps
60.0036 seconds passed, avg forwarding rate: 50399.389 pps
60.0038 seconds passed, avg forwarding rate: 50431.732 pps
60.0041 seconds passed, avg forwarding rate: 50403.777 pps
60.0036 seconds passed, avg forwarding rate: 50210.604 pps
60.0033 seconds passed, avg forwarding rate: 50279.220 pps
60.0036 seconds passed, avg forwarding rate: 50549.291 pps
60.0046 seconds passed, avg forwarding rate: 50437.615 pps
Cpu type: Athlon
Cpu speed was (MHz estimation) : 1394.26
Counter 0 counted RETIRED_INSNS events (Retired instructions (includes exceptions, interrupts, resyncs)) with a unit mask of 0x00 (No unit mask) count 697000
vma samples % symbol name
c02bf730 16019 33.2014 fn_hash_lookup
c0292b70 3882 8.04593 ip_route_input_slow
c0221710 2335 4.83958 tg3_rx
c02bd550 2004 4.15354 fib_validate_source
c0290d70 1955 4.05198 rt_hash_code
c0294e50 1670 3.46128 ip_rcv
c02933a0 1404 2.90997 ip_route_input
c01351b0 1349 2.79597 __kmalloc
c02885c0 1314 2.72343 netif_receive_skb
c02b8040 1168 2.42083 inet_select_addr
c0135270 1123 2.32756 kfree
c0284620 987 2.04568 alloc_skb
c028ec90 900 1.86536 eth_type_trans
c0135170 860 1.78246 kmem_cache_alloc
c02be8e0 844 1.7493 fib_semantic_match
c0135230 812 1.68297 kmem_cache_free
c0222330 652 1.35135 tg3_start_xmit
c02916b0 648 1.34306 rt_intern_hash
c02215c0 542 1.12336 tg3_recycle_rx
c010fc40 459 0.951335 do_gettimeofday
c028f220 422 0.874648 pfifo_fast_dequeue
c02be9b0 419 0.86843 __fib_res_prefsrc
c028eb20 417 0.864285 eth_header
c0295df0 386 0.800033 ip_forward
c028c520 363 0.752363 neigh_resolve_output
c0284860 345 0.715056 __kfree_skb
c0284840 311 0.644586 kfree_skbmem
c02847a0 295 0.611424 skb_release_data
c028b530 285 0.590698 neigh_lookup
c01adc80 276 0.572044 memcpy
c0134ff0 269 0.557536 free_block
c0291460 240 0.49743 rt_garbage_collect
c0134e20 236 0.489139 cache_alloc_refill
c02972d0 216 0.447687 ip_finish_output
c0114030 215 0.445614 get_offset_tsc
c0128a00 197 0.408307 call_rcu
c028ae20 193 0.400017 dst_alloc
c0288080 187 0.387581 dev_queue_xmit
c028af50 184 0.381363 dst_destroy
c028f1a0 175 0.362709 pfifo_fast_enqueue
c011f080 170 0.352346 local_bh_enable
c0221350 168 0.348201 tg3_tx
c0221e90 160 0.33162 tg3_set_txd
c0297570 152 0.315039 ip_output
c028c3a0 149 0.308821 neigh_hh_init
c028eeb0 141 0.29224 qdisc_restart
size IN: hit tot mc no_rt bcast madst masrc OUT: hit tot mc GC: tot ignored goal_miss ovrf
17929 13 214343 0 0 0 0 163822 0 0 0 50521 50519 0 0
18296 18 213694 0 0 0 0 163018 0 0 0 50676 50674 1 0
17616 11 214529 0 0 0 0 163993 0 0 0 50536 50534 0 0
17841 12 213816 0 0 0 0 163157 0 0 0 50659 50657 1 0
18272 7 214093 0 0 0 0 163583 0 0 0 50510 50508 1 0
18216 9 214843 0 0 0 0 164214 0 0 0 50629 50627 0 0
18318 16 214976 0 0 0 0 164299 0 0 0 50677 50675 0 0
18099 9 213447 0 0 0 0 162995 0 0 0 50452 50450 1 0
17610 14 216438 0 0 0 0 165408 0 0 0 51030 51028 1 0
17643 14 214638 0 0 0 0 163987 0 0 0 50651 50649 0 0
17516 7 213185 0 0 0 0 163016 0 0 0 50169 50167 1 0
18355 10 213894 0 0 0 0 163564 0 0 0 50330 50328 1 0
17723 11 214477 0 0 0 0 163705 0 0 0 50772 50770 0 0
17915 6 214342 0 0 0 0 163625 0 0 0 50717 50715 0 0
18166 19 213965 0 0 0 0 163521 0 0 0 50444 50442 0 0
17943 19 213417 0 0 0 0 162955 0 0 0 50462 50460 2 0
17515 5 214423 0 0 0 0 163718 0 0 0 50705 50703 0 0
18231 10 213434 0 0 0 0 162919 0 0 0 50515 50513 1 0
17523 8 213856 0 0 0 0 163385 0 0 0 50471 50469 0 0
18217 16 214940 0 0 0 0 164165 0 0 0 50775 50773 0 0
...recompiling with fibstats...
Erm. I can't get fib_stats2.pat to apply against 2.5.71, 2.5.71+davem's
join-two-diffs patch, 2.4.21-rc7, or 2.5.71+davem's rtcache changes.
What's it supposed to be against?
[sroot@debinst:/d/linux-2.5]# patch -p0 --dry < ../fib_stats2.pat
patching file include/net/ip_fib.h
Hunk #1 succeeded at 139 (offset 4 lines).
patching file net/ipv4/fib_hash.c
Hunk #3 succeeded at 305 (offset -11 lines).
Hunk #4 succeeded at 1110 with fuzz 1 (offset -14 lines).
Hunk #5 succeeded at 1166 (offset -14 lines).
patching file net/ipv4/route.c
Hunk #1 FAILED at 2754.
Hunk #2 succeeded at 2760 (offset -6 lines).
Hunk #3 succeeded at 2783 (offset -6 lines).
Hunk #4 FAILED at 2793.
2 out of 4 hunks FAILED -- saving rejects to file net/ipv4/route.c.rej
In any event, here is the profile of the single flow case with the full
routing table (probably identical to the empty routing table case). The
sender is pushing enough for NAPI to kick in, so there is a lot of tg3
overhead that would be with more traffic:
60.0041 seconds passed, avg forwarding rate: 329808.310 pps
Cpu type: Athlon
Cpu speed was (MHz estimation) : 1394.26
Counter 0 counted RETIRED_INSNS events (Retired instructions (includes
exceptions, interrupts, resyncs)) with a unit mask of 0x00 (No unit mask)
count 697000
vma samples % symbol name
c0222330 4470 8.51445 tg3_start_xmit
c0221710 3760 7.16204 tg3_rx
c0294e50 3142 5.98488 ip_rcv
c02885c0 2428 4.62485 netif_receive_skb
c0295df0 2065 3.93341 ip_forward
c028f220 2058 3.92007 pfifo_fast_dequeue
c02933a0 2033 3.87245 ip_route_input
c01351b0 1987 3.78483 __kmalloc
c0290d70 1904 3.62674 rt_hash_code
c02972d0 1752 3.33721 ip_finish_output
c01adc80 1649 3.14101 memcpy
c0134ff0 1626 3.0972 free_block
c0284620 1511 2.87815 alloc_skb
c0135270 1489 2.83624 kfree
c0288080 1461 2.78291 dev_queue_xmit
c028ec90 1351 2.57338 eth_type_trans
c0135170 1319 2.51243 kmem_cache_alloc
c028f1a0 1243 2.36766 pfifo_fast_enqueue
c0134e20 1172 2.23242 cache_alloc_refill
c0297570 1145 2.18099 ip_output
c0135230 1133 2.15814 kmem_cache_free
c0221350 1085 2.06671 tg3_tx
c0221a50 991 1.88766 tg3_poll
c02215c0 893 1.70098 tg3_recycle_rx
c0221e90 832 1.58479 tg3_set_txd
c0221b60 812 1.5467 tg3_interrupt
c028eeb0 755 1.43812 qdisc_restart
c010fc40 672 1.28002 do_gettimeofday
c011f080 578 1.10097 local_bh_enable
c0284840 492 0.937161 kfree_skbmem
c010a8b2 426 0.811444 restore_all
c02847a0 375 0.714299 skb_release_data
c010c6a0 327 0.622869 handle_IRQ_event
c010c910 288 0.548582 do_IRQ
c0284860 284 0.540963 __kfree_skb
c0114030 283 0.539058 get_offset_tsc
c021f4b0 270 0.514296 tg3_enable_ints
c0115e10 234 0.445723 end_level_ioapic_irq
c011f4a0 220 0.419056 cpu_raise_softirq
c028eb20 199 0.379055 eth_header
c0288500 188 0.358102 net_tx_action
c028c520 187 0.356197 neigh_resolve_output
c0134b40 179 0.340959 cache_init_objs
c0288900 171 0.32572 net_rx_action
c0132290 165 0.314292 buffered_rmqueue
c01321b0 122 0.232385 free_hot_cold_page
If I start two threads on the sender (Xeon w/HT), I'm able to push 420000
pps, which only partially starts to use NAPI on the Opteron box. Going
to try 2.4 again for a comparison (note: 2.5 seems to have an opposite
PCI scan order from 2.4 for the dual Tigon3s).
Simon-
[ Simon Kirby ][ Network Operations ]
[ sim@netnation.com ][ NetNation Communications Inc. ]
[ Opinions expressed are not necessarily those of my employer. ]
From: Robert Olsson <hidden> Date: 2003-06-17 20:12:14
Mr. James W. Laferriere writes:
>
> Is 'fibstat' only for 2.5.x kernels ?
Yes.
> The reason for that ? is there isn't a /proc/net/fib_stat .
> Under 2.4.21 , which has no mention of fib_stat anywhere in the
> sources .
The kernel part creates /proc/net/fib_stat. I should pretty straight
for 2.4.X too. If people find it useful it can be backported. Try. Look
at route.c and rt_cache_stat if you run into problems.
> The next ? is packet-generator.c (appears) to require kernel
> include files . Is there an updated user level net/ includes ?
It's a kernel module and should be compiled with the kernel.
Cheers.
--ro
From: Martin Josefsson <hidden> Date: 2003-06-17 20:17:14
On Tue, 2003-06-17 at 22:07, Simon Kirby wrote:
quoted
Also think Simon used only /32 routes... I took "real" Internet-routing
and made a script so it can be used for experiments. I can make it available.
Yes, I found that area less interesting since Dave M. fixed the hash
buckets. But yes, the prefix scanning will slow it down some.
Whoa. Uhm. A lot. I should compare with 2.4 again to see what's going
on here.
size IN: hit tot mc no_rt bcast madst masrc OUT: hit tot mc GC: tot ignored goal_miss ovrf
17929 13 214343 0 0 0 0 163822 0 0 0 50521 50519 0 0
Did you have rp_filter enabled? Looks like it.
--
/Martin
From: David S. Miller <hidden> Date: 2003-06-17 20:36:35
From: Simon Kirby [off-list ref]
Date: Tue, 17 Jun 2003 13:37:03 -0700
Forwarding rate more than doubles when I turn off
rp_filter off (Debian turns it on by default).
I have no idea why they do this, it's the stupidest thing
you can possibly do by default.
If we thought it was a good idea to turn this on by default
we would have done so in the kernel.
Does anyone have some cycles to spare to try and urge whoever is
repsponsible for this in Debian to leave the kernel's default setting
alone?
Thanks.
From: David S. Miller <hidden> Date: 2003-06-17 20:49:24
From: Simon Kirby [off-list ref]
Date: Tue, 17 Jun 2003 13:51:01 -0700
Specific firewall rules would have to be created otherwise. And
the overhead only really shows when the routing table is large,
right?
rp filter breaks things... just like firewalls break things...
so just like a user enables firewall rules by himself, he may
enable rp filter by himself...
From: Robert Olsson <hidden> Date: 2003-06-17 20:49:50
Simon Kirby writes:
> I changed Juno to send from a single IP, but it only spat out about
> 330000 pps, which the dual Tigon3 Opteron box forwarded completely.
> In order to do a single flow forwarding test, I need to be able to create
> more input traffic somehow. Seeing as you wrote pktgen.c, maybe you
> could help in this department. :)
OK. See below.
> > Also think Simon used only /32 routes... I took "real" Internet-routing
> > and made a script so it can be used for experiments. I can make it available.
>
> Yes, I found that area less interesting since Dave M. fixed the hash
> buckets. But yes, the prefix scanning will slow it down some.
Well I don't think it's so easy as there are 33 zones with prefixes if
you have all the routes in one zone I'm not sure what happens thats why
I suggested the comparison.
> Erm. I can't get fib_stats2.pat to apply against 2.5.71, 2.5.71+davem's
> join-two-diffs patch, 2.4.21-rc7, or 2.5.71+davem's rtcache changes.
> What's it supposed to be against?
Sorry. Our production system and lab uses very patched 2.5.66
I'll make a patch for 2.5.71....
> If I start two threads on the sender (Xeon w/HT), I'm able to push 420000
> pps, which only partially starts to use NAPI on the Opteron box. Going
> to try 2.4 again for a comparison (note: 2.5 seems to have an opposite
> PCI scan order from 2.4 for the dual Tigon3s).
Not bad. Replace net/core/pktgen.c in 2.5.X with the version from
ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/
and edit pktgen.sh to suit your needs.
And see what you got. I'm interested since you are using both different
processors as NIC's. Also packet generation itself is interesting as
it tests driver/HW xmit-path.
Cheers.
--ro
From: Simon Kirby <hidden> Date: 2003-06-17 20:51:01
On Tue, Jun 17, 2003 at 01:36:35PM -0700, David S. Miller wrote:
I have no idea why they do this, it's the stupidest thing
you can possibly do by default.
If we thought it was a good idea to turn this on by default
we would have done so in the kernel.
Does anyone have some cycles to spare to try and urge whoever is
repsponsible for this in Debian to leave the kernel's default setting
alone?
Sure, I can do this. But why is this stupid? It uses more CPU, but
stops IP spoofing by default. Specific firewall rules would have to be
created otherwise. And the overhead only really shows when the routing
table is large, right?
Simon-
From: Simon Kirby <hidden> Date: 2003-06-17 21:07:01
On Tue, Jun 17, 2003 at 01:07:21PM -0700, Simon Kirby wrote:
Whoa. Uhm. A lot. I should compare with 2.4 again to see what's going
on here.
60.0042 seconds passed, avg forwarding rate: 50759.683 pps
Ummmm, yeah, 2.5.71 is quite a bit slower than 2.4.21. I applied
Alexey's 2.5.71 rtcache fixes to 2.4.21 (changing "fl" to "key" in
the scoring function), and now I see:
60.0065 seconds passed, avg forwarding rate: 135379.152 pps
If reboot and don't fill the routing table:
60.0104 seconds passed, avg forwarding rate: 259027.200 pps
This is with standard juno (pseudo-random sources).
This is with CONFIG_IP_MULTIPLE_TABLES still on, too. I'll turn that off
and do some profiles. The only weird thing I'm seeing while doing this
is that the route cache table continues to grow slowly, and the pps
slowly falls off over a few minutes. "ip route flush cache" restores
performance again. I'll verify this is not happening in 2.5.
Simon-
[ Simon Kirby ][ Network Operations ]
[ sim@netnation.com ][ NetNation Communications Inc. ]
[ Opinions expressed are not necessarily those of my employer. ]
From: David S. Miller <hidden> Date: 2003-06-17 22:08:49
From: Ralph Doncaster [off-list ref]
Date: Tue, 17 Jun 2003 18:11:00 -0400 (EDT)
My (obviously incorrect) assumption would
be that fib_validate_source is responsible for rp_filter, and turning it
off would lead to only a 5% performance increase.
fib_validate_source() with rp_filter enabled causes an extra
fib_lookup() to occur for each packet.
If turning off rp_filter doubles your performance, then the profile
numbers above are misleading. My (obviously incorrect) assumption would
be that fib_validate_source is responsible for rp_filter, and turning it
off would lead to only a 5% performance increase.
Considering that, what kind of performance difference should removing the
route hashing make (i.e. going with r-trees or something like that). In
most of the profiles fn_hash_lookup has been at the top of the list.
-Ralph
From: Simon Kirby <hidden> Date: 2003-06-17 22:50:36
On Tue, Jun 17, 2003 at 02:07:01PM -0700, Simon Kirby wrote:
60.0104 seconds passed, avg forwarding rate: 259027.200 pps
This is with standard juno (pseudo-random sources).
This is with CONFIG_IP_MULTIPLE_TABLES still on, too.
From: David S. Miller <hidden> Date: 2003-06-17 23:07:21
From: Simon Kirby [off-list ref]
Date: Tue, 17 Jun 2003 15:50:36 -0700
so the problems in 2.4 are probably the result of me hacking in the
2.5 patch.
I have them in my pending 2.4.x tree, try this:
diff -Nru a/include/net/route.h b/include/net/route.h
--- a/include/net/route.h Tue Jun 17 16:08:06 2003+++ b/include/net/route.h Tue Jun 17 16:08:06 2003
--- a/net/ipv4/fib_hash.c Tue Jun 17 16:08:07 2003+++ b/net/ipv4/fib_hash.c Tue Jun 17 16:08:07 2003
@@ -89,7 +89,7 @@intfz_nent;/* Number of entries */intfz_divisor;/* Hash divisor */-u32fz_hashmask;/* (1<<fz_divisor) - 1 */+u32fz_hashmask;/* (fz_divisor - 1) */#define FZ_HASHMASK(fz) ((fz)->fz_hashmask)intfz_order;/* Zone order */
@@ -149,9 +149,19 @@staticrwlock_tfib_hash_lock=RW_LOCK_UNLOCKED;-#define FZ_MAX_DIVISOR 1024+#define FZ_MAX_DIVISOR ((PAGE_SIZE<<MAX_ORDER) / sizeof(struct fib_node *))-#ifdef CONFIG_IP_ROUTE_LARGE_TABLES+staticstructfib_node**fz_hash_alloc(intdivisor)+{+unsignedlongsize=divisor*sizeof(structfib_node*);++if(divisor<=1024){+returnkmalloc(size,GFP_KERNEL);+}else{+return(structfib_node**)+__get_free_pages(GFP_KERNEL,get_order(size));+}+}/* The fib hash lock must be held when this is called. */static__inline__voidfn_rebuild_zone(structfn_zone*fz,
@@ -185,24 +204,30 @@switch(old_divisor){case16:new_divisor=256;-new_hashmask=0xFF;break;case256:new_divisor=1024;-new_hashmask=0x3FF;break;default:-printk(KERN_CRIT"route.c: bad divisor %d!\n",old_divisor);-return;+if((old_divisor<<1)>FZ_MAX_DIVISOR){+printk(KERN_CRIT"route.c: bad divisor %d!\n",old_divisor);+return;+}+new_divisor=(old_divisor<<1);+break;}++new_hashmask=(new_divisor-1);+#if RT_CACHE_DEBUG >= 2printk("fn_rehash_zone: hash for zone %d grows from %d\n",fz->fz_order,old_divisor);#endif-ht=kmalloc(new_divisor*sizeof(structfib_node*),GFP_KERNEL);+ht=fz_hash_alloc(new_divisor);if(ht){memset(ht,0,new_divisor*sizeof(structfib_node*));+write_lock_bh(&fib_hash_lock);old_ht=fz->fz_hash;fz->fz_hash=ht;
@@ -365,6 +368,25 @@out:returnret;}+/* Bits of score are:+*31:veryvaluable+*30:notquiteuseless+*29..0:usagecounter+*/+staticinlineu32rt_score(structrtable*rt)+{+u32score=rt->u.dst.__use;++if(rt_valuable(rt))+score|=(1<<31);++if(!rt->key.iif||+!(rt->rt_flags&(RTCF_BROADCAST|RTCF_MULTICAST|RTCF_LOCAL)))+score|=(1<<30);++returnscore;+}+/* This runs via a timer and thus is always in BH context. */staticvoidSMP_TIMER_NAME(rt_check_expire)(unsignedlongdummy){
@@ -384,7 +406,7 @@while((rth=*rthp)!=NULL){if(rth->u.dst.expires){/* Entry is expired even if it is in use */-if((long)(now-rth->u.dst.expires)<=0){+if(time_before_eq(now,rth->u.dst.expires)){tmo>>=1;rthp=&rth->u.rt_next;continue;
@@ -650,9 +681,35 @@return0;}+if(!atomic_read(&rth->u.dst.__refcnt)){+u32score=rt_score(rth);++if(score<=min_score){+cand=rth;+candp=rthp;+min_score=score;+}+}++chain_length++;+rthp=&rth->u.rt_next;}+if(cand){+/* ip_rt_gc_elasticity used to be average length of chain+*length,whenexceededgcbecomesreallyaggressive.+*+*Thesecondlimitislesscertain.Atthemomentitallows+*only2entriesperbucket.Wewillsee.+*/+if(chain_length>ip_rt_gc_elasticity||+(chain_length>1&&!(min_score&(1<<31)))){+*candp=cand->u.rt_next;+rt_free(cand);+}+}+/* Try to bind route to arp only if it is outputrouteorunicastforwardingpath.*/
@@ -960,7 +1017,7 @@/* No redirected packets during ip_rt_redirect_silence;*resetthealgorithm.*/-if(jiffies-rt->u.dst.rate_last>ip_rt_redirect_silence)+if(time_after(jiffies,rt->u.dst.rate_last+ip_rt_redirect_silence))rt->u.dst.rate_tokens=0;/* Too many ignored redirects; do not send anything
@@ -974,8 +1031,9 @@/* Check for load limit; set rate_last to the latest sent*redirect.*/-if(jiffies-rt->u.dst.rate_last>-(ip_rt_redirect_load<<rt->u.dst.rate_tokens)){+if(time_after(jiffies,+(rt->u.dst.rate_last++(ip_rt_redirect_load<<rt->u.dst.rate_tokens)))){icmp_send(skb,ICMP_REDIRECT,ICMP_REDIR_HOST,rt->rt_gateway);rt->u.dst.rate_last=jiffies;++rt->u.dst.rate_tokens;
From: Pekka Savola <hidden> Date: 2003-06-18 05:50:53
On Tue, 17 Jun 2003, Simon Kirby wrote:
On Tue, Jun 17, 2003 at 01:36:35PM -0700, David S. Miller wrote:
quoted
I have no idea why they do this, it's the stupidest thing
you can possibly do by default.
If we thought it was a good idea to turn this on by default
we would have done so in the kernel.
Does anyone have some cycles to spare to try and urge whoever is
repsponsible for this in Debian to leave the kernel's default setting
alone?
Sure, I can do this. But why is this stupid? It uses more CPU, but
stops IP spoofing by default. Specific firewall rules would have to be
created otherwise. And the overhead only really shows when the routing
table is large, right?
Personally I think rp_filter by default is the only good choice
(security/operational-wise). It's typically not useful when you have a
lot of routes, though.. but as the 99.9% of users _don't_, it still seems
like a good default value.
--
Pekka Savola "You each name yourselves king, yet the
Netcore Oy kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings