From: Ben Greear <hidden> Date: 2005-08-25 04:52:05
Danial Thom wrote:
I think the concensus is that 2.6 has made trade
offs that lower raw throughput, which is what a
networking device needs. So as a router or
network appliance, 2.6 seems less suitable. A raw
bridging test on a 2.0Ghz operton system:
FreeBSD 4.9: Drops no packets at 900K pps
Linux 2.4.24: Starts dropping packets at 350K pps
Linux 2.6.12: Starts dropping packets at 100K pps
I ran some quick tests using kernel 2.6.11, 1ms tick (HZ=1000), SMP kernel.
Hardware is P-IV 3.0Ghz + HT on a new SuperMicro motherboard with 64/133Mhz
PCI-X bus. NIC is dual Intel pro/1000. Kernel is close to stock 2.6.11.
I used brctl to create a bridge with the two GigE adapters in it and
used pktgen to stream traffic through it (250kpps in one direction, 1kpps in
the other.)
I see a reasonable amount of drops at 250kpps (60 byte packets):
about 60,000,000 packets received, 20,700 dropped.
Interestingly, the system is about 60% idle according to top,
and still dropping pkts, so it would seem that the system could
be better utilized!
Ben
--
Ben Greear [off-list ref]
Candela Technologies Inc http://www.candelatech.com
networking device needs. So as a router or
network appliance, 2.6 seems less suitable. A
raw
quoted
bridging test on a 2.0Ghz operton system:
FreeBSD 4.9: Drops no packets at 900K pps
Linux 2.4.24: Starts dropping packets at 350K
pps
quoted
Linux 2.6.12: Starts dropping packets at 100K
pps
I ran some quick tests using kernel 2.6.11, 1ms
tick (HZ=1000), SMP kernel.
Hardware is P-IV 3.0Ghz + HT on a new
SuperMicro motherboard with 64/133Mhz
PCI-X bus. NIC is dual Intel pro/1000. Kernel
is close to stock 2.6.11.
I used brctl to create a bridge with the two
GigE adapters in it and
used pktgen to stream traffic through it
(250kpps in one direction, 1kpps in
the other.)
I see a reasonable amount of drops at 250kpps
(60 byte packets):
about 60,000,000 packets received, 20,700
dropped.
Interestingly, the system is about 60% idle
according to top,
and still dropping pkts, so it would seem that
the system could
be better utilized!
Ben
What GigE adapters did you use? Clearly every
driver is going to be different. My experience is
that a 3.4Ghz P4 is about the performance of a
2.0Ghz Opteron. I have to try your tuning script
tomorrow.
If your test is still set up, try compiling
something large while doing the test. The drops
go through the roof in my tests.
Danial
____________________________________________________
Start your day with Yahoo! - make it your home page
http://www.yahoo.com/r/hs
From: Ben Greear <hidden> Date: 2005-08-25 06:15:47
Danial Thom wrote:
quoted hunk
--- Ben Greear <greearb@candelatech.com> wrote:
quoted
Danial Thom wrote:
quoted
I think the concensus is that 2.6 has made
trade
quoted
offs that lower raw throughput, which is what
a
quoted
networking device needs. So as a router or
network appliance, 2.6 seems less suitable. A
raw
quoted
bridging test on a 2.0Ghz operton system:
FreeBSD 4.9: Drops no packets at 900K pps
Linux 2.4.24: Starts dropping packets at 350K
pps
quoted
Linux 2.6.12: Starts dropping packets at 100K
pps
I ran some quick tests using kernel 2.6.11, 1ms
tick (HZ=1000), SMP kernel.
Hardware is P-IV 3.0Ghz + HT on a new
SuperMicro motherboard with 64/133Mhz
PCI-X bus. NIC is dual Intel pro/1000. Kernel
is close to stock 2.6.11.
What GigE adapters did you use? Clearly every
driver is going to be different. My experience is
that a 3.4Ghz P4 is about the performance of a
2.0Ghz Opteron. I have to try your tuning script
tomorrow.
Intel pro/1000, as I mentioned. I haven't tried any other
NIC that comes close in performance to the e1000.
If your test is still set up, try compiling
something large while doing the test. The drops
go through the roof in my tests.
Installing RH9 on the box now to try some tests...
Disk access always robs networking, in my experience, so
I am not supprised you see bad ntwk performance while
compiling.
Ben
--
Ben Greear [off-list ref]
Candela Technologies Inc http://www.candelatech.com
From: Ben Greear <hidden> Date: 2005-08-25 06:34:56
Danial Thom wrote:
quoted hunk
--- Ben Greear <greearb@candelatech.com> wrote:
quoted
Danial Thom wrote:
quoted
I think the concensus is that 2.6 has made
trade
quoted
offs that lower raw throughput, which is what
a
quoted
networking device needs. So as a router or
network appliance, 2.6 seems less suitable. A
raw
quoted
bridging test on a 2.0Ghz operton system:
FreeBSD 4.9: Drops no packets at 900K pps
Linux 2.4.24: Starts dropping packets at 350K
pps
quoted
Linux 2.6.12: Starts dropping packets at 100K
pps
I ran some quick tests using kernel 2.6.11, 1ms
tick (HZ=1000), SMP kernel.
Hardware is P-IV 3.0Ghz + HT on a new
SuperMicro motherboard with 64/133Mhz
PCI-X bus. NIC is dual Intel pro/1000. Kernel
is close to stock 2.6.11.
I used brctl to create a bridge with the two
GigE adapters in it and
used pktgen to stream traffic through it
(250kpps in one direction, 1kpps in
the other.)
I see a reasonable amount of drops at 250kpps
(60 byte packets):
about 60,000,000 packets received, 20,700
dropped.
I get slightly worse performance on this system when running RH9
with kernel 2.4.29 (my hacks, HZ=1000, SMP). Tried increasing
e1000 descriptors to 2048 tx and rx, but that didn't help, or at least
not much.
Will try some other tunings, but I doubt it will affect performance
enough to come close to the discrepency that you show between 2.4
and 2.6 kernels...
I tried copying a 500MB CDROM to HD on my RH9 system, and only 6kpps
of the 250kpps get through the bridge...btw.
Ben
--
Ben Greear [off-list ref]
Candela Technologies Inc http://www.candelatech.com
networking device needs. So as a router or
network appliance, 2.6 seems less suitable.
A
quoted
quoted
raw
quoted
bridging test on a 2.0Ghz operton system:
FreeBSD 4.9: Drops no packets at 900K pps
Linux 2.4.24: Starts dropping packets at
350K
quoted
quoted
pps
quoted
Linux 2.6.12: Starts dropping packets at
100K
quoted
quoted
pps
I ran some quick tests using kernel 2.6.11,
1ms
quoted
quoted
tick (HZ=1000), SMP kernel.
Hardware is P-IV 3.0Ghz + HT on a new
SuperMicro motherboard with 64/133Mhz
PCI-X bus. NIC is dual Intel pro/1000.
Kernel
quoted
quoted
is close to stock 2.6.11.
I used brctl to create a bridge with the two
GigE adapters in it and
used pktgen to stream traffic through it
(250kpps in one direction, 1kpps in
the other.)
I see a reasonable amount of drops at 250kpps
(60 byte packets):
about 60,000,000 packets received, 20,700
dropped.
I get slightly worse performance on this system
when running RH9
with kernel 2.4.29 (my hacks, HZ=1000, SMP).
Tried increasing
e1000 descriptors to 2048 tx and rx, but that
didn't help, or at least
not much.
Will try some other tunings, but I doubt it
will affect performance
enough to come close to the discrepency that
you show between 2.4
and 2.6 kernels...
I tried copying a 500MB CDROM to HD on my RH9
system, and only 6kpps
of the 250kpps get through the bridge...btw.
The tests I reported where on UP systems. Perhaps
the default settings are better for this in 2.4,
since that is what I used, and you used your
hacks for both.
Are you getting drops or overruns (or both)? I
would assume drops is a decision to drop rather
than an overrun which is a ring overrun. Overruns
would imply more about performance than tuning,
I'd think.
I wouldn't think that HT would be appropriate for
this sort of setup...?
You're using a dual PCI-X NIC rather than the
onboard ports? Supermicro runs their onboard
controllers at 32bit/33mhz for some mindless
reason.
Danial
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
From: Ben Greear <hidden> Date: 2005-08-25 16:55:37
Danial Thom wrote:
The tests I reported where on UP systems. Perhaps
the default settings are better for this in 2.4,
since that is what I used, and you used your
hacks for both.
My modifications to the kernel are unlikely to speed anything
up, and probably will slow things down ever so slightly.
I can try with a UP kernel, but my machine at least has a single
processor. I'm using the SMP kernel to take advantage of HT.
Are you getting drops or overruns (or both)? I
would assume drops is a decision to drop rather
than an overrun which is a ring overrun. Overruns
would imply more about performance than tuning,
I'd think.
I was seeing lots of NIC errors...in fact, it was showing a great many
more errors than packets sent to it, so I just ignored them.
I increased the TxDescriptors and RxDescriptors and that helped a little.
Increasing the transmit queue for the NIC to 2000 also helped a little.
I wouldn't think that HT would be appropriate for
this sort of setup...?
2.6.11 seems to be faster when running SMP kernel on this system.
You're using a dual PCI-X NIC rather than the
onboard ports? Supermicro runs their onboard
Of course. Never found a motherboard yet with decent built-in
NICs. The built-ins on this board are tg3 and they must be on
a slow bus, because they cannot go faster than about 700Mbps
(using big pkts).
I'll benchmark things again when 2.6.13 comes out and try to
get some more detailed numbers...
Thanks,
Ben
--
Ben Greear [off-list ref]
Candela Technologies Inc http://www.candelatech.com
since that is what I used, and you used your
hacks for both.
My modifications to the kernel are unlikely to
speed anything
up, and probably will slow things down ever so
slightly.
I can try with a UP kernel, but my machine at
least has a single
processor. I'm using the SMP kernel to take
advantage of HT.
quoted
Are you getting drops or overruns (or both)?
I
quoted
would assume drops is a decision to drop
rather
quoted
than an overrun which is a ring overrun.
Overruns
quoted
would imply more about performance than
tuning,
quoted
I'd think.
I was seeing lots of NIC errors...in fact, it
was showing a great many
more errors than packets sent to it, so I just
ignored them.
I increased the TxDescriptors and RxDescriptors
and that helped a little.
Increasing the transmit queue for the NIC to
2000 also helped a little.
quoted
I wouldn't think that HT would be appropriate
for
quoted
this sort of setup...?
2.6.11 seems to be faster when running SMP
kernel on this system.
HT and SMP are not the same animal, are they? My
understanding is that an HT aware scheduler is
likely to make things worse most of the time,
particularly for systems not running a lot of
threads..
quoted
You're using a dual PCI-X NIC rather than the
onboard ports? Supermicro runs their onboard
Of course. Never found a motherboard yet with
decent built-in
NICs. The built-ins on this board are tg3 and
they must be on
a slow bus, because they cannot go faster than
about 700Mbps
(using big pkts).
If its the P8SCI or the same design they are on a
1X PCIE thats shared with the PCI-X. Pretty hokey
stuff. Its also a low-end controller amongst the
broadcom parts.
Danial
____________________________________________________
Start your day with Yahoo! - make it your home page
http://www.yahoo.com/r/hs
networking device needs. So as a router or
network appliance, 2.6 seems less suitable.
A
quoted
quoted
raw
quoted
bridging test on a 2.0Ghz operton system:
FreeBSD 4.9: Drops no packets at 900K pps
Linux 2.4.24: Starts dropping packets at
350K
quoted
quoted
pps
quoted
Linux 2.6.12: Starts dropping packets at
100K
quoted
quoted
pps
I ran some quick tests using kernel 2.6.11,
1ms
quoted
quoted
tick (HZ=1000), SMP kernel.
Hardware is P-IV 3.0Ghz + HT on a new
SuperMicro motherboard with 64/133Mhz
PCI-X bus. NIC is dual Intel pro/1000.
Kernel
quoted
quoted
is close to stock 2.6.11.
quoted
What GigE adapters did you use? Clearly every
driver is going to be different. My
experience is
quoted
that a 3.4Ghz P4 is about the performance of
a
quoted
2.0Ghz Opteron. I have to try your tuning
script
quoted
tomorrow.
Intel pro/1000, as I mentioned. I haven't
tried any other
NIC that comes close in performance to the
e1000.
quoted
If your test is still set up, try compiling
something large while doing the test. The
drops
quoted
go through the roof in my tests.
Installing RH9 on the box now to try some
tests...
Disk access always robs networking, in my
experience, so
I am not supprised you see bad ntwk performance
while
compiling.
Ben
It would be useful if there were some way to find
out "what" is getting "robbed". If networking has
priority, then what is keeping it from getting
back to processing the rx interrupts?
Ah, the e1000 has built-in interrupt moderation.
I can't get into my lab until tomorrow afternoon,
but if you get a chance try setting ITR in
e1000_main.c to something larger, like 20K. and
see if it makes a difference. At 200K pps that
would cause an interrupt every 10 packets, which
may allow the routine to grab back the cpu more
often.
Danial
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
From: Benjamin LaHaise <hidden> Date: 2005-08-26 19:10:52
On Thu, Aug 25, 2005 at 09:55:27AM -0700, Ben Greear wrote:
Of course. Never found a motherboard yet with decent built-in
NICs. The built-ins on this board are tg3 and they must be on
a slow bus, because they cannot go faster than about 700Mbps
(using big pkts).
There should be a number of decent boards out on the market these days.
Try picking up one with a CSA gige adapter (a dedicated high bandwidth
link to an e1000) or PCI Express (although that is harder to pick up on
from the specifications of most motherboards).
-ben
networking device needs. So as a router or
network appliance, 2.6 seems less
suitable.
quoted
A
quoted
quoted
raw
quoted
bridging test on a 2.0Ghz operton system:
FreeBSD 4.9: Drops no packets at 900K pps
Linux 2.4.24: Starts dropping packets at
350K
quoted
quoted
pps
quoted
Linux 2.6.12: Starts dropping packets at
100K
quoted
quoted
pps
I ran some quick tests using kernel 2.6.11,
1ms
quoted
quoted
tick (HZ=1000), SMP kernel.
Hardware is P-IV 3.0Ghz + HT on a new
SuperMicro motherboard with 64/133Mhz
PCI-X bus. NIC is dual Intel pro/1000.
Kernel
quoted
quoted
is close to stock 2.6.11.
quoted
What GigE adapters did you use? Clearly
every
quoted
quoted
driver is going to be different. My
experience is
quoted
that a 3.4Ghz P4 is about the performance
of
quoted
a
quoted
2.0Ghz Opteron. I have to try your tuning
script
quoted
tomorrow.
Intel pro/1000, as I mentioned. I haven't
tried any other
NIC that comes close in performance to the
e1000.
quoted
If your test is still set up, try compiling
something large while doing the test. The
drops
quoted
go through the roof in my tests.
Installing RH9 on the box now to try some
tests...
Disk access always robs networking, in my
experience, so
I am not supprised you see bad ntwk
performance
quoted
while
compiling.
Ben
It would be useful if there were some way to
find
out "what" is getting "robbed". If networking
has
priority, then what is keeping it from getting
back to processing the rx interrupts?
Ah, the e1000 has built-in interrupt
moderation.
I can't get into my lab until tomorrow
afternoon,
but if you get a chance try setting ITR in
e1000_main.c to something larger, like 20K. and
see if it makes a difference. At 200K pps that
would cause an interrupt every 10 packets,
which
may allow the routine to grab back the cpu more
often.
Danial
Just FYI, setting interrupt moderation to 20,000
didn't make much difference.
____________________________________________________
Start your day with Yahoo! - make it your home page
http://www.yahoo.com/r/hs
On Wed, Aug 24, 2005 at 11:08:43PM -0700,
Danial Thom wrote:
quoted
If your test is still set up, try compiling
something large while doing the test. The
drops
quoted
go through the roof in my tests.
Couldn't this happen because ksoftirqd by
default
has a nice value of 19?
~
:wq
With
best regards,
Vladimir Savkin.
renicing ksoftirqd improves it a bit, but
still tons of loss, even at only 120Kpps.
Danial
____________________________________________________
Start your day with Yahoo! - make it your home page
http://www.yahoo.com/r/hs