From: Troy Wilson <hidden> Date: 2002-09-05 18:25:51
I've got some early SPECWeb [*] results with 2.5.33 and TSO on e1000. I
get 2906 simultaneous connections, 99.2% conforming (i.e. faster than the
320 kbps cutoff), at 0% idle with TSO on. For comparison, with 2.5.25, I
got 2656, and with 2.5.29 I got 2662, (both 99+% conformance and 0% idle) so
TSO and 2.5.33 look like a Big Win.
I'm having trouble testing with TSO off (I changed the #define NETIF_F_TSO
to "0" in include/linux/netdevice.h to turn it off). I am getting errors.
NETDEV WATCHDOG: eth1: transmit timed out
e1000: eth1 NIC Link is Up 1000 Mbps Full Duplex
That's pushed my SPECWeb results down to below 2500 connections with TSO
off because of those adapter resets (It is only that one adapter, BTW) and
these results (with TSO off) shouldn't be considered valid.
eth1 is the only adapter with errors, and they all look like RX overruns.
For comparison:
eth1 Link encap:Ethernet HWaddr 00:02:B3:9C:F5:9E
inet addr:192.168.4.1 Bcast:192.168.4.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:48621378 errors:8890 dropped:8890 overruns:8890 frame:0
TX packets:64342993 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:3637004554 (3468.5 Mb) TX bytes:1377740556 (1313.9 Mb)
Interrupt:61 Base address:0x1200 Memory:fc020000-0
eth3 Link encap:Ethernet HWaddr 00:02:B3:A3:47:E7
inet addr:192.168.3.1 Bcast:192.168.3.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:37130540 errors:0 dropped:0 overruns:0 frame:0
TX packets:49061277 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:2774988658 (2646.4 Mb) TX bytes:3290541711 (3138.1 Mb)
Interrupt:44 Base address:0x2040 Memory:fe120000-0
I'm still working on getting a clean run with TSO off. If anyone has any
ideas for me about the timeout errors, I'd appreciate the clue.
Thanks,
- Troy
* SPEC(tm) and the benchmark name SPECweb(tm) are registered
trademarks of the Standard Performance Evaluation Corporation.
This benchmarking was performed for research purposes only,
and is non-compliant, with the following deviations from the
rules -
1 - It was run on hardware that does not meed the SPEC
availability-to-the-public criteria. The machine is
an engineering sample.
2 - access_log wasn't kept for full accounting. It was
being written, but deleted every 200 seconds.
Hey, thanks for crossposting to netdev
So if i understood correctly (looking at the intel site) the main value
add of this feature is probably in having the CPU avoid reassembling and
retransmitting. I am willing to bet that the real value in your results is
in saving on retransmits; I would think shoving the data down the NIC
and avoid the fragmentation shouldnt give you that much significant CPU
savings. Do you have any stats from the hardware that could show
retransmits etc; have you tested this with zero copy as well (sendfile)
again, if i am right you shouldnt see much benefit from that either?
I would think it probably works well with things like partial ACKs too?
(I am almost sure it does or someone needs to be spanked, so just
checking).
cheers,
jamal
So if i understood correctly (looking at the intel site) the main
value add of this feature is probably in having the CPU avoid
reassembling and retransmitting. I am willing to bet that the real
Er, even just assembling and transmitting? I'm thinking of the
reduction in things like separate memory allocation calls and looking
up the route, etc..??
value in your results is in saving on retransmits; I would think
shoving the data down the NIC and avoid the fragmentation shouldnt
give you that much significant CPU savings. Do you have any stats
Why do say that? Wouldnt the fact that youre now reducing the
number of calls down the stack by a significant number provide
a significant saving?
from the hardware that could show retransmits etc; have you tested
this with zero copy as well (sendfile) again, if i am right you
shouldnt see much benefit from that either?
value in your results is in saving on retransmits; I would think
shoving the data down the NIC and avoid the fragmentation shouldnt
give you that much significant CPU savings. Do you have any stats
Why do say that? Wouldnt the fact that youre now reducing the
number of calls down the stack by a significant number provide
a significant saving?
I am not sure; if he gets a busy system in a congested network, i can
see the offloading savings i.e i am not sure if the amortization of the
calls away from the CPU is sufficient enough savings if it doesnt
involve a lot of retransmits. I am also wondering how smart this NIC
in doing the retransmits; example i have doubts if this idea is briliant
to begin with; does it handle SACKs for example? What about
the du-jour algorithm, would you have to upgrade the NIC or can it be
taught some new trickes etc etc.
[also i can see why it makes sense to use this feature only with sendfile;
its pretty much useless for interactive apps]
Troy, i am not interested in the nestat -s data rather the TCP stats
this NIC has exposed. Unless those somehow show up magically in netstat.
cheers,
jamal
I am not sure; if he gets a busy system in a congested network, i
can see the offloading savings i.e i am not sure if the amortization
of the calls away from the CPU is sufficient enough savings if it
doesnt involve a lot of retransmits. I am also wondering how smart
this NIC in doing the retransmits; example i have doubts if this
idea is briliant to begin with; does it handle SACKs for example?
do you mean sack data being sent as a tcp option?
dont know, lots of other questions arise (like timestamp
on all the segments would be the same?).
Troy, i am not interested in the nestat -s data rather the TCP
stats this NIC has exposed. Unless those somehow show up magically
in netstat.
most recent (dont know how far back) versions of netstat
display /proc/net/snmp and /proc/net/netstat (with the
Linux TCP MIB), so netstat -s should show you most of
whats interesting. Or were you referring to something else?
ifconfig -a and netstat -rn would also be nice to have..
thanks,
Nivedita
From: David S. Miller <hidden> Date: 2002-09-06 03:49:52
From: jamal [off-list ref]
Date: Thu, 5 Sep 2002 16:59:47 -0400 (EDT)
I would think shoving the data down the NIC
and avoid the fragmentation shouldnt give you that much significant
CPU savings.
It's the DMA bandwidth saved, most of the specweb runs on x86 hardware
is limited by the DMA throughput of the PCI host controller. In
particular some controllers are limited to smaller DMA bursts to
work around hardware bugs.
Ie. the headers that don't need to go across the bus are the critical
resource saved by TSO.
I think I've said this a million times, perhaps the next person who
tries to figure out where the gains come from can just reply with
a pointer to a URL of this email I'm typing right now :-)
From: David S. Miller <hidden> Date: 2002-09-06 03:58:59
From: jamal [off-list ref]
Date: Thu, 5 Sep 2002 21:47:35 -0400 (EDT)
I am not sure; if he gets a busy system in a congested network, i can
see the offloading savings i.e i am not sure if the amortization of the
calls away from the CPU is sufficient enough savings if it doesnt
involve a lot of retransmits. I am also wondering how smart this NIC
in doing the retransmits; example i have doubts if this idea is briliant
to begin with; does it handle SACKs for example? What about
the du-jour algorithm, would you have to upgrade the NIC or can it be
taught some new trickes etc etc.
[also i can see why it makes sense to use this feature only with sendfile;
its pretty much useless for interactive apps]
Troy, i am not interested in the nestat -s data rather the TCP stats
this NIC has exposed. Unless those somehow show up magically in netstat.
There are no retransmits happening, the card does not analyze
activity on the TCP connection to retransmit things itself
it's just a simple header templating facility.
Read my other emails about where the benefits come from.
In fact when connection is sick (ie. retransmits and SACKs occur)
we disable TSO completely for that socket.
From: David S. Miller <hidden> Date: 2002-09-06 04:01:13
From: Nivedita Singhvi [off-list ref]
Date: Thu, 5 Sep 2002 20:38:10 -0700
most recent (dont know how far back) versions of netstat
display /proc/net/snmp and /proc/net/netstat (with the
Linux TCP MIB), so netstat -s should show you most of
whats interesting. Or were you referring to something else?
ifconfig -a and netstat -rn would also be nice to have..
TSO gets turned off during retransmits/SACK and the card does not do
retransmits.
Can we move on in this conversation now? :-)
ifconfig -a and netstat -rn would also be nice to have..
TSO gets turned off during retransmits/SACK and the card does not
do
retransmits.
Can we move on in this conversation now? :-)
Sure :). The motivation for seeing the stats though would
be to get an idea of how much retransmission/SACK etc
activity _is_ occurring during Troy's SpecWeb runs, which
would give us an idea of how often we're actually doing
segmentation offload, and better idea of how much gain
its possible to further get from this(ahem) DMA coalescing :).
Some of Troy's early runs had a very large number of
packets dropped by the card.
thanks,
Nivedita
From: David S. Miller <hidden> Date: 2002-09-06 04:19:35
From: Nivedita Singhvi [off-list ref]
Date: Thu, 5 Sep 2002 21:20:47 -0700
Sure :). The motivation for seeing the stats though would
be to get an idea of how much retransmission/SACK etc
activity _is_ occurring during Troy's SpecWeb runs, which
would give us an idea of how often we're actually doing
segmentation offload, and better idea of how much gain
its possible to further get from this(ahem) DMA coalescing :).
Some of Troy's early runs had a very large number of
packets dropped by the card.
One thing to do is make absolutely sure that flow control is
enabled and supported by all devices on the link from the
client to the test spedweb server.
Troy can do you do that for us along with the statistic
dumps?
Thanks.
From: Martin J. Bligh <hidden> Date: 2002-09-06 06:45:53
I would think shoving the data down the NIC
and avoid the fragmentation shouldnt give you that much significant
CPU savings.
It's the DMA bandwidth saved, most of the specweb runs on x86 hardware
is limited by the DMA throughput of the PCI host controller. In
particular some controllers are limited to smaller DMA bursts to
work around hardware bugs.
Ie. the headers that don't need to go across the bus are the critical
resource saved by TSO.
I'm not sure that's entirely true in this case - the Netfinity
8500R is slightly unusual in that it has 3 or 4 PCI buses, and
there's 4 - 8 gigabit ethernet cards in this beast spread around
different buses (Troy - are we still just using 4? ... and what's
the raw bandwidth of data we're pushing? ... it's not huge).
I think we're CPU limited (there's no idle time on this machine),
which is odd for an 8 CPU 900MHz P3 Xeon, but still, this is Apache,
not Tux. You mentioned CPU load as another advantage of TSO ...
anything we've done to reduce CPU load enables us to run more and
more connections (I think we started at about 260 or something, so
2900 ain't too bad ;-)).
Just to throw another firework into the fire whilst people are
awake, NAPI does not seem to scale to this sort of load, which
was disappointing, as we were hoping it would solve some of
our interrupt load problems ... seems that half the machine goes
idle, the number of simultaneous connections drop way down, and
everything's blocked on ... something ... not sure what ;-)
Any guesses at why, or ways to debug this?
M.
PS. Anyone else running NAPI on SMP? (ideally at least 4-way?)
From: David S. Miller <hidden> Date: 2002-09-06 06:54:33
From: "Martin J. Bligh" [off-list ref]
Date: Thu, 05 Sep 2002 23:48:42 -0700
Just to throw another firework into the fire whilst people are
awake, NAPI does not seem to scale to this sort of load, which
was disappointing, as we were hoping it would solve some of
our interrupt load problems ...
Stupid question, are you sure you have CONFIG_E1000_NAPI enabled?
NAPI is also not the panacea to all problems in the world.
I bet your greatest gain would be obtained from going to Tux
and using appropriate IRQ affinity settings and making sure
Tux threads bind to same cpu as device where they accept
connections.
It is standard method to obtain peak specweb performance.
From: David S. Miller <hidden> Date: 2002-09-06 07:25:29
From: Andrew Morton [off-list ref]
Date: Fri, 06 Sep 2002 00:36:04 -0700
"David S. Miller" wrote:
> NAPI is also not the panacea to all problems in the world.
Mala did some testing on this a couple of weeks back. It appears that
NAPI damaged performance significantly.
http://www-124.ibm.com/developerworks/opensource/linuxperf/netperf/results/july_02/netperf2.5.25results.htm
Unfortunately it is not listed what e1000 and core NAPI
patch was used. Also, not listed, are the RX/TX mitigation
and ring sizes given to the kernel module upon loading.
Robert can comment on optimal settings
Robert and Jamal can make a more detailed analysis of Mala's
graphs than I.
Mala did some testing on this a couple of weeks back. It appears that
NAPI damaged performance significantly.
http://www-124.ibm.com/developerworks/opensource/linuxperf/netperf/results/july_02/netperf2.5.25results.htm
Unfortunately it is not listed what e1000 and core NAPI
patch was used. Also, not listed, are the RX/TX mitigation
and ring sizes given to the kernel module upon loading.
Robert can comment on optimal settings
Robert and Jamal can make a more detailed analysis of Mala's
graphs than I.
I looked at those graphs, but the lack of information makes them useless.
For example there are too many variables to the tests -- what is the
effect the message size? and then look at the socket buffer size, would
you set it to 64K if you are trying to show perfomance numbers? What
other tcp settings are there?
Manfred Spraul about a year back complained about some performance issues
in low load setups (which is what this IBM setup seems to be if you count
the pps to the server); its one of those things that have been low in
the TODO deck.
The issue maybe legit not because NAPI is bad but because it is too good.
I dont have the e1000, but i have some Dlinks giges still in boxes and i
have a two-CPU SMP machine; I'll setup the testing this weekend.
In the case of Manfred, we couldnt reproduce the tests because he had this
odd weird NIC; in this case at least access to the e1000 doesnt require
a visit to the museum.
cheers,
jamal
From: Martin J. Bligh <hidden> Date: 2002-09-06 14:26:35
Stupid question, are you sure you have CONFIG_E1000_NAPI enabled?
NAPI is also not the panacea to all problems in the world.
No, but I didn't expect throughput to drop by 40% or so either,
which is (very roughly) what happened. Interrupts are a pain to
manage and do affinity with, so NAPI should (at least in theory)
be better for this kind of setup ... I think.
I bet your greatest gain would be obtained from going to Tux
and using appropriate IRQ affinity settings and making sure
Tux threads bind to same cpu as device where they accept
connections.
It is standard method to obtain peak specweb performance.
Ah, but that's not really our goal - what we're trying to do is
use specweb as a tool to simulate a semi-realistic customer
workload, and improve the Linux kernel performance, using that
as our yardstick for measuring ourselves. For that I like the
setup we have reasonably well, even though it won't get us the
best numbers.
To get the best benchmark numbers, you're absolutely right though.
M.
From: Dave Hansen <hidden> Date: 2002-09-06 15:26:40
Martin J. Bligh wrote:
Just to throw another firework into the fire whilst people are
awake, NAPI does not seem to scale to this sort of load, which
was disappointing, as we were hoping it would solve some of
our interrupt load problems ... seems that half the machine goes
idle, the number of simultaneous connections drop way down, and
everything's blocked on ... something ... not sure what ;-)
Any guesses at why, or ways to debug this?
I thought that I already tried to explain this to you. (although it could
have been on one of those too-much-coffee-days :)
Something strange happens to the clients when NAPI is enabled on the
Specweb clients. Somehow the start using a lot more CPU. The increased
idle time on the server is because the _clients_ are CPU maxed. I have
some preliminary oprofile data for the clients, but it appears that this is
another case of Specweb code just really sucking.
The real question is why NAPI causes so much more work for the client. I'm
not convinced that it is much, much greater, because I believe that I was
already at the edge of the cliff with my clients and NAPI just gave them a
little shove :). Specweb also takes a while to ramp up (even during the
real run), so sometimes it takes a few minutes to see the clients get
saturated.
--
Dave Hansen
haveblue@us.ibm.com
From: Dave Hansen <hidden> Date: 2002-09-06 15:34:52
Martin J. Bligh wrote:
quoted
Stupid question, are you sure you have CONFIG_E1000_NAPI enabled?
NAPI is also not the panacea to all problems in the world.
No, but I didn't expect throughput to drop by 40% or so either,
which is (very roughly) what happened. Interrupts are a pain to
manage and do affinity with, so NAPI should (at least in theory)
be better for this kind of setup ... I think.
No, no. Bad Martin! Throughput didn't drop, "Specweb compliance" dropped.
Those are two very, very different things. I've found that the server
can produce a lot more throughput, although it doesn't have the
characteristics that Specweb considers compliant. Just have Troy enable
mod-status and look at the throughput that Apache tells you that it is
giving during a run. _That_ is real throughput, not number of compliant
connections.
_And_ NAPI is for receive only, right? Also, my compliance drop occurs
with the NAPI checkbox disabled. There is something else in the new driver
that causes our problems.
--
Dave Hansen
haveblue@us.ibm.com
From: Martin J. Bligh <hidden> Date: 2002-09-06 16:08:15
No, no. Bad Martin! Throughput didn't drop, "Specweb compliance"
dropped. Those are two very, very different things. I've found
that the server can produce a lot more throughput, although it
doesn't have the characteristics that Specweb considers compliant.
Just have Troy enable mod-status and look at the throughput that
Apache tells you that it is giving during a run. _That_ is real
throughput, not number of compliant connections.
By throughput I meant number of compliant connections, not bandwidth.
It may well be latency that's going out the window, rather than
bandwidth. Yes, I should use more precise terms ...
_And_ NAPI is for receive only, right? Also, my compliance drop
occurs with the NAPI checkbox disabled. There is something else
in the new driver that causes our problems.
Not sure about that - I was told once that there were transmission
completion interrupts as well? What happens to those? Or am I
confused again ...
M.
No, no. Bad Martin! Throughput didn't drop, "Specweb compliance"
dropped. Those are two very, very different things. I've found that
the server can produce a lot more throughput, although it doesn't
have the characteristics that Specweb considers compliant.
Just have Troy enable mod-status and look at the throughput that
Apache tells you that it is giving during a run.
_That_ is real throughput, not number of compliant connections.
_And_ NAPI is for receive only, right? Also, my compliance drop
occurs with the NAPI checkbox disabled. There is something else in
the new driver that causes our problems.
Thanks, Dave, you saved me a bunch of typing...
Just looking at a networking benchmark result is worse than
useless. You really need to look at the stats, settings,
and the profiles. eg, for most of the networking stuff:
ifconfig -a
netstat -s
netstat -rn
/proc/sys/net/ipv4/
/proc/sys/net/core/
before and after the run.
Dave, although in your setup the clients are maxed out,
not sure thats the case for Mala and Troy's clients. (Dont
know, of course). But I'm fairly sure they arent using
single quad NUMAs and they may not be seeing the same
effects..
thanks,
Nivedita
From: Martin J. Bligh <hidden> Date: 2002-09-06 16:26:42
I thought that I already tried to explain this to you. (although
it could have been on one of those too-much-coffee-days :)
You told me, but I'm far from convinced this is the problem. I think
it's more likely this is a side-effect of a server issue - something
like a lot of dropped packets and retransmits, though not necessarily
that.
Something strange happens to the clients when NAPI is enabled on
the Specweb clients. Somehow the start using a lot more CPU.
The increased idle time on the server is because the _clients_ are
CPU maxed. I have some preliminary oprofile data for the clients,
but it appears that this is another case of Specweb code just
really sucking.
Hmmm ... if you change something on the server, and all the clients
go wild, I'm suspicious of whatever you did to the server. You need
to have a lot more data before leaping to the conclusion that it's
because the specweb client code is crap.
Troy - I think your UP clients weren't anywhere near maxed out on
CPU power, right? Can you take a peek at the clients under NAPI load?
Dave - did you ever try running 4 specweb clients bound to each of
the 4 CPUs in an attempt to make the clients scale better? I'm
suspicious that you're maxing out 4 4-way machines, and Troy's
16 UPs are cruising along just fine.
M.
From: Dave Hansen <hidden> Date: 2002-09-06 17:32:56
Martin J. Bligh wrote:
quoted
Something strange happens to the clients when NAPI is enabled on
the Specweb clients. Somehow the start using a lot more CPU.
The increased idle time on the server is because the _clients_ are
CPU maxed. I have some preliminary oprofile data for the clients,
but it appears that this is another case of Specweb code just
really sucking.
Hmmm ... if you change something on the server, and all the clients
go wild, I'm suspicious of whatever you did to the server.
Me too :) All that was changed was adding the new e1000 driver. NAPI was
disabled.
> You need
to have a lot more data before leaping to the conclusion that it's
because the specweb client code is crap.
Troy - I think your UP clients weren't anywhere near maxed out on
CPU power, right? Can you take a peek at the clients under NAPI load?
Make sure you wait a minute or two. The client tends to ramp up.
"vmstat 2" after the client has told the master that it is running:
U S I
----------
4 15 81
5 17 79
7 16 77
7 17 76
7 21 72
11 25 64
3 16 82
2 14 84
7 23 70
16 50 34
24 75 0
27 73 0
28 72 0
24 76 0
...
Dave - did you ever try running 4 specweb clients bound to each of
the 4 CPUs in an attempt to make the clients scale better? I'm
suspicious that you're maxing out 4 4-way machines, and Troy's
16 UPs are cruising along just fine.
No, but I'm not sure it will do any good. They don't run often enough and
I have the feeling that there are very few cache locality benefits to be had.
--
Dave Hansen
haveblue@us.ibm.com
The profile looks bogus. The NIC driver is nowhere in sight. Normally
its mmap IO for interrupts and device registers should show. I would
double check it (e.g. with normal profile)
In case it is no bogus:
Most of these are either atomic_inc/dec of reference counters or some
form of lock. The system_call could be the int 0x80 (using the SYSENTER
patches would help), which also does atomic operations implicitely.
restore_all is IRET, could also likely be speed up by using SYSEXIT.
If NAPI hurts here then it surely not because of eating CPU time.
-Andi
The profile looks bogus. The NIC driver is nowhere in sight. Normally
its mmap IO for interrupts and device registers should show. I would
double check it (e.g. with normal profile)
Actually, oprofile separated out the acenic module from the rest of the
kernel. I should have included that breakout as well. but it was only 1.3
of CPU:
1.3801 0.0000 /lib/modules/2.4.18+O1/kernel/drivers/net/acenic.o
--
Dave Hansen
haveblue@us.ibm.com
From: David S. Miller <hidden> Date: 2002-09-06 18:39:34
From: Dave Hansen [off-list ref]
Date: Fri, 06 Sep 2002 11:33:10 -0700
Actually, oprofile separated out the acenic module from the rest of the
kernel. I should have included that breakout as well. but it was only 1.3
of CPU:
1.3801 0.0000 /lib/modules/2.4.18+O1/kernel/drivers/net/acenic.o
We thought you were using e1000 in these tests?
From: Martin J. Bligh <hidden> Date: 2002-09-06 18:42:27
Actually, oprofile separated out the acenic module from the rest of the
kernel. I should have included that breakout as well. but it was only 1.3
of CPU:
1.3801 0.0000 /lib/modules/2.4.18+O1/kernel/drivers/net/acenic.o
We thought you were using e1000 in these tests?
e1000 on the server, those profiles were client side.
M.
From: David S. Miller <hidden> Date: 2002-09-06 18:46:22
From: "Martin J. Bligh" [off-list ref]
Date: Fri, 06 Sep 2002 11:45:17 -0700
> Actually, oprofile separated out the acenic module from the rest of the
> kernel. I should have included that breakout as well. but it was only 1.3
> of CPU:
> 1.3801 0.0000 /lib/modules/2.4.18+O1/kernel/drivers/net/acenic.o
>
> We thought you were using e1000 in these tests?
e1000 on the server, those profiles were client side.
Ok. BTW acenic is packet rate limited by the speed of the
MIPS cpus on the card.
It might be instramental to disable HW checksumming in the
acenic driver and see what this does to your results.
The profile looks bogus. The NIC driver is nowhere in sight.
Normally its mmap IO for interrupts and device registers
should show. I would double check it (e.g. with normal profile)
Separately compiled acenic..
I'm surprised by this profile a bit too - on the client side,
since the requests are small, and the client is receiving
all those files, I would have thought that __generic_copy_to_user
would have been way higher than *from_user.
inet_bind() and tcp_v4_get_port() are up there because
we have to grab the socket lock, the tcp_portalloc_lock,
then the head chain lock and traverse the hash table
which has now many hundred entries. Also, because
of the varied length of the connections, the clients
get freed not in the same order they are allocated
a port, hence the fragmentation of the port space..
Tthere is some cacheline thrashing hurting the NUMA
more than other systems here too..
If you just wanted to speed things up, you could get the
clients to specify ports instead of letting the kernel
cycle through for a free port..:)
thanks,
Nivedita
In case it is no bogus:
Most of these are either atomic_inc/dec of reference counters or
some form of lock. The system_call could be the int 0x80 (using the
SYSENTER patches would help), which also does atomic operations
implicitely. restore_all is IRET, could also likely be speed up by
using SYSEXIT.
If NAPI hurts here then it surely not because of eating CPU time.
-Andi
If you just wanted to speed things up, you could get the
clients to specify ports instead of letting the kernel
cycle through for a free port..:)
Better would be probably to change the kernel to keep a limited
list of free ports in a free list. The grabbing a free port would
be an O(1) operation.
I'm not entirely sure it is worth it in this case. The locks are
probably the majority of the cost.
-Andi
From: David S. Miller <hidden> Date: 2002-09-06 19:23:54
From: Nivedita Singhvi [off-list ref]
Date: Fri, 6 Sep 2002 12:19:14 -0700
inet_bind() and tcp_v4_get_port() are up there because
we have to grab the socket lock, the tcp_portalloc_lock,
then the head chain lock and traverse the hash table
which has now many hundred entries. Also, because
of the varied length of the connections, the clients
get freed not in the same order they are allocated
a port, hence the fragmentation of the port space..
Tthere is some cacheline thrashing hurting the NUMA
more than other systems here too..
There are methods to eliminate the centrality of the
port allocation locking.
Basically, kill tcp_portalloc_lock and make the port rover be per-cpu.
The only tricky case is the "out of ports" situation. Because there
is no centralized locking being used to serialize port allocation,
it is difficult to be sure that the port space is truly exhausted.
Another idea, which doesn't eliminate the tcp_portalloc_lock but
has other good SMP properties, is to apply a "cpu salt" to the
port rover value. For example, shift the local cpu number into
the upper parts of a 'u16', then 'xor' that with tcp_port_rover.
Alexey and I have discussed this several times but never became
bored enough to experiment :-)
From: David S. Miller <hidden> Date: 2002-09-06 19:27:42
From: Andi Kleen [off-list ref]
Date: Fri, 6 Sep 2002 21:26:19 +0200
I'm not entirely sure it is worth it in this case. The locks are
probably the majority of the cost.
You can more localize the lock accesses (since we use per-chain
locks) by applying a cpu salt to the port numbers you allocate.
See my other email.
There are methods to eliminate the centrality of the
port allocation locking.
Basically, kill tcp_portalloc_lock and make the port rover be
per-cpu.
Aha! Exactly what I started to do quite a while ago..
The only tricky case is the "out of ports" situation. Because
there is no centralized locking being used to serialize port
allocation, it is difficult to be sure that the port space is truly
exhausted.
I decided to use a stupid global flag to signal this..It did become
messy and I didnt finalize everything. Then my day job
intervened :). Still hoping for spare time*5 to complete
this if none comes up with something before then..
Another idea, which doesn't eliminate the tcp_portalloc_lock but
has other good SMP properties, is to apply a "cpu salt" to the
port rover value. For example, shift the local cpu number into
the upper parts of a 'u16', then 'xor' that with tcp_port_rover.
From: Martin J. Bligh <hidden> Date: 2002-09-06 19:42:36
Tthere is some cacheline thrashing hurting the NUMA
more than other systems here too..
There is no NUMA here ... the clients are 4 single node SMP
systems. We're using the old quads to make them, but they're
all split up, not linked together into one system.
Sorry if we didn't make that clear.
M.
From: Eric W. Biederman <hidden> Date: 2002-09-11 09:21:25
"Martin J. Bligh" [off-list ref] writes:
quoted
Ie. the headers that don't need to go across the bus are the critical
resource saved by TSO.
I'm not sure that's entirely true in this case - the Netfinity
8500R is slightly unusual in that it has 3 or 4 PCI buses, and
there's 4 - 8 gigabit ethernet cards in this beast spread around
different buses (Troy - are we still just using 4? ... and what's
the raw bandwidth of data we're pushing? ... it's not huge).
I think we're CPU limited (there's no idle time on this machine),
which is odd for an 8 CPU 900MHz P3 Xeon,
Quite possibly. The P3 has roughly an 800MB/s FSB bandwidth, that must
be used for both I/O and memory accesses. So just driving a gige card at
wire speed takes a considerable portion of the cpus capacity.
On analyzing this kind of thing I usually find it quite helpful to
compute what the hardware can theoretically to get a feel where the
bottlenecks should be.
Eric
From: Martin J. Bligh <hidden> Date: 2002-09-11 14:08:20
quoted
quoted
Ie. the headers that don't need to go across the bus are the critical
resource saved by TSO.
I'm not sure that's entirely true in this case - the Netfinity
8500R is slightly unusual in that it has 3 or 4 PCI buses, and
there's 4 - 8 gigabit ethernet cards in this beast spread around
different buses (Troy - are we still just using 4? ... and what's
the raw bandwidth of data we're pushing? ... it's not huge).
I think we're CPU limited (there's no idle time on this machine),
which is odd for an 8 CPU 900MHz P3 Xeon,
Quite possibly. The P3 has roughly an 800MB/s FSB bandwidth, that must
be used for both I/O and memory accesses. So just driving a gige card at
wire speed takes a considerable portion of the cpus capacity.
On analyzing this kind of thing I usually find it quite helpful to
compute what the hardware can theoretically to get a feel where the
bottlenecks should be.
We can push about 420MB/s of IO out of this thing (out of that
theoretical 800Mb/s). Specweb is only pushing about 120MB/s of
total data through it, so it's not bus limited in this case.
Of course, I should have given you that data to start with,
but ... ;-)
M.
PS. This thing actually has 3 system buses, 1 for each of the two
sets of 4 CPUs, and 1 for all the PCI buses, and the three buses
are joined by an interconnect in the middle. But all the IO goes
through 1 of those buses, so for the purposes of this discussion,
it makes no difference whatsoever ;-)
From: Eric W. Biederman <hidden> Date: 2002-09-11 15:16:19
"Martin J. Bligh" [off-list ref] writes:
quoted
quoted
quoted
Ie. the headers that don't need to go across the bus are the critical
resource saved by TSO.
I'm not sure that's entirely true in this case - the Netfinity
8500R is slightly unusual in that it has 3 or 4 PCI buses, and
there's 4 - 8 gigabit ethernet cards in this beast spread around
different buses (Troy - are we still just using 4? ... and what's
the raw bandwidth of data we're pushing? ... it's not huge).
I think we're CPU limited (there's no idle time on this machine),
which is odd for an 8 CPU 900MHz P3 Xeon,
Quite possibly. The P3 has roughly an 800MB/s FSB bandwidth, that must
be used for both I/O and memory accesses. So just driving a gige card at
wire speed takes a considerable portion of the cpus capacity.
On analyzing this kind of thing I usually find it quite helpful to
compute what the hardware can theoretically to get a feel where the
bottlenecks should be.
We can push about 420MB/s of IO out of this thing (out of that
theoretical 800Mb/s).
Sounds about average for a P3. I have pushed the full 800MiB/s out of
a P3 processor to memory but it was a very optimized loop. Is
that 420MB/sec of IO on this test?
Specweb is only pushing about 120MB/s of
total data through it, so it's not bus limited in this case.
Note quite. But you suck at least 240MB/s of your memory bandwidth with
DMA from disk, and then DMA to the nic. Unless there is a highly
cached component. So I doubt you can effectively use more than 1 gige
card, maybe 2. And you have 8?
Of course, I should have given you that data to start with,
but ... ;-)
PS. This thing actually has 3 system buses, 1 for each of the two
sets of 4 CPUs, and 1 for all the PCI buses, and the three buses
are joined by an interconnect in the middle. But all the IO goes
through 1 of those buses, so for the purposes of this discussion,
it makes no difference whatsoever ;-)
Wow the hardware designers really believed in over-subscription.
If the busses are just running 64bit/33Mhz you are oversubscribed.
And at 64bit/66Mhz the pci busses can easily swamp the system
533*4 ~= 2128MB/s.
What kind of memory bandwidth does the system have, and on which
bus are the memory controllers? I'm just curious
Eric
From: David S. Miller <hidden> Date: 2002-09-11 15:18:38
From: ebiederm@xmission.com (Eric W. Biederman)
Date: 11 Sep 2002 09:06:36 -0600
"Martin J. Bligh" [off-list ref] writes:
> We can push about 420MB/s of IO out of this thing (out of that
> theoretical 800Mb/s).
Sounds about average for a P3. I have pushed the full 800MiB/s out of
a P3 processor to memory but it was a very optimized loop.
You pushed that over the PCI bus of your P3? Just to RAM
doesn't count, lots of cpu's can do that.
That's what makes his number interesting.
From: Martin J. Bligh <hidden> Date: 2002-09-11 15:24:37
Sounds about average for a P3. I have pushed the full 800MiB/s out of
a P3 processor to memory but it was a very optimized loop. Is
that 420MB/sec of IO on this test?
Yup, Fibre channel disks. So we know we can push at least that.
Note quite. But you suck at least 240MB/s of your memory bandwidth with
DMA from disk, and then DMA to the nic. Unless there is a highly
cached component. So I doubt you can effectively use more than 1 gige
card, maybe 2. And you have 8?
Nope, it's operating totally out of pagecache, there's no real disk
IO to speak of.
Wow the hardware designers really believed in over-subscription.
If the busses are just running 64bit/33Mhz you are oversubscribed.
And at 64bit/66Mhz the pci busses can easily swamp the system
533*4 ~= 2128MB/s.
Two 32bit buses (or maybe it was just one) and two 64bit buses,
all at 66MHz. Yes, the PCI buses can push more than the backplane,
but things are never perfectly balanced in reality, so I'd prefer
it that way around ... it's not a perfect system, but hey, it's
Intel hardware - this is high volume market, not real high end ;-)
What kind of memory bandwidth does the system have, and on which
bus are the memory controllers? I'm just curious
Memory controllers are hung off the interconnect, slightly difficult
to describe. Look for docs on the Intel profusion chipset, or I can
send you a powerpoint (yeah, yeah) presentation when I get into work
later today if you can't find it. Theoretical mem bandwidth should
be 1600MB/s if you're balanced across the CPUs, in practice I'd
expect to be able to push somewhat over 800Mb/s.
M.
From: Eric W. Biederman <hidden> Date: 2002-09-11 15:41:34
"David S. Miller" [off-list ref] writes:
From: ebiederm@xmission.com (Eric W. Biederman)
Date: 11 Sep 2002 09:06:36 -0600
"Martin J. Bligh" [off-list ref] writes:
> We can push about 420MB/s of IO out of this thing (out of that
> theoretical 800Mb/s).
Sounds about average for a P3. I have pushed the full 800MiB/s out of
a P3 processor to memory but it was a very optimized loop.
You pushed that over the PCI bus of your P3? Just to RAM
doesn't count, lots of cpu's can do that.
That's what makes his number interesting.
I agree. Getting 420MB/s to the pci bus is nice, especially with a P3.
The 800MB/s to memory was just the test I happened to conduct about 2 years
ago when I was still messing with slow P3 systems. It was a proof of
concept test to see if we could plug in an I/O card into a memory
slot.
On a current P4 system with the E7500 chipset this kind of thing is
easy. I have gotten roughly 450MB/s to a single myrinet card. And there
is enough theoretical bandwidth to do 4 times that. I haven't had a
chance to get it working in practice. When I attempted to run to gige
cards simultaneously I had some weird problem (probably interrupt
related) where adding additional pci cards did not deliver any extra
performance.
On a P3 to get writes from the cpu to hit 800MB/s you use the special
cpu instructions that bypass the cache.
My point was that I have tested the P3 bus in question and I achieved
a real world 800MB/s over it. So I expect that on the system in
question unless another bottleneck is hit, it should be possible to
achieve a real world 800MB/s of I/O. There are enough pci busses
to support that kind of traffic.
Unless the memory controller is carefully placed on the system though
doing 400+MB/s could easily eat up most of the available memory
bandwidth and reduce the system to doing some very slow cache line fills.
Eric
folx,
sorry for the late reply. catching up on kernel mail.
so all this TSO stuff looks v. v. similar to the IP-only fragmentation
that patricia gilfeather and i implemented on alteon acenics a couple of
years ago (see http://www.cs.unm.edu/~maccabe/SSL/frag/FragPaper1/ for a
general overview). it's exciting to see someone else take a stab on
different hardware and approaching some of the tcp-specific issues.
the main different, though, is that general purpose kernel development
still focussed on the improvements in *sending* speed. for real high
performance networking, the improvements are necessary in *receiving* cpu
utilization, in our estimation. (see our analysis of interrupt overhead
and the effect on receivers at gigabit speeds--i hope that this has become
common understanding by now)
i guess i can't disagree with david miller that the improvments in TSO are
due entirely to header retransmission for sending, but that's only because
sending wasn't CPU-intensive in the first place. we were able to get a
significant reduction in receiver cpu-utilization by reassembling IP
fragments on the receiver side (sort of a standards-based interrupt
mitigation strategy that has the benefit of not increasing latency the way
interrupt coalescing does).
anyway, nice work,
t.
On Thu, 5 Sep 2002, David S. Miller wrote:
It's the DMA bandwidth saved, most of the specweb runs on x86 hardware
is limited by the DMA throughput of the PCI host controller. In
particular some controllers are limited to smaller DMA bursts to
work around hardware bugs.
Ie. the headers that don't need to go across the bus are the critical
resource saved by TSO.
I think I've said this a million times, perhaps the next person who
tries to figure out where the gains come from can just reply with
a pointer to a URL of this email I'm typing right now :-)
--
todd underwood, vp & cto
oso grande technologies, inc.
todd@osogrande.com
"Those who give up essential liberties for temporary safety deserve
neither liberty nor safety." - Benjamin Franklin
Good work. The first time i have seen someone say Linux's way of
reverse order is a GoodThing(tm). It was also great to see de-mything
some of the old assumption of the world.
BTW, TSO is not a intelligent as what you are suggesting.
If i am not mistaken you are not only suggesting fragmentation and
assembly at that level you are also suggesting retransmits at the NIC.
This could be dangerous for practical reasons (changes in TCP congestion
control algorithms etc). TSO as was pointed in earlier emails is just a
dumb sender of packets. I think even fragmentation is a misnomer.
Essentially you shove a huge buffer to the NIC and it breaks it into MTU
sized packets for you and sends them.
In regards to the receive side CPU utilization improvements: I think
that NAPI does a good job at least in getting ridding of the biggest
offender -- interupt overload. Also with NAPI also having got rid of
intermidiate queues to the socket level, facilitating of zero copy receive
should be relatively easy to add but there are no capable NICs in
existence (well, ok not counting the TIGONII/acenic that you can hack
and the fact that the tigon 2 is EOL doesnt help other than just for
experiments). I dont think theres any NIC that can offload reassembly;
that might not be such a bad idea.
Are you still continuing work on this?
cheers,
jamal
Good work. The first time i have seen someone say Linux's way of
reverse order is a GoodThing(tm). It was also great to see de-mything
some of the old assumption of the world.
thanks. although i'd love to take credit, i don't think that the
reverse-order fragmentation appreciation is all that original: who
wouldn't want their data sctructure size determined up-front? :-) (not to
mention getting header-overwriting for-free as part of the single copy.
BTW, TSO is not a intelligent as what you are suggesting.
If i am not mistaken you are not only suggesting fragmentation and
assembly at that level you are also suggesting retransmits at the NIC.
This could be dangerous for practical reasons (changes in TCP congestion
control algorithms etc). TSO as was pointed in earlier emails is just a
dumb sender of packets. I think even fragmentation is a misnomer.
Essentially you shove a huge buffer to the NIC and it breaks it into MTU
sized packets for you and sends them.
the biggest problem to our approach is that itis extremely difficult to
mix two very different kinds of workloads together: the regular
server-on-the-internet workload (SOI) and the large-cluster-member
workload (LCM). in the former case, SOI, you get dropped packets,
fragments, no fragments, out of order fragments, etc. in the LCM case you
basically never get any of that stuff--you're on a closed network with
1000-10000 of your closest cluster friends and that's just what you're
doing. no fragments (unless you put them there), no out of order
fragments (unless you send them) and basically no dropped packets ever.
obviously, if you can assume conditions like that, you can do things like:
only reassmble fragments in reverse order since you know you'll only send
them that way, e.g.
In regards to the receive side CPU utilization improvements: I think
that NAPI does a good job at least in getting ridding of the biggest
offender -- interupt overload. Also with NAPI also having got rid of
intermidiate queues to the socket level, facilitating of zero copy receive
should be relatively easy to add but there are no capable NICs in
existence (well, ok not counting the TIGONII/acenic that you can hack
and the fact that the tigon 2 is EOL doesnt help other than just for
experiments). I dont think theres any NIC that can offload reassembly;
that might not be such a bad idea.
i've done some reading about NAPI just recently (somehow i missed the
splash when it came out). the two things i like about it are the hardware
independent interrupt mitigation technique and using the DMA buffers as a
receive backlog. i'm concerned about the numbers posted by ibm folx
recently showing a slowdown under some conditions using NAPI and need to
read the rest of that discussion.
we are definitely aware of the fact that the more you want to put on the
NIC, the more the NIC will have to do (and the more expensive it will have
to be). right now the NICs, that people are developing on are the
TigonII/III and, even more closed/proprietary, the Myrinet NICs. i would
love to have a <$200 NIC with open firmware and a CPU/memory so that we
could offload some more of this functionality (where it makes sense).
Are you still continuing work on this?
definitely! we were just talking about some of these issues yesterday
(and trying to find hardware sepc info on the web for the e1000 platform
to see what else they might be able to do). patricia gilfeather is working
on finding parts of TCP that are separable from the rest of TCP, but the
problems you raise are serious: it would have to be on an
application-specific and socket-specific basis, so that the app would
*know* that functionality (like acks for synchronization packets or
whatever) was being offloaded.
the biggest difference in our perspective, versus the common kernel
developers, is that we're still looking for ways to get the OS out of the
way of the applications. if we can do large data transfers (with
pre-posted receives and pre-posted memory allocation, obviously) directly
from the nic into application memory and have a clean, relatively simple
and standard api to do that, we avoid all of the interrupt mitigation
techniques and save hugely on context switching overhead.
this may now be off-topic for linux-kernel and i'd be happy to chat
further in private email if others are getting bored :-).
cheers,
jamal
t.
--
todd underwood, vp & cto
oso grande technologies, inc.
todd@osogrande.com
"Those who give up essential liberties for temporary safety deserve
neither liberty nor safety." - Benjamin Franklin
On Thu, 2002-09-12 at 14:57, Todd Underwood wrote:
thanks. although i'd love to take credit, i don't think that the
reverse-order fragmentation appreciation is all that original: who
wouldn't want their data sctructure size determined up-front? :-) (not to
mention getting header-overwriting for-free as part of the single copy.
As far as I am aware it was original when Linux first did it (and we
broke cisco pix, some boot proms, some sco in the process). Credit goes
to Arnt Gulbrandsen probably better known nowdays for his work on Qt
alan,
good to know. it's a nice piece of engineering. it's useful to note that
linux has such a long and rich history of breaking de-facto standards in
order to make things work better.
t.
On 12 Sep 2002, Alan Cox wrote:
On Thu, 2002-09-12 at 14:57, Todd Underwood wrote:
quoted
thanks. although i'd love to take credit, i don't think that the
reverse-order fragmentation appreciation is all that original: who
wouldn't want their data sctructure size determined up-front? :-) (not to
mention getting header-overwriting for-free as part of the single copy.
As far as I am aware it was original when Linux first did it (and we
broke cisco pix, some boot proms, some sco in the process). Credit goes
to Arnt Gulbrandsen probably better known nowdays for his work on Qt
--
todd underwood, vp & cto
oso grande technologies, inc.
todd@osogrande.com
"Those who give up essential liberties for temporary safety deserve
neither liberty nor safety." - Benjamin Franklin
sorry for the late reply. catching up on kernel mail.
the main different, though, is that general purpose kernel
development still focussed on the improvements in *sending* speed.
for real high performance networking, the improvements are necessary
in *receiving* cpu utilization, in our estimation.
(see our analysis of interrupt overhead and the effect on receivers
at gigabit speeds--i hope that this has become common understanding
by now)
Some of that may be a byproduct of the "all the worlds' a webserver"
mindset - we are primarily focussed on the server side (aka
money side ;)), and there is some amount of automatic thinking that
this means we're going to be sending data and receiving small packets,
mostly acks) in return. There is much less emphasis given to solving
the problems on the other side (active connection scalability for
instance), or other issues that manifest themselves as
client side bottlenecks for most applications..
thanks,
Nivedita
From: David S. Miller <hidden> Date: 2002-09-12 23:15:51
From: jamal [off-list ref]
Date: Thu, 12 Sep 2002 08:30:44 -0400 (EDT)
In regards to the receive side CPU utilization improvements: I think
that NAPI does a good job at least in getting ridding of the biggest
offender -- interupt overload.
I disagree, at least for bulk receivers. We have no way currently to
get rid of the data copy. We desperately need sys_receivefile() and
appropriate ops all the way into the networking, then the necessary
driver level support to handle the cards that can do this.
Once 10gbit cards start hitting the shelves this will convert from a
nice perf improvement into a must have.
dave, all,
On Thu, 12 Sep 2002, David S. Miller wrote:
I disagree, at least for bulk receivers. We have no way currently to
get rid of the data copy. We desperately need sys_receivefile() and
appropriate ops all the way into the networking, then the necessary
driver level support to handle the cards that can do this.
not sure i understand what you're proposing, but while we're at it, why
not also make the api for apps to allocate a buffer in userland that (for
nics that support it) the nic can dma directly into? it seems likely
notification that the buffer was used would have to travel through the
kernel, but it would be nice to save the interrupts altogether.
this may be exactly what you were saying.
Once 10gbit cards start hitting the shelves this will convert from a
nice perf improvement into a must have.
totally agreed. this is a must for high-performance computing now (since
who wants to waste 80-100% of their CPU just running the network)?
t.
--
todd underwood, vp & cto
oso grande technologies, inc.
todd@osogrande.com
"Those who give up essential liberties for temporary safety deserve
neither liberty nor safety." - Benjamin Franklin
From: David S. Miller <hidden> Date: 2002-09-13 22:08:11
From: todd-lkml@osogrande.com
Date: Fri, 13 Sep 2002 15:59:15 -0600 (MDT)
not sure i understand what you're proposing
Cards in the future at 10gbit and faster are going to provide
facilities by which:
1) You register a IPV4 src_addr/dst_addr TCP src_port/dst_port cookie
with the hardware when TCP connections are openned.
2) The card scans TCP packets arriving, if the cookie matches, it
accumulated received data to fill full pages and wakes up the
networking when either:
a) full page has accumulated for a connection
b) connection cookie mismatch
c) configurable timer has expired
3) TCP ends up getting receive packets with skb->shinfo() fraglist
containing the data portion in full struct page *'s
This can be placed directly into the page cache via sys_receivefile
generic code in mm/filemap.c or f.e. NFSD/NFS receive side
processing.
not also make the api for apps to allocate a buffer in userland that (for
nics that support it) the nic can dma directly into? it seems likely
notification that the buffer was used would have to travel through the
kernel, but it would be nice to save the interrupts altogether.
This is already doable with sys_sendfile() for send today. The user
just does the following:
1) mmap()'s a file with MAP_SHARED to write the data
2) uses sys_sendfile() to send the data over the socket from that file
3) uses socket write space monitoring to determine if the portions of
the shared area are reclaimable for new writes
BTW Apache could make this, I doubt it does currently.
The corrolary with sys_receivefile would be that the use:
1) mmap()'s a file with MAP_SHARED to write the data
2) uses sys_receivefile() to pull in the data from the socket to that file
There is no need to poll the receive socket space as the successful
return from sys_receivefile() is the "data got received successfully"
event.
totally agreed. this is a must for high-performance computing now (since
who wants to waste 80-100% of their CPU just running the network)?
If send side is your bottleneck and you think zerocopy sends of
user anonymous data might help, see the above since we can do it
today and you are free to experiment.
Franks a lot,
David S. Miller
davem@redhat.com
dave, all,
not sure i understand what you're proposing, but while we're at it,
why not also make the api for apps to allocate a buffer in userland
that (for nics that support it) the nic can dma directly into? it
I believe thats exactly what David was referring to - reverse
direction sendfile() so to speak..
seems likely notification that the buffer was used would have to
travel through the kernel, but it would be nice to save the
interrupts altogether.
However, I dont think what youre saving are interrupts as
much as the extra copy, but I could be wrong..
thanks,
Nivedita
10 gige becomes more of an interesting beast. Not sure if we would see
servers with 10gige real soon now. Your proposal does make sense although
compute power would still be a player. I think the key would be
parallelization;
Now if it wasnt for the stupid way TCP options were designed
you could easily do remote DMA instead. Would be relatively easy to add
NIC support for that. Maybe SCTP would save us ;-> however, if history
could be used to predict the future, i think TCP will continue to be
"hacked" and fit the throughput requirements so no chance for SCTP to be
a big player i am afraid .
cheers,
jamal
On Fri, 13 Sep 2002, David S. Miller wrote:
From: todd-lkml@osogrande.com
Date: Fri, 13 Sep 2002 15:59:15 -0600 (MDT)
not sure i understand what you're proposing
Cards in the future at 10gbit and faster are going to provide
facilities by which:
1) You register a IPV4 src_addr/dst_addr TCP src_port/dst_port cookie
with the hardware when TCP connections are openned.
From: David S. Miller <hidden> Date: 2002-09-16 04:27:12
From: jamal [off-list ref]
Date: Sun, 15 Sep 2002 16:16:13 -0400 (EDT)
Your proposal does make sense although compute power would still be
a player. I think the key would be parallelization;
Oh I forgot to mention that some of these cards also compute a cookie
for you on receive packets, and your meant to point the input
processing for that packet to a cpu whose number is derived from that
cookie it gives you.
Lockless per-cpu packet input queues make this sort of hard for us
to implement currently.
david,
comments/questions below...
On Fri, 13 Sep 2002, David S. Miller wrote:
1) You register a IPV4 src_addr/dst_addr TCP src_port/dst_port cookie
with the hardware when TCP connections are openned.
intriguing architecture. are there any standards in progress to support
this. bascially, people doing high performance computing have been
customizing non-commodity nics (acenic, myrinet, quadrics, etc.) to do
some of this cookie registration/scanning. it would be nice if there were
a standard API/hardware capability that took care of at least this piece.
(frankly, it would also be nice if customizable, almost-commodity nics
based on processor/memory/firmware architecture rather than just asics
(like the acenic) continued to exist).
not also make the api for apps to allocate a buffer in userland that (for
nics that support it) the nic can dma directly into? it seems likely
notification that the buffer was used would have to travel through the
kernel, but it would be nice to save the interrupts altogether.
This is already doable with sys_sendfile() for send today. The user
just does the following:
1) mmap()'s a file with MAP_SHARED to write the data
2) uses sys_sendfile() to send the data over the socket from that file
3) uses socket write space monitoring to determine if the portions of
the shared area are reclaimable for new writes
BTW Apache could make this, I doubt it does currently.
The corrolary with sys_receivefile would be that the use:
1) mmap()'s a file with MAP_SHARED to write the data
2) uses sys_receivefile() to pull in the data from the socket to that file
There is no need to poll the receive socket space as the successful
return from sys_receivefile() is the "data got received successfully"
event.
the send case has been well described and seems work well for the people
for whom that is the bottleneck. that has not been the case in HPC, since
sends are relatively cheaper (in terms of cpu) than receives.
who is working on this architecture for receives? i know quite a few
people who would be interested in working on it and willing to prototype
as well.
totally agreed. this is a must for high-performance computing now (since
who wants to waste 80-100% of their CPU just running the network)?
If send side is your bottleneck and you think zerocopy sends of
user anonymous data might help, see the above since we can do it
today and you are free to experiment.
for many of the applications that i care about, receive is the bottleneck,
so zerocopy sends are somewhat of a non-issue (not that they're not nice,
they just don't solve the primary waste of processor resources).
is there a beginning implementation yet of zerocopy receives as you
describe above, or you you be interested in entertaining implementations
that work on existing (1Gig-e) cards?
what i'm thinking is something that prototypes the api to the nic that you
are proposing and implements the NIC-side functionality in firmware on the
acenic-2's (which have available firmware in at least two
implementations--the alteon version and pete wyckoff's version (which may
be less license-encumbered).
this is obviously only feasible if there already exists some consensus on
what the os-to-hardware API should look like (or there is willingness to
try to build a consensus around that now).
t.
--
todd underwood, vp & cto
oso grande technologies, inc.
todd@osogrande.com
"Those who give up essential liberties for temporary safety deserve
neither liberty nor safety." - Benjamin Franklin
From: David S. Miller <hidden> Date: 2002-09-16 19:56:02
From: todd-lkml@osogrande.com
Date: Mon, 16 Sep 2002 08:16:47 -0600 (MDT)
are there any standards in progress to support this.
Your question makes no sense, it is a hardware optimization
of an existing standard. The chip merely is told what flows
exist and it concatenates TCP data from consequetive packets
for that flow if they arrive in sequence.
who is working on this architecture for receives?
Once cards with the feature exist, probably Alexey and myself
will work on it.
Basically, who ever isn't busy with something else once the technology
appears.
is there a beginning implementation yet of zerocopy receives
No.
Franks a lot,
David S. Miller
davem@redhat.com
folx,
perhaps i was insufficiently clear.
On Mon, 16 Sep 2002, David S. Miller wrote:
are there any standards in progress to support this.
Your question makes no sense, it is a hardware optimization
of an existing standard. The chip merely is told what flows
exist and it concatenates TCP data from consequetive packets
for that flow if they arrive in sequence.
hardware optimizations can be standardized. in fact, when they are, it is
substantially easier to implement to them.
my assumption (perhaps incorrect) is that some core set of functionality
is necessary for a card to support zero-copy receives (in particular, the
ability to register cookies of expected data flows and the memory location
to which they are to be sent). what 'existing standard' is this
kernel<->api a standardization of?
who is working on this architecture for receives?
Once cards with the feature exist, probably Alexey and myself
will work on it.
Basically, who ever isn't busy with something else once the technology
appears.
so if we wrote and distributed firmware for alteon acenics that supported
this today, you would be willing to incorporate the new system calls into
the networking code (along with the new firmware for the card, provided we
could talk jes into accepting the changes, assuming he's still the
maintainer of the driver)? that's great.
is there a beginning implementation yet of zerocopy receives
No.
thanks for your feedback.
t.
--
todd underwood, vp & cto
oso grande technologies, inc.
todd@osogrande.com
"Those who give up essential liberties for temporary safety deserve
neither liberty nor safety." - Benjamin Franklin
From: David S. Miller <hidden> Date: 2002-09-16 21:33:25
From: todd-lkml@osogrande.com
Date: Mon, 16 Sep 2002 15:32:56 -0600 (MDT)
new system calls into the networking code
The system calls would go into the VFS, sys_receivefile is not
networking specific in any way shape or form.
And to answer your question, if I had the time I'd work on it yes.
Right now the answer to "well do you have the time" is no, I am
working on something much more important wrt. Linux networking. I've
hinted at what this is in previous postings, and if people can't
figure out what it is I'm not going to mention this explicitly :-)
From: David S. Miller <hidden> Date: 2002-09-16 22:50:38
From: David Woodhouse [off-list ref]
Date: Mon, 16 Sep 2002 23:53:00 +0100
Er, surely the same goes for sys_sendfile? Why have a new system call
rather than just swapping the 'in' and 'out' fds?
There is an assumption that one is a linear stream of output (in this
case a socket) and the other one is a page cache based file.
It would be nice to extend sys_sendfile to work properly in both
ways in a manner that Linus would accept, want to work on that?
From: David Woodhouse <dwmw2@infradead.org> Date: 2002-09-16 22:58:24
davem@redhat.com said:
quoted
Er, surely the same goes for sys_sendfile? Why have a new system
call rather than just swapping the 'in' and 'out' fds?
There is an assumption that one is a linear stream of output (in this
case a socket) and the other one is a page cache based file.
That's an implementation detail and it's not clear we should be exposing it
to the user. It's not entirely insane to contemplate socket->socket or
file->file sendfile either -- would we invent new system calls for those
too? File descriptors are file descriptors.
It would be nice to extend sys_sendfile to work properly in both ways
in a manner that Linus would accept, want to work on that?
Yeah -- I'll add it to the TODO list. Scheduled for some time in 2007 :)
More seriously though, I'd hope that whoever implemented what you call
'sys_receivefile' would solve this issue, as 'sys_receivefile' isn't really
useful as anything more than a handy nomenclature for describing the
process in question.
--
dwmw2
From: Jeff Garzik <hidden> Date: 2002-09-16 23:03:51
David Woodhouse wrote:
davem@redhat.com said:
quoted
quoted
Er, surely the same goes for sys_sendfile? Why have a new system
call rather than just swapping the 'in' and 'out' fds?
quoted
There is an assumption that one is a linear stream of output (in this
case a socket) and the other one is a page cache based file.
That's an implementation detail and it's not clear we should be exposing it
to the user. It's not entirely insane to contemplate socket->socket or
file->file sendfile either -- would we invent new system calls for those
too? File descriptors are file descriptors.
I was rather disappointed when file->file sendfile was [purposefully?]
broken in 2.5.x...
Jeff
From: David S. Miller <hidden> Date: 2002-09-16 23:06:09
From: Jeff Garzik [off-list ref]
Date: Mon, 16 Sep 2002 19:08:15 -0400
I was rather disappointed when file->file sendfile was [purposefully?]
broken in 2.5.x...
What change made this happen?
From: Jeff Garzik <hidden> Date: 2002-09-16 23:44:13
David S. Miller wrote:
From: Jeff Garzik [off-list ref]
Date: Mon, 16 Sep 2002 19:08:15 -0400
I was rather disappointed when file->file sendfile was [purposefully?]
broken in 2.5.x...
What change made this happen?
I dunno when it happened, but 2.5.x now returns EINVAL for all
file->file cases.
In 2.4.x, if sendpage is NULL, file_send_actor in mm/filemap.c faked a
call to fops->write().
In 2.5.x, if sendpage is NULL, EINVAL is unconditionally returned.
From: David S. Miller <hidden> Date: 2002-09-16 23:47:45
From: Jeff Garzik [off-list ref]
Date: Mon, 16 Sep 2002 19:48:37 -0400
I dunno when it happened, but 2.5.x now returns EINVAL for all
file->file cases.
In 2.4.x, if sendpage is NULL, file_send_actor in mm/filemap.c faked a
call to fops->write().
In 2.5.x, if sendpage is NULL, EINVAL is unconditionally returned.
What if source and destination file and offsets match?
Sounds like 2.4.x might deadlock.
In fact it sounds similar to the "read() with buf pointed to same
page in MAP_WRITE mmap()'d area" deadlock we had ages ago.
From: Jeff Garzik <hidden> Date: 2002-09-16 23:57:00
David S. Miller wrote:
From: Jeff Garzik [off-list ref]
Date: Mon, 16 Sep 2002 19:48:37 -0400
I dunno when it happened, but 2.5.x now returns EINVAL for all
file->file cases.
In 2.4.x, if sendpage is NULL, file_send_actor in mm/filemap.c faked a
call to fops->write().
In 2.5.x, if sendpage is NULL, EINVAL is unconditionally returned.
What if source and destination file and offsets match?
The same data is written out. No deadlock.
(unless the attached test is wrong)
Jeff
From: todd-lkml@osogrande.com
Date: Mon, 16 Sep 2002 08:16:47 -0600 (MDT)
are there any standards in progress to support this.
Your question makes no sense, it is a hardware optimization
of an existing standard. The chip merely is told what flows
exist and it concatenates TCP data from consequetive packets
for that flow if they arrive in sequence.
Hrm. Again, the big Q:
How "thmart" is this NIC going to be (think congestion control and
the du-jour flavor).
cheers,
jamal