From: Werner Almesberger <hidden> Date: 2003-08-02 21:49:27
Jeff Garzik wrote:
jabbering at the same time. TCP is a "one size fits all" solution, but
it doesn't work well for everyone.
But then, ten "optimized xxPs" that work well in two different
scenarios each, but not so good in the 98 others, wouldn't be
much fun either.
It's been tried a number of times. Usually, real life sneaks
in at one point or another, leaving behind a complex mess.
When they've sorted out these problems, regular TCP has caught
up with the great optimized transport protocols. At that point,
they return to their niche, sometimes tail between legs and
muttering curses, sometimes shaking their fist and boldly
proclaiming how badly they'll rub TCP in the dirt in the next
round. Maybe they shed off some of the complexity, and trade it
for even more aggressive optimization, which puts them into
their niche even more firmly. Eventually, they fade away.
There are cases where TCP doesn't work well, like a path of
badly mismatched link layers, but such paths don't treat any
protocol following the end-to-end principle kindly.
Another problem of TCP is that it has grown a bit too many
knobs you need to turn before it works over your really fast
really long pipe. (In one of the OLS after dinner speeches,
this was quite appropriately called the "wizard gap".)
It's obviously not over a WAN...
That's why NFS turned off UDP checksums ;-) As soon as you put
it on IP, it will crawl to distances you didn't imagine in your
wildest dreams. It always does.
So, fix the other end of the pipeline too, otherwise this fast network
stuff is flashly but pointless. If you want to serve up data from disk,
then start creating PCI cards that have both Serial ATA and ethernet
connectors on them :) Cut out the middleman of the host CPU and host
memory bus instead of offloading portions of TCP that do not need to be
offloaded.
That's a good point. A hierarchical memory structure can help
here. Moving one end closer to the hardware, and letting it
know (e.g. through sendfile) that also the other end is close
(or can be reached more directly that through some hopelessly
crowded main bus) may help too.
- Werner
--
_________________________________________________________________________
/ Werner Almesberger, Buenos Aires, Argentina werner@almesberger.net /
/_http://www.almesberger.net/____________________________________________/
From: Jeff Garzik <hidden> Date: 2003-08-03 06:40:48
Werner Almesberger wrote:
Jeff Garzik wrote:
quoted
jabbering at the same time. TCP is a "one size fits all" solution, but
it doesn't work well for everyone.
But then, ten "optimized xxPs" that work well in two different
scenarios each, but not so good in the 98 others, wouldn't be
much fun either.
It's been tried a number of times. Usually, real life sneaks
in at one point or another, leaving behind a complex mess.
When they've sorted out these problems, regular TCP has caught
up with the great optimized transport protocols. At that point,
they return to their niche, sometimes tail between legs and
muttering curses, sometimes shaking their fist and boldly
proclaiming how badly they'll rub TCP in the dirt in the next
round. Maybe they shed off some of the complexity, and trade it
for even more aggressive optimization, which puts them into
their niche even more firmly. Eventually, they fade away.
There are cases where TCP doesn't work well, like a path of
badly mismatched link layers, but such paths don't treat any
protocol following the end-to-end principle kindly.
Another problem of TCP is that it has grown a bit too many
knobs you need to turn before it works over your really fast
really long pipe. (In one of the OLS after dinner speeches,
this was quite appropriately called the "wizard gap".)
quoted
It's obviously not over a WAN...
That's why NFS turned off UDP checksums ;-) As soon as you put
it on IP, it will crawl to distances you didn't imagine in your
wildest dreams. It always does.
Really fast, really long pipes in practice don't exist for 99.9% of all
Internet users.
When you approach traffic levels that push you want to offload most of
the TCP net stack, then TCP isn't the right solution for you anymore,
all things considered.
The Linux net stack just isn't built to be offloaded. TOE engines will
either need to (1) fall back to Linux software for all-but-the-common
case (otherwise netfilter, etc. break), or, (2) will need to be
hideously complex beasts themselves. And I can't see ASIC and firmware
designers being excited about implementing netfilter on a PCI card :)
Unfortunately some vendors seem to choosing TOE option #3: TCP offload
which introduces many limitations (connection limits, netfilter not
supported, etc.) which Linux never had before. Vendors don't seem to
realize TOE has real potential to damage the "good network neighbor"
image the net stack has. The Linux net stack's behavior is known,
documented, predictable. TOE changes all that.
There is one interesting TOE solution, that I have yet to see created:
run Linux on an embedded processor, on the NIC. This stripped-down
Linux kernel would perform all the header parsing, checksumming, etc.
into the NIC's local RAM. The Linux OS driver interface becomes a
virtual interface with a large MTU, that communicates from host CPU to
NIC across the PCI bus using jumbo-ethernet-like data frames.
Management frames would control the ethernet interface on the other side
of the PCI bus "tunnel".
quoted
So, fix the other end of the pipeline too, otherwise this fast network
stuff is flashly but pointless. If you want to serve up data from disk,
then start creating PCI cards that have both Serial ATA and ethernet
connectors on them :) Cut out the middleman of the host CPU and host
memory bus instead of offloading portions of TCP that do not need to be
offloaded.
That's a good point. A hierarchical memory structure can help
here. Moving one end closer to the hardware, and letting it
know (e.g. through sendfile) that also the other end is close
(or can be reached more directly that through some hopelessly
crowded main bus) may help too.
From: Eric W. Biederman <hidden> Date: 2003-08-03 19:24:36
Werner Almesberger [off-list ref] writes:
Jeff Garzik wrote:
quoted
jabbering at the same time. TCP is a "one size fits all" solution, but
it doesn't work well for everyone.
But then, ten "optimized xxPs" that work well in two different
scenarios each, but not so good in the 98 others, wouldn't be
much fun either.
The optimized for low latency cases seem to have a strong
market in clusters. And they are currently keeping alive
quite a few technologies. Myrinet, Infiniband, Quadric's Elan, etc.
Having low latency and switch technologies that scale is quite
rare currently.
Another problem of TCP is that it has grown a bit too many
knobs you need to turn before it works over your really fast
really long pipe. (In one of the OLS after dinner speeches,
this was quite appropriately called the "wizard gap".)
Does anyone know which knobs to turn to make TCP go fast over
Infiniband. (A low latency high bandwidth network?) I get to
deal with them on a regular basis...
There is one place in low latency communications that I can think
of where TCP/IP is not the proper solution. For low latency
communication the checksum is at the wrong end of the packet.
IB gets this one correct and places the checksum at the tail end of
the packet. This allows the packet to start transmitting before
the checksum is computed, possibly even having the receive start
at the other end before the tail of the packet is transmitted.
Would it make any sense to do a low latency variation on TCP that
fixes that problem? For the IP header we are fine as the data
precedes the checksum. But the problem appears to affect all
of the upper level protocols that ride on IP, UDP, TCP, SCTP...
quoted
So, fix the other end of the pipeline too, otherwise this fast network
stuff is flashly but pointless. If you want to serve up data from disk,
then start creating PCI cards that have both Serial ATA and ethernet
connectors on them :) Cut out the middleman of the host CPU and host
memory bus instead of offloading portions of TCP that do not need to be
offloaded.
That's a good point. A hierarchical memory structure can help
here. Moving one end closer to the hardware, and letting it
know (e.g. through sendfile) that also the other end is close
(or can be reached more directly that through some hopelessly
crowded main bus) may help too.
On that score it is worth noting that the next generation of
peripheral busses (Hypertransport, PCI Express, etc) are all switched.
Which means that device to device communication may be more
reasonable. Going from a bussed interconnect to a switched
interconnect is certainly a dramatic change in infrastructure. How
that will affect the tradeoffs I don't know.
Eric
From: Andre Hedrick <hidden> Date: 2003-08-06 07:23:11
Jeff,
Do be sure to check that your data payload is correct.
Everyone knows that a router/gateway/switch with a sticky bit in its
memory will recompute the net crc16 checksum insure it pass the to the nic
regardless. It is amazing how much data can be corrupted by a network
environment via all the NFS/NBD/etc wantabie storage products out there.
Just a chuckle for you to ponder.
--a
On Sun, 3 Aug 2003, Jeff Garzik wrote:
Werner Almesberger wrote:
quoted
Jeff Garzik wrote:
quoted
jabbering at the same time. TCP is a "one size fits all" solution, but
it doesn't work well for everyone.
But then, ten "optimized xxPs" that work well in two different
scenarios each, but not so good in the 98 others, wouldn't be
much fun either.
It's been tried a number of times. Usually, real life sneaks
in at one point or another, leaving behind a complex mess.
When they've sorted out these problems, regular TCP has caught
up with the great optimized transport protocols. At that point,
they return to their niche, sometimes tail between legs and
muttering curses, sometimes shaking their fist and boldly
proclaiming how badly they'll rub TCP in the dirt in the next
round. Maybe they shed off some of the complexity, and trade it
for even more aggressive optimization, which puts them into
their niche even more firmly. Eventually, they fade away.
There are cases where TCP doesn't work well, like a path of
badly mismatched link layers, but such paths don't treat any
protocol following the end-to-end principle kindly.
Another problem of TCP is that it has grown a bit too many
knobs you need to turn before it works over your really fast
really long pipe. (In one of the OLS after dinner speeches,
this was quite appropriately called the "wizard gap".)
quoted
It's obviously not over a WAN...
That's why NFS turned off UDP checksums ;-) As soon as you put
it on IP, it will crawl to distances you didn't imagine in your
wildest dreams. It always does.
Really fast, really long pipes in practice don't exist for 99.9% of all
Internet users.
When you approach traffic levels that push you want to offload most of
the TCP net stack, then TCP isn't the right solution for you anymore,
all things considered.
The Linux net stack just isn't built to be offloaded. TOE engines will
either need to (1) fall back to Linux software for all-but-the-common
case (otherwise netfilter, etc. break), or, (2) will need to be
hideously complex beasts themselves. And I can't see ASIC and firmware
designers being excited about implementing netfilter on a PCI card :)
Unfortunately some vendors seem to choosing TOE option #3: TCP offload
which introduces many limitations (connection limits, netfilter not
supported, etc.) which Linux never had before. Vendors don't seem to
realize TOE has real potential to damage the "good network neighbor"
image the net stack has. The Linux net stack's behavior is known,
documented, predictable. TOE changes all that.
There is one interesting TOE solution, that I have yet to see created:
run Linux on an embedded processor, on the NIC. This stripped-down
Linux kernel would perform all the header parsing, checksumming, etc.
into the NIC's local RAM. The Linux OS driver interface becomes a
virtual interface with a large MTU, that communicates from host CPU to
NIC across the PCI bus using jumbo-ethernet-like data frames.
Management frames would control the ethernet interface on the other side
of the PCI bus "tunnel".
quoted
quoted
So, fix the other end of the pipeline too, otherwise this fast network
stuff is flashly but pointless. If you want to serve up data from disk,
then start creating PCI cards that have both Serial ATA and ethernet
connectors on them :) Cut out the middleman of the host CPU and host
memory bus instead of offloading portions of TCP that do not need to be
offloaded.
That's a good point. A hierarchical memory structure can help
here. Moving one end closer to the hardware, and letting it
know (e.g. through sendfile) that also the other end is close
(or can be reached more directly that through some hopelessly
crowded main bus) may help too.