TOE brain dump

At OLS, there was a bit of discussion on (true and false *) TOEs
(TCP Offload Engines). In the course of this discussion, I've
suggested what might be a novel approach, so in case this is a
good idea, I'd like to dump my thoughts on it, before someone
tries to patent my ideas. (Most likely, some of this has already
been done or tried elsewhere, but it can't hurt to try to err on
the safe side.)

(*) The InfiniBand people unfortunately call also their TCP/IP
    bypass "TOE" (for which they promptly get shouted down,
    every time they use that word). This is misleading, because
    there is no TCP that's getting offloaded, but TCP is simply
    never done. I would consider it to be more accurate to view
    this as a separate networking technology, with semantics
    different from TCP/IP, similar to ATM and AAL5.

While I'm not entirely convinced about the usefulness of TOE in
all the cases it's been suggested for, I can see value in certain
areas, e.g. when TCP per-packet overhead becomes an issue.

However, I consider the approach of putting a new or heavily
modified stack, which duplicates a considerable amount of the
functionality in the main kernel, on a separate piece of hardware
questionable at best. Some of the issues:

 - if this stack is closed source or generally hard to modify,
   security fixes will be slowed down

 - if this stack is closed source or generally hard to modify,
   TOE will not be available to projects modifying the stack,
   e.g. any of the research projects trying to make TCP work at
   gigabit speeds

 - this stack either needs to implement all administrative
   interfaces of the regular kernel, or such a system would have
   non-uniform configuration/monitoring across interfaces

 - in some cases, administrative interfaces will require a
   NIC/TOE-specific switch in the kernel (netlink helps here)

 - route changes on multi-homed hosts (or any similar kind of
   failover) are difficult if the state of TCP connections is
   tied to specific NICs (I've discussed some issues when
   "migrating" TCP connections in the documentation of tcpcp,
   http://www.almesberger.net/tcpcp/)

 - new kernel features will always lag behind on this kind of
   TOE, and different kernels will require different "firmware"

 - last but not least, keeping TOE firmware up to date with the
   TCP/IP stack in the mainstream kernel will require - for each
   such TOE device - a significant and continuous effort over a
   long period of time

In short, I think such a solution is either a pain to use, or
unmaintainable, or - most likely - both.

So, how to do better ? Easy: use the Source, Luke. Here's my
idea:

 - instead of putting a different stack on the TOE, a
   general-purpose processor (probably with some enhancements,
   and certainly with optimized data paths) is added to the NIC

 - that processor runs the same Linux kernel image as the host,
   acting like a NUMA system

 - a selectable part of TCP/IP is handled on the NIC, and the
   rest of the system runs on the host processor

 - instrumentation is added to the mainstream kernel to ensure
   that as little data as possible is shared between the main
   CPU and such peripheral CPUs. Note that such instrumentation
   would be generic, outlining possible boundaries, and not tied
   to a specific TOE design.

 - depending on hardware details (cache coherence, etc.), the
   instrumentation mentioned above may even be necessary for
   correctness. This would have the unfortunate effect of making
   the design very fragile with respect to changes in the
   mainstream kernel. (Performance loss in the case of imperfect
   instrumentation would be preferable.)

 - further instrumentation may be needed to let the kernel switch
   CPUs (i.e. host to NIC, and vice versa) at the right time

 - since the NIC would probably use a CPU design different from
   the host CPU, we'd need "fat" kernel binaries:

   - data structures are the same, i.e. word sizes, byte order,
     bit numbering, etc. are compatible, and alignments are
     chosen such that all CPUs involved are reasonably happy

   - kernels live in the same address space

   - function pointers become arrays, with one pointer per
     architecture. When comparing pointers, the first element is
     used.

 - if one should choose to also run parts of user space on the
   NIC, fat binaries would also be needed for this (along with
   other complications)

Benefits:

 - putting the CPU next to the NIC keeps data paths short, and
   allows for all kinds of optimizations (e.g. a pipelined
   memory architecture)

 - the design is fairly generic, and would equally apply to
   other areas of the kernel than TCP/IP

 - using the same kernel image eliminates most maintenance
   problems, and encourages experimenting with the stack

 - using the same kernel image (and compatible data structures)
   guarantees that administrative interfaces are uniform in the
   entire system

 - such a design is likely to be able to allow TCP state to be
   moved to a different NIC, if necessary

Possible problems, that may kill this idea:

 - it may be too hard to achieve correctness

 - it may be too hard to switch CPUs properly

 - it may not be possible to express copy operations efficiently
   in such a context

 - there may be no way to avoid sharing of hardware-specific
   data structures, such as page tables, or to emulate their use

 - people may consider the instrumentation required for this,
   although fairly generic, too intrusive

 - all this instrumentation may eat too much performance

 - nobody may be interested in building hardware for this

 - nobody may be patient enough to pursue such long-termish
   development, with uncertain outcome

 - something I haven't thought of

I lack the resources (hardware, financial, and otherwise) to
actually do something with these ideas, so please feel free to
put them to some use.

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina     werner@almesberger.net /
/_http://www.almesberger.net/____________________________________________/

Werner Almesberger wrote:

(*) The InfiniBand people unfortunately call also their TCP/IP
    bypass "TOE" (for which they promptly get shouted down,
    every time they use that word). This is misleading, because

Thank you! Yes! All in favor say Aye..AYE!!! Motion passes,
the infiniband people don't get to call it TOE anymore..

While I'm not entirely convinced about the usefulness of TOE in
all the cases it's been suggested for, I can see value in certain
areas, e.g. when TCP per-packet overhead becomes an issue.

Ditto, but I see it being used to rollout the idea and process,
rather than anything of value now, and the lessons are being
learned for the future, when we reach 20Gb, 40Gb, even faster
networks of tommorow. The processors might keep up, but nothing
else will, for sure.

However, I consider the approach of putting a new or heavily
modified stack, which duplicates a considerable amount of the
functionality in the main kernel, on a separate piece of hardware
questionable at best. Some of the issues:

 - if this stack is closed source or generally hard to modify,
   security fixes will be slowed down

as will bug fixes, and debugging becomes a right royal pain.

Also, most profiles of networking applications show the
largest blip is essentially the user<->kernel transfer, and
that would still remain the unaddressed bottleneck.

So, how to do better ? Easy: use the Source, Luke. Here's my
idea:

 - instead of putting a different stack on the TOE, a
   general-purpose processor (probably with some enhancements,
   and certainly with optimized data paths) is added to the NIC

The thing is, all the TOE efforts are propietary ones, to
my limited knowledge. Thus all the design is occurring in
confidential, vendor internal forums. How will they/we
come up with really the needed, _common_ design approach?

Or is this not so needed?

thanks,
Nivedita

My own brain dump:

If one wants to go straight from disk to network, why is anyone 
bothering to involve the host CPU and host memory bus at all?  Memory 
bandwidth and PCI bus bandwidth are still bottlenecks, no much how much 
of the net stack you offload.


Regardless of how fast your network zooms packets, you've gotta keep 
that pipeline full to make use of it.  And you've gotta do something 
intelligent with it, which in TCP's case involves the host CPU quite a 
bit.  TCP is sufficiently complex, for a reason.  It has to handle all 
manner of disturbingly slow and disturbing fast net connections, all 
jabbering at the same time.  TCP is a "one size fits all" solution, but 
it doesn't work well for everyone.

The "TCP Offload Everything" people really need to look at what data 
your users want to push, at such high speeds.  It's obviously not over a 
WAN...  so steer users away from TCP, to an IP protocol that is tuned 
for your LAN needs, and more friendly to some sort of h/w offloading 
solution.

A "foo over ipv6" protocol that was designed for h/w offloading from the 
start, would be a far better idea than full TCP offload will ever be.

In any case, when you approach these high speeds, you really must take a 
good look at the other end of the pipeline:  what are you serving at 
10Gb/s, 20Gb/s, 40Gb/s?  For some time, I think the answer will be 
"highly specialized stuff"  At some point, Intel networking gear will be 
able to transfer more bits per second than there exist atoms on planet 
Earth :)  Garbage in, garbage out.

So, fix the other end of the pipeline too, otherwise this fast network 
stuff is flashly but pointless.  If you want to serve up data from disk, 
then start creating PCI cards that have both Serial ATA and ethernet 
connectors on them :)  Cut out the middleman of the host CPU and host 
memory bus instead of offloading portions of TCP that do not need to be 
offloaded.

	Jeff

On Sad, 2003-08-02 at 18:04, Werner Almesberger wrote:

 - last but not least, keeping TOE firmware up to date with the
   TCP/IP stack in the mainstream kernel will require - for each
   such TOE device - a significant and continuous effort over a
   long period of time

or even the protocol and protocol refinements..

 - instead of putting a different stack on the TOE, a
   general-purpose processor (probably with some enhancements,
   and certainly with optimized data paths) is added to the NIC

Like say an opteron in the 2nd socket on the motherboard

Benefits:

 - putting the CPU next to the NIC keeps data paths short, and
   allows for all kinds of optimizations (e.g. a pipelined
   memory architecture)

It moves the cost it doesnt make it vanish

If I read you right you are arguing for a second processor running
Linux.with its own independant memory bus. AMD make those already its
called AMD64. I don't know anyone thinking at that level about
partitioning one as an I/O processor.

Jeff Garzik wrote:

So, fix the other end of the pipeline too, otherwise this fast network 
stuff is flashly but pointless.  If you want to serve up data from disk, 
then start creating PCI cards that have both Serial ATA and ethernet 
connectors on them :)  Cut out the middleman of the host CPU and host 

I for one would love to see something like this, and not just Serial ATA..
but maybe 8x Serial ATA and RAID :)

Ben


-- 
Ben Greear [off-list ref]
Candela Technologies Inc  http://www.candelatech.com

A DMA xfer that fills the NIC pipe with IDE source. That's not very hard...
need a lot of bufferring/FIFO though.  May require large modification to the
file serving applications?

Alan

-----Original Message-----
From: linux-kernel-owner@vger.kernel.org
[mailto:linux-kernel-owner@vger.kernel.org]On Behalf Of Ben Greear
Sent: Saturday, August 02, 2003 9:02 PM
To: Jeff Garzik
Cc: Nivedita Singhvi; Werner Almesberger; netdev@oss.sgi.com;
linux-kernel@vger.kernel.org
Subject: Re: TOE brain dump


Jeff Garzik wrote:

So, fix the other end of the pipeline too, otherwise this fast network
stuff is flashly but pointless.  If you want to serve up data from disk,
then start creating PCI cards that have both Serial ATA and ethernet
connectors on them :)  Cut out the middleman of the host CPU and host

I for one would love to see something like this, and not just Serial ATA..
but maybe 8x Serial ATA and RAID :)

Ben


--
Ben Greear [off-list ref]
Candela Technologies Inc  http://www.candelatech.com

Alan Shih wrote:

A DMA xfer that fills the NIC pipe with IDE source. That's not very hard...
need a lot of bufferring/FIFO though.  May require large modification to the
file serving applications?

Nope, that's using the existing sendfile(2) facility.

	Jeff

do you really want the processor on the card to be tunning
apache/NFS/Samba/etc ?

putting enough linux on the card to act as a router (which would include
the netfilter stuff) is one thing. putting the userspace code that
interfaces with the outside world for file transfers is something else.

if you really want the disk connected to your network card you are just
talking a low-end linux box. forget all this stuff about it being on a
card and just use a full box (economys of scale will make this cheaper)

making a firewall that's a core system with a dozen slave systems attached
to it (the network cards) sounds like the type of clustering that Linux
has been used for for compute nodes. complicated to setup, but extremely
powerful and scalable once configured.

if you want more then a router on the card then Alan Cox is right, just
add another processor to the system, it's easier and cheaper.

David Lang


 On
Sat, 2 Aug 2003, Alan Shih wrote:

Date: Sat, 2 Aug 2003 23:22:52 -0700
From: Alan Shih <redacted>
To: Ben Greear <redacted>, Jeff Garzik <redacted>
Cc: Nivedita Singhvi <redacted>,
     Werner Almesberger [off-list ref], netdev@oss.sgi.com,
     linux-kernel@vger.kernel.org
Subject: RE: TOE brain dump

A DMA xfer that fills the NIC pipe with IDE source. That's not very hard...
need a lot of bufferring/FIFO though.  May require large modification to the
file serving applications?

Alan

-----Original Message-----
From: linux-kernel-owner@vger.kernel.org
[mailto:linux-kernel-owner@vger.kernel.org]On Behalf Of Ben Greear
Sent: Saturday, August 02, 2003 9:02 PM
To: Jeff Garzik
Cc: Nivedita Singhvi; Werner Almesberger; netdev@oss.sgi.com;
linux-kernel@vger.kernel.org
Subject: Re: TOE brain dump


Jeff Garzik wrote:

quoted

So, fix the other end of the pipeline too, otherwise this fast network
stuff is flashly but pointless.  If you want to serve up data from disk,
then start creating PCI cards that have both Serial ATA and ethernet
connectors on them :)  Cut out the middleman of the host CPU and host

I for one would love to see something like this, and not just Serial ATA..
but maybe 8x Serial ATA and RAID :)

Ben


--
Ben Greear [off-list ref]
Candela Technologies Inc  http://www.candelatech.com


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

On Sul, 2003-08-03 at 05:01, Ben Greear wrote:

Jeff Garzik wrote:

quoted

So, fix the other end of the pipeline too, otherwise this fast network 
stuff is flashly but pointless.  If you want to serve up data from disk, 
then start creating PCI cards that have both Serial ATA and ethernet 
connectors on them :)  Cut out the middleman of the host CPU and host 

I for one would love to see something like this, and not just Serial ATA..
but maybe 8x Serial ATA and RAID :)

There is a protocol floating around for ATA over ethernet, no TCP layer
or nasty latency eating complexities in the middle

On an embedded system, no processor will be fast enough to compete with
direct DMA xfer.  So just provide sendfile hooks that allow the kernel to
initiate data filling from source to dest then allow TSO to take place.
Kernel still needs to take care of the TCP stack.

I don't see this as building extensive customization though.

Alan

-----Original Message-----
From: David Lang [mailto:david.lang@digitalinsight.com]
Sent: Sunday, August 03, 2003 1:26 AM
To: Alan Shih
Cc: Ben Greear; Jeff Garzik; Nivedita Singhvi; Werner Almesberger;
netdev@oss.sgi.com; linux-kernel@vger.kernel.org
Subject: RE: TOE brain dump


do you really want the processor on the card to be tunning
apache/NFS/Samba/etc ?

putting enough linux on the card to act as a router (which would include
the netfilter stuff) is one thing. putting the userspace code that
interfaces with the outside world for file transfers is something else.

if you really want the disk connected to your network card you are just
talking a low-end linux box. forget all this stuff about it being on a
card and just use a full box (economys of scale will make this cheaper)

making a firewall that's a core system with a dozen slave systems attached
to it (the network cards) sounds like the type of clustering that Linux
has been used for for compute nodes. complicated to setup, but extremely
powerful and scalable once configured.

if you want more then a router on the card then Alan Cox is right, just
add another processor to the system, it's easier and cheaper.

David Lang


 On
Sat, 2 Aug 2003, Alan Shih wrote:

Date: Sat, 2 Aug 2003 23:22:52 -0700
From: Alan Shih <redacted>
To: Ben Greear <redacted>, Jeff Garzik <redacted>
Cc: Nivedita Singhvi <redacted>,
     Werner Almesberger [off-list ref], netdev@oss.sgi.com,
     linux-kernel@vger.kernel.org
Subject: RE: TOE brain dump

A DMA xfer that fills the NIC pipe with IDE source. That's not very

hard...

need a lot of bufferring/FIFO though.  May require large modification to

the

file serving applications?

Alan

-----Original Message-----
From: linux-kernel-owner@vger.kernel.org
[mailto:linux-kernel-owner@vger.kernel.org]On Behalf Of Ben Greear
Sent: Saturday, August 02, 2003 9:02 PM
To: Jeff Garzik
Cc: Nivedita Singhvi; Werner Almesberger; netdev@oss.sgi.com;
linux-kernel@vger.kernel.org
Subject: Re: TOE brain dump


Jeff Garzik wrote:

quoted

So, fix the other end of the pipeline too, otherwise this fast network
stuff is flashly but pointless.  If you want to serve up data from disk,
then start creating PCI cards that have both Serial ATA and ethernet
connectors on them :)  Cut out the middleman of the host CPU and host

I for one would love to see something like this, and not just Serial ATA..
but maybe 8x Serial ATA and RAID :)

Ben


--
Ben Greear [off-list ref]
Candela Technologies Inc  http://www.candelatech.com


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/