TOE is evil, read this:
http://www.usenix.org/events/hotos03/tech/full_papers/mogul/mogul.pdf
TOE is exactly suboptimal for the very things performance
matters, high connection rates.
Your return is also absolutely questionable. Servers "serve" data
and we offload all of the send side TCP processing that can
reasonably be done (segmentation, checksumming).
I've never seen an impartial benchmark showing that TCP send
side performance goes up as a result of using TOE vs. the usual
segmentation + checksum offloading offered today.
On receive side, clever RX buffer flipping tricks are the way
to go and require no protocol changes and nothing gross like
TOE or weird buffer ownership protocols like RDMA requires.
I've made postings showing how such a scheme can work using a limited
flow cache on the networking card. I don't have a reference handy,
but I suppose someone else does.
And finally, this discussion belongs on the "networking" lists.
Nearly all of the "networking" developers don't have time to sift
through linux-kernel every day.
From: Roland Dreier <hidden> Date: 2003-07-13 16:08:16
David> TOE is evil, read this:
David> http://www.usenix.org/events/hotos03/tech/full_papers/mogul/mogul.pdf
David> TOE is exactly suboptimal for the very things performance
David> matters, high connection rates.
David> Your return is also absolutely questionable. Servers
David> "serve" data and we offload all of the send side TCP
David> processing that can reasonably be done (segmentation,
David> checksumming).
David> I've never seen an impartial benchmark showing that TCP
David> send side performance goes up as a result of using TOE
David> vs. the usual segmentation + checksum offloading offered
David> today.
David> On receive side, clever RX buffer flipping tricks are the
David> way to go and require no protocol changes and nothing gross
David> like TOE or weird buffer ownership protocols like RDMA
David> requires.
David> I've made postings showing how such a scheme can work using
David> a limited flow cache on the networking card. I don't have
David> a reference handy, but I suppose someone else does.
Your ideas are certainly very interesting, and I would be happy to see
hardware that supports flow identification. But the Usenix paper
you're citing completely disagrees with you! For example, Mogul writes:
"Nevertheless, copy-avoidance designs have not been widely adopted,
due to significant limitations. For example, when network maximum
segment size (MSS) values are smaller than VM page sizes, which is
often the case, page-remapping techniques are insufficient (and
page-remapping often imposes overheads of its own.)"
In fact, his conclusion is:
"However, as hardware trends change the feasibility and economics of
network-based storage connections, RDMA will become a significant
and appropriate justification for TOEs."
- Roland
Your ideas are certainly very interesting, and I would be happy to see
hardware that supports flow identification. But the Usenix paper
you're citing completely disagrees with you! For example, Mogul writes:
Take a look at who holds the official internet land speed record. Its
not a TOE using system.
"Nevertheless, copy-avoidance designs have not been widely adopted,
due to significant limitations. For example, when network maximum
segment size (MSS) values are smaller than VM page sizes, which is
often the case, page-remapping techniques are insufficient (and
page-remapping often imposes overheads of its own.)"
Page remapping is adequate for send of data when the MSS is below the
VM page size since you don't have to send all of the page you pinned
or set COW/SOW (sleep on write)
For receive if your hardware can do demux from the tcp headers and
expecting sequence then page remapping isn't needed either.
Finally if you are streaming objects by non mapped references (eg
sendfile or see LM's paper from long ago on splice()) then the problem
goes away.
In fact, his conclusion is:
"However, as hardware trends change the feasibility and economics of
network-based storage connections, RDMA will become a significant
and appropriate justification for TOEs."
- Roland
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Jeff Garzik <hidden> Date: 2003-07-13 16:43:29
Alan Cox wrote:
Finally if you are streaming objects by non mapped references (eg
sendfile or see LM's paper from long ago on splice()) then the problem
goes away.
As an aside, I really like sendfile's semantics except for
* People occasionally want to add a receivefile(2). I disagree...
sendfile(2) interface should be really be considered a universal
"fdcopy" interface, regardless of what the 'to' and 'from' file
descriptors are attached to. File to socket. Socket to file. File to
file. socket to socket. All should be supported, even if the fallback
is a stupid (but small!) in-kernel copy loop.
* Copy-until-EOF semantics are either undefined, or, unclear to me
personally.
Jeff
Your ideas are certainly very interesting, and I would be happy to see
hardware that supports flow identification. But the Usenix paper
you're citing completely disagrees with you!
I didn't say I agree with all of Moguls ideas, just his anti-TOE
arguments. For example, I also think RDMA sucks too yet he thinks
it's a good iea.
For example, Mogul writes:
"Nevertheless, copy-avoidance designs have not been widely adopted,
due to significant limitations. For example, when network maximum
segment size (MSS) values are smaller than VM page sizes, which is
often the case, page-remapping techniques are insufficient (and
page-remapping often imposes overheads of its own.)"
On send this doesn't matter, on receive you use my clever receive
buffer handling + flow cache idea to accumulate the data portion of
packets into page sized chunks for the networking to flip.
You obviously don't understand my ideas if you think that it matters
whether there is some relationship between the MTU and the system
page size necessary for the scheme to work.
From: Larry McVoy <hidden> Date: 2003-07-13 23:20:35
On Sun, Jul 13, 2003 at 04:02:00PM -0700, David S. Miller wrote:
On send this doesn't matter, on receive you use my clever receive
buffer handling + flow cache idea to accumulate the data portion of
packets into page sized chunks for the networking to flip.
Please don't. I think page flipping was a bad idea. I think you'd be
better off to try and make the data flow up the stack in small enough
windows that it all sits in the cache.
One thing SGI taught me (not that they wanted to do so) is that infinitely
large packets are infinitely stupid, for lots of reasons. One is that
you have to buffer them somewhere and another is that the bigger they
are the bigger your cache needs to be to go fast.
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
From: David S. Miller <hidden> Date: 2003-07-13 23:34:22
On Sun, 13 Jul 2003 16:35:03 -0700
Larry McVoy [off-list ref] wrote:
On Sun, Jul 13, 2003 at 04:02:00PM -0700, David S. Miller wrote:
quoted
On send this doesn't matter, on receive you use my clever receive
buffer handling + flow cache idea to accumulate the data portion of
packets into page sized chunks for the networking to flip.
Please don't. I think page flipping was a bad idea. I think you'd be
better off to try and make the data flow up the stack in small enough
windows that it all sits in the cache.
At 10GB/sec nothing fits in the cache :-)
One thing SGI taught me (not that they wanted to do so) is that infinitely
large packets are infinitely stupid, for lots of reasons. One is that
you have to buffer them somewhere and another is that the bigger they
are the bigger your cache needs to be to go fast.
The whole point is to not touch any of this data.
The idea is to push the pages directly into the page cache
of the filesystem.
I'm not talking about doing this for userspace normal sys_recvmsg()
type reads, that's an entirely different topic but if we ever did
all agree to do something like that we'd have the network level
infrastructure to do it already.
From: Larry McVoy <hidden> Date: 2003-07-13 23:39:56
The whole point is to not touch any of this data.
The idea is to push the pages directly into the page cache
of the filesystem.
It doesn't work. Measure the cost of the VM operations before you go
down this path. Just set up a system call that swaps a page with a
kernel allocated buffer and then see how many of those you can do a
second. Maybe Linux is so blindingly fast this makes sense but IRIX
certainly wasn't, the VM overhead hurt like crazy.
Every time I tried to push the page flip idea or offloading or any of
that crap, Andy Bechtolsheim would tell "the CPUs will get faster faster
than you can make that work". He was right.
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
From: David S. Miller <hidden> Date: 2003-07-13 23:47:40
On Sun, 13 Jul 2003 16:54:24 -0700
Larry McVoy [off-list ref] wrote:
Every time I tried to push the page flip idea or offloading or any of
that crap, Andy Bechtolsheim would tell "the CPUs will get faster faster
than you can make that work". He was right.
I really don't see why receive is so much of a big deal
compared to send, and we do a send side version of this
stuff already with zero problems.
The NFS code is already basically ready to handle a fragmented packet
(headers + pages), and could stick the page part into the page cache
easily on receive.
And it's not the CPUs that really limit us here, it's memory
bandwidth. It's one thing to have a PCI-X bus fast enough
to service 10Ggb/sec rates, it's yet another thing to have
a memory bus and RAM underneath that which can handle moving
that data over it _twice_.
The infrastructure needed to support this on the networking side
help us support other useful things, such as driver local packet
buffer recycling.
From: Roland Dreier <hidden> Date: 2003-07-14 00:06:05
David> I didn't say I agree with all of Moguls ideas, just his
David> anti-TOE arguments. For example, I also think RDMA sucks
David> too yet he thinks it's a good iea.
Sure, he talks about some weaknesses of TOE, but his conclusion is
that the time has come for OS developers to start working on TCP
offload (for storage).
David> You obviously don't understand my ideas if you think that
David> it matters whether there is some relationship between the
David> MTU and the system page size necessary for the scheme to
David> work.
I was just quoting part of Mogul's paper that seemed to directly
contradict your original post. I also said it would be great to see
NIC hardware with support for flow classification.
Look, I pretty much agree with you about TOE hardware. Every chip
I've seen either requires a bunch of dedicated expensive memory
(including a giant CAM) or is just firmware running on a
low-performance embedded CPU. But I also think Mogul is right: iSCSI
HBAs are going to force OS designers to deal with TCP offload.
My whole point was just that it doesn't make much sense to dismiss the
whole idea by saying "TOE is evil" and then cite as support a paper
that explains why TOEs now make sense and need to be supported.
- Roland
From: Larry McVoy <hidden> Date: 2003-07-14 00:07:40
On Sun, Jul 13, 2003 at 04:53:23PM -0700, David S. Miller wrote:
On Sun, 13 Jul 2003 16:54:24 -0700
Larry McVoy [off-list ref] wrote:
quoted
Every time I tried to push the page flip idea or offloading or any of
that crap, Andy Bechtolsheim would tell "the CPUs will get faster faster
than you can make that work". He was right.
I really don't see why receive is so much of a big deal
compared to send, and we do a send side version of this
stuff already with zero problems.
Hey, maybe it isn't, but could you please quantify the cost of the VM
operations? How hard is that?
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
From: David S. Miller <hidden> Date: 2003-07-14 00:18:34
On Sun, 13 Jul 2003 17:22:00 -0700
Larry McVoy [off-list ref] wrote:
Hey, maybe it isn't, but could you please quantify the cost of the VM
operations? How hard is that?
Ok.
So the page is in a non-uptodate state, NFS would have it locked,
and anyone else trying to get at it would sleep.
This page we have currently is "dummy" in that it is only a place
holder in case we don't get a full page from the networking.
We have all the infrastructure to do everything up to this point.
Next, if the networking gave us a full page, we'd "replace"
the dummy page with this one, which would involve:
1) delete the dummy page from the lookup, insert the networking's
page
2) arrange so that all sleepers on the dummy page will do a relookup
and find the new page
And when we're done with the operation we wake everyone up.
I can't see any part of this turning out to be expensive.
From: David S. Miller <hidden> Date: 2003-07-14 00:22:53
On 13 Jul 2003 17:20:41 -0700
Roland Dreier [off-list ref] wrote:
David> I didn't say I agree with all of Moguls ideas, just his
David> anti-TOE arguments. For example, I also think RDMA sucks
David> too yet he thinks it's a good iea.
Sure, he talks about some weaknesses of TOE, but his conclusion is
that the time has come for OS developers to start working on TCP
offload (for storage).
The bad assumption here is that this belongs in the OS.
Let me ask you this, how many modern scsi drivers have to speak every
piece of the SCSI bus protocol. Or fibre channel? All of it is
done on the cards, and that is what I think the iSCSI people should
be doing instead of putting garbage into the OS.
And I've presented a solution to the problem at the OS level that
doesn't require broken things like TOE and RDMA yet arrives at
the same solution.
But I also think Mogul is right: iSCSI HBAs are going to force OS
designers to deal with TCP offload.
You don't need to offload TCP, it's the segmentation and checksuming
that has the high cost not the actual TCP logic in the operating
system.
RDMA and TOE both add unnecessary complications. My solution requires
no protocol changes, just smart hardware which needs to be designed
for any of the presented ideas anyways.
From: Larry McVoy <hidden> Date: 2003-07-14 00:33:56
On Sun, Jul 13, 2003 at 05:24:14PM -0700, David S. Miller wrote:
I can't see any part of this turning out to be expensive.
In theory, practice and theory are the same...
I think the point I'm trying to make is that the VM stuff costs something
and it shouldn't be that hard to dummy up a system call to measure it.
It was counterintuitive as hell at SGI that the VM stuff would cost that
much and the reasons are subtle. Part of the problem turned out to be
falling out of the instruction cache - the network stack and the VM system
didn't fit and that left no room at all for the app.
If you are trading instruction cache misses for data misses, err, dude,
I think that might be a problem. The point is to process all the data
with less, not more, cache misses, right? In fact, if we agree on that
then that leads you to considering the various ways you could do this
and maybe your way is the right way but maybe there is a less cache
intensive way.
If you're right you're right, so peace. But I'd like the definition of
"right" to be "less cache misses to do the same thing". In fact, if
I managed to communicate only one thing in my entire set of rants and
it was "pay attention to cache misses", hey, that'd be cool with me.
That's how you make things go fast and I like fast.
Think about it, a 3GHz machine is a .3ns clock cycle and the suckers are
super scalar and hyper threaded and all that crud. Memory is about 133ns
away. That's 400 clocks of stall for each cache miss. Lotta code can run
in 400 clocks of super scalar/hyper threaded/fully buzzword enabled processors.
--
---
Larry McVoy lm at bitmover.com http://www.bitmover.com/lm