Re: [RFC][PATCH 1/1] cxgb3i: cxgb3 iSCSI initiator

 > > Probably depends on whether or not the iSCSI offload solutions are doing 
 > > zero-copy receive into the filecache?
 > 
 > That's a data placement issue, which also can be solved with
 > stateless offloading.

How can you place iSCSI data properly with only stateless offloads?

 - R.

From: Roland Dreier <redacted>
Date: Mon, 11 Aug 2008 14:41:16 -0700

 > > Probably depends on whether or not the iSCSI offload solutions are doing 
 > > zero-copy receive into the filecache?
 > 
 > That's a data placement issue, which also can be solved with
 > stateless offloading.

How can you place iSCSI data properly with only stateless offloads?

By teaching the stateless offload how to parse the iSCSI headers
on the flow and place the data into pages at the correct offsets
such that you can place the pages hanging off of the SKB directly
into the page cache.

On Monday 11 August 2008 02:53:13 pm David Miller wrote:

From: Roland Dreier <redacted>
Date: Mon, 11 Aug 2008 14:41:16 -0700

quoted

 > > Probably depends on whether or not the iSCSI offload solutions are
 > > doing zero-copy receive into the filecache?
 >
 > That's a data placement issue, which also can be solved with
 > stateless offloading.

How can you place iSCSI data properly with only stateless offloads?

By teaching the stateless offload how to parse the iSCSI headers
on the flow and place the data into pages at the correct offsets
such that you can place the pages hanging off of the SKB directly
into the page cache.

Hi Dave,

iSCSI PDUs might spawn over multiple TCP segments, it is unclear to me how to 
do placement without keeping some state of the transactions.

In any case, such a stateless solution is not yet designed, whereas 
accelerated iSCSI is available now, from us and other companies.
The accelerated iSCSI streams benefit from the performance TOE provides, 
outlined in the following third party papers:
http://www.chelsio.com/assetlibrary/pdf/redhat-chelsio-toe-final_v2.pdf
http://www.chelsio.com/assetlibrary/pdf/RMDS6BNTChelsioRHEL5.pdf

iSCSI is primarily targeted to the data center, where the SW stack's traffic 
shaping features might be redundant with specialized equipment. It should 
however be possible to integrate security features on a per offoaded 
connection basis, and TOEs - at least ours :) - are capable of rate control 
and traffic shaping.

While CPU and - to a far lesser extent - memory performance improves, so does 
ethernet's. 40G, 100G are not too far ahead. It is not obvious at all that 
TOE is a point of time solution, especially for heavy load traffic as in a 
storage environment. It is quite the opposite actually.

There is room for co-existence of the SW managed traffic and accelerated 
traffic. As our submission shows, enabling accelerated iSCSI is not intrusive 
code wise to the stack. The port stealing issue is solved if we can grab a 
port from the stack.

Cheers,
Divy

From: Divy Le Ray <redacted>
Date: Tue, 12 Aug 2008 14:57:09 -0700

iSCSI PDUs might spawn over multiple TCP segments, it is unclear to me how to 
do placement without keeping some state of the transactions.

You keep a flow table with buffer IDs and offsets.

The S2IO guys did something similar for one of their initial LRO
impelementations.

It's still strictly stateless, and best-effort.  Entries can fall out
of the flow cache which makes upcoming data use new buffers and
offsets.

But these are the kinds of tricks you hardware folks should be
more than adequately able to design, rather than me. :-)

From: Divy Le Ray <redacted>
Date: Tue, 12 Aug 2008 14:57:09 -0700

In any case, such a stateless solution is not yet designed, whereas 
accelerated iSCSI is available now, from us and other companies.

So, WHAT?!

There are TOE pieces of crap out there too.

It's strictly not our problem.

Like Herbert said, this is the TOE discussion all over again.
The results will be the same, and as per our decisions wrt.
TOE, history speaks for itself.

On Tuesday 12 August 2008 03:02:46 pm David Miller wrote:

From: Divy Le Ray <redacted>
Date: Tue, 12 Aug 2008 14:57:09 -0700

quoted

In any case, such a stateless solution is not yet designed, whereas
accelerated iSCSI is available now, from us and other companies.

So, WHAT?!

There are TOE pieces of crap out there too.

Well, there is demand for accerated iscsi out there, which is the driving 
reason of our driver submission. 

It's strictly not our problem.

Like Herbert said, this is the TOE discussion all over again.
The results will be the same, and as per our decisions wrt.
TOE, history speaks for itself.

Herbert requested some benchmark numbers, I consequently obliged.

Cheers,
Divy

Divy Le Ray [off-list ref] wrote:

Herbert requested some benchmark numbers, I consequently obliged.

Have you posted a hardware-accelerated iSCSI vs. LRO comparison?

Thanks,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} [off-list ref]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Divy Le Ray wrote:

On Tuesday 12 August 2008 03:02:46 pm David Miller wrote:

quoted

From: Divy Le Ray <redacted>
Date: Tue, 12 Aug 2008 14:57:09 -0700

quoted

In any case, such a stateless solution is not yet designed, whereas
accelerated iSCSI is available now, from us and other companies.

So, WHAT?!

There are TOE pieces of crap out there too.

Well, there is demand for accerated iscsi out there, which is the driving 
reason of our driver submission. 

I'm, as an iSCSI target developer, strongly voting for hardware iSCSI 
offload. Having possibility of the direct data placement is a *HUGE* 
performance gain.

For example, according to measurements done by one iSCSI-SCST user in 
system with iSCSI initiator and iSCSI target (with iSCSI-SCST 
(http://scst.sourceforge.net/target_iscsi.html) running), both with 
identical modern high speed hardware and 10GbE cards, the _INITIATOR_ is 
the bottleneck for READs (data transfers from target to initiator). This 
is because the target sends data in a zero-copy manner, so its CPU is 
capable to deal with the load, but on the initiator there are additional 
data copies from skb's to page cache and from page cache to application. 
As the result, in the measurements initiator got near 100% CPU load and 
only ~500MB/s throughput. Target had ~30% CPU load. For the opposite 
direction (WRITEs), where there is no the application data copy on the 
target, throughput was ~800MB/s with also near 100% CPU load, but in 
this case on the target. The initiator ran Linux with open-iscsi. The 
test was with real backstorage: target ran BLOCKIO (direct BIOs to/from 
backstorage) with 3ware card. Locally on the target the backstorage was 
able to provide 900+MB/s for READs and about 1GB/s for WRITEs. The 
commands queue in both cases was sufficiently big to eliminate the link 
and processing latencies (20-30 outstanding commands).

Vlad

Vladislav Bolkhovitin wrote:

Divy Le Ray wrote:

quoted

On Tuesday 12 August 2008 03:02:46 pm David Miller wrote:

quoted

From: Divy Le Ray <redacted>
Date: Tue, 12 Aug 2008 14:57:09 -0700

quoted

In any case, such a stateless solution is not yet designed, whereas
accelerated iSCSI is available now, from us and other companies.

So, WHAT?!

There are TOE pieces of crap out there too.

Well, there is demand for accerated iscsi out there, which is the 
driving reason of our driver submission. 

I'm, as an iSCSI target developer, strongly voting for hardware iSCSI 
offload. Having possibility of the direct data placement is a *HUGE* 
performance gain.

Well, two responses here:

* no one is arguing against hardware iSCSI offload.  Rather, it is a 
problem with a specific implementation, one that falsely assumes two 
independent TCP stacks can co-exist peacefully on the same IP address 
and MAC.

* direct data placement is possible without offloading the entire TCP 
stack onto a firmware/chip.

There is plenty of room for hardware iSCSI offload...

	Jeff

From: Jeff Garzik <redacted>
Date: Wed, 13 Aug 2008 15:29:55 -0400

* direct data placement is possible without offloading the entire TCP 
stack onto a firmware/chip.

I've even described in this thread how that's possible.

From: Vladislav Bolkhovitin <redacted>
Date: Wed, 13 Aug 2008 22:35:34 +0400

This is because the target sends data in a zero-copy manner, so its
CPU is capable to deal with the load, but on the initiator there are
additional data copies from skb's to page cache and from page cache
to application.

If you've actually been reading at all what I've been saying in this
thread you'll see that I've described a method to do this copy
avoidance in a completely stateless manner.

You don't need to implement a TCP stack in the card in order to do
data placement optimizations.  They can be done completely stateless.

Also, large portions of the cpu overhead are transactional costs,
which are significantly reduced by existing technologies such as
LRO.

Jeff Garzik wrote:

Vladislav Bolkhovitin wrote:

quoted

Divy Le Ray wrote:

quoted

On Tuesday 12 August 2008 03:02:46 pm David Miller wrote:

quoted

From: Divy Le Ray <redacted>
Date: Tue, 12 Aug 2008 14:57:09 -0700

quoted

In any case, such a stateless solution is not yet designed, whereas
accelerated iSCSI is available now, from us and other companies.

So, WHAT?!

There are TOE pieces of crap out there too.

Well, there is demand for accerated iscsi out there, which is the 
driving reason of our driver submission. 

I'm, as an iSCSI target developer, strongly voting for hardware iSCSI 
offload. Having possibility of the direct data placement is a *HUGE* 
performance gain.

Well, two responses here:

* no one is arguing against hardware iSCSI offload.  Rather, it is a 
problem with a specific implementation, one that falsely assumes two 
independent TCP stacks can co-exist peacefully on the same IP address 
and MAC.

* direct data placement is possible without offloading the entire TCP 
stack onto a firmware/chip.

There is plenty of room for hardware iSCSI offload...

Sure, nobody is arguing against that. My points are:

1. All those are things not for near future. I don't think it can be 
implemented earlier than in a year time, but there is a huge demand for 
high speed and low CPU overhead iSCSI _now_. Nobody's satisfied by the 
fact that with the latest high end hardware he can saturate 10GbE link 
on only less than 50%(!). Additionally, for me, as an iSCSI target 
developer, it looks especially annoying that hardware requirements for 
_clients_ (initiators) are significantly higher than for _server_ 
(target). This situation for me looks as a nonsense.

2. I believe, that iSCSI/TCP pair is sufficiently heavy weighted 
protocol to be completely offloaded to hardware. All partial offloads 
will never make it comparably efficient. It still would consume a lot of 
CPU. For example, consider digests. Even if they computed by new CRC32C 
instruction, the computation still would need a chunk of CPU power. I 
think, at least as much as to copy the computed block to new location. 
Can we save it? Sure, with hardware offload. The additional CPU load can 
be acceptable if only data are transferred and there are no other 
activities, but in real life this is quite rare. Consider, for instance, 
a VPS server, like VMware. It always lacks CPU power and 30% CPU load 
during data transfers makes a huge difference. Another example is a 
target doing some processing of transferred data, like encryption or 
de-duplication.

Note, I'm not advocating this particular cxgb3 driver. I have not 
examined it closely enough and don't have sufficient knowledge about the 
hardware to judge it. But I'm advocating the concept of full offload 
HBAs, because they provide a real gain, which IMHO can't be reached by 
any partial offloads.

Actually, in the Fibre Channel world from the very beginning the entire 
FC protocol has been implemented on hardware and everybody have been 
happy with that. Now FCoE is coming, which means that Linux kernel is 
going to have implemented in software a big chunk of FC protocol. Then, 
hopefully, nobody would declare all existing FC cards as a crap and 
force FC vendors redesign their hardware to use Linux FC implementation 
and make partial offloads for it? ;) Instead, several implementations 
would live in a peace. The situation is the same with iSCSI. What we 
need is only to find an acceptable way for two TCP implementations to 
coexist. Then iSCSI on 10GbE hardware would have good chances to 
outperform 8Gbps FC in both performance and CPU efficiency.

Vlad

David Miller wrote:

From: Vladislav Bolkhovitin <redacted>
Date: Wed, 13 Aug 2008 22:35:34 +0400

quoted

This is because the target sends data in a zero-copy manner, so its
CPU is capable to deal with the load, but on the initiator there are
additional data copies from skb's to page cache and from page cache
to application.

If you've actually been reading at all what I've been saying in this
thread you'll see that I've described a method to do this copy
avoidance in a completely stateless manner.

You don't need to implement a TCP stack in the card in order to do
data placement optimizations.  They can be done completely stateless.

Sure, I read what you wrote before writing (although, frankly, didn't 
get the idea). But I don't think that overall it would be as efficient 
as full hardware offload. See my reply to Jeff Garzik about that.

Also, large portions of the cpu overhead are transactional costs,
which are significantly reduced by existing technologies such as
LRO.

The test used Myricom Myri-10G cards (myri10ge driver), which support 
LRO. And from ethtool -S output I conclude it was enabled. Just in case, 
I attached it, so you can recheck me.

Thus, apparently, LRO doesn't make a fundamental difference. Maybe this 
particular implementation isn't too efficient, I don't know. I don't 
have enough information for that.

Vlad

Vladislav Bolkhovitin wrote:

David Miller wrote:

quoted

From: Vladislav Bolkhovitin <redacted>
Date: Wed, 13 Aug 2008 22:35:34 +0400

quoted

This is because the target sends data in a zero-copy manner, so its
CPU is capable to deal with the load, but on the initiator there are
additional data copies from skb's to page cache and from page cache
to application.

If you've actually been reading at all what I've been saying in this
thread you'll see that I've described a method to do this copy
avoidance in a completely stateless manner.

You don't need to implement a TCP stack in the card in order to do
data placement optimizations.  They can be done completely stateless.

Sure, I read what you wrote before writing (although, frankly, didn't 
get the idea). But I don't think that overall it would be as efficient 
as full hardware offload. See my reply to Jeff Garzik about that.

quoted

Also, large portions of the cpu overhead are transactional costs,
which are significantly reduced by existing technologies such as
LRO.

The test used Myricom Myri-10G cards (myri10ge driver), which support 
LRO. And from ethtool -S output I conclude it was enabled. Just in case, 
I attached it, so you can recheck me.

Also, there wasn't big difference between MTU 1500 and 9000, which is 
another point to think that LRO was working.

Thus, apparently, LRO doesn't make a fundamental difference. Maybe this 
particular implementation isn't too efficient, I don't know. I don't 
have enough information for that.

Vlad

On Thu, 2008-08-14 at 22:24 +0400, Vladislav Bolkhovitin wrote:

Jeff Garzik wrote:

quoted

Vladislav Bolkhovitin wrote:

quoted

Divy Le Ray wrote:

quoted

On Tuesday 12 August 2008 03:02:46 pm David Miller wrote:

quoted

From: Divy Le Ray <redacted>
Date: Tue, 12 Aug 2008 14:57:09 -0700

quoted

In any case, such a stateless solution is not yet designed, whereas
accelerated iSCSI is available now, from us and other companies.

So, WHAT?!

There are TOE pieces of crap out there too.

Well, there is demand for accerated iscsi out there, which is the 
driving reason of our driver submission. 

I'm, as an iSCSI target developer, strongly voting for hardware iSCSI 
offload. Having possibility of the direct data placement is a *HUGE* 
performance gain.

Well, two responses here:

* no one is arguing against hardware iSCSI offload.  Rather, it is a 
problem with a specific implementation, one that falsely assumes two 
independent TCP stacks can co-exist peacefully on the same IP address 
and MAC.

* direct data placement is possible without offloading the entire TCP 
stack onto a firmware/chip.

There is plenty of room for hardware iSCSI offload...

Sure, nobody is arguing against that. My points are:

1. All those are things not for near future. I don't think it can be 
implemented earlier than in a year time, but there is a huge demand for 
high speed and low CPU overhead iSCSI _now_.

Well, the first step wrt to this for us software folks is getting the
Slicing by 8 algoritim CRC32C into the kernel..  This would be a great
benefit for not just traditional iSCSI/TCP, but Linux/SCTP and
Linux/iWARP software codebases.

 Nobody's satisfied by the 
fact that with the latest high end hardware he can saturate 10GbE link 
on only less than 50%(!). Additionally, for me, as an iSCSI target 
developer, it looks especially annoying that hardware requirements for 
_clients_ (initiators) are significantly higher than for _server_ 
(target). This situation for me looks as a nonsense.

I have always found this to be the historical case wrt iSCSI on x86
hardware.  The rough estimate was that given identical hardware and
network configuration, an iSCSI target talking to a SCSI subsystem layer
would be able to handle 2x throughput compared to an iSCSI Initiator,
obviously as long as the actual storage could handle it.

2. I believe, that iSCSI/TCP pair is sufficiently heavy weighted 
protocol to be completely offloaded to hardware.

Heh, I think the period of designing news ASICs for traditional iSCSI
offload is probably slowing.  Aside from the actual difficulting of
doing this and competing with software iSCSI on commodity x86 4x & 8x
core (8x and 16x thread) micropressors with highly efficent software
implementation, that can do BOTH traditional iSCSI offload (where
available) and real deal OS independent connection recovery
(ErrorRecoveryLevel=2) between multiple stateless iSER iWARP/TCP
connections across both hardware *AND* software iWARP RNICs.

 All partial offloads will never make it comparably efficient.

With traditional iSCSI, I definately agree on this.

With iWARP and iSER however, I believe the end balance of simplicity is
greater for both hardware and software, and allows both hardware and
software to scale more effectively because  The simple gain of having a
Framed PDU on top of legacy TCP with RFC 504[0-4] in order to determine
the offload of the received packet that will be mapped to storage
subsystem later memory for eventual hardware DMA on a vast array of
Linux supported storage hardware and CPU architectures.

 It still would consume a lot of 
CPU. For example, consider digests. Even if they computed by new CRC32C 
instruction, the computation still would need a chunk of CPU power. I 
think, at least as much as to copy the computed block to new location. 
Can we save it? Sure, with hardware offload.

So yes, we are talking about quite a few possible cases:

I) Traditional iSCSI:

1) Complete hardware offload for legacy HBAs

2) Hybrid of hardware/software 

As mentioned, reducing application layer checksum overhead for current
software implementations is very important for our quickly increase user
base.  Using the Slicing by 8 CRC32C will help the current code, but I
think the only other real optimization by network ASIC design folks
would be to do something along the lines with traditional iSCSI with the
application layer that the say the e1000 driver does with transport and
network layer checksums today.  I believe the complexity and time to
market considerations of a complete traditional iSCSI offload solution
compared to highly optimized software iSCSI on dedicated commodity cores
still outweighs the benefit IMHO.

Not that I am saying there is no room for improvement from the current
set iSCSI Initiator TOEs.  Again I could build a children's fortress
from iSCSI TOE's and their retail boxes that I have in my office that I
have gotten over the years.   I would definately like to see them
running on the LIO production fabric and VHACS bare-metal storage clouds
at some point for validation purposes, et al.  But as for new designs,
this is still a very difficult proposition,  I am glad to see it being
discussed here..

II) iWARP/TCP and iSER

1) Hardware RNIC w/ iWARP/TCP with software iSER

2) Software RNIC w/ iWARP/TCP with software iSER

3) More possible iSER logic in hardware for latency/performance
optimizations (We won't know this until #1 and #2 happen)

Ahh, now this is the interesting case for scaling vendor independent IP
storage fabric to multiple port full duplex 10 Gb/sec fabrics.  As this
hardware on PCIe gets out (yes, I have some AMSO1100 goodness too
Steve :-), and iSER initiator/targets on iWARP/TCP come online, I
believe the common code between the different flavours of implemenations
will be much larger here.   For example, I previously mentioned ERL=2 in
the context of traditional iSCSI/iSER.  This logic is independent of
what RFC5045 knows a network fabric capable of of direct data placement.
I will also make this code independent in lio-target-2.6.git for my
upstream work.

 The additional CPU load can 
be acceptable if only data are transferred and there are no other 
activities, but in real life this is quite rare. Consider, for instance, 
a VPS server, like VMware. It always lacks CPU power and 30% CPU load 
during data transfers makes a huge difference. Another example is a 
target doing some processing of transferred data, like encryption or 
de-duplication.

Well, I think alot of this depends on hardware.  For example, there is
the X3100 adapter from Neterion today that can do 10 Gb/sec line rate
with x86_64 virtualization.  Obviously, the Linux kernel (and my
project, Linux-iSCSI.org) wants to be able to support this as vendor
neutral as possible, which is why we make extensive use of multiple
technologies in our production fabrics, and in the VHACS stack. :-)

Also, the Nested Page Tables would be a big win for this particular
case, but I am not familar with the exact numbers..

Actually, in the Fibre Channel world from the very beginning the entire 
FC protocol has been implemented on hardware and everybody have been 
happy with that. Now FCoE is coming, which means that Linux kernel is 
going to have implemented in software a big chunk of FC protocol. Then, 
hopefully, nobody would declare all existing FC cards as a crap and 
force FC vendors redesign their hardware to use Linux FC implementation 
and make partial offloads for it? ;) Instead, several implementations 
would live in a peace. The situation is the same with iSCSI. What we 
need is only to find an acceptable way for two TCP implementations to 
coexist. Then iSCSI on 10GbE hardware would have good chances to 
outperform 8Gbps FC in both performance and CPU efficiency.

<nod> :-)

--nab

Vlad

--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html