Re: [PATCH v2 04/13] Connection Manager

 > Could you convince network core developers that it is not own TCP
 > implementation which will mess with existing one?

I'm not qualified to comment on this...

 > This and a lot of other changes in this driver definitely says you
 > implement your own stack of protocols on top of infiniband hardware.

...but I do know this driver is for 10-gig ethernet HW.

 - R.

On Mon, 2006-12-04 at 07:45 -0800, Roland Dreier wrote:

 > Could you convince network core developers that it is not own TCP
 > implementation which will mess with existing one?

I'm not qualified to comment on this...

I don't understand your question?

 > This and a lot of other changes in this driver definitely says you
 > implement your own stack of protocols on top of infiniband hardware.

...but I do know this driver is for 10-gig ethernet HW.

There is no SW TCP stack in this driver.  The HW supports RDMA over
TCP/IP/10GbE in HW and this is required for zero-copy RDMA over Ethernet
(aka iWARP).  The device is a 10 GbE device, not Infiniband.  The
Ethernet driver, upon which the rdma driver depends, acts both like a
traditional Ethernet NIC for the Linux stack as well as a TCP offload
device for the RDMA driver allowing establishment of RDMA connections.
The Connection Manager (patch 04/13) sends/receives messages from the
Ethernet driver that sets up HW TCP connections for doing RDMA.  While
this is indeed implementing TCP offload, it is _not_ integrating it with
the sockets layer nor the linux stack and offloading sockets
connections.  Its only supporting offload connections for the RDMA
driver to do iWARP.   The Ammasso device is another example of this
(drivers/infiniband/hw/amso1100).  Deep iSCSI adapters are another
example of this.


Steve.

On Mon, Dec 04, 2006 at 10:20:51AM -0600, Steve Wise (swise@opengridcomputing.com) wrote:

quoted

 > This and a lot of other changes in this driver definitely says you
 > implement your own stack of protocols on top of infiniband hardware.

...but I do know this driver is for 10-gig ethernet HW.

There is no SW TCP stack in this driver.  The HW supports RDMA over
TCP/IP/10GbE in HW and this is required for zero-copy RDMA over Ethernet
(aka iWARP).  The device is a 10 GbE device, not Infiniband.  The
Ethernet driver, upon which the rdma driver depends, acts both like a
traditional Ethernet NIC for the Linux stack as well as a TCP offload
device for the RDMA driver allowing establishment of RDMA connections.
The Connection Manager (patch 04/13) sends/receives messages from the
Ethernet driver that sets up HW TCP connections for doing RDMA.  While
this is indeed implementing TCP offload, it is _not_ integrating it with
the sockets layer nor the linux stack and offloading sockets
connections.  Its only supporting offload connections for the RDMA
driver to do iWARP.   The Ammasso device is another example of this
(drivers/infiniband/hw/amso1100).  Deep iSCSI adapters are another
example of this.

So what will happen when application will create a socket, bind it to
that NIC, and then try to establish a TCP connection? How NIC will
decide that received packets are from socket but not for internal TCP
state machine handled by that device?

As a side note, does all iwarp devices _require_ to have very
limited TCP engine implemented it in its hardware, or it is possible
to work with external SW stack?
 

Steve.

-- 
	Evgeniy Polyakov

On Tue, 2006-12-05 at 08:13 +0300, Evgeniy Polyakov wrote:

On Mon, Dec 04, 2006 at 10:20:51AM -0600, Steve Wise (swise@opengridcomputing.com) wrote:

quoted

quoted

 > This and a lot of other changes in this driver definitely says you
 > implement your own stack of protocols on top of infiniband hardware.

...but I do know this driver is for 10-gig ethernet HW.

There is no SW TCP stack in this driver.  The HW supports RDMA over
TCP/IP/10GbE in HW and this is required for zero-copy RDMA over Ethernet
(aka iWARP).  The device is a 10 GbE device, not Infiniband.  The
Ethernet driver, upon which the rdma driver depends, acts both like a
traditional Ethernet NIC for the Linux stack as well as a TCP offload
device for the RDMA driver allowing establishment of RDMA connections.
The Connection Manager (patch 04/13) sends/receives messages from the
Ethernet driver that sets up HW TCP connections for doing RDMA.  While
this is indeed implementing TCP offload, it is _not_ integrating it with
the sockets layer nor the linux stack and offloading sockets
connections.  Its only supporting offload connections for the RDMA
driver to do iWARP.   The Ammasso device is another example of this
(drivers/infiniband/hw/amso1100).  Deep iSCSI adapters are another
example of this.

So what will happen when application will create a socket, bind it to
that NIC, and then try to establish a TCP connection? How NIC will
decide that received packets are from socket but not for internal TCP
state machine handled by that device?

The HW knows which TCP connections are offloaded by virtue of the fact
that they were setup via the RDMA subsystem.  Any other TCP traffic (and
all other non TCP traffic) gets passed to the host stack.

As a side note, does all iwarp devices _require_ to have very
limited TCP engine implemented it in its hardware, or it is possible
to work with external SW stack?

It is possible, but not very interesting.

One could implement an all-software iWARP stack.  The iWARP protocols
are just TCP payload and _could_ be implemented in user mode on top of a
socket.  However, this isn't very interesting:  the goal of iWARP (and
RDMA for that matter) is to allow direct placement of data into user
memory with 0 copies done by the host CPU.  low latency.

Steve.

Steve Wise wrote:

There is no SW TCP stack in this driver.  The HW supports RDMA over
TCP/IP/10GbE in HW and this is required for zero-copy RDMA over Ethernet
(aka iWARP).  The device is a 10 GbE device, not Infiniband.

Then, I wonder why the driver goes in drivers/infiniband/ :)

Is there really no way to only keep the actual hw infiniband there, move
iwarp/rdma drivers in drivers/net/something/ and the core stuff in
net/something/ ?

Brice

On Tue, 2006-12-05 at 11:45 +0100, Brice Goglin wrote:

Steve Wise wrote:

quoted

There is no SW TCP stack in this driver.  The HW supports RDMA over
TCP/IP/10GbE in HW and this is required for zero-copy RDMA over Ethernet
(aka iWARP).  The device is a 10 GbE device, not Infiniband.

Then, I wonder why the driver goes in drivers/infiniband/ :)

drivers/infiniband support both IB and IWARP transports.

Is there really no way to only keep the actual hw infiniband there, move
iwarp/rdma drivers in drivers/net/something/ and the core stuff in
net/something/ ?

Sure, this _could_ be done, but what I think you're missing is that
applications use the interface exported by drivers/infiniband over both
IB -and- IWARP transports.  The application can be written to not care
which transport is used.   Examples of apps that can run over both
transports using the same common interface: 

user mode: MVAPICH2, OMPI, IMPI, HPMPI, 
kernel mode: NFS-RDMA, iSER.  

Note that the include directory used by drivers/infiniband is now
include/rdma.  Perhaps drivers/infiniband should be renamed to
drivers/rdma as well at some point...



Steve.

On Tue, 2006-12-05 at 10:02 -0600, Steve Wise wrote:

On Tue, 2006-12-05 at 11:45 +0100, Brice Goglin wrote:

quoted

Steve Wise wrote:

quoted

There is no SW TCP stack in this driver.  The HW supports RDMA over
TCP/IP/10GbE in HW and this is required for zero-copy RDMA over Ethernet
(aka iWARP).  The device is a 10 GbE device, not Infiniband.

Then, I wonder why the driver goes in drivers/infiniband/ :)

drivers/infiniband support both IB and IWARP transports.

quoted

Is there really no way to only keep the actual hw infiniband there, move
iwarp/rdma drivers in drivers/net/something/ and the core stuff in
net/something/ ?

Sure, this _could_ be done, but what I think you're missing is that
applications use the interface exported by drivers/infiniband over both
IB -and- IWARP transports.  The application can be written to not care
which transport is used.   Examples of apps that can run over both
transports using the same common interface: 

user mode: MVAPICH2, OMPI, IMPI, HPMPI, 
kernel mode: NFS-RDMA, iSER.  

Note that the include directory used by drivers/infiniband is now
include/rdma.  Perhaps drivers/infiniband should be renamed to
drivers/rdma as well at some point...

By the way, FYI:  The Chelsio T3 device support is split into 2 driver
modules: the Ethernet driver and the RDMA driver.  The Ethernet driver
lives in drivers/net/cxgb3 while the RDMA driver lives in
drivers/infiniband/hw/cxgb3.  The Ethernet driver can be used
stand-alone as a 10GbE high-performance NIC driver.  The RDMA driver has
a config-time dependency on the Ethernet driver.

The 2nd version of the Ethernet driver was posted yesterday.  See:

http://www.spinics.net/lists/netdev/msg20464.html



Steve.

On Mon, Dec 04, 2006 at 07:45:52AM -0800, Roland Dreier (rdreier@cisco.com) wrote:

 > This and a lot of other changes in this driver definitely says you
 > implement your own stack of protocols on top of infiniband hardware.

...but I do know this driver is for 10-gig ethernet HW.

It is for iwarp/rdma from description.
If it is 10ge, then why does it parse incomping packet headers and
implements initial tcp state machine?

 - R.

-- 
	Evgeniy Polyakov

On Tue, 2006-12-05 at 08:07 +0300, Evgeniy Polyakov wrote:

On Mon, Dec 04, 2006 at 07:45:52AM -0800, Roland Dreier (rdreier@cisco.com) wrote:

quoted

 > This and a lot of other changes in this driver definitely says you
 > implement your own stack of protocols on top of infiniband hardware.

...but I do know this driver is for 10-gig ethernet HW.

It is for iwarp/rdma from description.
If it is 10ge, then why does it parse incomping packet headers and
implements initial tcp state machine?

Its not implementing the TCP state machine at all. Its implementing the
MPA state machine (see the iWARP internet drafts).  These packets are
TCP payload.  MPA is used to negotiate RDMA mode on a TCP connection.
This entails an exchange of 2 messages on the TCP connection.  Once this
is exchanged and both side agree, the connection is bound to an RDMA QP
and the connection moved into RDMA mode.  From that point on, all IO is
done via the post_send() and post_recv().


Steve. 

On Tue, Dec 05, 2006 at 09:02:05AM -0600, Steve Wise (swise@opengridcomputing.com) wrote:

quoted

quoted

 > This and a lot of other changes in this driver definitely says you
 > implement your own stack of protocols on top of infiniband hardware.

...but I do know this driver is for 10-gig ethernet HW.

It is for iwarp/rdma from description.
If it is 10ge, then why does it parse incomping packet headers and
implements initial tcp state machine?

Its not implementing the TCP state machine at all. Its implementing the
MPA state machine (see the iWARP internet drafts).  These packets are
TCP payload.  MPA is used to negotiate RDMA mode on a TCP connection.
This entails an exchange of 2 messages on the TCP connection.  Once this
is exchanged and both side agree, the connection is bound to an RDMA QP
and the connection moved into RDMA mode.  From that point on, all IO is
done via the post_send() and post_recv().

And why does rdma require window scaling, keep alive, nagle and other
interesting options from TCP spec?

This really looks like initial implementation of TCP in hardware - you
setup flags like doing the same using setsockopt() and then hardware
manages the flow like network stack manages TCP state machine changes.

According to draft-culley-iwarp-mpa-03.txt this layer can do a lot of
things with valid TCP flow like

   5.  The TCP sender puts the FPDUs into the TCP stream.  If the TCP
       Sender is MPA-aware, it segments the TCP stream in such a way
       that a TCP Segment boundary is also the boundary of an FPDU.  
       TCP then passes each segment to the IP layer for transmission.

Phrases like "MPA-aware TCP" rises a lot of questions - briefly saying
that hardware (even if it is called ethernet driver) can create and work
with own TCP flows potentially modified in the way it likes which is seen 
in driver. Likely such flows will not be seen by upper layers like OS 
network stack according to hardware descriptions.

Is it correct?

Steve. 

-- 
	Evgeniy Polyakov

On Tue, 2006-12-05 at 18:19 +0300, Evgeniy Polyakov wrote:

On Tue, Dec 05, 2006 at 09:02:05AM -0600, Steve Wise (swise@opengridcomputing.com) wrote:

quoted

quoted

quoted

 > This and a lot of other changes in this driver definitely says you
 > implement your own stack of protocols on top of infiniband hardware.

...but I do know this driver is for 10-gig ethernet HW.

It is for iwarp/rdma from description.
If it is 10ge, then why does it parse incomping packet headers and
implements initial tcp state machine?

Its not implementing the TCP state machine at all. Its implementing the
MPA state machine (see the iWARP internet drafts).  These packets are
TCP payload.  MPA is used to negotiate RDMA mode on a TCP connection.
This entails an exchange of 2 messages on the TCP connection.  Once this
is exchanged and both side agree, the connection is bound to an RDMA QP
and the connection moved into RDMA mode.  From that point on, all IO is
done via the post_send() and post_recv().

And why does rdma require window scaling, keep alive, nagle and other
interesting options from TCP spec?

The connection setup messages sent to the hardware need to have these
parameters so the TCP engine on the HW knows how to do connection
options, windows, etc.

This really looks like initial implementation of TCP in hardware - you
setup flags like doing the same using setsockopt() and then hardware
manages the flow like network stack manages TCP state machine changes.

According to draft-culley-iwarp-mpa-03.txt this layer can do a lot of
things with valid TCP flow like

   5.  The TCP sender puts the FPDUs into the TCP stream.  If the TCP
       Sender is MPA-aware, it segments the TCP stream in such a way
       that a TCP Segment boundary is also the boundary of an FPDU.  
       TCP then passes each segment to the IP layer for transmission.

Phrases like "MPA-aware TCP" rises a lot of questions - briefly saying
that hardware (even if it is called ethernet driver) can create and work
with own TCP flows potentially modified in the way it likes which is seen 
in driver. Likely such flows will not be seen by upper layers like OS 
network stack according to hardware descriptions.

Is it correct?

I don't quite get your point about the driver aspect of this?

The HW manages the iWARP connection including data flow.  It adheres to
the MPA, RDDP, and RDMAP protocol specification IDs from the IETF.  The
HW manages how data gets pushed out in the RDMA stream.   The RDMA
Driver just requests a TCP connection and does the MPA exchange.  Then
tells the hardware to move the connection into RDMA mode.  From that
point on, the driver simply suffles IO work requests from the consumer
application to the hardware and handles asynchronous events while the
connection is up and running.

Steve.

On Tue, Dec 05, 2006 at 09:39:58AM -0600, Steve Wise (swise@opengridcomputing.com) wrote:

quoted

Phrases like "MPA-aware TCP" rises a lot of questions - briefly saying
that hardware (even if it is called ethernet driver) can create and work
with own TCP flows potentially modified in the way it likes which is seen 
in driver. Likely such flows will not be seen by upper layers like OS 
network stack according to hardware descriptions.

Is it correct?

I don't quite get your point about the driver aspect of this?

The HW manages the iWARP connection including data flow.  It adheres to
the MPA, RDDP, and RDMAP protocol specification IDs from the IETF.  The
HW manages how data gets pushed out in the RDMA stream.   The RDMA
Driver just requests a TCP connection and does the MPA exchange.  Then
tells the hardware to move the connection into RDMA mode.  From that
point on, the driver simply suffles IO work requests from the consumer
application to the hardware and handles asynchronous events while the
connection is up and running.

My main concern about this is the fact, that protocol handling is
splitted into SF and HW parts, and actually until negotiation is
completed those parts are completely unrelated to each other, so
requested TCP connection can leak into main stack and main stack can
send some packets which can be considered as MPA negotiation.

Steve.

-- 
	Evgeniy Polyakov

On Tue, 2006-12-05 at 18:59 +0300, Evgeniy Polyakov wrote:

On Tue, Dec 05, 2006 at 09:39:58AM -0600, Steve Wise (swise@opengridcomputing.com) wrote:

quoted

quoted

Phrases like "MPA-aware TCP" rises a lot of questions - briefly saying
that hardware (even if it is called ethernet driver) can create and work
with own TCP flows potentially modified in the way it likes which is seen 
in driver. Likely such flows will not be seen by upper layers like OS 
network stack according to hardware descriptions.

Is it correct?

I don't quite get your point about the driver aspect of this?

The HW manages the iWARP connection including data flow.  It adheres to
the MPA, RDDP, and RDMAP protocol specification IDs from the IETF.  The
HW manages how data gets pushed out in the RDMA stream.   The RDMA
Driver just requests a TCP connection and does the MPA exchange.  Then
tells the hardware to move the connection into RDMA mode.  From that
point on, the driver simply suffles IO work requests from the consumer
application to the hardware and handles asynchronous events while the
connection is up and running.

My main concern about this is the fact, that protocol handling is
splitted into SF and HW parts, and actually until negotiation is
completed those parts are completely unrelated to each other, so
requested TCP connection can leak into main stack and main stack can
send some packets which can be considered as MPA negotiation.

Ah.  Data from an offloaded connection cannot leak into the main stack
nor vice-verse.  We can take an active RDMA connection establishment as
an example if you want:  Once the message is sent to the HW to "setup a
TCP connection from addr/port a.b to addr/port c.d", then packets on
that connection (that 4-tuple) will always be delivered to the RDMA
driver, not the native stack.  If the the packet received after the
connection is setup is -not- an MPA reply (in this example), then the
connection is aborted.  Once the connection is aborted.  So no leaking
can happen.

On Tue, 2006-12-05 at 10:12 -0600, Steve Wise wrote:

On Tue, 2006-12-05 at 18:59 +0300, Evgeniy Polyakov wrote:

quoted

On Tue, Dec 05, 2006 at 09:39:58AM -0600, Steve Wise (swise@opengridcomputing.com) wrote:

quoted

quoted

Phrases like "MPA-aware TCP" rises a lot of questions - briefly saying
that hardware (even if it is called ethernet driver) can create and work
with own TCP flows potentially modified in the way it likes which is seen 
in driver. Likely such flows will not be seen by upper layers like OS 
network stack according to hardware descriptions.

Is it correct?

I don't quite get your point about the driver aspect of this?

The HW manages the iWARP connection including data flow.  It adheres to
the MPA, RDDP, and RDMAP protocol specification IDs from the IETF.  The
HW manages how data gets pushed out in the RDMA stream.   The RDMA
Driver just requests a TCP connection and does the MPA exchange.  Then
tells the hardware to move the connection into RDMA mode.  From that
point on, the driver simply suffles IO work requests from the consumer
application to the hardware and handles asynchronous events while the
connection is up and running.

My main concern about this is the fact, that protocol handling is
splitted into SF and HW parts, and actually until negotiation is
completed those parts are completely unrelated to each other, so
requested TCP connection can leak into main stack and main stack can
send some packets which can be considered as MPA negotiation.

Ah.  Data from an offloaded connection cannot leak into the main stack
nor vice-verse.  We can take an active RDMA connection establishment as
an example if you want:  Once the message is sent to the HW to "setup a
TCP connection from addr/port a.b to addr/port c.d", then packets on
that connection (that 4-tuple) will always be delivered to the RDMA
driver, not the native stack.  If the the packet received after the
connection is setup is -not- an MPA reply (in this example), then the
connection is aborted.  Once the connection is aborted.  

                                                       ^ the 4 tuple can
then be reused for rdma or native stack tcp connections.

On Tue, Dec 05, 2006 at 10:12:42AM -0600, Steve Wise (swise@opengridcomputing.com) wrote:

Ah.  Data from an offloaded connection cannot leak into the main stack
nor vice-verse.  We can take an active RDMA connection establishment as
an example if you want:  Once the message is sent to the HW to "setup a
TCP connection from addr/port a.b to addr/port c.d", then packets on
that connection (that 4-tuple) will always be delivered to the RDMA
driver, not the native stack.  If the the packet received after the
connection is setup is -not- an MPA reply (in this example), then the
connection is aborted.  Once the connection is aborted.  So no leaking
can happen.

 
And if there were a dataflow between addr/port a.b to addr/port c.d
already, it will either terminated?

Considering the following sequence:
handlers->t3c_handlers->sched()->work_queue->work_handlers()->for
example CPL_PASS_ACCEPT_REQ->pass_accept_req() - it just parses incoming
skb and sets port/addr/route and other fields to be used as a base for rdma
connection. What if it just a usual network packet from kernelspace or 
userspace with the same payload as should be sent by remote rdma system?

-- 
	Evgeniy Polyakov

On Tue, 2006-12-05 at 19:31 +0300, Evgeniy Polyakov wrote:

On Tue, Dec 05, 2006 at 10:12:42AM -0600, Steve Wise (swise@opengridcomputing.com) wrote:

quoted

Ah.  Data from an offloaded connection cannot leak into the main stack
nor vice-verse.  We can take an active RDMA connection establishment as
an example if you want:  Once the message is sent to the HW to "setup a
TCP connection from addr/port a.b to addr/port c.d", then packets on
that connection (that 4-tuple) will always be delivered to the RDMA
driver, not the native stack.  If the the packet received after the
connection is setup is -not- an MPA reply (in this example), then the
connection is aborted.  Once the connection is aborted.  So no leaking
can happen.

 
And if there were a dataflow between addr/port a.b to addr/port c.d
already, it will either terminated?

Considering the following sequence:
handlers->t3c_handlers->sched()->work_queue->work_handlers()->for
example CPL_PASS_ACCEPT_REQ->pass_accept_req() - it just parses incoming
skb and sets port/addr/route and other fields to be used as a base for rdma
connection. What if it just a usual network packet from kernelspace or 
userspace with the same payload as should be sent by remote rdma system?

That skb isn't a network packet.  Its a CPL_PASS_ACCEPT_REQ message (see
struct cpl_pass_accept_req in the Ethernet driver t3_cpl.h).  If the
RDMA driver hadn't registered to listen on that addr/port, it would
never get this skb.  Once a connection is established, the MPA messages
(and any TCP payload data) is delivered to the RDMA driver in the form
of skb's containing struct cpl_rx_data.  So these skbs aren't just TCP
packets at all.  They either control messages or TCP payload. Either way
they are encapsulated in CPL message structures.

Does this make sense?

On Tue, Dec 05, 2006 at 10:47:25AM -0600, Steve Wise (swise@opengridcomputing.com) wrote:

quoted

And if there were a dataflow between addr/port a.b to addr/port c.d
already, it will either terminated?

Considering the following sequence:
handlers->t3c_handlers->sched()->work_queue->work_handlers()->for
example CPL_PASS_ACCEPT_REQ->pass_accept_req() - it just parses incoming
skb and sets port/addr/route and other fields to be used as a base for rdma
connection. What if it just a usual network packet from kernelspace or 
userspace with the same payload as should be sent by remote rdma system?

That skb isn't a network packet.  Its a CPL_PASS_ACCEPT_REQ message (see
struct cpl_pass_accept_req in the Ethernet driver t3_cpl.h).  If the
RDMA driver hadn't registered to listen on that addr/port, it would
never get this skb.  Once a connection is established, the MPA messages
(and any TCP payload data) is delivered to the RDMA driver in the form
of skb's containing struct cpl_rx_data.  So these skbs aren't just TCP
packets at all.  They either control messages or TCP payload. Either way
they are encapsulated in CPL message structures.

Does this make sense?

 
Almost - except the case about where those skbs are coming from?
It looks like they are obtained from network, since it is ethernet
driver, and if they match some set of rules, they are considered as valid 
MPA negotiation protocol.

If it is correct, it means that any packet in the network can be
potentially 'stolen' by rdma hardware, although it was part of the usual
dataflow. 
If that packets are not from ethernet network, but from different
low-level, then there is a question (besides why this driver is called
ethernet if it manages different hardware) about how connection over
that different media is being setup and since packets contain perfectly
valid IP addresses and ports.

And, btw, not related question - does postponing the whole skb multiplexing 
to work queue result in lower latency and/or higher speed?
Since there are a lot of tricks introduced to minimize gap between
interrupt/napi polling and protocol processing, so such huge postponing
with the whole context switch looks strange.

-- 
	Evgeniy Polyakov

On Tue, Dec 05, 2006 at 08:26:49PM +0300, Evgeniy Polyakov (johnpol@2ka.mipt.ru) wrote:

On Tue, Dec 05, 2006 at 10:47:25AM -0600, Steve Wise (swise@opengridcomputing.com) wrote:

quoted

quoted

And if there were a dataflow between addr/port a.b to addr/port c.d
already, it will either terminated?

Considering the following sequence:
handlers->t3c_handlers->sched()->work_queue->work_handlers()->for
example CPL_PASS_ACCEPT_REQ->pass_accept_req() - it just parses incoming
skb and sets port/addr/route and other fields to be used as a base for rdma
connection. What if it just a usual network packet from kernelspace or 
userspace with the same payload as should be sent by remote rdma system?

That skb isn't a network packet.  Its a CPL_PASS_ACCEPT_REQ message (see
struct cpl_pass_accept_req in the Ethernet driver t3_cpl.h).  If the
RDMA driver hadn't registered to listen on that addr/port, it would
never get this skb.  Once a connection is established, the MPA messages
(and any TCP payload data) is delivered to the RDMA driver in the form
of skb's containing struct cpl_rx_data.  So these skbs aren't just TCP
packets at all.  They either control messages or TCP payload. Either way
they are encapsulated in CPL message structures.

Does this make sense?

 
Almost - except the case about where those skbs are coming from?
It looks like they are obtained from network, since it is ethernet
driver, and if they match some set of rules, they are considered as valid 
MPA negotiation protocol.

If it is correct, it means that any packet in the network can be
potentially 'stolen' by rdma hardware, although it was part of the usual
dataflow. 
If that packets are not from ethernet network, but from different
low-level, then there is a question (besides why this driver is called
ethernet if it manages different hardware) about how connection over
that different media is being setup and since packets contain perfectly
valid IP addresses and ports.

It looks like I've answered myself - it is _not_ ethernet driver, but
rdma one, and although it gets all data through skbs from ethernet
driver, the latter gets them not from ethernet network.
And thus addresses and ports and all other information can not be mixed
between the two.

-- 
	Evgeniy Polyakov

On Tue, 2006-12-05 at 20:26 +0300, Evgeniy Polyakov wrote:

On Tue, Dec 05, 2006 at 10:47:25AM -0600, Steve Wise (swise@opengridcomputing.com) wrote:

quoted

quoted

And if there were a dataflow between addr/port a.b to addr/port c.d
already, it will either terminated?

Considering the following sequence:
handlers->t3c_handlers->sched()->work_queue->work_handlers()->for
example CPL_PASS_ACCEPT_REQ->pass_accept_req() - it just parses incoming
skb and sets port/addr/route and other fields to be used as a base for rdma
connection. What if it just a usual network packet from kernelspace or 
userspace with the same payload as should be sent by remote rdma system?

That skb isn't a network packet.  Its a CPL_PASS_ACCEPT_REQ message (see
struct cpl_pass_accept_req in the Ethernet driver t3_cpl.h).  If the
RDMA driver hadn't registered to listen on that addr/port, it would
never get this skb.  Once a connection is established, the MPA messages
(and any TCP payload data) is delivered to the RDMA driver in the form
of skb's containing struct cpl_rx_data.  So these skbs aren't just TCP
packets at all.  They either control messages or TCP payload. Either way
they are encapsulated in CPL message structures.

Does this make sense?

 
Almost - except the case about where those skbs are coming from?
It looks like they are obtained from network, since it is ethernet
driver, and if they match some set of rules, they are considered as valid 
MPA negotiation protocol.

They come from the Ethernet driver, but that driver manages multiple HW
queues and these packets come from an offload queue, not the NIC queue.
So the HW demultiplexes.

Perhaps Divy or Felix from Chelsio can expand on how the Ethernet driver
manages this?

If it is correct, it means that any packet in the network can be
potentially 'stolen' by rdma hardware, although it was part of the usual
dataflow. 
If that packets are not from ethernet network, but from different
low-level, then there is a question (besides why this driver is called
ethernet if it manages different hardware) about how connection over
that different media is being setup and since packets contain perfectly
valid IP addresses and ports.

The HW has different queues for offload vs native Ethernet frames.  I'm
not an expert on the Ethernet driver, so you'll have to consult that
code and ask questions of Divy and/or Felix.

And, btw, not related question - does postponing the whole skb multiplexing 
to work queue result in lower latency and/or higher speed?
Since there are a lot of tricks introduced to minimize gap between
interrupt/napi polling and protocol processing, so such huge postponing
with the whole context switch looks strange.

Neither.   The work queue makes the RDMA driver's life easier because it
has context to allocate skbs, for instance.  Note all the work queue
stuff is done _only_ for RDMA connection setup and teardown.  Once the
connection is in RDMA mode, there's no work queues at all for IO, and CQ
notifications happen in interrupt context.  RDMA operations are
submitted to the hardware via iwch_post_send().  Completion notification
is done in the interrupt context via iwch_ev_dispatch().  And completion
entries reaped by the consumer application via iwch_poll_cq().


Steve.

On Tue, Dec 05, 2006 at 11:51:40AM -0600, Steve Wise (swise@opengridcomputing.com) wrote:

quoted

Almost - except the case about where those skbs are coming from?
It looks like they are obtained from network, since it is ethernet
driver, and if they match some set of rules, they are considered as valid 
MPA negotiation protocol.

They come from the Ethernet driver, but that driver manages multiple HW
queues and these packets come from an offload queue, not the NIC queue.
So the HW demultiplexes.

Ok, thanks for explaination.

-- 
	Evgeniy Polyakov

If you require more details on how this all works - it was fully explored 
in the IETF RDDP workgroup - may I suggest a reading of the RDMA Security 
Considerations draft which goes through many of the issues on how one 
relates to a host stack.   This complements the MPA spec and supports much 
of what Steve has already responded to during this string of e-mails.  We 
took a great deal of time and debate to insure this can work efficiently 
and without confusion in terms of who owns what and when.

Mike



At 10:09 AM 12/5/2006, Evgeniy Polyakov wrote:

On Tue, Dec 05, 2006 at 11:51:40AM -0600, Steve Wise 
(swise@opengridcomputing.com) wrote:

quoted

quoted

Almost - except the case about where those skbs are coming from?
It looks like they are obtained from network, since it is ethernet
driver, and if they match some set of rules, they are considered as 

valid

quoted

quoted

MPA negotiation protocol.

They come from the Ethernet driver, but that driver manages multiple HW
queues and these packets come from an offload queue, not the NIC queue.
So the HW demultiplexes.

Ok, thanks for explaination.

--
        Evgeniy Polyakov

_______________________________________________
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit 
http://openib.org/mailman/listinfo/openib-general