Re: [ewg] Re: [PATCH] IB/ehca: Serialize HCA-related hCalls on POWER5

 > What is the fix you suggest, to add a device query that tells you for 
 > which verbs the documentation does not apply? or enhance the code of the 
 >   map_phys_fmr verb within the ehca driver to return error if called 
 > from non-sleepable context?

I think the right fix for iSER would be to make iSER work even for
devices that don't support FMRs.  For example cxgb3 doesn't implement
FMRs so if anyone ever updates iSER to work on iWARP and not just IB,
then this is something that has to be tackled anyway.  Then ehca could
just get rid of the FMR support it has.

Roland Dreier wrote:

I think the right fix for iSER would be to make iSER work even for
devices that don't support FMRs.  For example cxgb3 doesn't implement
FMRs so if anyone ever updates iSER to work on iWARP and not just IB,
then this is something that has to be tackled anyway.  Then ehca could
just get rid of the FMR support it has.

OK, The iSER design took into account the case of many initiators 
running on strong/modern machines talking to possibly lightweight 
embedded target for which the processing cost per I/O at the target side 
should be minimized, that is at most --one-- RDMA operation should be 
issued by the target to serve an I/O request.

For that end, iSER works with one descriptor (called stag in iWARP and 
rkey in IB) per I/O direction sent from the initiator to the target and 
hence can't work without some sort of FMR implementation.

The current implementation of the open iscsi initiator makes sure to 
issue commands in thread (sleepable) context, see iscsi_xmitworker and 
references to it in drivers/scsi/libiscsi.c , so this keeps ehca users 
safe for the time being.

Or.

On Dec 13, 2007 12:30 AM, Or Gerlitz [off-list ref] wrote:

Roland Dreier wrote:

quoted

I think the right fix for iSER would be to make iSER work even for
devices that don't support FMRs.  For example cxgb3 doesn't implement
FMRs so if anyone ever updates iSER to work on iWARP and not just IB,
then this is something that has to be tackled anyway.  Then ehca could
just get rid of the FMR support it has.

OK, The iSER design took into account the case of many initiators
running on strong/modern machines talking to possibly lightweight
embedded target for which the processing cost per I/O at the target side
should be minimized, that is at most --one-- RDMA operation should be
issued by the target to serve an I/O request.

For that end, iSER works with one descriptor (called stag in iWARP and
rkey in IB) per I/O direction sent from the initiator to the target and
hence can't work without some sort of FMR implementation.

The current implementation of the open iscsi initiator makes sure to
issue commands in thread (sleepable) context, see iscsi_xmitworker and
references to it in drivers/scsi/libiscsi.c , so this keeps ehca users
safe for the time being.

Or.

I agree, *some* form of FMR support is important for iSER (and probably
for NFS over RDMA as well). Rather than adding a crippled NO FMR
mode it would make more sense to add support for FMR Work Requests.
I'm not certain what, if any, impact that would have on the Power5 problem,
but that's certainly a cleaner path for iWARP.

caitlin.bestler@gmail.com wrote on 13.12.2007 20:22:49:

On Dec 13, 2007 12:30 AM, Or Gerlitz [off-list ref] wrote:

quoted

The current implementation of the open iscsi initiator makes sure to
issue commands in thread (sleepable) context, see iscsi_xmitworker and
references to it in drivers/scsi/libiscsi.c , so this keeps ehca users
safe for the time being.

I agree, *some* form of FMR support is important for iSER (and probably
for NFS over RDMA as well). Rather than adding a crippled NO FMR
mode it would make more sense to add support for FMR Work Requests.
I'm not certain what, if any, impact that would have on the Power5 

problem,

but that's certainly a cleaner path for iWARP.

Well, FMR WRs wouldn't change the eHCA issue -- the driver would have to 
make an hCall in any case, and the architecture says that the hCalls used 
in this scenario might return H_LONG_BUSY, causing the driver to sleep. No 
way around that. Because of this, eHCA's FMRs are actually standard MRs 
with a different API.

If, as Or said, the iSCSI initiator issues commands in sleepable context 
anyway, nothing would be lost by using standard MRs as a fallback solution 
if FMRs aren't available, would it?

J.

-----Original Message-----
From: Joachim Fenkes [mailto:FENKES@de.ibm.com]
Sent: Thursday, December 13, 2007 1:00 PM
To: Caitlin Bestler
Cc: Arnd Bergmann; caitlin.bestler@gmail.com; OF-General; LKML;
linuxppc-dev@ozlabs.org; Or Gerlitz; Roland Dreier; Stefan Roscher
Subject: Re: [ofa-general] Re: [ewg] Re: [PATCH] IB/ehca: Serialize
HCA-related hCalls on POWER5
=20
caitlin.bestler@gmail.com wrote on 13.12.2007 20:22:49:
=20

quoted

On Dec 13, 2007 12:30 AM, Or Gerlitz [off-list ref] wrote:

quoted

The current implementation of the open iscsi initiator makes sure

to

quoted

quoted

issue commands in thread (sleepable) context, see iscsi_xmitworker

and

quoted

quoted

references to it in drivers/scsi/libiscsi.c , so this keeps ehca

users

quoted

quoted

safe for the time being.

=20

quoted

I agree, *some* form of FMR support is important for iSER (and

probably

quoted

for NFS over RDMA as well). Rather than adding a crippled NO FMR
mode it would make more sense to add support for FMR Work Requests.
I'm not certain what, if any, impact that would have on the Power5

problem,

quoted

but that's certainly a cleaner path for iWARP.

=20
Well, FMR WRs wouldn't change the eHCA issue -- the driver would have
to
make an hCall in any case, and the architecture says that the hCalls
used
in this scenario might return H_LONG_BUSY, causing the driver to

sleep.

No
way around that. Because of this, eHCA's FMRs are actually standard

MRs

with a different API.
=20
If, as Or said, the iSCSI initiator issues commands in sleepable
context
anyway, nothing would be lost by using standard MRs as a fallback
solution
if FMRs aren't available, would it?
=20

To clarify, an FMR Work Request is simply posted to the SendQ like
any other Work Request (of course the QP has to be privileged, or
it will complete in error). An SQ Post should never block.

But yes, if the current iSCSI initiator always does all call-based
FMRs in a sleepable context then I would agree then any changes can
wait for the first vendor that wants to support FMR Work Requests.

FMR Work Requests can be pipelined, so anyone with hardware that
supported them would have strong motivation to enable the open
iSCSI initiator to take advantage of this.

"Caitlin Bestler" [off-list ref] wrote on 13.12.2007 
22:08:34:

To clarify, an FMR Work Request is simply posted to the SendQ like
any other Work Request (of course the QP has to be privileged, or
it will complete in error). An SQ Post should never block.

This would require hardware support, wouldn't it? eHCA2 doesn't have this 
kind of support, so FMR WRs are not an option here.

J.

To clarify, an FMR Work Request is simply posted to the SendQ like
any other Work Request (of course the QP has to be privileged, or
it will complete in error). An SQ Post should never block.

FMR's as defined by the IB spec and that created by Mellanox are not the same.
They, unfortunately, use the same name and acronym only.  Mellanox FMRs use an
API that is more like that of standard MRs. 

- Sean