RE: [PATCH v3 05/11] xprtrdma: Do not wait if ib_post_send() fails

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

Moving the QP into error state right after with rdma_disconnect
you are not sure that none of the subset of the invalidations
that _were_ posted completed and you get the corresponding MRs
in a bogus state...

Moving the QP to error state and then draining the CQs means
that all LOCAL_INV WRs that managed to get posted will get
completed or flushed. That's already handled today.

It's the WRs that didn't get posted that I'm worried about
in this patch.

Are there RDMA consumers in the kernel that use that third
argument to recover when LOCAL_INV WRs cannot be posted?

None :)

quoted

quoted

quoted

I suppose I could reset these MRs instead (that is,
pass them to ib_dereg_mr).

Or, just wait for a completion for those that were posted
and then all the MRs are in a consistent state.

When a LOCAL_INV completes with IB_WC_SUCCESS, the associated
MR is in a known state (ie, invalid).

The WRs that flush mean the associated MRs are not in a known
state. Sometimes the MR state is different than the hardware
state, for example. Trying to do anything with one of these
inconsistent MRs results in IB_WC_BIND_MW_ERR until the thing
is deregistered.

Correct.

It is legal to invalidate an MR that is not in the valid state.  So

you

quoted

quoted

quoted

quoted

quoted

don't

quoted

quoted

have to deregister it, you can assume it is valid and post another

LINV

quoted

quoted

quoted

WR.

quoted

quoted

quoted

I've tried that. Once the MR is inconsistent, even LOCAL_INV
does not work.

Maybe IB Verbs don't mandate that invalidating an invalid MR must be

allowed?

quoted

quoted

(looking at the verbs spec now).

IB Verbs doesn't have specify this requirement.  iW verbs does.  So

transport

quoted

quoted

independent applications cannot rely on it.  So ib_dereg_mr() seems to be

the

quoted

quoted

only thing you can do.

quoted

If the MR is truly invalid, then there is no issue, and
the second LOCAL_INV completes successfully.

The problem is after a flushed LOCAL_INV, the MR state
sometimes does not match the hardware state. The MR is
neither registered or invalid.

There is a difference, at least with iWARP devices, between the MR state:

VALID

quoted

quoted

vs INVALID, and if the MR is allocated or not.

quoted

A flushed LOCAL_INV tells you nothing more than that the
LOCAL_INV didn't complete. The MR state at that point is
unknown.

With respect to iWARP and cxgb4: when you allocate a fastreg MR, HW has an

entry

quoted

for that MR and it is marked "allocated".  The MR record in HW also has a

state:

quoted

quoted

VALID or INVALID.  While the MR is "allocated" you can post WRs to

invalidate it

quoted

quoted

which changes the state to INVALID, or fast-register memory which makes it
VALID.  Regardless of what happens on any given QP, the MR remains

"allocated"

quoted

quoted

until you call ib_dereg_mr().  So at least for cxgb4, you could in fact

just

quoted

quoted

quoted

post another LINV to get it back to a known state that allows subsequent
fast-reg WRs.

Perhaps IB devices don't work this way.

What error did you get when you tried just doing an LINV after a flush?

With CX-2 and CX-3, after a flushed LOCAL_INV, trying either
a FASTREG or LOCAL_INV on that MR can sometimes complete with
IB_WC_MW_BIND_ERR.

I wonder if you post a FASREG+LINV+LINV if you'd get the same failure?  IE
invalidate the same rkey twice.  Just as an experiment...

Once the MR is in this state, FASTREG does not work either.
All FASTREG and LINV flush with IB_WC_MW_BIND_ERR until
the MR is deregistered.

Mellanox can probably tell us why. 

I was just wondering if posting a double LINV on a valid working FRMR would fail
with these devices.  But its moot.  As you've concluded, looks like the only
safe was to handle this is to dereg them and reallocate...


--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html