On Jun 27, 2017, at 12:07 PM, Sagi Grimberg [off-list ref] wrote:

quoted

Go back and browse the end of the thread: there's no need to change
xprtrdma, and maybe no need to change the others either.

I think there is, even with inline, xprtrdma dma maps the immediate
buffers, also the message head and tail so unmapping these buffers
without waiting for the send completion would trigger a IOMMU access
error (the HCA (re)tries to access an already unmapped buffer).

Thinking out loud:

IIRC the message head and tail reside in the persistently registered
and DMA mapped buffers with few exceptions.

However, when page cache pages are involved, xprtrdma will do a DMA
unmap as you say.

So:
- we don't have a problem transmitting a garbled request thanks to
  exactly-once receive semantics
- we don't have a problem with the timing of registration and
  invalidation on the initiator because the PD's DMA lkey is used
- we do have a problem with DMA unmap

Using only persistently registered and DMA mapped Send buffers
should avoid the need to signal all Sends. However, when page
cache pages are involved, then the Send needs to be signaled,
and the pages unmapped only after Send completion, to be completely
safe.

Or, the initiator should just use RDMA Read and Write, and stick
with small inline sizes.


--
Chuck Lever



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html