Am 24.08.21 um 11:06 schrieb Gal Pressman:

On 23/08/2021 13:43, Christian König wrote:

quoted

Am 21.08.21 um 11:16 schrieb Gal Pressman:

quoted

On 20/08/2021 17:32, Jason Gunthorpe wrote:

quoted

On Fri, Aug 20, 2021 at 03:58:33PM +0300, Gal Pressman wrote:

quoted

Though it would've been nicer if we could agree on a solution that could work
for more than 1-2 RDMA devices, using the existing tools the RDMA subsystem
has.

I don't think it can really be done, revoke is necessary, and isn't a
primitive we have today.

Revoke is sort of like rereg MR, but with a guaranteed no-change to
the lkey/rkey

Then there is the locking complexity of linking the mr creation and
destruction to the lifecycle of the pages, which is messy and maybe
not general. For instance mlx5 would call its revoke_mr, disconnect
the dmabuf then destroy the mkey - but this is only safe because mlx5
HW can handle concurrent revokes.

Thanks, that makes sense.

quoted

quoted

That's why I tried to approach this by denying such attachments for non-ODP
importers instead of exposing a "limited" dynamic importer.

That is fine if there is no revoke - once revoke exists we must have
driver and HW support.

Agree.
IIUC, we're talking about three different exporter "types":
- Dynamic with move_notify (requires ODP)
- Dynamic with revoke_notify
- Static

Which changes do we need to make the third one work?

Basically none at all in the framework.

You just need to properly use the dma_buf_pin() function when you start using a
buffer (e.g. before you create an attachment) and the dma_buf_unpin() function
after you are done with the DMA-buf.

I replied to your previous mail, but I'll ask again.
Doesn't the pin operation migrate the memory to host memory?

Sorry missed your previous reply.

And yes at least for the amdgpu driver we migrate the memory to host 
memory as soon as it is pinned and I would expect that other GPU drivers 
do something similar.

This is intentional since we don't want any P2P to video memory with 
pinned objects and want to avoid to run into a situation where one 
device is doing P2P to video memory while another device needs the 
DMA-buf in host memory.

You can still do P2P with pinned object, it's just up to the exporting 
driver if it is allowed or not.

The other option is what Daniel suggested that we have some kind of 
revoke. This is essentially what our KFD is doing as well when doing 
interop with 3D GFX, but from Jasons responses I have a bit of doubt 
that this will actually work on the hardware level for RDMA.

Regards,
Christian.