On Fri, May 10, 2019 at 01:12:49PM -0700, Santosh Shilimkar wrote:

On 5/10/2019 12:47 PM, Jason Gunthorpe wrote:

quoted

On Fri, May 10, 2019 at 12:38:31PM -0700, Santosh Shilimkar wrote:

quoted

On 5/10/2019 12:20 PM, Jason Gunthorpe wrote:

quoted

On Fri, May 10, 2019 at 11:58:42AM -0700, santosh.shilimkar@oracle.com wrote:

quoted

On 5/10/19 11:07 AM, Jason Gunthorpe wrote:

quoted

On Fri, May 10, 2019 at 11:02:35AM -0700, santosh.shilimkar@oracle.com wrote:

[...]

quoted

quoted

quoted

Why would you need to detect FS DAX memory? GUP users are not supposed
to care.

GUP is supposed to work just 'fine' - other than the usual bugs we
have with GUP and any FS backed memory.

Am not saying there is any issue with GUP. Let me try to explain the
issue first. You are aware of various discussions about doing DMA
or RDMA on FS DAX memory. e.g [1] [2] [3]

One of the proposal to do safely RDMA on FS DAX memory is/was ODP

It is not about safety. ODP is required in all places that would have
used gup_longterm because ODP avoids th gup_longterm entirely.

quoted

Currently RDS doesn't have support for ODP MR registration
and hence we don't want user application to do RDMA using
fastreg/fmr on FS DAX memory which isn't safe.

No, it is safe.

The only issue is you need to determine if this use of GUP is longterm
or short term. Longterm means userspace is in control of how long the
GUP lasts, short term means the kernel is in control.

ie posting a fastreg, sending the data, then un-GUP'ing on completion
is a short term GUP and it is fine on any type of memory.

So if it is a long term pin then it needs to be corrected and the only
thing the comment needs to explain is that it is a long term pin.

Thanks for clarification. At least the distinction is clear to me now. Yes
the key can be valid for long term till the remote RDMA IO is issued and
finished. After that user can issue an invalidate/free key or
upfront specify a flag to free/invalidate the key on remote IO
completion.

Again, the test is if *userspace* controls this. So if userspace is
the thing that does the invalidate/free then it is long term. Sounds
like if it specifies the free/invalidate flag then it short term.

At this point you'd probably be better to keep both options.

Thats possible using the provided flag state but I am still not sure
whether its guaranteed to be safe when marked as short term even with
flag which tells kernel to invalidate/free the MR on remote IO
completion. Till the remote server finishes the IO, 

This is fine.

there is still a window where userspace on local server can modify
the file mappings. Registered file handle say was ftuncated to zero
by another process and the backing memory was allocated by some
other process as part of fallocate.  

The FS is supposed to maintain sane semantics across GUP - fallocate
should block until GUP is done. This is normal.

How do we avoid such an issue without GUP_longterm ?

You don't, there is no problem using GUP for short term - ie a time
frame entirely under control of the kernel.

Jason