On Aug 31, 2021, at 4:42 PM, J. Bruce Fields [off-list ref] wrote:

On Tue, Aug 31, 2021 at 03:05:09PM -0400, Chuck Lever wrote:

quoted

Hi Bruce-

Here is part of what we discussed recently about trying to align
pages in NFS WRITE requests so that splice can be used. This
series updates server-side RPC and svcrdma to ensure that an
aligned xdr_buf::pages array is presented to NFSD, which is then
converted into an aligned rq_vec for the VFS layer.

Seems sensible to me.

Do you have a git tree?

I don't yet, but can set up a topic branch somewhere where we're
a little further along.

It didn't apply cleanly to 5.14 when I tried,
but I didn't stop to figure out why.

It might apply to 5.15-pre now that nfsd-5.15 has been merged.

quoted

The next step would be to look at how to make the best use of the
aligned rq_vec.

Have you done any performance comparison just with this?

Not yet. I just got it behaving correctly with the usual tests.
Baby steps, sir.

Doesn't seem like it should make a significant difference, but it might
be interesting to check anyway.

I expect it to add a small amount of latency to NFS WRITEs, since
RDMA Reads are done just a little later than before.

--b.

quoted

My naive thought is that where there is a PAGE_SIZE
entry in rq_vec and there is no page in the file's page cache at
that offset, the transport-provided page can be flipped into place.
Might work for replacing whole pages as well, but baby steps first.

This series has been exercised a bit with both TCP and RDMA, but no
guarantees that it is completely bug-free. NFSv4 compounds with
multiple WRITE payloads on RDMA are treated like TCP: the RPC
message is contained in an unstructured stream of unaligned pages.

Comments encouraged.

---

Chuck Lever (6):
     SUNRPC: Capture value of xdr_buf::page_base
     SUNRPC: xdr_stream_subsegment() must handle non-zero page_bases
     NFSD: Have legacy NFSD WRITE decoders use xdr_stream_subsegment()
     SUNRPC: svc_fill_write_vector() must handle non-zero page_bases
     NFSD: Add a transport hook for pulling argument payloads
     svcrdma: Pull Read chunks in ->xpo_argument_payload


fs/nfsd/nfs3proc.c                       |   3 +-
fs/nfsd/nfs3xdr.c                        |  16 +--
fs/nfsd/nfs4proc.c                       |   3 +-
fs/nfsd/nfs4xdr.c                        |   6 +
fs/nfsd/nfsproc.c                        |   3 +-
fs/nfsd/nfsxdr.c                         |  13 +--
fs/nfsd/xdr.h                            |   2 +-
fs/nfsd/xdr3.h                           |   2 +-
include/linux/sunrpc/svc.h               |   6 +-
include/linux/sunrpc/svc_rdma.h          |   8 ++
include/linux/sunrpc/svc_xprt.h          |   3 +
include/trace/events/rpcrdma.h           |  26 +++++
include/trace/events/sunrpc.h            |  20 +++-
net/sunrpc/svc.c                         |  38 +++++--
net/sunrpc/svcsock.c                     |   8 ++
net/sunrpc/xdr.c                         |  32 +++---
net/sunrpc/xprtrdma/svc_rdma_recvfrom.c  |  37 +++++-
net/sunrpc/xprtrdma/svc_rdma_rw.c        | 139 ++++++++++++++++++++---
net/sunrpc/xprtrdma/svc_rdma_transport.c |   1 +
19 files changed, 292 insertions(+), 74 deletions(-)

--
Chuck Lever

--
Chuck Lever