Hi Timo,

Can you get a network trace? Also, you say that the copy_file_range()
(after what looks like a successful copy) never returns (and
application hangs), can you get a sysrq output of what the process's
stack (echo t > /proc/sysrq-trigger  and see what gets dumped into the
var log messages and locate your application and report what the stack
says)?

On Sat, Feb 13, 2021 at 10:41 PM Timo Rothenpieler
[off-list ref] wrote:

On our Fileserver, running a few weeks old 5.10, we are running into a
weird issue with NFS 4.2 Server-Side Copy and RDMA (and ZFS, though I'm
not sure how relevant that is to the issue).
The servers are connected via InfiniBand, on a Mellanox ConnectX-4 card,
using the mlx5 driver.

Anything using the copy_file_range() syscall to copy stuff just hangs.
In strace, the syscall never returns.

Simple way to reproduce on the client:
 > xfs_io -fc "pwrite 0 1M" testfile
 > xfs_io -fc "copy_range testfile" testfile.copy

The second call just never exits. It sits in S+ state, with no CPU
usage, and can easily be killed via Ctrl+C.
I let it sit for a couple hours as well, it does not seem to ever complete.

Some more observations about it:

If I do a fresh reboot of the client, the operation works fine for a
short while (like, 10~15 minutes). No load is on the system during that
time, it's effectively idle.

The operation actually does successfully copy all data. The size and
checksum of the target file is as expected. It just never returns.

This only happens when mounting via RDMA. Mounting the same NFS share
via plain TCP has the operation work reliably.

Had this issue with Kernel 5.4 already, and had hoped that 5.10 might
have fixed it, but unfortunately it didn't.

I tried two server and 30 different client machines, they all exhibit
the exact same behaviour. So I'd carefully rule out a hardware issue.


Any pointers on how to debug or maybe even fix this?



Thanks,
Timo