From Israel,

The purpose of this patchset is to add support for inline
encryption/decryption of the data at storage protocols like nvmf over
RDMA (at a similar way like integrity is used via unique mkey).

This patchset adds support for plaintext keys. The patches were tested
on BF-3 HW with fscrypt tool to test this feature, which showed reduce
in CPU utilization when comparing at 64k or more IO size. The CPU utilization
was improved by more than 50% comparing to the SW only solution at this case.

How to configure fscrypt to enable plaintext keys:
  # mkfs.ext4 -O encrypt /dev/nvme0n1
  # mount /dev/nvme0n1 /mnt/crypto -o inlinecrypt
  # head -c 64 /dev/urandom > /tmp/master_key
  # fscryptctl add_key /mnt/crypto/ < /tmp/master_key
  # mkdir /mnt/crypto/test1
  # fscryptctl set_policy 152c41b2ea39fa3d90ea06448456e7fb /mnt/crypto/test1
    ** “152c41b2ea39fa3d90ea06448456e7fb” is the output of the
       “fscryptctl add_key” command.
  # echo foo > /mnt/crypto/test1/foo

Notes:
  - At plaintext mode only, the user set a master key and the fscrypt
    driver derived from it the DEK and the key identifier.
  - 152c41b2ea39fa3d90ea06448456e7fb is the derived key identifier
  - Only on the first IO, nvme-rdma gets a callback to load the derived DEK.

There is no special configuration to support crypto at nvme modules.

Hey, this looks sane to me in a very first glance.

Few high level questions:
- what happens with multipathing? when if not all devices are
capable. SW fallback?
- Does the crypt stuff stay intact when bio is requeued?

I'm assuming you tested this with multipathing? This is not very
useful if it is incompatible with it.