quoted

quoted

Like I said already, I think Job needs to be a first class RDMA
object that is used by all transports that have job semantics.

How do you handle or expose device specific resource allocations or
restrictions, which may be needed?  Should a kernel 'RDMA job manager'
abstract device level resources?

Consider a situation where a MR or MW should only be accessible by a
specific job.  When the MR is created, the device specific job
resource may be needed.  Should drivers need to query the job manager
to map some global object to a device specific resource?

I imagine for cases like that the job would be linked to the PD and then MR ->
PD -> Job.

The kernel side would create any HW object for the job when the PD is created
for a specific HW device.

The PD security semantic for the MR would be a little bit different in that the
PD is more like a shared PD.

The PD is a problem, as it's not a transport function.  It's a hardware implementation component; one which may NOT exist for a UEC NIC.  (I know there are NICs which do not implement PDs and have secure RDMA transfers.)  I have a proposal to rework/redefine PDs to support a more general model, which I think will work for NICs that need a PD and ones that don't.  It can support MR -> PD -> Job, but I considered the PD -> job relationship as 1 to many.  I can't immediately think of a reason why a 1:1 'job-based PD' wouldn't work in theory.

It's challenging in that a UET endpoint (QP) may communicate with multiple jobs, and a MR may be accessible by a single job, all jobs, or only a few.

Basically, the RDMA PD model forces a HW implementation.  Some, but not all, NICs will implement this.  But in general, there's not a clean {PD, QP, MR, job} relationship.

- Sean