quoted

 I have a proposal to rework/redefine PDs to support a more general
model,

It would certainly be good to have some text explaining some of the mappings
to different technologies.

quoted

which I think will work for NICs that
need a PD and ones that don't.  It can support MR -> PD -> Job, but I
considered the PD -> job relationship as 1 to many.

Yes, and the 1:1 is degenerate.

quoted

Sure, It's challenging in that a UET endpoint (QP) may communicate
with multiple jobs, and a MR may be accessible by a single job, all
jobs, or only a few.

I would suggest that the PD is a superset of all jobs and the objects (endpoint,
mr, etc) get to choose a subset of the PD's jobs during allocation?

Or you keep job/pd as 1:1 and allow specifying multiple PDs during object
allocation.

But to be clear, this is largely verbs modeling stuff - however there is a certain
practicality to trying to fit this multi-job ability into a PD because it allow
reusing alot of existing uAPI kernel code.

Especially if people are going to take existing RDMA HW and tweak it to some
level of UET (ie support only single job) and still require a HW level PD under
the covers.

Yes, I'm trying to ensure that the existing RDMA model continues to work but also support NICs/transports which implement the equivalent security model at the QP (endpoint) level, reusing the PD for both.

Specifically, I want to *allow* separating the different functions that a single PD provides into separate PDs.  The functions being page mapping (registration), local (lkey) access, and remote (rkey) access.  The RDMA model limits a QP to a single PD for all.  To support job-based transports, I propose allowing a QP to use 1 PD for local access (PD specified at QP creation) and multiple PDs for remote access.  Each PD used for remote access would correspond to a different job.

Note: a NIC may limit a QP to being used with a single job and require the local and remote PD be the same (i.e. 1 pd per qp).  So, the RDMA model still fits.

As an optimization, registration can be a separate function, so that the same page mapping can be re-used across different jobs as they start and end.  This requires some ability to import a MR from one PD into another.  This is probably just an optimization and not required for a job model.

I was still envisioning a job manager allocating device specific resources for a job and sharing those with the local processes.  I.e. it shares a set of fd's, with each fd associated with a device, which restricts the job to those devices.  A job may also have device specific resource limits or allocations (limit on number of MRs, specific endpoint addresses, etc.)  A global job object could work, but a subsequent user to device flow will need to access and translate the global object.  Either way, there's uABI requirement(s).

- Sean