Re: [PATCH for-next V2 02/22] IB/core: change pkey table lookups to support full and partial membership for the same pkey

On Wednesday 12 September 2012 19:48, Doug Ledford wrote:

quoted

On the Hypervisor, however, we assume that if both versions of the pkey are in its pkey table,
then for its own infiniband operation (as opposed to performing its pkey virtualizing function),
it should operate with the highest membership type in its table for a given 15-bit pkey.

That's what I was looking for.  So, how can you know this assumption is
correct?  It seems to me that if someone wanted to restrict membership
of the hypervisor as part of a security lockdown, then give full
membership to a guest because that guest is some high security, single
task guest, then this assumption would break things (the user would be
able to assign the full membership key to the guest OK, but regardless
of how they wanted the hypervisor to be subscribed to that particular
pkey, it would always get the full membership from the guest).

This issue, unfortunately, opens up a real "can of worms".
ib_find_cached_pkey() is used by the CM in determining the (15-bit) p_key
to be used for the connection (although 16-bit pkeys are placed in the
CM_REQ message (see IB Spec 1.2.1, table 99, page 667). The REQ handler on
the remote side finds the index in its pkey table which contains the
(15-bit) pkey enclosed in the REQ message.  This index is then used
when creating the local RC qp as its pkey_index.
AFAIK, no check is performed regarding compatible membership forms (i.e.,
that at least one of the two sides has the full membership form of the
15-bit pkey) -- it is the network administrator's responsibility to see
that the pkey configurations are correct.

Up to now, the ib_core driver has assumed that only 1 membership form per
15-bit pkey is contained in the local pkey table.

Now that both forms may exist in the Hypervisor's table, we do have a problem.
1. We cannot depend on ib_find_cached_pkey simply finding the first 15-bit pkey
   which matches its search, since we cannot depend on the order of full vs limited
   membership forms appearing in the pkey table (the order is not controllable -- see
   the example at the end of this post).
2. We therefore need a consistent policy for preferentially retrieving either the full
   or the partial member for a given 15-bit pkey.
3. This policy must be configurable by the administrator per host, per HCA.

The problems that I see:
1. What if the Hypervisor needs to have limited membership for some connections, but
   full membership for others?  Such a split policy is complex to implement -- it would
   require specifying PER PKEY whether the preferential return should be full, or should
   be limited.
2. What if the Hypervisor has several HCAs, and wishes a different policy for each HCA?

There may be more problems as well.  Liran?

In any event, the issue of multiple pkey forms in a given pkey table is actually separate
from SRIOV (it's just that SRIOV needs multiple forms, so the issue came up).
Solving it will require changes in the ib core driver.

I don't yet have a concrete proposal to fix this at present.  Any ideas would be appreciated.

-Jack

P.S. We have the same issue in procedure ib_find_pkey(), also in this patch, in core/device.c .
This procedure is used in lots of places: ipoib, core/multicast, sa_query.

I seem to recall that there were problems with IPoIB when partial membership pkeys are used.
Liran, do you recall something?

======================================
It is also impossible to demand that the pkey table be ordered so that if both limited
and full pkeys are present, then the full membership value always appears first.

A simple example:
  1. Fill the entire pkey table with full-membership pkeys. Say pkey A and pkey B are both
     in the table, and that pkey A appears first.

  2. Delete full-membership pkey A, and add the limited-membership value of Pkey B. This
     new limited-pkey-B will be entered into the pkey table by OpenSM in the position just
     vacated by the deleted pkey A -- it is the only available slot!.

  3. The Limited-membership pkey-B value will therefore unavoidably appear in the table
     before the full-membership pkey-B value.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html