From: Saleem, Shiraz <redacted>
Sent: Friday, March 26, 2021 1:40 AM

quoted

Subject: RE: [PATCH v2 05/23] ice: Add devlink params support

[..]

quoted

Resources are better represented as devlink resource.
Such as,

$ devlink resource set pci/0000:06:00.0 /rdma/max_qps 16384 $ devlink
resource set pci/0000:06:00.0 /rdma/max_cqs 8192 $ devlink resource
set pci/0000:06:00.0 /rdma/max_mrs 16384

Hi Parav - Thank you for the feedback.

Maybe I am missing something but I see that a devlink hot reload is required
to enforce the update?

It isn't mandatory to reload, but yes either reload or driver unbind/bind is needed as you suggested below.

There isn't really a de-init required of PCI driver entities in this case for this
rdma param.
But only an unplug, plug of the auxdev with new value. Intuitively it feels
more runtime-ish.

Driver unbind/bind to reflect new limits is ok for cases where it is not time sensitive.
For mlx5 use cases, user expects device to be provisioned in < few msecs.
And driver unbind/bind are sub-optimal to achieve it from time and memory wise.
So mlx5 driver prefers to stay away from driver unbind/bind steps.
So I am working on series to not create class aux devices by default for SFs.
Rather to create them on reload.
Something like, 
$ devlink dev param set pci/0000:06:00.0 name pcisf_classes value false cmode driverinit
$ devlink dev class set auxiliary/mlx5_core.sf.4 rdma true
$ devlink resource set auxiliary/mlx5_core.sf.4 path rdma/max_qps size 200000
$ devlink dev reload auxiliary/mlx5_core.sf.4
This last command will create the mlx5_core.rdma.4 aux device and when its bound to driver, it will create IB device with right max_qp.
So rdma device is created only once, instead of twice using unbind/bind sequence.

This may not be possible for the PF/VF device due to backward compatibility and their different usage in system.

There is also a device powerof2 requirement on the maxqp which I don't see
enforceable as it stands.

Right, but similar to size_params.size_max, size_granularity, I believe size_params can be extended for alignment restriction.

This is not super-critical for the initial submission but a nice to have. But I do
want to brainstorm options..

If max_qp is truly a dynamic value that doesn't require device recreation,
extending existing rdma resource command seems more useful to end user than doing unbind/bind.