Re: [PATCH net-next 0/2] devlink: Add port function attribute to enable/disable roce

On Wed, 03 Feb 2021 11:22:44 -0800 Saeed Mahameed wrote:

On Wed, 2021-02-03 at 10:51 -0800, Jakub Kicinski wrote:

quoted

On Tue, 02 Feb 2021 20:13:48 -0800 Saeed Mahameed wrote:  

quoted

yes, user in this case is the admin, who controls the provisioned
network function SF/VFs.. by turning off this knob it allows to
create
more of that resource in case the user/admin is limited by memory.  

Ah, so in case of the SmartNIC this extra memory is allocated on the
control system, not where the function resides?

most of the memeory are actually allocated from where the function
resides, some are on the management system but it is not as critical.
SFs for now can only be probed on the management system, so the main
issue will be on the SmartNIC side for now.

Why not leave the decision whether to allocate that memory or not to
the SF itself? If user never binds the RDMA driver to the SF they
clearly don't care for RDMA. No extra knobs needed.

quoted

My next question is regarding the behavior on the target system -
what
does "that user" see? Can we expect they will understand that the
limitation was imposed by the admin and not due to some
initialization
failure or SW incompatibility?

the whole thing works with only real HW capabilities, there is no
synthetic SW capabilities. 

when mlx5 instance driver loads, it doesn't assume anything about
underlying HW, and it queries for the advertised FW capability
according to the HW spec before it enables a feature.

so this patch adds the ability for admin to enforce a specific HW cap
"off" for a VF/SF hca slice.

quoted

quoted

RAW eth QP, i think you already know this one, it is a very thin
layer
that doesn't require the whole rdma stack.  

Sorry for asking a leading question. You know how we'll feel about
that one, do we need to talk this out or can we save ourselves the
battle? :S  

I know, I know :/

So, there is no rdma bit/cap in HW.. to disable non-RoCE commands we
will have to disable etherent capability. 

It's your driver, you can make it do what you need to. Why does 
the RDMA driver bind successfully to a non-RoCE Ethernet device 
in the first place?

The user interface here has no synthetic semantics, all knobs will
eventually be mapped to real HW/FW capabilities to get disabled.

the whole feature is about allowing admin to ship network functions
with different capabilities that are actually enforced by FW/HW.. 
so the user of the VF will see, RDMA/ETH only cards or both.

RDMA-only, ETH-only, RDMA+ETH makes sense to me. Having an ETH-only
device also exposed though rdma subsystem does not.