Re: [PATCH net-next 0/2] devlink: Add port function attribute to enable/disable roce

On Wed, 2021-02-03 at 13:26 -0800, Jakub Kicinski wrote:

On Wed, 03 Feb 2021 11:22:44 -0800 Saeed Mahameed wrote:

quoted

On Wed, 2021-02-03 at 10:51 -0800, Jakub Kicinski wrote:

quoted

On Tue, 02 Feb 2021 20:13:48 -0800 Saeed Mahameed wrote:  

quoted

yes, user in this case is the admin, who controls the
provisioned
network function SF/VFs.. by turning off this knob it allows to
create
more of that resource in case the user/admin is limited by
memory.  

Ah, so in case of the SmartNIC this extra memory is allocated on
the
control system, not where the function resides?

most of the memeory are actually allocated from where the function
resides, some are on the management system but it is not as
critical.
SFs for now can only be probed on the management system, so the
main
issue will be on the SmartNIC side for now.

Why not leave the decision whether to allocate that memory or not to
the SF itself? If user never binds the RDMA driver to the SF they
clearly don't care for RDMA. No extra knobs needed.

Sorry about the late response.

But FW is already setup for RDMA weather SW wants it or not for this
specific function.
a system admin may want to deploy a leaner function to the client. we
can disable RoCE in FW before we even instantiate this function.

quoted

quoted

My next question is regarding the behavior on the target system -
what
does "that user" see? Can we expect they will understand that the
limitation was imposed by the admin and not due to some
initialization
failure or SW incompatibility?

the whole thing works with only real HW capabilities, there is no
synthetic SW capabilities. 

when mlx5 instance driver loads, it doesn't assume anything about
underlying HW, and it queries for the advertised FW capability
according to the HW spec before it enables a feature.

so this patch adds the ability for admin to enforce a specific HW
cap
"off" for a VF/SF hca slice.

quoted

quoted

RAW eth QP, i think you already know this one, it is a very
thin
layer
that doesn't require the whole rdma stack.  

Sorry for asking a leading question. You know how we'll feel
about
that one, do we need to talk this out or can we save ourselves
the
battle? :S  

I know, I know :/

So, there is no rdma bit/cap in HW.. to disable non-RoCE commands
we
will have to disable etherent capability. 

It's your driver, you can make it do what you need to. Why does 
the RDMA driver bind successfully to a non-RoCE Ethernet device 
in the first place?

because RDMA people are greedy :), and they can create QPs that are not
RoCE..

this patch is only basic enable/disable RoCE feature on a specific
device slice, how rdma driver initializes itself is not related to this
patch at all, and it is just how rdma works, with or without this
patch.

quoted

The user interface here has no synthetic semantics, all knobs will
eventually be mapped to real HW/FW capabilities to get disabled.

the whole feature is about allowing admin to ship network functions
with different capabilities that are actually enforced by FW/HW.. 
so the user of the VF will see, RDMA/ETH only cards or both.

RDMA-only, ETH-only, RDMA+ETH makes sense to me. Having an ETH-only
device also exposed though rdma subsystem does not.

But this has nothing to do with this patch. this is the rdma driver
implementation.