RE: [PATCH RESEND iproute2-next] devlink: Add optional controller user input

From: Yunsheng Lin <redacted>
Sent: Tuesday, June 8, 2021 8:58 AM

On 2021/6/7 19:12, Parav Pandit wrote:

quoted

quoted

From: Yunsheng Lin <redacted>
Sent: Monday, June 7, 2021 4:27 PM

[..]

quoted

quoted

quoted

quoted

2. each PF's devlink instance has three types of port, which is
   FLAVOUR_PHYSICAL, FLAVOUR_PCI_PF and

FLAVOUR_PCI_VF(supposing I

quoted

quoted

understand
   port flavour correctly).

FLAVOUR_PCI_{PF,VF,SF} belongs to eswitch (representor) side on

switchdev device.

If devlink instance or eswitch is in DEVLINK_ESWITCH_MODE_LEGACY
mode, the FLAVOUR_PCI_{PF,VF,SF} port instance does not need to

created?

quoted

No. in eswitch legacy, there are no representor netdevice or devlink ports.

It seems each devlink port instance corresponds to a netdevice.
More specificly, the devlink instance is created in the struct pci_driver' probe
function of a pci function, a devlink port instance is created and registered to
that devlink instance when a netdev of that pci function is created?

Yes.

As in diagram [1], the devlink port instance(flavour FLAVOUR_PHYSICAL) for
ctrl-0-pf0 is created when the netdev of ctrl-0-pf0 is created in the host of
smartNIC, the devlink port instance(flavour FLAVOUR_VIRTUAL) for ctrl-0-
pf0vfN is created when the netdev of ctrl-0-pf0vfN is created in the host of
smartNIC, right?

Ctrl-0-pf0vfN, ctrl-0-pf0 ports are eswitch ports. They are created where there is eswitch.
Usually in smartnic where eswitch is located.

When eswitch mode is set to DEVLINK_ESWITCH_MODE_SWITCHDEV, the
representor netdev for PF/VF in "controller_num=1" is created in the host of
smartNIC, so is the devlink port instance(FLAVOUR_PCI_{PF,VF,SF})
corresponding to that representor netdev just created in the host of
smartNIC? More specificly, devlink port instance(flavour FLAVOUR_PCI_PF)
for ctrl-1-pf0 and devlink port instance (flavour FLAVOUR_PCI_VF)for ctrl-1-
pf0vfN?

I do not know what you mean by "host of smartnic". But as clarified in diagram and above,
They are created on the eswitch device.

When "controller_num=1" is plugged to a server, the server host creates
devlink instance and devlink port instance in the host of server as similar as
the ctrl-0-pf0 and ctrl-0-pf0vfN in the host of smartNIC?

Since eswitch is not in the controller=1, controller 1 is just hosting PCI PF, VF, SF functions.
Either it has PHYSICAL or VIRTUAL ports along with its netdevice and devlink instance.

quoted

quoted

Or devlink instance does exist in the host corresponding to
"controller_num=1", but the mode of that devlink instance is
DEVLINK_ESWITCH_MODE_LEGACY in diagram [1]?

As you can see that eswitch is located only on controller=0.
This eswitch is serving PF, VF, SFs of controller=1 + controlloler=0 as well.

How do we decide where eswitch is located? through some fw/hw
configuration?

This is usually decided by the hardware.

It seems if the eswitch is enabled on "controller=1", that is a nested eswitch
too, which you mentioned below?

Yes.

quoted

quoted

Also, eswitch mode can only be set on the devlink instance
corresponding to PF, but not for VF/SF(supposing that VF/SF could
have it's own devlink instance too), right?

Yes. Eswitch can be located on the VF too. Mlx5 driver doesn't have it yet

on VF.

quoted

This may be some nested eswitch in future. I do not know when.

quoted

by the network/sysadmin.

quoted

While devlink instance of a given PF,VF,SF is managed by the user of
such

function itself.

'devlink port function' means "struct devlink_port", right?

'function' is the object managing the function connected on the otherside

of this port.

quoted

This includes its hw_addr, rate, state, operational state.

Does "other side of this port" means the pci function that is most likely have
been passed through to a VM?

Yes.

"devlink port" without the "function" represents the representor netdev on
the host where eswitch is located?

Yes.

quoted

quoted

It seems 'devlink port function' in the host is representing a VF
when devlink instance of that VF is in the VM?

Right.

quoted

quoted

For example when a VF is mapped to a VM, devlink instance of this VF

resides in the VM managed by the guest VM.

Does the user in VM really care about devlink info or configuration
when network/sysadmin has configured the VF through 'devlink port

function'

quoted

quoted

in the host?

Yes. devlink instance offers many knobs in uniform way on PF, VF, SF.
They are in use in mlx5 for devlink params, reload, net ns.

"net ns" refer to "net namespace", right?

Yes.

I am not sure how devlink instance related to net namespace yet.
I thought devlink is not limited to networking, it can be used in other pcie
device other than ethernet device?

It can be. And it can be used for networking devices too and hence it is attached to single net ns.