RE: [PATCH RESEND iproute2-next] devlink: Add optional controller user input

From: Yunsheng Lin <redacted>
Sent: Monday, June 7, 2021 4:27 PM

On 2021/6/7 14:10, Parav Pandit wrote:

quoted

quoted

From: Yunsheng Lin <redacted>
Sent: Monday, June 7, 2021 9:01 AM

On 2021/6/6 15:10, Parav Pandit wrote:

quoted

Hi Yunsheng,

quoted

From: Yunsheng Lin <redacted>
Sent: Friday, June 4, 2021 7:05 AM

On 2021/6/3 19:19, Parav Pandit wrote:

quoted

A user optionally provides the external controller number when
user wants to create devlink port for the external controller.

Hi, Parav
   I was planing to use controller id to solve the devlink instance
representing problem for multi-function which shares common
resource in the same ASIC, see [1].

It seems the controller id used here is to differentiate the
function used in different host?

That’s correct. Controller holds one or more PCI functions (PF,VF,SF).

I am not sure I understand the exact usage of controller and why
controller id is in "devlink_port_*_attrs".

Let's consider a simplified case where there is two PF(supposing both
have VF enabled), and each PF has different controller and each PF
corresponds to a different physical port(Or it is about multi-host
case multi PF may sharing the same physical port?):

Typically single host with two PFs have their own physical ports.
Multi-host with two PFs, on each host they share respective physical ports.

Single host:
Pf0.physical_port = p0
Pf1.physical_port = p1.

Multi-host (two) host setup
H1.pf0.phyical_port = p0.
H1.pf1.phyical_port = p1.
H2.pf0.phyical_port = p0.
H2.pf1.phyical_port = p1.

Multi-host (two) host setup with separate physical port for each host:
H1.pf0.phyical_port = p0
H2.pf0.phyical_port = p1

Does above use case make sense for mlx, it seems a common case for our
internal use.

It is mlx5 use case too.

quoted

quoted

1. I suppose each PF has it's devlink instance for mlx case(I suppose each
   VF can not have it's own devlink instance for VF shares the same physical
   port with PF, right?).

VF and SF ports are of flavour VIRTUAL.

Which devlink instance does the flavour VIRTUAL port instance for VF and SF
is registered to?

To the devlink instance of the VF/SF.

Does it mean VF has it's own devlink instance in VM when it is passed a VM,
and flavour VIRTUAL port instance for that VF is registered to that devlink
instance in the VM too？

Yes.

Even in the same host as PF, the VF also has it's own devlink instance?

Yes.

quoted

quoted

2. each PF's devlink instance has three types of port, which is
   FLAVOUR_PHYSICAL, FLAVOUR_PCI_PF and

FLAVOUR_PCI_VF(supposing I

quoted

quoted

understand
   port flavour correctly).

FLAVOUR_PCI_{PF,VF,SF} belongs to eswitch (representor) side on

switchdev device.

If devlink instance or eswitch is in DEVLINK_ESWITCH_MODE_LEGACY mode,
the FLAVOUR_PCI_{PF,VF,SF} port instance does not need to created?

No. in eswitch legacy, there are no representor netdevice or devlink ports.

quoted

quoted

If I understand above correctly, all ports in the same devlink
instance should have the same controller id, right? If yes, why not
put the controller id in the devlink instance?

Need not be. All PCI_{PF,VF,SF} can have controller id different for

different controllers.

The point is that two VF from different PF may be in the different host, all VF
of a specific PF need to be in the same host, right?
otherwise it may break PCI enumeration process?

Sure. VFs belong to PF, PF belong to controller, controller is plugged into a host root complex.

If yes, as PCI_{PF,VF,SF} belongs to eswitch (representor) side on switchdev
device(which means PCI_{PF,VF,SF} port instance is in the same host, as the
host corresponding to "controller_num=0" in diagram [1]), so it seems all the
PCI_{PF,VF,SF} of a specific PF should have the same controller id, 

Yes.

and using
a controller id of the devlink instance in "controller_num=0" in diagram [1]
seems enough?

Yes.

quoted

Usually each multi-host is a different controller.
Refer to this diagram [1] and detailed description.

devlink instance does not exist in the host corresponding to
"controller_num=1" in diagram [1]?

Devlink instance do exist for controller=1 related PCI PF,VF,SF devices when those functions are plugged in the host.

Or devlink instance does exist in the host corresponding to
"controller_num=1", but the mode of that devlink instance is
DEVLINK_ESWITCH_MODE_LEGACY in diagram [1]?

As you can see that eswitch is located only on controller=0.
This eswitch is serving PF, VF, SFs of controller=1 + controlloler=0 as well.

Also, eswitch mode can only be set on the devlink instance corresponding to
PF, but not for VF/SF(supposing that VF/SF could have it's own devlink
instance too), right?

Yes. Eswitch can be located on the VF too. Mlx5 driver doesn't have it yet on VF.
This may be some nested eswitch in future. I do not know when.

by the network/sysadmin.

quoted

While devlink instance of a given PF,VF,SF is managed by the user of such

function itself.

'devlink port function' means "struct devlink_port", right?

'function' is the object managing the function connected on the otherside of this port.
This includes its hw_addr, rate, state, operational state.

It seems 'devlink port function' in the host is representing a VF when devlink
instance of that VF is in the VM?

Right.

quoted

For example when a VF is mapped to a VM, devlink instance of this VF

resides in the VM managed by the guest VM.

Does the user in VM really care about devlink info or configuration when
network/sysadmin has configured the VF through 'devlink port function'
in the host?

Yes. devlink instance offers many knobs in uniform way on PF, VF, SF.
They are in use in mlx5 for devlink params, reload, net ns.

which devlink info or configuration does user need to query or configure in a
VM?

Usually not much.
Few examples that mlx5 users do with devlink instance of VF in a VM are, devlink params, devlink reload, board info. Health reporters, health recovery to name few.