RE: [PATCH RESEND iproute2-next] devlink: Add optional controller user input

From: Yunsheng Lin <redacted>
Sent: Monday, June 7, 2021 9:01 AM

On 2021/6/6 15:10, Parav Pandit wrote:

quoted

Hi Yunsheng,

quoted

From: Yunsheng Lin <redacted>
Sent: Friday, June 4, 2021 7:05 AM

On 2021/6/3 19:19, Parav Pandit wrote:

quoted

A user optionally provides the external controller number when user
wants to create devlink port for the external controller.

Hi, Parav
   I was planing to use controller id to solve the devlink instance
representing problem for multi-function which shares common resource
in the same ASIC, see [1].

It seems the controller id used here is to differentiate the function
used in different host?

That’s correct. Controller holds one or more PCI functions (PF,VF,SF).

I am not sure I understand the exact usage of controller and why controller id
is in "devlink_port_*_attrs".

Let's consider a simplified case where there is two PF(supposing both have
VF enabled), and each PF has different controller and each PF corresponds to
a different physical port(Or it is about multi-host case multi PF may sharing
the same physical port?):

Typically single host with two PFs have their own physical ports.
Multi-host with two PFs, on each host they share respective physical ports.

Single host:
Pf0.physical_port = p0
Pf1.physical_port = p1.

Multi-host (two) host setup
H1.pf0.phyical_port = p0.
H1.pf1.phyical_port = p1.
H2.pf0.phyical_port = p0.
H2.pf1.phyical_port = p1.

1. I suppose each PF has it's devlink instance for mlx case(I suppose each
   VF can not have it's own devlink instance for VF shares the same physical
   port with PF, right?).

VF and SF ports are of flavour VIRTUAL.

2. each PF's devlink instance has three types of port, which is
   FLAVOUR_PHYSICAL, FLAVOUR_PCI_PF and FLAVOUR_PCI_VF(supposing I
understand
   port flavour correctly).

FLAVOUR_PCI_{PF,VF,SF} belongs to eswitch (representor) side on switchdev device.

If I understand above correctly, all ports in the same devlink instance should
have the same controller id, right? If yes, why not put the controller id in the
devlink instance?

Need not be. All PCI_{PF,VF,SF} can have controller id different for different controllers.
Usually each multi-host is a different controller. 
Refer to this diagram [1] and detailed description.

quoted

In your case if there is single devlink instance representing ASIC, it is better

to have health reporters under this single instance.

quoted

Devlink parameters do not span multiple devlink instance.

Yes, that is what I try to do: shared status/parameters in devlink instance,
physical port specific status/parameters in devlink port instance.

quoted

So if you need to control devlink instance parameters of each function

byitself, you likely need devlink instance for each.

quoted

And still continue to have ASIC wide health reporters under single instance

that represents whole ASIC.

I do not think each function need a devlink instance if there is a devlink
instance representing a whole ASIC, using the devlink port instance to
represent the function seems enough?

'devlink port function's is equivalent of hypervisor/nicvisor entity controlled by the network/sysadmin.
While devlink instance of a given PF,VF,SF is managed by the user of such function itself.
For example when a VF is mapped to a VM, devlink instance of this VF resides in the VM managed by the guest VM.

Before this VF is given to VM, usually hypervisor/nicvisor admin programs this function (such as mac address) explained in [3].
So that a given VM always gets the same mac address regardless of which VF {a or b).

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/networking/devlink/devlink-port.rst?h=v5.13-rc5#n72
[2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/networking/devlink/devlink-port.rst?h=v5.13-rc5#n60
[3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/networking/devlink/devlink-port.rst?h=v5.13-rc5#n110

quoted

quoted

1. https://lkml.org/lkml/2021/5/31/296