RE: [EXTERNAL] Re: [net-next PATCH v7 00/10] Introduce RVU representors

-----Original Message-----
From: Michal Swiatkowski <redacted>
Sent: Friday, July 5, 2024 2:04 PM
To: Geethasowjanya Akula <gakula@marvell.com>
Cc: netdev@vger.kernel.org; linux-kernel@vger.kernel.org; kuba@kernel.org;
davem@davemloft.net; pabeni@redhat.com; edumazet@google.com; Sunil
Kovvuri Goutham [off-list ref]; Subbaraya Sundeep Bhatta
[off-list ref]; Hariprasad Kelam [off-list ref]
Subject: Re: [EXTERNAL] Re: [net-next PATCH v7 00/10] Introduce RVU
representors

On Thu, Jul 04, 2024 at 01: 48: 23PM +0000, Geethasowjanya Akula wrote: > > >

quoted

-----Original Message----- > >From: Michal Swiatkowski

<michal. swiatkowski@ linux. intel. com> > >Sent: Thursday, July 4, 2024 12: 45
PM 
On Thu, Jul 04, 2024 at 01:48:23PM +0000, Geethasowjanya Akula wrote:

quoted

quoted

-----Original Message-----
From: Michal Swiatkowski <redacted>
Sent: Thursday, July 4, 2024 12:45 PM
To: Geethasowjanya Akula <gakula@marvell.com>
Cc: netdev@vger.kernel.org; linux-kernel@vger.kernel.org;
kuba@kernel.org; davem@davemloft.net; pabeni@redhat.com;
edumazet@google.com; Sunil Kovvuri Goutham

[off-list ref];

quoted

quoted

Subbaraya Sundeep Bhatta [off-list ref]; Hariprasad Kelam
[off-list ref]
Subject: Re: [EXTERNAL] Re: [net-next PATCH v7 00/10] Introduce RVU
representors

On Wed, Jul 03, 2024 at 02: 34: 03PM +0000, Geethasowjanya Akula
wrote: > >

quoted

quoted

-----Original Message----- > >From: Michal Swiatkowski

<michal. swiatkowski@ linux. intel. com> > >Sent: Wednesday, July 3,
2024 4: 14 PM On Wed, Jul 03, 2024 at 02:34:03PM +0000,
Geethasowjanya Akula wrote:

quoted

quoted

-----Original Message-----
From: Michal Swiatkowski <redacted>
Sent: Wednesday, July 3, 2024 4:14 PM
To: Geethasowjanya Akula <gakula@marvell.com>
Cc: netdev@vger.kernel.org; linux-kernel@vger.kernel.org;
kuba@kernel.org; davem@davemloft.net; pabeni@redhat.com;
edumazet@google.com; Sunil Kovvuri Goutham

[off-list ref];

quoted

quoted

Subbaraya Sundeep Bhatta [off-list ref]; Hariprasad Kelam
[off-list ref]
Subject: [EXTERNAL] Re: [net-next PATCH v7 00/10] Introduce RVU
representors On Fri, Jun 28, 2024 at 07:05:07PM +0530, Geetha
sowjanya

wrote:

quoted

quoted

quoted

This series adds representor support for each rvu devices.
When switchdev mode is enabled, representor netdev is registered
for each rvu device. In implementation of representor model, one
NIX HW LF with multiple SQ and RQ is reserved, where each RQ and
SQ of the LF are mapped to a representor. A loopback channel is
reserved to support packet path between representors and VFs.
CN10K silicon supports 2 types of MACs, RPM and SDP. This patch
set adds representor support for both RPM and SDP MAC interfaces.

- Patch 1: Refactors and exports the shared service functions.
- Patch 2: Implements basic representor driver.
- Patch 3: Add devlink support to create representor netdevs that
  can be used to manage VFs.
- Patch 4: Implements basec netdev_ndo_ops.
- Patch 5: Installs tcam rules to route packets between representor and
	   VFs.
- Patch 6: Enables fetching VF stats via representor interface
- Patch 7: Adds support to sync link state between representors and VFs

.

quoted

quoted

quoted

quoted

quoted

- Patch 8: Enables configuring VF MTU via representor netdevs.
- Patch 9: Add representors for sdp MAC.
- Patch 10: Add devlink port support.

Command to create VF representor #devlink dev eswitch set
pci/0002:1c:00.0 mode switchdev VF representors are created for
each VF when switch mode is set switchdev on representor PCI
device

Does it mean that VFs needs to be created before going to
switchdev mode? (in legacy mode). Keep in mind that in both
mellanox and ice driver assume that VFs are created after chaning
mode to switchdev (mode can't be changed if VFs).

No. RVU representor driver implementation is similar to mellanox
and ice

drivers.

quoted

It assumes that VF gets created only after switchdev mode is enabled.
Sorry, if above commit description is confusing. Will rewrite it.

Ok, but why the rvu_rep_create() is called in switching mode to
switchdev function? In this function you are creating netdevs, only
for PF representor? It looks like it doesn't called from other
context in this patchset, so where the port representor netdevs for VFs are

created?

quoted

quoted

RVU representors for PF/VFs are created when switchdev mode is set, similar

to the bnxt and nfp drivers.

quoted

rvu_rep_create() will create representors based on rep_cnt (which
include both PFs and VFs count)

Sorry, I don't understand now. You wrote in previous message that VFs should
be created after switchdev mode is enabled. Now you are writting that they are
created based on rep_cnt (so assuming VFs have been created before).

What is the correct order?
# echo 1 x > sriov_numvfs
# devlink dev eswitch set pci/0000:ca:00.0 mode switchdev or # devlink dev
eswitch set pci/0000:ca:00.0 mode switchdev # echo 1 x > sriov_numvfs

or you can create VFs before and after? From looking at the drivers code it
looks like driver bnxt supports that.

Actually, we support VF creation before and after representor creation. 
Representor will not able send or receive any traffic untill corresponding representee
is created.

I am not familiar with nfp and bnxt driver but based on code in current kernel
version it looks like nfp creates representor when switching mode only for
physical and PF types, VF representor type is created when the VF is created.
Bnxt support VF representor creation before and after switching mode. In ice
port representors are created during VFs creation only.

I am not against your solution, only pointing that it can lead to some problems.

Thanks for sharing the information.

Geetha.

Thanks,
Michal

quoted

Thanks,
Geetha.

quoted

Thanks,
Michal

quoted

quoted

Different order can be problematic. For example (AFAIK) kubernetes
scripts for switchdev assume that first is switching to switchdev
and VFs creation is done after that.

Thanks,
Michal

quoted

# devlink dev eswitch set pci/0002:1c:00.0  mode switchdev # ip
link show
25: r0p1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN

mode

quoted

quoted

DEFAULT group default qlen 1000

quoted

    link/ether 32:0f:0f:f0:60:f1 brd ff:ff:ff:ff:ff:ff
26: r1p1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN

mode

quoted

quoted

DEFAULT group default qlen 1000

quoted

    link/ether 3e:5d:9a:4d:e7:7b brd ff:ff:ff:ff:ff:ff

#devlink dev
pci/0002:01:00.0
pci/0002:02:00.0
pci/0002:03:00.0
pci/0002:04:00.0
pci/0002:05:00.0
pci/0002:06:00.0
pci/0002:07:00.0

~# devlink port
pci/0002:1c:00.0/0: type eth netdev r0p1v0 flavour pcipf
controller
0 pfnum 1 vfnum 0 external false splittable false
pci/0002:1c:00.0/1: type eth netdev r1p1v1 flavour pcivf
controller
0 pfnum 1 vfnum 1 external false splittable false
pci/0002:1c:00.0/2: type eth netdev r2p1v2 flavour pcivf
controller
0 pfnum 1 vfnum 2 external false splittable false
pci/0002:1c:00.0/3: type eth netdev r3p1v3 flavour pcivf
controller
0 pfnum 1 vfnum 3 external false splittable false

-----------
v1-v2:
 -Fixed build warnings.
 -Address review comments provided by "Kalesh Anakkur Purayil".

v2-v3:
 - Used extack for error messages.
 - As suggested reworked commit messages.
 - Fixed sparse warning.

v3-v4:
 - Patch 2 & 3: Fixed coccinelle reported warnings.
 - Patch 10: Added devlink port support.

v4-v5:
  - Patch 3: Removed devm_* usage in rvu_rep_create()
  - Patch 3: Fixed build warnings.

v5-v6:
  - Addressed review comments provided by "Simon Horman".
  - Added review tag.

v6-v7:
  - Rebased on top net-next branch.

Geetha sowjanya (10):
  octeontx2-pf: Refactoring RVU driver
  octeontx2-pf: RVU representor driver
  octeontx2-pf: Create representor netdev
  octeontx2-pf: Add basic net_device_ops
  octeontx2-af: Add packet path between representor and VF
  octeontx2-pf: Get VF stats via representor
  octeontx2-pf: Add support to sync link state between representor and
    VFs
  octeontx2-pf: Configure VF mtu via representor
  octeontx2-pf: Add representors for sdp MAC
  octeontx2-pf: Add devlink port support

 .../net/ethernet/marvell/octeontx2/Kconfig    |   8 +
 .../ethernet/marvell/octeontx2/af/Makefile    |   3 +-
 .../ethernet/marvell/octeontx2/af/common.h    |   2 +
 .../net/ethernet/marvell/octeontx2/af/mbox.h  |  74 ++
 .../net/ethernet/marvell/octeontx2/af/npc.h   |   1 +
 .../net/ethernet/marvell/octeontx2/af/rvu.c   |  11 +
 .../net/ethernet/marvell/octeontx2/af/rvu.h   |  30 +-
 .../marvell/octeontx2/af/rvu_debugfs.c        |  27 -
 .../marvell/octeontx2/af/rvu_devlink.c        |   6 +
 .../ethernet/marvell/octeontx2/af/rvu_nix.c   |  81 ++-
 .../marvell/octeontx2/af/rvu_npc_fs.c         |   5 +
 .../ethernet/marvell/octeontx2/af/rvu_reg.h   |   4 +
 .../ethernet/marvell/octeontx2/af/rvu_rep.c   | 464 ++++++++++++
 .../marvell/octeontx2/af/rvu_struct.h         |  26 +
 .../marvell/octeontx2/af/rvu_switch.c         |  20 +-
 .../ethernet/marvell/octeontx2/nic/Makefile   |   2 +
 .../ethernet/marvell/octeontx2/nic/cn10k.c    |   4 +-
 .../ethernet/marvell/octeontx2/nic/cn10k.h    |   2 +-
 .../marvell/octeontx2/nic/otx2_common.c       |  56 +-
 .../marvell/octeontx2/nic/otx2_common.h       |  84 ++-
 .../marvell/octeontx2/nic/otx2_devlink.c      |  49 ++
 .../ethernet/marvell/octeontx2/nic/otx2_pf.c  | 305 +++++---
 .../marvell/octeontx2/nic/otx2_txrx.c         |  38 +-
 .../marvell/octeontx2/nic/otx2_txrx.h         |   3 +-
 .../ethernet/marvell/octeontx2/nic/otx2_vf.c  |  19 +-
.../net/ethernet/marvell/octeontx2/nic/rep.c  | 684
++++++++++++++++++ .../net/ethernet/marvell/octeontx2/nic/rep.h
++++++++++++++++++ |
53 ++
 27 files changed, 1834 insertions(+), 227 deletions(-)  create
mode
100644 drivers/net/ethernet/marvell/octeontx2/af/rvu_rep.c
 create mode 100644
drivers/net/ethernet/marvell/octeontx2/nic/rep.c
 create mode 100644
drivers/net/ethernet/marvell/octeontx2/nic/rep.h

--
2.25.1