-----Original Message-----
From: Wang, Haiyue <redacted>
Sent: Tuesday, February 23, 2021 7:46 PM
To: Stephen Hemminger <stephen@networkplumber.org>; Xueming(Steven) Li <redacted>
Cc: dev@dpdk.org; Slava Ovsiienko <redacted>; Asaf Penso <redacted>
Subject: RE: [dpdk-dev] [PATCH v6 0/9] ethdev: support SubFunction representor

quoted

-----Original Message-----
From: dev <redacted> On Behalf Of Stephen Hemminger
Sent: Tuesday, February 23, 2021 09:55
To: Xueming Li <redacted>
Cc: dev@dpdk.org; Viacheslav Ovsiienko <redacted>; Asaf
Penso [off-list ref]
Subject: Re: [dpdk-dev] [PATCH v6 0/9] ethdev: support SubFunction
representor

On Sun, 14 Feb 2021 03:21:30 +0000
Xueming Li [off-list ref] wrote:

quoted

SubFunction [1] is a portion of the PCI device, a SF netdev has its
own dedicated queues(txq, rxq). A SF netdev supports E-Switch
representation offload similar to existing PF and VF representors. A
SF shares PCI level resources with other SFs and/or with its parent PCI function.

From SmartNIC perspective, when PCI device is shared for multi-host,
representors for host controller and host PF is required.

This patch set introduces new representor types in addtion to
existing VF representor. Syntax:

[[c#]pf#]vf#: VF port representor/s from controller/pf
[[c#]pf#]sf#: SF port representor/s from controller/pf
#: VF representor - for backwards compatibility

"#" is number instance, list or range, valid examples:
  1, [1,3,5], [0-3], [0,2-4,6]

For backward compatibility, this patch also introduces new netdev
capability to indicate the capability of supportting SF representor.

Version history:
 RFC:
 	initial version [2]
 V2:
    - separate patch for represnetor infrastructure, controller, pf and
      sf.
    - replace representor ID macro with functions:
      rte_eth_representor_id_encode()
      rte_eth_representor_id_parse()
    - new patch to allow devargs with same PCI BDF but different
      representors.
    - other minor code updates according to comments, thanks Andrew!
    - update document
 V3:
    - improve probing of allowed devargs with same name.
    - parse single word of kvargs as key.
    - update kvargs test cases.
 V4:
    - split first representor refactor patch into
      1: add representor type
      2: refector representor list parsing
    - push the patch supporting multi-devargs for same device.
 V5:
    - add comments for parsing functions
    - update switch_representation.rst - Thanks Ajit
 V6:
    - split representor types into different patches, move to
      rte_ethdev.h
    - improvements of rte_eth_devargs_process_list() according to
      Andrew's suggestion
    - fixed PF probe failure for Intel i40e
    - replace ethdev SF capability with rte_eth_representor_info_get()
    - add new ethdev ops api to get representor info from PMD
    - replace representor ID encode/decode with conversion from
      representor info
    - change ethdev representor iterator to use new ID encoding


Xueming Li (9):
  ethdev: introduce representor type
  ethdev: support representor port list
  ethdev: support new VF representor syntax
  ethdev: support sub function representor
  ethdev: support PF index in representor
  ethdev: support multi-host in representor
  ethdev: new API to get representor info
  ethdev: representor iterator compare complete info
  kvargs: update parser to support lists

 app/test/test_kvargs.c                        |  46 ++++-
 config/rte_config.h                           |   1 +
 doc/guides/prog_guide/poll_mode_drv.rst       |  13 +-
 .../prog_guide/switch_representation.rst      |  35 +++-
 drivers/net/bnxt/bnxt_ethdev.c                |   7 +
 drivers/net/enic/enic_ethdev.c                |   6 +
 drivers/net/i40e/i40e_ethdev.c                |   7 +
 drivers/net/ixgbe/ixgbe_ethdev.c              |   7 +
 drivers/net/mlx5/linux/mlx5_os.c              |  11 ++
 lib/librte_ethdev/ethdev_driver.h             |  49 ++++-
 lib/librte_ethdev/ethdev_private.c            | 173 ++++++++++++------
 lib/librte_ethdev/ethdev_private.h            |   3 -
 lib/librte_ethdev/rte_class_eth.c             |  40 ++--
 lib/librte_ethdev/rte_ethdev.c                | 102 ++++++++++-
 lib/librte_ethdev/rte_ethdev.h                |  53 ++++++
 lib/librte_ethdev/version.map                 |   4 +
 lib/librte_kvargs/rte_kvargs.c                | 101 +++++++---
 17 files changed, 535 insertions(+), 123 deletions(-)

A couple of higher level design questions:
1. How does Linux and other OS handle this in their API?

Hi Stephen, thanks for looking this!
Representor was a type of ethdev, OS independent.
SF representor is a new type besides existing VF representor type.

quoted

2. This solution seems quite tied into a specific implementation on your hardware.
   Is there a PCI standard for this?

SF has been carved in kernel, supported by devlink. The devlink document in the link Haiyue posted should be helpful to understand the overall picture:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=24a790da0ac4d9bcce2a9becc8799241716672f6

quoted

3. Maybe a more general solution where these were represented as buses would
   allow for other connection methods in the future?

It should be "auxiliary bus", I think. ;-) https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/base/auxiliary.c

Good link, Mellanox has exactly implemented SF based auxiliary bus, I'm going to post another RFC to support auxiliary bus.
But this patchset still needed to support representor of SF.

mlx5 subfunction support
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=24a790da0ac4d9bcce2a9becc8799241716672f6