Hi Kirti,

-----Original Message-----
From: Kirti Wankhede <kwankhede@nvidia.com>
Sent: Tuesday, March 5, 2019 4:40 PM
To: Parav Pandit <redacted>; Jakub Kicinski
[off-list ref]
Cc: Or Gerlitz <redacted>; netdev@vger.kernel.org; linux-
kernel@vger.kernel.org; michal.lkml@markovi.net; davem@davemloft.net;
gregkh@linuxfoundation.org; Jiri Pirko [off-list ref]
Subject: Re: [RFC net-next 0/8] Introducing subdev bus and devlink extension

quoted

quoted

On Mon, 4 Mar 2019 04:41:01 +0000, Parav Pandit wrote:

quoted

quoted

-----Original Message-----
From: Jakub Kicinski <redacted>
Sent: Friday, March 1, 2019 2:04 PM
To: Parav Pandit <redacted>; Or Gerlitz
[off-list ref]
Cc: netdev@vger.kernel.org; linux-kernel@vger.kernel.org;
michal.lkml@markovi.net; davem@davemloft.net;
gregkh@linuxfoundation.org; Jiri Pirko [off-list ref]
Subject: Re: [RFC net-next 0/8] Introducing subdev bus and devlink
extension

On Thu, 28 Feb 2019 23:37:44 -0600, Parav Pandit wrote:

quoted

Requirements for above use cases:
--------------------------------
1. We need a generic user interface & core APIs to create sub
devices from a parent pci device but should be generic enough for
other parent devices 2. Interface should be vendor agnostic 3.
User should be able to set device params at creation time 4. In
future if needed, tool should be able to create passthrough device
to map to a virtual machine

Like a mediated device?

Yes.

quoted

https://www.kernel.org/doc/Documentation/vfio-mediated-device.txt
https://www.dpdk.org/wp-

content/uploads/sites/35/2018/06/Mediated-

quoted

quoted

quoted

quoted

Devices-Better-Userland-IO.pdf

Other than pass-through it is entirely unclear to me why you'd need
a

bus.

quoted

quoted

(Or should I say VM pass through or DPDK?)  Could you clarify why
the need for a bus?

A bus follow standard linux kernel device driver model to attach a
driver to specific device. Platform device with my limited
understanding looks a hack/abuse of it based on documentation [1],
but it can possibly be an alternative to bus if it looks fine to
Greg and others.

I grok from this text that the main advantage you see is the ability
to choose a driver for the subdevice.

Yes.

quoted

quoted

quoted

My thinking is that we should allow spawning subports in devlink
and if user specifies "passthrough" the device spawned would be an

mdev.

quoted

quoted

quoted

devlink device is much more comprehensive way to create sub-devices
than sub-ports for at least below reasons.

1. devlink device already defines device->port relation which
enables to create multiport device.

I presume that by devlink device you mean devlink instance?  Yes,
this part I'm following.

Yes -> 'struct devlink'

quoted

quoted

subport breaks that.

Breaks what?  The ability to create a devlink instance with multiple ports?

Right.

quoted

quoted

2. With bus model, it enables us to load driver of same vendor or
generic one such a vfio in future.

You can achieve this with mdev as well.

quoted

quoted

Yes, sorry, I'm not an expert on mdevs, but isn't that the goal of those?
Could you go into more detail why not just use mdevs?

I am novice at mdev level too. mdev or vfio mdev.
Currently by default we bind to same vendor driver, but when it was

created as passthrough device, vendor driver won't create netdevice or rdma
device for it.

quoted

And vfio/mdev or whatever mature available driver would bind at that

point.

quoted

Using mdev framework, if you want to partition a physical device into
multiple logic devices, you can bind those devices to same vendor driver
through vfio-mdev, where as if you want to passthrough the device bind it to
vfio-pci. If I understand correctly, that is what you are looking for.

We cannot bind a whole PCI device to vfio-pci, reason is,
A given PCI device has existing protocol devices on it such as netdevs and rdma dev.
This device is partitioned while those protocol devices exist and
mlx5_core, mlx5_ib drivers are loaded on it.
And we also need to connect these objects rightly to eswitch exposed 
by devlink interface (net/core/devlink.c) that supports
eswitch binding, health, registers, parameters, ports support.
It also supports existing PCI VFs.

I don’t think we want to replicate all of this again in mdev subsystem [1].

[1] https://www.kernel.org/doc/Documentation/vfio-mediated-device.txt

So devlink interface to migrate users from managing VFs to
non_VF sub device is natural progression.

However, in future, I believe we would be creating mediated devices on user request,
to use mdev modules and map them to VM.

Also 'mdev_bus' is created as a class and not as a bus. This limits to not use 
devlink interface whose handle is bus+device name.

So one option is to change mdev from class to bus.
devlink will create mdevs on the bus, mdev driver can probe these devices on host system by default.
And if told to do passthrough, a different driver exposes them to VM.
How feasible is this?

quoted

quoted

quoted

3. Devices live on the bus, mapping a subport to 'struct device' is
not intuitive.

Are you saying that the main devlink instance would not have any port
information for the subdevices?

Right, this newly created devlink device is the control point of its port(s).

quoted

Devices live on a bus.  Software constructs - depend on how one wants
to model them - don't have to.

quoted

4. sub-device allows to use existing devlink port, registers, health
infrastructure to sub devices, which otherwise need to be duplicated
for ports.

Health stuff is not tied to a port, I'm not following you.  You can
create a reporter per port, per ACL rule or per SB or per whatever your

heart desires..

quoted

quoted

Instead of creating multiple reporters and inventing these reporter
naming schemes, creating devlink instance leverage all health reporting

done for a devliink instance.

quoted

So whatever is done for instance A (parent), can be available for instance B

(subdev).

quoted

quoted

quoted

5. Even though current devlink devices are networking devices, there
is nothing restricts it to be that way. So subport is a restricted
view.
6. devlink device already covers
port sub-object, hence creating devlink device is desired.

quoted

quoted

5. A device can have multiple ports

What does this mean, in practice?  You want to spawn a subdev which
can access both ports?  That'd be for RDMA use cases, more than
Ethernet, right?  (Just clarifying :))

Yep, you got it right. :-)

quoted

quoted

So how is it done?
------------------
(a) user in control
To address above requirements, a generic tool iproute2/devlink is
extended for sub device's life cycle.
However a devlink tool and its kernel counter part is not
sufficient to create protocol agnostic devices on a existing PCI
bus.

"Protocol agnostic"?...  What does that mean?

Devlink works on bus,device model. It doesn't matter what class of
device is. For example, for pci class can be anything. So newly
created sub-devices are not limited to netdev/rdma devices. Its
agnostic to protocol. More importantly, we don't want to create
these sub-devices who bus type is 'pci'. Because as described below,
PCI has its addressing scheme and pci bus must not have mix-n match

devices.

quoted

quoted

quoted

So probably better wording should be, 'a devlink tool and its kernel
counterpart is not sufficient to create sub-devices of same class as
that of PCI device.

Let me clarify - for networking devices the partition will most
likely end up as a subport, but its not a requirement that each partition

must be a subport..

quoted

quoted

The question was about the necessity to invent a new bus, and have
every resource have a struct device..

A device object and bus connecting all software objects correctly.
This includes, 1. devlink bus/name handle based access 2. matching
such device in sysfs 3. parent child hierarchy in sysfs 4. ability to
bind different driver 5. multi-ports per device 6. still usable for
single port use case 7. parameters setting at devlink instance level
8. parent-child relation handling power mgmt 9. follows standard linux
driver model

Some are achievable to through mfd too, instead of subdev bus.
Will follow Greg's guidance on this.

I think you can achieve all the above points with mdev framework as well.
Check samples at samples/vfio-mdev/ in kernel for quick understanding.

Thanks,
Kirti