Hi Parav,

-----Original Message-----
From: Parav Pandit <redacted>
Sent: Tuesday, June 15, 2021 1:43 PM
To: Xia, Chenbo <redacted>; NBU-Contact-Thomas Monjalon
[off-list ref]; Yigit, Ferruh [off-list ref]
Cc: Xueming(Steven) Li <redacted>; Slava Ovsiienko
[off-list ref]; dev@dpdk.org; Jason Gunthorpe [off-list ref]
Subject: RE: [dpdk-dev] [RFC 00/14] mlx5: support SubFunction

quoted

From: Xia, Chenbo <redacted>
Sent: Tuesday, June 15, 2021 11:03 AM

Hi Parav,

quoted

-----Original Message-----
From: Parav Pandit <redacted>
Sent: Tuesday, June 15, 2021 12:05 PM
To: Xia, Chenbo <redacted>; NBU-Contact-Thomas Monjalon
[off-list ref]; Yigit, Ferruh [off-list ref]
Cc: Xueming(Steven) Li <redacted>; Slava Ovsiienko
[off-list ref]; dev@dpdk.org; Jason Gunthorpe
[off-list ref]
Subject: RE: [dpdk-dev] [RFC 00/14] mlx5: support SubFunction

Hi Chenbo,

quoted

From: Xia, Chenbo <redacted>
Sent: Tuesday, June 15, 2021 7:41 AM

Hi Thomas,

quoted

From: Thomas Monjalon <redacted>
Sent: Friday, June 11, 2021 3:54 PM

[..]

quoted

Yes. In our term it's called Assignable Device Interface (ADI)
introduced in Intel Scalable IOV
(https://01.org/blogs/2019/assignable-interfaces-intel-
scalable-i/o-virtualization-linux)

And vfio-mdev is chosen to be the software framework for it. I start
to

realize

quoted

there is difference between SF and ADI: SF considers multi-function
devices which may include net/regex/vdpa/...

Yes. net, rdma, vdpa, regex ++.
And eventually vfio_device to map to VM too.

Non mdev framework is chosen so that all the use cases of kernel only,
or user only or mix modes can be supported.

OK. Got it.

quoted

quoted

But ADI only focuses on the
virtualization of the devices and splitting devices to logic parts
and

providing

quoted

huge number of interfaces to host APP. I think SF also considers
this but is mainly used for multi-function devices (like DPU in your

term?

quoted

quoted

quoted

Correct me if I'm wrong).

SF also supports DPU mode too but it is in addition to above use cases.
SF will expose mdev (or a vfio_device) to map to a VM.

So your SW actually supports vfio-mdev? I suppose the device-specific mdev
Kernel module is out-of-tree?

mlx5 driver doesn't support vfio_device for SFs.
Kernel plumbing for PASID assignment to SF is WIP currently kernel community.
We do not have any out-of-tree kernel module.

quoted

Just FYI:

We are introducing a new mdev bus for DPDK:
http://patchwork.dpdk.org/project/dpdk/cover/20210601030644.3318-1-
chenbo.xia@intel.com/

I am yet to read about it. But I am not sure what value does it add.
A user can open a vfio device using vfio subsystem and operate on it.
A vfio device can be a create as a result of binding PCI VF/PF to vfio-pci
driver or a SF by binding SF to vfio_foo driver.

Yes, in general it is the way. For vfio-mdev, it works as binding the vfio-mdev
to parent device and echo uuid to create a virtual device. VFIO APP like DPDK,
as you said, should work similar with VFIO UAPI for vfio-pci devices or mdev-based
devices. But currently DPDK only cares about vfio-pci devices and does not care
things for other cases like mdev-based pci devices. For example, it does not scan
/sys/bus/mdev and it always uses pci bdf as device address, which mdev-based pci
devices do not have. Therefore I sent that patchset.

There is kernel work in progress to use vfio core as library.

OK. Could you share me some link to it? Much appreciated.

So we do not anticipate to use add mdev layer and uuid to create a vfio device
for a SF.

OK. For now, we are following the vfio-mdev standard, using UUID to create vfio
devices.

For Intel, ADI will never has any netdevs or rdma dev?

I think technically it could have. But for some devices like our dma devices, it's
just using mdev:

https://www.spinics.net/lists/kvm/msg244417.html

Thanks,
Chenbo