On Fri, Jun 11, 2021 at 11:21:42AM -0700, Dave Jiang wrote:

On 6/8/2021 9:02 AM, Dave Jiang wrote:

quoted

On 6/7/2021 12:11 PM, Jason Gunthorpe wrote:

quoted

On Mon, Jun 07, 2021 at 11:13:04AM -0700, Dave Jiang wrote:

quoted

So in step 1, we 'tag' the wq to be dedicated to guest usage and
put the
hardware wq into enable state. For a dedicated mode wq, we can
definitely
just register directly and skip the mdev step. For a shared wq
mode, we can
have multiple mdev running on top of a single wq. So we need
some way to
create more mdevs. We can either go with the existing
established creation
path by mdev, or invent something custom for the driver as Jason
suggested
to accomodate additional virtual devices for guests. We
implemented the mdev
path originally with consideration of mdev is established and
has a known
interface already.

It sounds like you could just as easially have a 'create new vfio'
file under the idxd sysfs.. Especially since you already have a bus
and dynamic vfio specific things being created on this bus.

Will explore this and using of 'struct vfio_device' without mdev.

Hi Jason. I hacked the idxd driver to remove mdev association and use
vfio_device directly. Ran into some issues. Specifically mdev does some
special handling when it comes to iommu domain.

Yes, I know of this, it this needs fixing.

effect of special handling for iommu_attach_group. And in addition, it ends
up switching the bus to pci_bus_type before iommu_domain_alloc() is called. 
Do we need to provide similar type of handling for vfio_device that are not
backed by an entire PCI device like vfio_pci? Not sure it's the right thing
to do to attach these devices to pci_bus_type directly.

Yes, type1 needs to be changed so it somehow knows that the struct
device is 'sw only', for instance because it has no direct HW IOMMU
connection, then all the mdev hackery should be deleted from type1.

I haven't investigated closely exactly how to do this.

Jason