Re: [PATCH v2 04/19] iommufd: Allow pt_id to carry viommu_id for IOMMU_HWPT_ALLOC

On Fri, Sep 27, 2024 at 08:59:25AM -0300, Jason Gunthorpe wrote:

On Thu, Sep 26, 2024 at 11:02:37PM -0700, Nicolin Chen wrote:

quoted

On Fri, Sep 27, 2024 at 01:38:08PM +0800, Yi Liu wrote:

quoted

quoted

quoted

Does it mean each vIOMMU of VM can only have
one s2 HWPT?

Giving some examples here:
  - If a VM has 1 vIOMMU, there will be 1 vIOMMU object in the
    kernel holding one S2 HWPT.
  - If a VM has 2 vIOMMUs, there will be 2 vIOMMU objects in the
    kernel that can hold two different S2 HWPTs, or share one S2
    HWPT (saving memory).

So if you have two devices assigned to a VM, then you may have two
vIOMMUs or one vIOMMU exposed to guest. This depends on whether the two
devices are behind the same physical IOMMU. If it's two vIOMMUs, the two
can share the s2 hwpt if their physical IOMMU is compatible. is it?

Yes.

quoted

To achieve the above, you need to know if the physical IOMMUs of the
assigned devices, hence be able to tell if physical IOMMUs are the
same and if they are compatible. How would userspace know such infos?

My draft implementation with QEMU does something like this:
 - List all viommu-matched iommu nodes under /sys/class/iommu: LINKs
 - Get PCI device's /sys/bus/pci/devices/0000:00:00.0/iommu: LINK0
 - Compare the LINK0 against the LINKs

We so far don't have an ID for physical IOMMU instance, which can
be an alternative to return via the hw_info call, otherwise.

We could return the sys/class/iommu string from some get_info or
something

I had a patch doing an ida alloc for each iommu_dev and returning
the ID via hw_info. It wasn't useful at that time, as we went for
fail-n-retry for S2 HWPT allocations on multi-pIOMMU platforms.

Perhaps that could be cleaner than returning a string?

quoted

For compatibility to share a stage-2 HWPT, basically we would do
a device attach to one of the stage-2 HWPT from the list that VMM
should keep. This attach has all the compatibility test, down to
the IOMMU driver. If it fails, just allocate a new stage-2 HWPT.

Ideally just creating the viommu should validate the passed in hwpt is
compatible without attaching.

I think I should add a validation between hwpt->domain->owner and
dev_iommu_ops(idev->dev) then!

Thanks
Nicolin