Re: [PATCH v2 04/19] iommufd: Allow pt_id to carry viommu_id for IOMMU_HWPT_ALLOC

On Fri, Sep 27, 2024 at 08:12:40PM +0800, Yi Liu wrote:

External email: Use caution opening links or attachments


On 2024/9/27 14:02, Nicolin Chen wrote:

quoted

On Fri, Sep 27, 2024 at 01:38:08PM +0800, Yi Liu wrote:

quoted

quoted

quoted

Does it mean each vIOMMU of VM can only have
one s2 HWPT?

Giving some examples here:
   - If a VM has 1 vIOMMU, there will be 1 vIOMMU object in the
     kernel holding one S2 HWPT.
   - If a VM has 2 vIOMMUs, there will be 2 vIOMMU objects in the
     kernel that can hold two different S2 HWPTs, or share one S2
     HWPT (saving memory).

So if you have two devices assigned to a VM, then you may have two
vIOMMUs or one vIOMMU exposed to guest. This depends on whether the two
devices are behind the same physical IOMMU. If it's two vIOMMUs, the two
can share the s2 hwpt if their physical IOMMU is compatible. is it?

Yes.

quoted

To achieve the above, you need to know if the physical IOMMUs of the
assigned devices, hence be able to tell if physical IOMMUs are the
same and if they are compatible. How would userspace know such infos?

My draft implementation with QEMU does something like this:
  - List all viommu-matched iommu nodes under /sys/class/iommu: LINKs
  - Get PCI device's /sys/bus/pci/devices/0000:00:00.0/iommu: LINK0
  - Compare the LINK0 against the LINKs

We so far don't have an ID for physical IOMMU instance, which can
be an alternative to return via the hw_info call, otherwise.

intel platform has a kind of ID for the physical IOMMUs.

ls /sys/class/iommu/
dmar0  dmar1  dmar10  dmar11  dmar12  dmar13  dmar14  dmar15  dmar16
dmar17  dmar18  dmar19  dmar2  dmar3  dmar4  dmar5  dmar6  dmar7  dmar8
dmar9  iommufd_selftest_iommu.0

Wow, that's a lot of IOMMU devices. I somehow had an impression
that Intel uses one physical IOMMU..

Yea, we need something in the core. I had one patch previously:
https://github.com/nicolinc/iommufd/commit/b7520901184fd9fa127abb88c1f0be16b9967cff

quoted

QEMU then does the routing to assign PCI buses and IORT (or DT).
This part is suggested now to move to libvirt though. So, I think
at the end of the day, libvirt would run the sys check and assign
a device to the corresponding pci bus backed by the correct IOMMU.

and also give the correct viommu for the device.

In this design, a pxb bus is exclusively created for a viommu
instance, meaning so long as device is assigned to the correct
bus number, it'll be linked to the correct viommu.

quoted

This gives an example showing two devices behind iommu0 and third
device behind iommu1 are assigned to a VM:
   -device pxb-pcie.id=pcie.viommu0,bus=pcie.0.... \   # bus for viommu0
   -device pxb-pcie.id=pcie.viommu1,bus=pcie.0.... \   # bus for viommu1
   -device pcie-root-port,id=pcie.viommu0p0,bus=pcie.viommu0... \
   -device pcie-root-port,id=pcie.viommu0p1,bus=pcie.viommu0... \
   -device pcie-root-port,id=pcie.viommu1p0,bus=pcie.viommu1... \
   -device vfio-pci,bus=pcie.viommu0p0... \  # connect to bus for viommu0
   -device vfio-pci,bus=pcie.viommu0p1... \  # connect to bus for viommu0
   -device vfio-pci,bus=pcie.viommu1p0...    # connect to bus for viommu1

is the viommu# an "-object" or just hints to describe the relationship
between device and viommu and build the IORT?

Yes. Eric actually suggested something better for the relationship
between pxb-pcie with viommu:

-device
pxb-pcie,bus_nr=100,id=pci.12,numa_node=0,bus=pcie.0,addr=0x3,iommu=<id>
from:
https://lore.kernel.org/qemu-devel/9c3e95c2-1035-4a55-89a3-97165ef32f18@redhat.com/ (local)

This would likely help the IORT or Device Tree building.

Currently, ARM VIRT machine doesn't create a vSMMU via a "-device"
string, i.e. not a plugable module yet. I recall Intel does. So,
you guys are one step ahead.

I'm considering how it would look like if the QEMU Intel vIOMMU is going
to use the viommu obj. Currently, we only support one virtual VT-d due to
some considerations like hot-plug. Per your conversation with Kevin, it
seems to be supported. So there is no strict connection between vIOMMU
and vIOMMU obj. But the vIOMMU obj can only be connected with one pIOMMU.
right?

Yes. Most of my earlier vSMMU versions did something similar, e.g.
one shared vSMMU instance in the QEMU holding a list of S2 hwpts.
With this new iommufd viommu object, it would be a list of viommu
objs. Eric suggested that HostIOMMUDevice could store any pIOMMU
info. So, compatibility check can be done with that (or the old
fashioned way of trying an device attach).

The invalidation on the other hand needs to identify each trapped
invalidation request to distribute it to the correct viommu. This
is also one of the cons of this shared viommu model: invalidation
inefficiency -- there can be some cases where we fail to identify
which viommu to distribute so we have to broadcast to all viommus.
With a multi-viommu-instance model, invalidations are distributed
naturally by the guest kernel.

Thanks
Nicolin