On Mon, Oct 26, 2026 at 11:49:17AM -0700, Jacob Pan wrote:
Hi Mostafa,
On Wed, 21 Oct 2026 10:55:36 +0000
Mostafa Saleh [off-list ref] wrote:
quoted
On Fri, Sep 25, 2026 at 12:07:33PM -0700, Jacob Pan wrote:
quoted
Hi,
This RFC supports guest VM device assignment from a Linux host
using the Microsoft Hypervisor (MSHV). The goal is that a common
set of flows and infrastructure can be applied for Xen and other
type 1 hypervisors.
Background:
The IOMMUFD/VFIO stack used for device assignment has largely
evolved around the KVM/QEMU model, at least as the first enabling
target. In that model, the Linux host owns the host-side physical
IOMMU programming as well as the physical PCI devices.
Architecturally, MSHV and Xen differ from this KVM/QEMU ownership
model. At a high level:
+------------------+
| Guest VM |
| virtual IOMMU |
+--------+---------+
| vIOMMU ops
v
+------------------+ external attach +------------------+
| MSHV |<-----------------------| Linux host |
+--------+---------+ +--------+---------+
| programs | owns
v v
+------------------+ +------------------+
| HW IOMMU |<---------------------->| PCI dev |
+------------------+ DMA +------------------+
For MSHV, the specific ownership model is as follows:
1. The Linux host OS owns the physical device and exposes it
through VFIO/IOMMUFD, but it does not directly own any of the IOMMU
hardware. 2. The device node (/dev/mshv) owns the VM partition file
used by userspace. 3. MSHV owns most of the physical IOMMU
programming, including stage-2 I/O page tables.
4. The Linux MSHV root pvIOMMU driver bridges IOMMUFD/VFIO UAPIs
to hardware IOMMU programming through MSHV hypercalls.
5. Guest virtual IOMMU emulation is done in MSHV rather than the
userspace VMM.
6. Stage-2 IOPT may be shared between CPU and IOMMU. It is
managed by MSHV and abstracted from the Linux host, including TLB
synchronization.
These structural differences from the KVM/QEMU model create several
needs for IOMMUFD support:
1. Move a device to a guest whose I/O page table is externally
managed. This patchset refers to that operation as external attach.
Unlike conventional IOMMU vendor drivers, where translation is
managed within the driver through paging domains, external attach
means that translation is managed outside the driver, such as by
the hypervisor. 2. Identify the guest-visible device across VMM,
host pvIOMMU driver, and hypervisor namespaces.
3. Identify the guest partition in the host pvIOMMU driver for
device attach hypercalls.
4. Tie the device-attachment lifetime to the hypervisor partition.
5. Coordinate guest-memory pinning and mapping for DMA between
MSHV and the external IOMMU domain attachment.
External attach has a few caveats compared with KVM/QEMU, where
nesting is typically handled by the host IOMMU driver in one
transaction after the userspace VMM uses IOMMUFD to form nested
domains:
1. MSHV creates a temporal split between host and guest
device-domain attach. The host first attaches S2 through a
hypercall; later, if virtual IOMMU is present, the guest
allocates S1 and establishes nesting with MSHV directly
through a pvIOMMU or emulated vIOMMU.
2. Guest-memory pinning before mapping is covered by the
MSHV/userspace VMM contract, not by the VFIO/IOMMUFD pin-and-map
DMA flow. 3. Device capabilities such as ATS and PASID need a
contract among MSHV, the userspace VMM, and the host pvIOMMU
driver. At the time of host external attach, only S2 is attached,
so the guest's later ATS/PASID usage is not yet known. For example,
whether to expose and enable PASID in the device could depend on
whether the userspace VMM creates a guest virtual IOMMU.
This RFC proposes an IOMMUFD external attach model for those cases.
The normal Linux-managed paging or nested domain is not the object
that ultimately owns the translation context. Instead, userspace
identifies a VM/domain object, binds a physical device to a
VM-visible device identity, and asks the host pvIOMMU driver to
attach the device to an externally managed address space.
The series makes that relationship explicit in IOMMUFD. The model
is anchored around a hypervisor vIOMMU object that represents the
per-VM partition abstraction. It introduces:
1. IOMMU_DOMAIN_EXTERNAL for externally managed translation
domains. 2. IOMMU_VIOMMU_TYPE_HYPERVISOR for the file-backed VM
partition. 3. IOMMU_HWPT_EXTERNAL for user-visible external type
IOMMU domains.
It reuses the IOMMUFD vDEVICE object to carry the guest-VM-scoped
logical device ID visible to the hypervisor.
This is very similar to pKVM on arm64 also, where the IOMMU is managed
by the hypervisor, and for device assignment, the host doesn't control
it for the guest, but the hypervisor (in EL2), and there will be a
channel between the guest and pKVM directly after the guest runs.
The host only sets the virtual to physical topology (how many vIOMMUs,
vSID -> SID mappings) and describes that for the hypervisor through
hypercalls.
In Android, we have an out of tree uAPI through KVM/VFIO bridge
IOCTLs that proxies those hypercalls.
In that case, the host just creates a virtual IOMMU and the guest sees
a paravirtual one which runs with single stage collapsed translation,
although I don't see a difference on the host side if nested
translation was used beyond an argument change to the hypercalls.
By "argument change", do you mean that the host would pass the guest S1
context in the device-attach hypercall for nested translation?
IMHO, I don't see any change needed for the host-side attach between hw
nested and collapsed single stage, if guest and host attach are
split. I think it is up to pKVM or MSHV to handle guest virtual iommu
mapping for both cases.
Although the hypervisor (pKVM, MSHV) is managing the guest IOMMU in that
case, I believe the host should be able to decide the guest virtual
toplology, as that will impact the firmware tables populated from the
host for example, as in the nested case the guest will see an
architectural SMMUv3 while in the collapsed case it will see a simple
HVC based IOMMU.
quoted
My goal is to converge to IOMMUFD and I believe this series is the way
forward, By relying on the vDEVICE virtual toplogy uAPI and the
external attach introduced here.
I will try to prototype something based on that to prove that it
works. However that work has a dependency on the host SMMUv3 support
[1] which needs to land first.
[1] https://lore.kernel.org/all/20261015200903.2735565-1-smostafa@google.com/ (local)
Same here, our pviommu root driver must land first.
https://lore.kernel.org/linux-iommu/20260929223622.778617-5-mrathor@linux.microsoft.com/ (local)
My next step is to rebase on Aneesh's v7 such that we have a converged
iommufd viommu layer, i will remove mshv specific helper registration
in this series.
https://lore.kernel.org/linux-iommu/20261008055955.4014342-1-aneesh.kumar@kernel.org/ (local)
I see, I will add this to my review list also.
Thanks,
Mostafa