Re: [RFC PATCH v4 03/16] iommu/arm-smmu-v3: Add initial pSMMU realm viommu plumbing

Jason Gunthorpe [off-list ref] writes:

On Wed, Sep 02, 2026 at 06:45:42PM +0530, Aneesh Kumar K.V wrote:

quoted

To reiterate, for this configuration:

- The viommu will use a stage-1 bypass configuration.
- A new IOMMU_VIOMMU_TYPE_ARM_REALM_SMMUV3 type will create the viommu.
  The psmmu will be activated at this point to avoid creating psmmu
  objects early. We will reference-count it to ensure that the same
  psmmu is shared across realm guests.
- Creating a vdevice will invoke SMC_RMI_PSMMU_ST_L2_CREATE.

I am unclear about the vdev_create suggestion. Creating a vdevice
requires an RD, which is created later in the flow above. How do you
suggest linking vdevice_alloc to vdev_create?

I was thinking we'd make sure the KVM is associated with the viommu
and/or possibly the S2 domain. That was always sort of broadly the
idea in this space. There are several topics unrelated to CC that
needed this.
In any case, when you create the iommufd viommu you should also do
VSMMU_CREATE which needs the RD. It seems reasonable to assume the RD
is available during vdevice create.

[There is an aside here I will mention: several other use cases need
this idea of an "external" domain where the HWPT would be created but
not controlled by iommufd or the iommu subsystem. Xen and Hyperv for
example. There is probably some merit in thinking more about exactly
what the nested parent domain should be for this viommu, but it isn't
critical.]

quoted

From the RMM's perspective, the sequence is as follows:

viommu alloc

[   rmm ] SMC_RMI_PSMMU_ACTIVATE            2b400000 8819bb000 > RMI_INCOMPLETE 0 10008
[   rmm ]       SMC_RMI_OP_MEM_DONATE       0 881bac098 1 > RMI_INCOMPLETE 2 10004
[   rmm ]       SMC_RMI_OP_MEM_DONATE       0 881bac098 1 > RMI_INCOMPLETE 1 10004
[   rmm ]       SMC_RMI_OP_MEM_DONATE       0 881bac098 1 > RMI_INCOMPLETE 1 0
[   rmm ] L1 StrTab: PA 0x881baa000 VA 0x80003c0000 size 0x2000
[   rmm ] CMDQ: PA 0x88066a000 VA 0x80003c2000
[   rmm ] EVTQ: PA 0x8815c5000 VA 0x80003c3000
[   rmm ] PSMMU 0x2b400000 activated
[   rmm ]       SMC_RMI_OP_CONTINUE         0 0 > RMI_SUCCESS 0 0

[   rmm ] SMC_RMI_PSMMU_ST_L2_CREATE        2b400000 300 > RMI_INCOMPLETE 0 10004
[   rmm ]       SMC_RMI_OP_MEM_DONATE       0 881bac098 1 > RMI_INCOMPLETE 1 0
[   rmm ] smmu->strtab_base[12] 0x0 @0x80003c0060
[   rmm ] L1STD[12] 0x8819bb007 for SID 0x300: L2 table VA 0x80003d0000 PA 0x8819bb000
[   rmm ]       SMC_RMI_OP_CONTINUE         0 0 > RMI_SUCCESS 0 0

What was this one for during viommu alloc?

quoted

vdevice alloc

[   rmm ] SMC_RMI_PSMMU_ST_L2_CREATE        2b400000 200 > RMI_INCOMPLETE 0 10004
[   rmm ]       SMC_RMI_OP_MEM_DONATE       0 882648098 1 > RMI_INCOMPLETE 1 0
[   rmm ] smmu->strtab_base[8] 0x0 @0x80003c0040
[   rmm ] L1STD[8] 0x882506007 for SID 0x200: L2 table VA 0x80003cc000 PA 0x882506000
[   rmm ]       SMC_RMI_OP_CONTINUE         0 0 > RMI_SUCCESS 0 0

RD gets allocated here

[   rmm ] SMC_RMI_REALM_CREATE              881b03000 8819a1000 > RMI_INCOMPLETE 0 24
[   rmm ]       SMC_RMI_OP_MEM_DONATE       0 881605098 9 > RMI_INCOMPLETE 9 0
[   rmm ]       SMC_RMI_OP_CONTINUE         0 0 > RMI_SUCCESS 0 0

Why? The VMM needs to create the kvm before starting the iommu stuff.

I would expect that KVM knows it is going to a realm very early on? I
had assumed the RD would also be created early by KVM?

What triggers RD creation? Why can't the VMM do it earlier?

Your followup said:
  - The realm is created during viommu allocation, ensuring that the
    vSMMU can be created here.

Do you mean the SMMUv3 driver triggers RD creation? That feels
wrong. Did you mean the VMM just does it earlier?

Currently, CCA creates the realm lazily as part of another operation.
Realm creation is triggered by kvm_arm_rmi_populate() or
kvm_arch_vcpu_run_pid_change()

The change looks like this:

modified   arch/arm64/include/asm/kvm_rmi.h

@@ -102,6 +102,14 @@ u64 kvm_realm_reset_id_aa64dfr0_el1(const struct kvm_vcpu *vcpu, u64 val);
 bool kvm_rmi_supports_sve(void);
 
 int kvm_init_realm(struct kvm *kvm);
+#ifdef CONFIG_KVM
+int kvm_realm_ensure_created(struct kvm *kvm);
+#else
+static inline int kvm_realm_ensure_created(struct kvm *kvm)
+{
+	return -EOPNOTSUPP;
+}
+#endif
 int kvm_activate_realm(struct kvm *kvm);
 void kvm_destroy_realm(struct kvm *kvm);
 int kvm_realm_teardown_stage2(struct kvm *kvm);

modified   arch/arm64/kvm/rmi.c

@@ -1120,6 +1120,20 @@ static int realm_ensure_created(struct kvm *kvm)
 	return realm_create_rd(kvm);
 }
 
+int kvm_realm_ensure_created(struct kvm *kvm)
+{
+	int ret;
+
+	if (!kvm_is_realm(kvm))
+		return -EINVAL;
+
+	guard(mutex)(&kvm->arch.config_lock);
+	ret = realm_ensure_created(kvm);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(kvm_realm_ensure_created);
+
 int kvm_arm_rmi_init_ripas(struct kvm *kvm,
 			   struct kvm_arm_rmi_init_ripas *args)
 {

modified   drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-realm.c

@@ -253,8 +253,9 @@ int arm_realm_smmu_v3_init(struct iommufd_viommu *viommu,
 		return -EINVAL;
 
 	kvm = viommu->kvm_file->private_data;
-	if (!kvm_is_realm(kvm))
-		return -EINVAL;
+	ret = kvm_realm_ensure_created(kvm);
+	if (ret)
+		return ret;
 
 	if (!(smmu->features & ARM_SMMU_FEAT_RME))
 		return -EOPNOTSUPP;

The draft in the next message looks promising, did you discover any
other gotchas when exploring it?

Nothing significant. I still have questions about moving TDI flows such
as vdev creation and guest requests into the SMMU driver instead of
keeping them in arm-cca-host.ko, but I will follow up in the other
thread.

When you repost this can you take some care to explain the general
idea of this modelling in the commit messages so AMD and Intel can
confirm they can use it for both their non-viommu and viommu cases?

Thanks,
Jason

-aneesh