On Thu, 12 Sept 2024 at 12:29, Baolu Lu [off-list ref] wrote:

On 9/12/24 11:42 AM, Zhangfei Gao wrote:

quoted

On Wed, 28 Aug 2024 at 05:32, Nicolin Chen[off-list ref]  wrote:

quoted

On Tue, Aug 27, 2024 at 12:51:30PM -0300, Jason Gunthorpe wrote:

quoted

This brings support for the IOMMFD ioctls:

  - IOMMU_GET_HW_INFO
  - IOMMU_HWPT_ALLOC_NEST_PARENT
  - IOMMU_DOMAIN_NESTED
  - ops->enforce_cache_coherency()

This is quite straightforward as the nested STE can just be built in the
special NESTED domain op and fed through the generic update machinery.

The design allows the user provided STE fragment to control several
aspects of the translation, including putting the STE into a "virtual
bypass" or a aborting state. This duplicates functionality available by
other means, but it allows trivially preserving the VMID in the STE as we
eventually move towards the VIOMMU owning the VMID.

Nesting support requires the system to either support S2FWB or the
stronger CANWBS ACPI flag. This is to ensure the VM cannot bypass the
cache and view incoherent data, currently VFIO lacks any cache flushing
that would make this safe.

Yan has a series to add some of the needed infrastructure for VFIO cache
flushing here:

  https://lore.kernel.org/linux-iommu/20240507061802.20184-1-yan.y.zhao@intel.com/ (local)

Which may someday allow relaxing this further.

Remove VFIO_TYPE1_NESTING_IOMMU since it was never used and superseded by
this.

This is the first series in what will be several to complete nesting
support. At least:
  - IOMMU_RESV_SW_MSI related fixups
     https://lore.kernel.org/linux-iommu/cover.1722644866.git.nicolinc@nvidia.com/ (local)
  - VIOMMU object support to allow ATS and CD invalidations
     https://lore.kernel.org/linux-iommu/cover.1723061377.git.nicolinc@nvidia.com/ (local)
  - vCMDQ hypervisor support for direct invalidation queue assignment
     https://lore.kernel.org/linux-iommu/cover.1712978212.git.nicolinc@nvidia.com/ (local)
  - KVM pinned VMID using VIOMMU for vBTM
     https://lore.kernel.org/linux-iommu/20240208151837.35068-1-shameerali.kolothum.thodi@huawei.com/ (local)
  - Cross instance S2 sharing
  - Virtual Machine Structure using VIOMMU (for vMPAM?)
  - Fault forwarding support through IOMMUFD's fault fd for vSVA

The VIOMMU series is essential to allow the invalidations to be processed
for the CD as well.

It is enough to allow qemu work to progress.

This is on github:https://github.com/jgunthorpe/linux/commits/smmuv3_nesting

v2:

As mentioned above, the VIOMMU series would be required to test
the entire nesting feature, which now has a v2 rebasing on this
series. I tested it with a paring QEMU branch. Please refer to:
https://lore.kernel.org/linux-iommu/cover.1724776335.git.nicolinc@nvidia.com/ (local)
Also, there is another new VIRQ series on top of the VIOMMU one
and this nesting series. And I tested it too. Please refer to:
https://lore.kernel.org/linux-iommu/cover.1724777091.git.nicolinc@nvidia.com/ (local)

With that,

Tested-by: Nicolin Chen<redacted>

Have you tested the user page fault?

I got an issue, when a user page fault happens,
  group->attach_handle = iommu_attach_handle_get(pasid)
return NULL.

A bit confused here, only find IOMMU_NO_PASID is used when attaching

  __fault_domain_replace_dev
ret = iommu_replace_group_handle(idev->igroup->group, hwpt->domain,
&handle->handle);
curr = xa_store(&group->pasid_array, IOMMU_NO_PASID, handle, GFP_KERNEL);

not find where the code attach user pasid with the attach_handle.

Have you set iommu_ops::user_pasid_table for SMMUv3 driver?

Thanks Baolu, Nico

Yes, after arm_smmu_ops = {
+       .user_pasid_table       = 1,

find_fault_handler can go inside attach_handle =
iommu_attach_handle_get(IOMMU_NO_PASID);
qemu handler also gets called.

But hardware reports errors and needs reset, still in check.
[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 0

Thanks