Re: [PATCH v1 4/5] iommu/arm-smmu-v3: Keep track of attached ssids

Oh nice, this is exactly what I was looking for (minus the missing ATC
inv, but that's somewhat easier to reason from code)! Thanks for the
detailed guide Jean!
I finally got around to trying it out and was able to see the page
fault followed by invalidations on this patch-series as it is. This
will be super useful to start refactoring SVA for a v2 of this patch.

On Fri, May 12, 2023 at 3:59 AM Jean-Philippe Brucker
[off-list ref] wrote:

On Thu, May 11, 2023 at 11:26:48PM +0800, Michael Shavit wrote:

quoted

quoted

You should be getting rid of mm->pasid in this series as well.

When each domain keeps track of what STE/CD entries that point to it then
*ALL* invalidation should iterate over the list of pointing entires
and generate the correct invalidation for that pointer.

Completely agree. The arm_smmu_atc_inv_domain_ssid function introduced
by this patch is a stopgap to decompose this patch from the SVA
refactor that's required to stop using ssid in these calls.
I also agree that such a refactoring probably belongs in the same
patch series. @Jean-Philippe Brucker and others: is there any way I
can about testing or at least exercising the SVA flow without physical
hardware that supports SVA?

Yes, there is a model with a simple test device that supports PASID and
I/O page faults. It's not completely straightforward to setup and the
driver needs to be rewritten from scratch, but it's the best we have at
the moment.  I'd like to do something equally useful for QEMU, so we can
have proper regression tests, but that requires a lot of preliminary work
to add PASID+PRI to PCI, virtio and IOMMUs.

You'll need a kernel with the driver and a rootfs with the smmute
tool [1]; the RevC model [2] and a boot-wrapper [3].

  $ ${BOOTWRAPPER}/configure --host=aarch64-linux-gnu
     --with-dtb=${KERNEL}/arch/arm64/boot/dts/arm/fvp-base-revc.dts
     --with-kernel-dir=${KERNEL}
     --with-initrd=${BUILDROOT}/images/rootfs.cpio
     --with-cmdline="console=ttyAMA0"
     --enable-psci --enable-gicv3
  $ make        # produces linux-system.axf

Run the model:
  $ FVP_Base_RevC-2xAEMvA
     -C bp.secure_memory=0
     -C 'pctl.startup=0.*.*.*'
     -C bp.refcounter.non_arch_start_at_default=1
     -C cache_state_modelled=0
     -C bp.vis.disable_visualisation=1
     -C bp.virtio_net.enabled=1
     -C bp.virtio_net.hostbridge.userNetPorts=8022=22
     -C bp.virtio_net.hostbridge.userNetworking=1
     -C pci.pci_smmuv3.mmu.SMMU_IDR0=135100351
     -C pci.pci_smmuv3.mmu.SMMU_IDR3=4116
     -C pci.pci_smmuv3.mmu.SMMU_IDR5=8389749
     -C cluster0.NUM_CORES=4
     -C cluster1.NUM_CORES=4
     -a 'cluster*.cpu*=linux-system.axf'

Then run a job using the tool. The process allocates two buffers and
passes their VA to the device (via the kernel driver). The device memcpies
one buffer to the other:

   $ smmute -u mmap
   ... Success

With smmu and iommu trace events enabled, a trace should contain
smmu_mm_invalidate and dev_fault/dev_page_response events.

It's not entirely representative of SVA flow, where an assignable device
interface is mapped into the process and the process launches jobs
directly without going through the kernel (that would now use
drivers/misc/uacce), but it does exercise IOMMU SVA: sva_bind(), device
accessing the process address space with PASID and some IOPFs, which I
think is what you're looking for. However this model doesn't have a PCI
test device so you won't be able to test ATC invalidations with PASID.

Other useful tests would be enabling lockdep (some intricate locking
between IOPF, the driver and mm), killing bound processes (-k), triggering
invalid accesses to verify TLB invalidations (-f tlb, I think). There is a
lot more to test, like thp and oom, but I don't have those in this branch.

Thanks,
Jean

[1] https://jpbrucker.net/git/linux/log/?h=sva/smmute-revc
[2] https://developer.arm.com/downloads/-/arm-ecosystem-models
[3] https://git.kernel.org/pub/scm/linux/kernel/git/mark/boot-wrapper-aarch64.git/

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel