RE: [PATCH v2 2/8] iommu/arm-smmu-v3: Use S2FWB when available

-----Original Message-----
From: Jason Gunthorpe <jgg@nvidia.com>
Sent: Tuesday, August 27, 2024 4:52 PM
To: acpica-devel@lists.linux.dev; Guohanjun (Hanjun Guo)
[off-list ref]; iommu@lists.linux.dev; Joerg Roedel
[off-list ref]; Kevin Tian [off-list ref];
kvm@vger.kernel.org; Len Brown [off-list ref]; linux-
acpi@vger.kernel.org; linux-arm-kernel@lists.infradead.org; Lorenzo Pieralisi
[off-list ref]; Rafael J. Wysocki [off-list ref]; Robert
Moore [off-list ref]; Robin Murphy
[off-list ref]; Sudeep Holla [off-list ref]; Will
Deacon [off-list ref]
Cc: Alex Williamson <redacted>; Eric Auger
[off-list ref]; Jean-Philippe Brucker <jean-
philippe@linaro.org>; Moritz Fischer [off-list ref]; Michael Shavit
[off-list ref]; Nicolin Chen [off-list ref];
patches@lists.linux.dev; Shameerali Kolothum Thodi
[off-list ref]; Mostafa Saleh
[off-list ref]
Subject: [PATCH v2 2/8] iommu/arm-smmu-v3: Use S2FWB when available

Force Write Back (FWB) changes how the S2 IOPTE's MemAttr field
works. When S2FWB is supported and enabled the IOPTE will force cachable
access to IOMMU_CACHE memory when nesting with a S1 and deny cachable
access otherwise.

When using a single stage of translation, a simple S2 domain, it doesn't
change anything as it is just a different encoding for the exsting mapping
of the IOMMU protection flags to cachability attributes.

However, when used with a nested S1, FWB has the effect of preventing the
guest from choosing a MemAttr in it's S1 that would cause ordinary DMA to
bypass the cache. Consistent with KVM we wish to deny the guest the
ability to become incoherent with cached memory the hypervisor believes is
cachable so we don't have to flush it.

Turn on S2FWB whenever the SMMU supports it and use it for all S2
mappings.

Signed-off-by: Jason Gunthorpe <jgg@nvidia.com>
---

(...)

This should be added to arm_64_lpae_alloc_pgtable_s2(), not here.

With the above fixed, I was able to assign a n/w VF dev to a Guest on a
test hardware that supports S2FWB.

However host kernel has this WARN message:
[ 1546.165105] WARNING: CPU: 5 PID: 7047 at drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c:1086 arm_smmu_entry_qword_diff+0x124/0x138
....
[ 1546.330312]  arm_smmu_entry_qword_diff+0x124/0x138
[ 1546.335090]  arm_smmu_write_entry+0x38/0x22c
[ 1546.339346]  arm_smmu_install_ste_for_dev+0x158/0x1ac
[ 1546.344383]  arm_smmu_attach_dev+0x138/0x240
[ 1546.348639]  __iommu_device_set_domain+0x7c/0x11c
[ 1546.353330]  __iommu_group_set_domain_internal+0x60/0x134
[ 1546.358714]  iommu_group_replace_domain+0x3c/0x68
[ 1546.363404]  iommufd_device_do_replace+0x334/0x398
[ 1546.368181]  iommufd_device_change_pt+0x26c/0x650
[ 1546.372871]  iommufd_device_replace+0x18/0x24
[ 1546.377214]  vfio_iommufd_physical_attach_ioas+0x28/0x68
[ 1546.382514]  vfio_df_ioctl_attach_pt+0x98/0x170


And when I tried to use the assigned n/w dev, it seems to do a reset
continuously.

root@localhost:/# ping 150.0.124.42
PING 150.0.124.42 (150.0.124.42): 56 data bytes
64 bytes from 150.0.124.42: seq=0 ttl=64 time=47.648 ms
[ 1395.958630] hns3 0000:c2:00.0 eth1: NETDEV WATCHDOG: CPU: 1: transmit queue 10 timed out 5260 ms
[ 1395.960187] hns3 0000:c2:00.0 eth1: DQL info last_cnt: 42, queued: 42, adj_limit: 0, completed: 0
[ 1395.961758] hns3 0000:c2:00.0 eth1: queue state: 0x6, delta msecs: 5260
[ 1395.962925] hns3 0000:c2:00.0 eth1: tx_timeout count: 1, queue id: 10, SW_NTU: 0x1, SW_NTC: 0x0, napi state: 16
[ 1395.964677] hns3 0000:c2:00.0 eth1: tx_pkts: 0, tx_bytes: 0, sw_err_cnt: 0, tx_pending: 0
[ 1395.966114] hns3 0000:c2:00.0 eth1: seg_pkt_cnt: 0, tx_more: 0, restart_queue: 0, tx_busy: 0
[ 1395.967598] hns3 0000:c2:00.0 eth1: tx_push: 1, tx_mem_doorbell: 0
[ 1395.968687] hns3 0000:c2:00.0 eth1: BD_NUM: 0x7f HW_HEAD: 0x0, HW_TAIL: 0x0, BD_ERR: 0x0, INT: 0x1
[ 1395.970291] hns3 0000:c2:00.0 eth1: RING_EN: 0x1, TC: 0x0, FBD_NUM: 0x0 FBD_OFT: 0x0, EBD_NUM: 0x400, EBD_OFT: 0x0
[ 1395.972134] hns3 0000:c2:00.0: received reset request from VF enet

All this works fine on a hardware without S2FWB though.

Also on this test hardware, it works fine with legacy VFIO assignment.

Not debugged further. Please let me know if you have any hunch.

Thanks,
Shameer