Re: [RFC 09/13] iommu/arm-smmu: Make use of dev_64bit_mmio_supported()

Hi Robin, thanks for taking the time to look at this.

On Tue, 2021-03-02 at 11:07 +0000, Robin Murphy wrote:

On 2021-02-26 14:03, Nicolas Saenz Julienne wrote:

quoted

Some arm SMMU implementations might sit on a bus that doesn't support
64bit memory accesses. In that case default to using hi_lo_{readq,
writeq}() and BUG if such platform tries to use AArch64 formats as they
rely on writeq()'s atomicity.

Signed-off-by: Nicolas Saenz Julienne <redacted>
---
  drivers/iommu/arm/arm-smmu/arm-smmu.c | 9 +++++++++
  drivers/iommu/arm/arm-smmu/arm-smmu.h | 9 +++++++--
  2 files changed, 16 insertions(+), 2 deletions(-)

diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.c b/drivers/iommu/arm/arm-smmu/arm-smmu.c
index d8c6bfde6a61..239ff42b20c3 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu.c
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu.c

@@ -1889,6 +1889,15 @@ static int arm_smmu_device_cfg_probe(struct arm_smmu_device *smmu)

  			smmu->features |= ARM_SMMU_FEAT_FMT_AARCH64_64K;
  	}
  

+	/*
+	 * 64bit accesses not possible through the interconnect, AArch64
+	 * formats depend on it.
+	 */
+	BUG_ON(!dev_64bit_mmio_supported(smmu->dev) &&
+	       smmu->features & (ARM_SMMU_FEAT_FMT_AARCH64_4K |
+				ARM_SMMU_FEAT_FMT_AARCH64_16K |
+				ARM_SMMU_FEAT_FMT_AARCH64_64K));

No. Crashing the kernel in a probe routine which is free to fail is 
unacceptable either way, but guaranteeing failure in the case that the 
workaround *would* be required is doubly so.

Basically, this logic is backwards - if you really wanted to handle it 
generically, this would be the point at which you'd need to actively 
suppress all the detected hardware features which depend on 64-bit 
atomicity, not complain about them.

Understood.

quoted

+
  	if (smmu->impl && smmu->impl->cfg_probe) {
  		ret = smmu->impl->cfg_probe(smmu);
  		if (ret)

diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.h b/drivers/iommu/arm/arm-smmu/arm-smmu.h
index d2a2d1bc58ba..997d13a21717 100644
--- a/drivers/iommu/arm/arm-smmu/arm-smmu.h
+++ b/drivers/iommu/arm/arm-smmu/arm-smmu.h

@@ -477,15 +477,20 @@ static inline void arm_smmu_writel(struct arm_smmu_device *smmu, int page,

  {
  	if (smmu->impl && unlikely(smmu->impl->write_reg))
  		smmu->impl->write_reg(smmu, page, offset, val);
-	else
+	else if (dev_64bit_mmio_supported(smmu->dev))
  		writel_relaxed(val, arm_smmu_page(smmu, page) + offset);
+	else
+		hi_lo_writeq_relaxed(val, arm_smmu_page(smmu, page) + offset);

As Arnd pointed out, this is in completely the wrong place. Also, in 

Yes, sorry for that, not too proud of it.

general it doesn't work if the implementation already needs a hook to 
filter or override register accesses for any other reason. TBH I'm not 

I'm not sure I get your point here, 'smmu->impl' has precedence over the MMIO
capability check. Custom implementations would still get their callbacks.

convinced that this isn't *more* of a mess than handling it on a 
SoC-specific basis...

I see your point.

Just to explain why I went to these lengths: my understanding is that the
specifics of how to perform 32bit accesses to SMMU's 64bit registers is defined
in spec. So it made sense to move it into the non implementation dependent side
of the driver.

All in all, I'll think of something simpler.

Regards,
Nicolas