-----Original Message-----
From: Robin Murphy [mailto:robin.murphy@arm.com]
Sent: Tuesday, June 2, 2020 1:14 AM
To: Song Bao Hua (Barry Song) <redacted>; will@kernel.org;
hch@lst.de; m.szyprowski@samsung.com; iommu@lists.linux-foundation.org
Cc: Linuxarm <redacted>; linux-arm-kernel@lists.infradead.org
Subject: Re: [PATCH] iommu/arm-smmu-v3: expose numa_node attribute to
users in sysfs

On 2020-05-30 10:15, Barry Song wrote:

quoted

As tests show the latency of dma_unmap can increase dramatically while
calling them cross NUMA nodes, especially cross CPU packages, eg.
300ns vs 800ns while waiting for the completion of CMD_SYNC in an
empty command queue. The large latency causing by remote node will
in turn make contention of the command queue more serious, and enlarge
the latency of DMA users within local NUMA nodes.

Users might intend to enforce NUMA locality with the consideration of
the position of SMMU. The patch provides minor benefit by presenting
this information to users directly, as they might want to know it without
checking hardware spec at all.

Hmm, given that dev-to_node() is a standard driver model thing, is there
not already some generic device property that can expose it - and if
not, should there be? Presumably if userspace cares enough to want to
know whereabouts in the system an IOMMU is, it probably also cares where
the actual endpoint devices are too.

At the very least, it doesn't seem right for it to be specific to one
single IOMMU driver.

Right now pci devices have generally got the numa_node in sysfs by drivers/pci/pci-sysfs.c

static ssize_t numa_node_store(struct device *dev,
                               struct device_attribute *attr, const char *buf,
                               size_t count)
{
        ...

        add_taint(TAINT_FIRMWARE_WORKAROUND, LOCKDEP_STILL_OK);
        pci_alert(pdev, FW_BUG "Overriding NUMA node to %d.  Contact your vendor for updates.",
                  node);

        dev->numa_node = node;
        return count;
}

static ssize_t numa_node_show(struct device *dev, struct device_attribute *attr,
                              char *buf)
{
        return sprintf(buf, "%d\n", dev->numa_node);
}
static DEVICE_ATTR_RW(numa_node);

for other devices who care about numa information, the specific drivers are doing that, for example:

drivers/dax/bus.c:      if (a == &dev_attr_numa_node.attr && !IS_ENABLED(CONFIG_NUMA))
drivers/dax/bus.c:      &dev_attr_numa_node.attr,
drivers/dma/idxd/sysfs.c:       &dev_attr_numa_node.attr,
drivers/hv/vmbus_drv.c: &dev_attr_numa_node.attr,
drivers/nvdimm/bus.c:   &dev_attr_numa_node.attr,
drivers/nvme/host/core.c:       &dev_attr_numa_node.attr,

smmu is usually a platform device, we can actually expose numa_node for platform_device, or globally expose numa_node
for general "device" if people don't opposite.

Barry

Robin.

quoted

Signed-off-by: Barry Song <redacted>
---
  drivers/iommu/arm-smmu-v3.c | 40

++++++++++++++++++++++++++++++++++++-

quoted

  1 file changed, 39 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
index 82508730feb7..754c4d59498b 100644
--- a/drivers/iommu/arm-smmu-v3.c
+++ b/drivers/iommu/arm-smmu-v3.c

@@ -4021,6 +4021,44 @@ err_reset_pci_ops: __maybe_unused;
  	return err;
  }

+static ssize_t numa_node_show(struct device *dev,
+		struct device_attribute *attr, char *buf)
+{
+	return sprintf(buf, "%d\n", dev_to_node(dev));
+}
+static DEVICE_ATTR_RO(numa_node);
+
+static umode_t arm_smmu_numa_attr_visible(struct kobject *kobj, struct

attribute *a,

quoted

+		int n)
+{
+	struct device *dev = container_of(kobj, typeof(*dev), kobj);
+
+	if (!IS_ENABLED(CONFIG_NUMA))
+		return 0;
+
+	if (a == &dev_attr_numa_node.attr &&
+			dev_to_node(dev) == NUMA_NO_NODE)
+		return 0;
+
+	return a->mode;
+}
+
+static struct attribute *arm_smmu_dev_attrs[] = {
+	&dev_attr_numa_node.attr,
+	NULL
+};
+
+static struct attribute_group arm_smmu_dev_attrs_group = {
+	.attrs          = arm_smmu_dev_attrs,
+	.is_visible     = arm_smmu_numa_attr_visible,
+};
+
+
+static const struct attribute_group *arm_smmu_dev_attrs_groups[] = {
+	&arm_smmu_dev_attrs_group,
+	NULL,
+};
+
  static int arm_smmu_device_probe(struct platform_device *pdev)
  {
  	int irq, ret;

@@ -4097,7 +4135,7 @@ static int arm_smmu_device_probe(struct

platform_device *pdev)

quoted

  		return ret;

  	/* And we're up. Go go go! */
-	ret = iommu_device_sysfs_add(&smmu->iommu, dev, NULL,
+	ret = iommu_device_sysfs_add(&smmu->iommu, dev,

arm_smmu_dev_attrs_groups,

quoted

  				     "smmu3.%pa", &ioaddr);
  	if (ret)
  		return ret;

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel