+CC Brice.  

On Sun, 5 Jul 2020 09:53:58 +0000
"Song Bao Hua (Barry Song)" [off-list ref] wrote:

quoted

-----Original Message-----
From: Will Deacon [mailto:will@kernel.org]
Sent: Saturday, July 4, 2020 4:22 AM
To: Song Bao Hua (Barry Song) <redacted>
Cc: robin.murphy@arm.com; hch@lst.de; m.szyprowski@samsung.com;
iommu@lists.linux-foundation.org; linux-arm-kernel@lists.infradead.org;
Linuxarm [off-list ref]
Subject: Re: [PATCH] iommu/arm-smmu-v3: expose numa_node attribute to
users in sysfs

On Sat, May 30, 2020 at 09:15:05PM +1200, Barry Song wrote:  

quoted

As tests show the latency of dma_unmap can increase dramatically while
calling them cross NUMA nodes, especially cross CPU packages, eg.
300ns vs 800ns while waiting for the completion of CMD_SYNC in an
empty command queue. The large latency causing by remote node will
in turn make contention of the command queue more serious, and enlarge
the latency of DMA users within local NUMA nodes.

Users might intend to enforce NUMA locality with the consideration of
the position of SMMU. The patch provides minor benefit by presenting
this information to users directly, as they might want to know it without
checking hardware spec at all.  

I don't think that's a very good reason to expose things to userspace.
I know sysfs shouldn't be treated as ABI, but the grim reality is that
once somebody relies on this stuff then we can't change it, so I'd
rather avoid exposing it unless it's absolutely necessary.  

Will, thanks for taking a look!

I am not sure if it is absolutely necessary, but it is useful to users. The whole story started
from some users who wanted to know the hardware topology very clear by reading some
sysfs node just like they are able to do that for pci devices. The intention is that users can
know hardware topology of various devices easily from linux since they maybe don't know
all the hardware details.

For pci devices, kernel has done that. And there are some other drivers out of pci
exposing numa_node as well. It seems it is hard to say it is absolutely necessary
for them too since sysfs shouldn't be treated as ABI. 

Brice,

Given hwloc is probably the most demanding user of topology information
currently...

How useful would this info be for hwloc and hwloc users?
Sort of feels like it might be useful in some cases.

The very brief description of what we have here is exposing the numa node
of an IOMMU.  The discussion also diverted into whether it just makes sense
to expose this for all platform devices or even do it at the device level.

Jonathan

I got some input from Linux users who also wanted to know the numa node for
other devices which are not PCI, for example, platform devices. And I thought the
requirement is kind of reasonable. So I also had another patch to generally support
this kind of requirements, with the below patch, this smmu patch is not necessary
any more:
https://lkml.org/lkml/2020/6/18/1257

for platform device created by ARM ACPI/IORT and general acpi_create_platform_device()
drivers/acpi/scan.c:
static void acpi_default_enumeration(struct acpi_device *device)
{
	...
	if (!device->flags.enumeration_by_parent) {
		acpi_create_platform_device(device, NULL);
		acpi_device_set_enumerated(device);
	}
}

struct platform_device *acpi_create_platform_device(struct acpi_device *adev,
					struct property_entry *properties)
{
	...

	pdev = platform_device_register_full(&pdevinfo);
	if (IS_ERR(pdev))
		...
	else {
		set_dev_node(&pdev->dev, acpi_get_node(adev->handle));
		...
	}
	...
}
numa_node is set for this kind of devices.

Anyway, just want to explain to you the background some people want to know the 
hardware topology from Linux in same simple way. And it seems it is a reasonable
requirement to me :-)

quoted

Thanks,

Will  

Thanks
barry
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel