Hi Mark,

I have tried to answer your comments, in the meantime we are waiting for
Ben to share the details.

On Fri, Aug 28, 2015 at 6:02 PM, Mark Rutland [off-list ref] wrote:

Hi,

On Fri, Aug 14, 2015 at 05:39:32PM +0100, Ganapatrao Kulkarni wrote:

quoted

DT bindings for numa map for memory, cores and IOs using
arm,associativity device node property.

Given this is just a copy of ibm,associativity, I'm not sure I see much
point in renaming the properties.

However, (somewhat counter to that) I'm also concerned that this isn't
sufficient for systems we're beginning to see today (more on that
below), so I don't think a simple copy of ibm,associativity is good
enough.

it is just copy right now, however it can evolve when we come across more
arm64 numa platforms

quoted

Signed-off-by: Ganapatrao Kulkarni <redacted>
---
 Documentation/devicetree/bindings/arm/numa.txt | 212

+++++++++++++++++++++++++

quoted

 1 file changed, 212 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/arm/numa.txt

diff --git a/Documentation/devicetree/bindings/arm/numa.txt

b/Documentation/devicetree/bindings/arm/numa.txt

quoted

new file mode 100644
index 0000000..dc3ef86

--- /dev/null
+++ b/Documentation/devicetree/bindings/arm/numa.txt

@@ -0,0 +1,212 @@

+==============================================================================

quoted

+NUMA binding description.

+==============================================================================

quoted

+

+==============================================================================

quoted

+1 - Introduction

+==============================================================================

quoted

+
+Systems employing a Non Uniform Memory Access (NUMA) architecture

contain

quoted

+collections of hardware resources including processors, memory, and I/O

buses,

quoted

+that comprise what is commonly known as a NUMA node.
+Processor accesses to memory within the local NUMA node is generally

faster

quoted

+than processor accesses to memory outside of the local NUMA node.
+DT defines interfaces that allow the platform to convey NUMA node
+topology information to OS.
+

+==============================================================================

quoted

+2 - arm,associativity

+==============================================================================

quoted

+The mapping is done using arm,associativity device property.
+this property needs to be present in every device node which needs to

to be

quoted

+mapped to numa nodes.

Can't there be some inheritance? e.g. all devices on a bus with an
arm,associativity property being assumed to share that value?

yes there is inheritance and respective bus drivers should take care of it,
like pci driver does at present.

quoted

+
+arm,associativity property is set of 32-bit integers which defines

level of

s/set/list/ -- the order is important.

ok

quoted

+topology and boundary in the system at which a significant difference in
+performance can be measured between cross-device accesses within
+a single location and those spanning multiple locations.
+The first cell always contains the broadest subdivision within the

system,

quoted

+while the last cell enumerates the individual devices, such as an SMT

thread

quoted

+of a CPU, or a bus bridge within an SoC".

While this gives us some hierarchy, this doesn't seem to encode relative
distances at all. That seems like an oversight.

distance is computed, will add the details to document.
local nodes will have distance as 10(LOCAL_DISTANCE) and every level, the
distance multiplies by 2.
for example, for level 1 numa topology, distance from local node to remote
node will be 20.

Additionally, I'm somewhat unclear on how what you'd be expected to
provide for this property in cases like ring or mesh interconnects,
where there isn't a strict hierarchy (see systems with ARM's own CCN, or
Tilera's TILE-Mx), but there is some measure of closeness.

IIUC, as per ARMs CCN architecture, all core/clusters are at equal distance
of DDR, i dont see any NUMA topology.
however, if there are 2 SoC connected thorough the CCN, then it is very
much similar to cavium topology.

Must all of these have the same length? If so, why not have a

#(whatever)-cells property in the root to describe the expected length?
If not, how are they to be interpreted relative to each other?

yes, all are of default size.
IMHO, there is no need to add cells property.

quoted

+
+ex:

s/ex/Example:/, please. There's no need to contract that.

quoted

+       /* board 0, socket 0, cluster 0, core 0  thread 0 */
+       arm,associativity = <0 0 0 0 0>;
+

+==============================================================================

quoted

+3 - arm,associativity-reference-points

+==============================================================================

quoted

+This property is a set of 32-bit integers, each representing an index

into

Likeise, s/set/list/

ok

quoted

+the arm,associativity nodes. The first integer is the most significant
+NUMA boundary and the following are progressively less significant

boundaries.

quoted

+There can be more than one level of NUMA.

I'm not clear on why this is necessary; the arm,associativity property
is already ordered from most significant to least significant per its
description.

first entry in arm,associativity-reference-points is used to find which
entry in associativity defines node id.
also entries in arm,associativity-reference-points defines,
how many entries(depth) in associativity can be used to calculate node
distance
in both level 1 and  multi level(hierarchical) numa topology.

What does this property achieve?

The description also doesn't describe where this property is expected to
live. The example isn't sufficient to disambiguate that, especially as
it seems like a trivial case.

sure, will add one more example to describe the
arm,associativity-reference-points

Is this only expected at the root of the tree? Can it be re-defined in
sub-nodes?

yes it is defined only at the root.

quoted

+
+Ex:

s/Ex/Example:/, please

sure.

quoted

+       arm,associativity-reference-points = <0 1>;
+       The board Id(index 0) used first to calculate the associativity

(node

quoted

+       distance), then follows the  socket id(index 1).
+
+       arm,associativity-reference-points = <1 0>;
+       The socket Id(index 1) used first to calculate the associativity,
+       then follows the board id(index 0).
+
+       arm,associativity-reference-points = <0>;
+       Only the board Id(index 0) used to calculate the associativity.
+
+       arm,associativity-reference-points = <1>;
+       Only socket Id(index 1) used to calculate the associativity.
+

+==============================================================================

quoted

+4 - Example dts

+==============================================================================

quoted

+
+Example: 2 Node system consists of 2 boards and each board having one

socket

quoted

+and 8 core in each socket.
+
+       arm,associativity-reference-points = <0>;
+
+       memory@00c00000 {
+               device_type = "memory";
+               reg = <0x0 0x00c00000 0x0 0x80000000>;
+               /* board 0, socket 0, no specific core */
+               arm,associativity = <0 0 0xffff>;
+       };
+
+       memory@10000000000 {
+               device_type = "memory";
+               reg = <0x100 0x00000000 0x0 0x80000000>;
+               /* board 1, socket 0, no specific core */
+               arm,associativity = <1 0 0xffff>;
+       };
+
+       cpus {
+               #address-cells = <2>;
+               #size-cells = <0>;
+
+               cpu@000 {
+                       device_type = "cpu";
+                       compatible =  "arm,armv8";
+                       reg = <0x0 0x000>;
+                       enable-method = "psci";
+                       /* board 0, socket 0, core 0*/
+                       arm,associativity = <0 0 0>;

We should specify w.r.t. memory and CPUs how the property is expected to
be used (e.g. in the CPU nodes rather than the cpu-map, with separate
memory nodes, etc). The generic description of arm,associativity isn't
sufficient to limit confusion there.

ok, will add the details like which nodes can use this property.

Thanks,
Mark.

thanks
Ganapat