On 6/17/21 1:20 PM, David Gibson wrote:

On Tue, Jun 15, 2021 at 01:10:27PM +0530, Aneesh Kumar K.V wrote:

quoted

David Gibson [off-list ref] writes:

....

quoted

quoted

quoted

PAPR defines "most significant" as below

When the “ibm,architecture-vec-5” property byte 5 bit 0 has the value of one, the “ibm,associativ-
ity-reference-points” property indicates boundaries between associativity domains presented by the
“ibm,associativity” property containing “near” and “far” resources. The
first such boundary in the list represents the 1 based ordinal in the
associativity lists of the most significant boundary, with subsequent
entries indicating progressively less significant boundaries

No... that's not a definition.  Like your draft PAPR uses the term
while entirely failing to define it.  From what I can tell about how
it is used the "most significant" boundary corresponds to what Linux
simply thinks of as the node id.  But intuitively, I'd think of that
as the "least significant" boundary, since that's basically the
smallest granularity at which we care about NUMA distances.

quoted

I would interpret it as the boundary where we start defining NUMA
nodes.

That isn't any clearer to me.

How about calling it least significant boundary then?

Heck, no.  My whole point here is that the meaning is unclear: my
first guess at the meaning is different from whoever wrote that text.
We need to come up with a way of describing it that's clearer.

quoted

The “ibm,associativity-reference-points” property contains one or more list of numbers
(domainID index) that represents the 1 based ordinal in the associativity lists of the
least significant boundary, with subsequent entries indicating progressively higher
significant boundaries.

ex:
{ primary domainID index, secondary domainID index, tertiary domainID index.. }

Linux kernel uses the domainID of the least significant boundary (aka primary domain)
as the NUMA node id. Linux kernel computes NUMA distance between two domains by
recursively comparing if they belong to the same higher-level domains. For mismatch
at every higher level of the resource group, the kernel doubles the NUMA distance between
the comparing domains.

Any suggestion on how to reword the above section then? We could say
associativity-reference-points is list of domainID index representing 
increasing hierarchy of resource group. I am not sure that explains it 
any better?

....

quoted

quoted

quoted

For ex: With domainID 0, 4, 5 we could do a 5x5 matrix to represent the
numa distance. Instead ibm,numa-lookup-index-table allows us to present
the same in a 3x3 matrix  distance[index0][index1] is the  distance
between NUMA node 0 and 4 and distance[index0][index2] is the distance
between NUMA node 0 and 5

Right, I get the purpose of it, and I realized I misphrashed my
question.  My point is that in a Form2 world, the *only* thing the
associativity array is used for is to deduce its position in
lookup-index-table.  Once you have have that for each resource, you
have everything you need, yes?

ibm,associativity is used find the domainID/NUMA node id of the
resource.

ibm,lookup-index-table is used compute the distance information between
NUMA nodes using ibm,numa-distance-table.

I get that you need to use lookup-index-table to work out how to
interpret numa-distance-table.  My point is that IIUC once you've done
the lookup in lookup-index-table once for each associativity array
value, the number you get out (which just a compacted version of the
node id) should be all you need ever again.

That is correct. We will continue to use the index to nodeid map during 
DLPAR, if such an operation adds a new numa node. update_numa_distance() 
shows the detail.

-aneesh