Re: [RFC PATCH 8/8] powerpc/papr_scm: Use FORM2 associativity details

On Thu, Jun 17, 2021 at 04:29:01PM +0530, Aneesh Kumar K.V wrote:

On 6/17/21 1:16 PM, David Gibson wrote:

quoted

On Tue, Jun 15, 2021 at 12:35:17PM +0530, Aneesh Kumar K.V wrote:

quoted

David Gibson [off-list ref] writes:

quoted

On Tue, Jun 15, 2021 at 11:27:50AM +0530, Aneesh Kumar K.V wrote:

quoted

David Gibson [off-list ref] writes:

...

quoted

quoted

quoted

It's weird to me that you'd want to consider them in different nodes
for those different purposes.

    --------------------------------------
   |                            NUMA node0 |
   |    ProcA -----> MEMA                  |
   |     |                                 |
   |	|                                 |
   |	-------------------> PMEMB        |
   |                                       |
    ---------------------------------------

    ---------------------------------------
   |                            NUMA node1 |
   |                                       |
   |    ProcB -------> MEMC                |
   |	|                                 |
   |	-------------------> PMEMD        |
   |                                       |
   |                                       |
    ---------------------------------------

For a topology like the above application running of ProcA wants to find out
persistent memory mount local to its NUMA node. Hence when using it as
pmem fsdax mount or devdax device we want PMEMB to have associativity
of NUMA node0 and PMEMD to have associativity of NUMA node 1. But when
we want to use it as memory using dax kmem driver, we want both PMEMB
and PMEMD to appear as memory only NUMA node at a distance that is
derived based on the latency of the media.

I'm still not understanding why the latency we care about is different
in the two cases.  Can you give an example of when this would result
in different actual node assignments for the two different cases?

In the above example in order allow use of PMEMB and PMEMD as memory only
NUMA nodes
we need platform to represent them in its own domainID. Let's assume that
platform assigned id 40 and 41 and hence both PMEMB and PMEMD will have
associativity array like below

{ 4, 6, 0}  -> PROCA/MEMA
{ 4, 6, 40} -> PMEMB
{ 4, 6, 41} -> PMEMD
{ 4, 6, 1} ->  PROCB/MEMB

When we want to use this device PMEMB and PMEMD as fsdax/devdax devices, we
essentially look for the first nearest online node. Which means both PMEMB
and PMEMD will appear as devices attached to node0. That is not ideal for
for many applications.

Not if you actually look at the distance table which tells you that
PMEMB is closer to node0 and PMEMD is closer to node1.  That's exactly
what the distance table is for - making this information explicit,
rather than intuited from a confusing set of nested domains.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson