RE: [PATCH v3 0/3] create sysfs representation of ACPI HMAT

-----Original Message-----
From: Linux-nvdimm [mailto:linux-nvdimm-bounces@lists.01.org] On Behalf O=

f

Ross Zwisler

...

=20
On Wed, Dec 20, 2017 at 10:19:37AM -0800, Matthew Wilcox wrote:

...

quoted

initiator is a CPU?  I'd have expected you to expose a memory controlle=

r

quoted

abstraction rather than re-use storage terminology.

=20
Yea, I agree that at first blush it seems weird.  It turns out that
looking at it in sort of a storage initiator/target way is beneficial,
though, because it allows us to cut down on the number of data values
we need to represent.
=20
For example the SLIT, which doesn't differentiate between initiator and
target proximity domains (and thus nodes) always represents a system
with N proximity domains using a NxN distance table.  This makes sense
if every node contains both CPUs and memory.
=20
With the introduction of the HMAT, though, we can have memory-only
initiator nodes and we can explicitly associate them with their local=20
CPU.  This is necessary so that we can separate memory with different
performance characteristics (HBM vs normal memory vs persistent memory,
for example) that are all attached to the same CPU.
=20
So, say we now have a system with 4 CPUs, and each of those CPUs has 3
different types of memory attached to it.  We now have 16 total proximity
domains, 4 CPU and 12 memory.

The CPU cores that make up a node can have performance restrictions of
their own; for example, they might max out at 10 GB/s even though the
memory controller supports 120 GB/s (meaning you need to use 12 cores
on the node to fully exercise memory).  It'd be helpful to report this,
so software can decide how many cores to use for bandwidth-intensive work.

If we represent this with the SLIT we end up with a 16 X 16 distance tabl=

e

(256 entries), most of which don't matter because they are memory-to-
memory distances which don't make sense.
=20
In the HMAT, though, we separate out the initiators and the targets and
put them into separate lists.  (See 5.2.27.4 System Locality Latency and
Bandwidth Information Structure in ACPI 6.2 for details.)  So, this same
config in the HMAT only has 4*12=3D48 performance values of each type, al=

l

of which convey meaningful information.
=20
The HMAT indeed even uses the storage "initiator" and "target"
terminology. :)

Centralized DMA engines (e.g., as used by the "DMA based blk-mq pmem
driver") have performance differences too.  A CPU might include
CPU cores that reach 10 GB/s, DMA engines that reach 60 GB/s, and
memory controllers that reach 120 GB/s.  I guess these would be
represented as extra initiators on the node?


---
Robert Elliott, HPE Persistent Memory