On Tue, Jul 16, 2019 at 7:08 PM Aneesh Kumar K.V
[off-list ref] wrote:
This is similar to what ACPI does. Nvdimm layer doesn't bring the SCM device
numa node online. Hence we need to make sure we always use an online node
as ndr_desc.numa_node. Otherwise this result in kernel crashes. The target
node is used by dax/kmem and that will bring up the numa node online correctly.
Without this patch, we do hit kernel crash as below because we try to access
uninitialized NODE_DATA in different code paths.
Right, so we're getting a crash due to libnvdimm (via devm_kmalloc)
trying to to node local allocations to an offline node. Using a
different node fixes that problem, but what else does changing
ndr_desc.numa_node do?
@@ -299,8 +319,11 @@ static int papr_scm_nvdimm_init(struct papr_scm_priv *p)memset(&ndr_desc,0,sizeof(ndr_desc));ndr_desc.attr_groups=region_attr_groups;-ndr_desc.numa_node=dev_to_node(&p->pdev->dev);-ndr_desc.target_node=ndr_desc.numa_node;+target_nid=dev_to_node(&p->pdev->dev);+online_nid=papr_scm_node(target_nid);+set_dev_node(&p->pdev->dev,online_nid);+ndr_desc.numa_node=online_nid;+ndr_desc.target_node=target_nid;
Annoying question: Say I have two SCM regions with slightly different
numa affinities and I online one, then the other. If the first is
setup to be used as kmem could we have the target_node of the first
being used as the online_node of the second? If so, what happens if I
unbind kmem from the first region?
I'm mostly wondering if we need to set online_node to the nearest node
with actual memory on it.
On Tue, Jul 16, 2019 at 7:08 PM Aneesh Kumar K.V
[off-list ref] wrote:
quoted
This is similar to what ACPI does. Nvdimm layer doesn't bring the SCM device
numa node online. Hence we need to make sure we always use an online node
as ndr_desc.numa_node. Otherwise this result in kernel crashes. The target
node is used by dax/kmem and that will bring up the numa node online correctly.
Without this patch, we do hit kernel crash as below because we try to access
uninitialized NODE_DATA in different code paths.
Right, so we're getting a crash due to libnvdimm (via devm_kmalloc)
trying to to node local allocations to an offline node. Using a
different node fixes that problem, but what else does changing
ndr_desc.numa_node do?
@@ -299,8 +319,11 @@ static int papr_scm_nvdimm_init(struct papr_scm_priv *p)memset(&ndr_desc,0,sizeof(ndr_desc));ndr_desc.attr_groups=region_attr_groups;-ndr_desc.numa_node=dev_to_node(&p->pdev->dev);-ndr_desc.target_node=ndr_desc.numa_node;+target_nid=dev_to_node(&p->pdev->dev);+online_nid=papr_scm_node(target_nid);+set_dev_node(&p->pdev->dev,online_nid);+ndr_desc.numa_node=online_nid;+ndr_desc.target_node=target_nid;
Annoying question: Say I have two SCM regions with slightly different
numa affinities and I online one, then the other. If the first is
setup to be used as kmem could we have the target_node of the first
being used as the online_node of the second? If so, what happens if I
unbind kmem from the first region?
kmem, for now, doesn't allow to unregister the mem region. So this won't
happen. But when we start supporting hot unplugging memory added via
kmem, I would expect we continue to keep the SCM region 1 attached to
the node that had kmem before.
I'm mostly wondering if we need to set online_node to the nearest node
with actual memory on it.
@@ -318,6 +341,9 @@ static int papr_scm_nvdimm_init(struct papr_scm_priv *p) ndr_desc.res, p->dn); goto err; }+ if (target_nid != online_nid)+ dev_info(dev, "Region registered with target node %d and online node %d",+ target_nid, online_nid); return 0;
There was a related question that Michael asked w.r.t how do we
define closest NUMA node if we have two NUMA nodes (say `x` and `y`)
with the same distance from the nvdimm region NUMA node. For now, we
pick the first one. But then that also means we lose the information
that nvdimm region access latency is going to be same from both nodes
`x` and `y`.
I guess if application want to find those detail, they will have to use
/sys/devices/system/node/nodeX/accessY/ introduced as part of
08d9dbe72b1f8 ("node: Link memory nodes to their compute nodes")
-aneesh
From: Dan Williams <hidden> Date: 2019-07-24 04:23:40
On Thu, Jul 18, 2019 at 12:49 AM Aneesh Kumar K.V
[off-list ref] wrote:
"Oliver O'Halloran" [off-list ref] writes:
quoted
On Tue, Jul 16, 2019 at 7:08 PM Aneesh Kumar K.V
[off-list ref] wrote:
quoted
This is similar to what ACPI does. Nvdimm layer doesn't bring the SCM device
numa node online. Hence we need to make sure we always use an online node
as ndr_desc.numa_node. Otherwise this result in kernel crashes. The target
node is used by dax/kmem and that will bring up the numa node online correctly.
Without this patch, we do hit kernel crash as below because we try to access
uninitialized NODE_DATA in different code paths.
Right, so we're getting a crash due to libnvdimm (via devm_kmalloc)
trying to to node local allocations to an offline node. Using a
different node fixes that problem, but what else does changing
ndr_desc.numa_node do?
@@ -299,8 +319,11 @@ static int papr_scm_nvdimm_init(struct papr_scm_priv *p)memset(&ndr_desc,0,sizeof(ndr_desc));ndr_desc.attr_groups=region_attr_groups;-ndr_desc.numa_node=dev_to_node(&p->pdev->dev);-ndr_desc.target_node=ndr_desc.numa_node;+target_nid=dev_to_node(&p->pdev->dev);+online_nid=papr_scm_node(target_nid);+set_dev_node(&p->pdev->dev,online_nid);+ndr_desc.numa_node=online_nid;+ndr_desc.target_node=target_nid;
Annoying question: Say I have two SCM regions with slightly different
numa affinities and I online one, then the other. If the first is
setup to be used as kmem could we have the target_node of the first
being used as the online_node of the second? If so, what happens if I
unbind kmem from the first region?
kmem, for now, doesn't allow to unregister the mem region. So this won't
happen. But when we start supporting hot unplugging memory added via
kmem, I would expect we continue to keep the SCM region 1 attached to
the node that had kmem before.
kmem grew unregister support in v5.3-rc1, see these commits:
9f960da72b25 device-dax: "Hotremove" persistent memory that is used
like normal RAM
eca499ab3749 mm/hotplug: make remove_memory() interface usable
31e4ca92a7dd device-dax: fix memory and resource leak if hotplug fails
quoted
I'm mostly wondering if we need to set online_node to the nearest node
with actual memory on it.
@@ -318,6 +341,9 @@ static int papr_scm_nvdimm_init(struct papr_scm_priv *p) ndr_desc.res, p->dn); goto err; }+ if (target_nid != online_nid)+ dev_info(dev, "Region registered with target node %d and online node %d",+ target_nid, online_nid); return 0;
There was a related question that Michael asked w.r.t how do we
define closest NUMA node if we have two NUMA nodes (say `x` and `y`)
with the same distance from the nvdimm region NUMA node. For now, we
pick the first one. But then that also means we lose the information
that nvdimm region access latency is going to be same from both nodes
`x` and `y`.
This 'equidistant' case is a general problem and x86 currently just
happens to pick the lowest id as the value to store in the 'numa_node'
attribute. It impacts system software in practice when performing an
operation like:
numactl --preferred block:pmem0
Ideally numactl would be updated to check for equidistant nodes
relative to the 'numa_node' attribute and return a mask/list of
cpu-initiator nodes rather a single node.
I guess if application want to find those detail, they will have to use
/sys/devices/system/node/nodeX/accessY/ introduced as part of
08d9dbe72b1f8 ("node: Link memory nodes to their compute nodes")
The HMAT only links a single node as the node that houses the memory
controller for the given address range, however if another range /
proximity domain has identical performance characteristics it can be
expressed here:
/sys/devices/system/node/nodeX/accessY/initiators/