On Wed, Oct 18, 2017 at 7:38 AM, Stephen Bates [off-list ref] wrote=
:
Hi All
I am hoping someone can help shed some light on an issue I am seeing with=
my attempt to add p2pmem [1] to the ppc64el kernel. p2pmem is a (currently=
) out-of-tree patchset that allows one to add device memory with struct pag=
e backing into the Linux kernel. It leverages MEMORY_HOTPLUG and ZONE_DEVIC=
E which were added to ppc64 in 4.14 so I thought it would be fun to try it =
out.
We constructed a patchset based off 4.14-rc1 [1] and build a kernel based=
off the pseries defconfig and ran this on upstream qemu-system-ppc64. The =
exact command to run QEMU was:
-device nvme,drive=3Dnvme2,serial=3Dnvme2,cmb_size_mb=3D64
This resulted in the following extract from dmesg when registering the p2=
pmem associated with one of the NVMe SSDs.
[ 3.508497] nvme 0000:00:03.0: enabling device (0100 -> 0102)
[ 3.510743] nvme 0000:00:03.0: Using 64-bit direct DMA at offset 80000=
0000000000
[ 3.535706] p2pmem p2pmem0: registered
[ 3.537780] lpar: Attempting to resize HPT to shift 21
[ 3.539251] Unable to resize hash page table to target order 21: -1
I am guessing that the hotplug of ZONE_DEVICE memory was done
incorrectly as it lead to HPT resizing (the system thinking this is
normal memory). Ideally one would expect that the driver would online
ZONE_DEVICE memory and not go through the HOTPLUG path. Are you using
devm_memremap_pages() path to add these pages?
Balbir Singh.
On Sat, 21 Oct 2017 15:03:29 +0000
"Stephen Bates" [off-list ref] wrote:
quoted
I am guessing that the hotplug of ZONE_DEVICE memory was done
incorrectly as it lead to HPT resizing (the system thinking this is
normal memory). Ideally one would expect that the driver would online
ZONE_DEVICE memory and not go through the HOTPLUG path. Are you using
devm_memremap_pages() path to add these pages?
Thanks for the response Balbir. Yes we use devm_memremap_pages() to add these pages and it does call arch_add_memory(). We do have an alternate set of patches which still calls devm_memremap_pages() but can take a flag to indicate the memory being added is io memory and uses io_remap() rather than arch_add_memory() for that type of memory [1]. Would that be a better approach for this arch? I can try and apply this patch but __add_pages() has gone through some changes recently so it will take me a few days to get to that.
I just double checked, for pmem you do need to come in via arch_add_memory(). I was confused
by what we do for HMM, which is call __add_pages(), but we do need a section mapping so
the interface is correct.
The following
[ 3.537780] lpar: Attempting to resize HPT to shift 21
[ 3.539251] Unable to resize hash page table to target order 21: -1
[ 3.541079] Unable to create mapping for hot added memory 0xc000210000000000..0xc000210004000000: -2
Needs to be debugged further. For #1 above please check if your qemu supports
H_RESIZE_HPT_* hcalls? For create mapping failures, the rc is -ENOENT. Can
you help debug this further? We could do hcall tracing or enable debugging.
Balbir Singh.
From: Oliver <oohall@gmail.com> Date: 2017-10-25 14:34:28
On Tue, Oct 24, 2017 at 7:17 AM, Stephen Bates [off-list ref] wrote=
:
quoted
[ 3.537780] lpar: Attempting to resize HPT to shift 21
[ 3.539251] Unable to resize hash page table to target order 21: -1
[ 3.541079] Unable to create mapping for hot added memory 0xc00021000=
0000000..0xc000210004000000: -2
quoted
For #1 above please check if your qemu supports H_RESIZE_HPT_* hcalls?
Balbir do you have any suggestions as to how to test for this support? No=
te I am running this on my x86_64 host so there is no virtualization hardwa=
re in my QEMU. My qemu is very recent (QEMU emulator version 2.10.50 (v2.10=
.0-1026-gd8f932c-dirty)).
Honestly I'd just ignore the resize error. The hash table stores PTE
entries so it should be sized based on the amount of memory in the
system. If it's drastically under sized there'll be a performance hit,
but everything should still work.
quoted
For create mapping failures, the rc is -ENOENT. Can you help debug this =
further? We could do hcall tracing or enable debugging.
Sure I can help debug. My original email also had all you needed to recre=
ate this issue so that=E2=80=99s an option too?
I'm not too sure what's happening there. My hunch is that the
hypervisor (qemu in this case) is rejecting the attempt to map the PCI
device MMIO space as cachable memory. On bare metal systems this can
result in cache paradoxes which will kill the system so the hypervisor
has an incentive to prevent that situation.
From: Oliver <oohall@gmail.com> Date: 2017-10-27 08:07:10
On Thu, Oct 26, 2017 at 1:34 AM, Oliver [off-list ref] wrote:
On Tue, Oct 24, 2017 at 7:17 AM, Stephen Bates [off-list ref] wro=
te:
quoted
quoted
[ 3.537780] lpar: Attempting to resize HPT to shift 21
[ 3.539251] Unable to resize hash page table to target order 21: -1
[ 3.541079] Unable to create mapping for hot added memory 0xc0002100=
00000000..0xc000210004000000: -2
quoted
quoted
For #1 above please check if your qemu supports H_RESIZE_HPT_* hcalls?
Balbir do you have any suggestions as to how to test for this support? N=
ote I am running this on my x86_64 host so there is no virtualization hardw=
are in my QEMU. My qemu is very recent (QEMU emulator version 2.10.50 (v2.1=
0.0-1026-gd8f932c-dirty)).
Honestly I'd just ignore the resize error. The hash table stores PTE
entries so it should be sized based on the amount of memory in the
system. If it's drastically under sized there'll be a performance hit,
but everything should still work.
quoted
quoted
For create mapping failures, the rc is -ENOENT. Can you help debug this=
further? We could do hcall tracing or enable debugging.
quoted
Sure I can help debug. My original email also had all you needed to recr=
eate this issue so that=E2=80=99s an option too?
I'm not too sure what's happening there. My hunch is that the
hypervisor (qemu in this case) is rejecting the attempt to map the PCI
device MMIO space as cachable memory. On bare metal systems this can
result in cache paradoxes which will kill the system so the hypervisor
has an incentive to prevent that situation.
So I had a deeper look and found the hypervisor interface spec (PAPR)
says the hypervisor should reject attempts to map memory with
inappropriate attributes for the type of memory being mapped. The
pseries model in qemu interprets this by only allowing cacheable
mappings on memory ranges that it considers as RAM. While KVM will
allow any mappings provided they have the same cachable attribute as
the hypervisor's mapping. Either way trying to use
devm_memremap_pages() like this on pseries is fundementally broken.
The alternative approach you mentioned that uses ioremap() should work
fine though.
Also, Alexy (+cc) said he was interested in trying this on some real
hardware. Is there a test suite for p2pmem floating around that he can
use?
Thanks,
Oliver