On Wed, Jan 27, 2021 at 1:50 PM Pavel Tatashin
[off-list ref] wrote:

On Wed, Jan 27, 2021 at 4:09 PM David Hildenbrand [off-list ref] wrote:

quoted

On 27.01.21 21:43, Pavel Tatashin wrote:

quoted

This is something that Dan Williams and I discussed off the mailing
list sometime ago, but I want to have a broader discussion about this
problem so I could send out a fix that would be acceptable.

We have a 2G pmem device that is carved out of regular memory that we
use to pass data across reboots. After the machine is rebooted we

Ordinary reboots or kexec-style reboots? I assume the latter, because
otherwise there is no guarantee about persistence, right?

Both, our firmware supports cold and warm reboot. When we do warm
reboot, memory content is not initialized. However, for performance
reasons, we mostly do kexec reboots.

quoted

I remember for kexec-style reboots there is a different approach (using
tmpfs) on the list.

Right, we are using a similar approach to that tmpfs, but that tmpfs
approach was never upstreamed.

quoted

quoted

hotplug that memory back, so we do not lose 2G of system memory
(machine is small, only 8G of RAM total).

In order to hotplug pmem memory it first must be converted to devdax.
Devdax has a label 2M in size that is placed at the beginning of the
pmem device memory which brings the problem.

The section size is a hotplugging unit on Linux. Whatever gets
hot-plugged or hot-removed must be section size aligned. On x86
section size is 128M on arm64 it is 1G (because arm64 supports 64K
pages, and 128M does not work with 64K pages). Because the first 2M

Note that it's soon 128M with 4k and 16k base pages and 512MB with 64k.
The arm64 patch for that is already queued.

This is great. Do you have a pointer to that series? It means we can
get rid of our special section size workaround patch, and use the 128M
section size for 4K pages. However, we still can't move to 64K because
losing 510M is too much.

quoted

quoted

are subtracted from the pmem device to create devdax, that actual
hot-pluggable memory is not 1G/128M aligned, and instead we lose 126M
on x86 or 1022M on arm64 of memory that is getting hot-plugged, the
whole first section is skipped when memory gets hot plugged because of
2M label.

As a  workaround, so we do not lose 1022M out of 8G of memory on arm64
we have section size reduced to 128M. We are using this patch [1].
This way we are losing 126M (which I still hate!)

I would like to get rid of this workaround. First, because I would
like us to switch to 64K pages to gain performance, and second so we
do not depend on an unofficial patch which already has given us some
headache with kdump support.

I'd want to see 128M sections on arm64 with 64k base pages. "How?" you
might ask. One idea would be to switch from 512M THP to 2MB THP (using
cont pages), and instead implement 512MB gigantic pages. Then we can
reduce pageblock_order / MAX_ORDER - 1 and no longer have the section
limitations. Stuff for the future, though (if even ever).

Interesting, but this is not something that would address the
immediate issue. Because, even losing 126M is something I would like
to fix. However, what other benefits reducing section size on arm64
would bring? Do we have requirement where reducing section size is
actually needed?

quoted

quoted

Here are some solutions that I think we can do:

1. Instead of carving the memory at 1G aligned address, do it at 1G -
2M address, this way when devdax is created it is perfectly 1G
aligned. On ARM64 it causes a panic because there is a 2M hole in
memory. Even if panic is fixed, I do not think this is a proper fix.
This is simply a workaround to the underlying problem.

I remember arm64 already has to deal with all different kinds of memory
holes (including huge ones). I don't think this should be a fundamental
issue.

Perhaps not. I can root cause, and report here what actually happens.

quoted

I think it might be a reasonable thing to do for such a special use
case. Does it work on x86-64?

It does.

quoted

quoted

2.  Dan Williams introduced subsections [2]. They, however do not work
with devdax, and hot-plugging in general. Those patches take care of
__add_pages() side of things, and not add_memory(). Also, it is
unclear what kind of user interface changes need to be made in order
to enable subsection features to online/offline pages.

I am absolutely no fan of teaching add_memory() and friends in general
about sub-sections.

quoted

3. Allow to hot plug daxdev together with the label, but teach the
kernel not to touch label (i.e. allocate its memory). IMO, kind of
ugly solution, because when devdax is hot-plugged it is not even aware
of label size. But, perhaps that can be changed.

I mean, we could teach add_memory() to "skip the first X pages" when
onlining/offlining, not exposing them to the buddy. Something similar we
already do with Oscars vmemmap-on-memory series.

But I guess the issue is that the memmap for the label etc. is already
allocated? Is the label memremapped ZONE_DEVICE memory or what is it? Is
the label exposed in the resource tree?

It is exposed:

# ndctl create-namespace --mode raw -e namespace0.0 -f

Since we last talked about this the enabling for EFI "Special Purpose"
/ Soft Reserved Memory has gone upstream and instantiates device-dax
instances for address ranges marked with EFI_MEMORY_SP attribute.
Critically this way of declaring device-dax removes the consideration
of it as persistent memory and as such no metadata reservation. So, if
you are willing to maintain the metadata external to the device (which
seems reasonable for your environment) and have your platform firmware
/ kernel command line mark it as EFI_CONVENTIONAL_MEMORY +
EFI_MEMORY_SP, then these reserve-free dax-devices will surface.

See efi_fake_mem for how to apply that range to existing
EFI_CONVENTIONAL_MEMORY ranges, it requires CONFIG_EFI_SOFT_RESERVE=y.

The daxctl utility has grown mechanisms to subdivide such ranges.

   daxctl create-device

...starting with v71.

{
  "dev":"namespace0.0",
  "mode":"raw",
  "size":"2.00 GiB (2.15 GB)",
  "sector_size":512,
  "blockdev":"pmem0"
}

The raw device is exactly 2G

# cat /proc/iomem | grep 'dax\|namespace'
980000000-9ffffffff : namespace0.0

namespace0.0 is 2G, and there is dax0.0.

Create devdax device:
# ndctl create-namespace --mode devdax --map mem -e namespace0.0 -f
{
  "dev":"namespace0.0",
  "mode":"devdax",
  "map":"mem",
  "size":"2046.00 MiB (2145.39 MB)",
  "uuid":"ed4d6a34-6a11-4ced-8a4f-b2487bddf5d7",
  "daxregion":{
    "id":0,
    "size":"2046.00 MiB (2145.39 MB)",
    "align":2097152,
    "devices":[
      {
        "chardev":"dax0.0",
        "size":"2046.00 MiB (2145.39 MB)",
        "mode":"devdax"
      }
    ]
  },
  "align":2097152
}

Now, the device is 2046M in size instead of 2G.

root@dplat-cp22:/# cat /proc/iomem | grep 'namespace\|dax'
980000000-9801fffff : namespace0.0
980200000-9ffffffff : dax0.0

We can see the namespace0.0 is 2M, which is label, and dax0.0 is 2046M.

quoted

In case "it's just untouched/unexposed memory", it's fairly simple. In
case the label is exposed as ZONE_DEVICE already, it's more of an issue
and might require further tweaks.

quoted

4. Other ideas? (move dax label to the end? a special case without a
label? label outside of data?)

What does the label include in your example? Sorry, I have no idea about
devdax labels.

I read "ndctl-create-namespace" - "--no-autolabel: Manage labels for
legacy NVDIMMs that do not support labels". So I assume there is at
least some theoretical way to not have a label on the memory?

Right, but I do not think it is possible to do for dax devices (as of
right now). I assume, it contains information about what kind of
device it is: devdax, fsdax, sector, uuid etc.
See [1] namespaces tabel. It contains summary of pmem devices types,
and which of them have label (all except for raw).

[1] https://nvdimm.wiki.kernel.org/

quoted

quoted

Thank you,
Pasha

[1] https://lore.kernel.org/lkml/20190423203843.2898-1-pasha.tatashin@soleen.com (local)
[2] https://lore.kernel.org/lkml/156092349300.979959.17603710711957735135.stgit@dwillia2-desk3.amr.corp.intel.com (local)

--
Thanks,

David / dhildenb