From: David Hildenbrand <hidden> Date: 2021-06-07 19:54:51
Hi,
this series aims at improving in-kernel auto-online support. It tackles the
fundamental problems that:
1) We can create zone imbalances when onlining all memory blindly to
ZONE_MOVABLE, in the worst case crashing the system. We have to know
upfront how much memory we are going to hotplug such that we can
safely enable auto-onlining of all hotplugged memory to ZONE_MOVABLE
via "online_movable". This is far from practical and only applicable in
limited setups -- like inside VMs under the RHV/oVirt hypervisor which
will never hotplug more than 3 times the boot memory (and the
limitation is only in place due to the Linux limitation).
2) We see more setups that implement dynamic VM resizing, hot(un)plugging
memory to resize VM memory. In these setups, we might hotplug a lot of
memory, but it might happen in various small steps in both directions
(e.g., 2 GiB -> 8 GiB -> 4 GiB -> 16 GiB ...). virtio-mem is the
primary driver of this upstream right now, performing such dynamic
resizing NUMA-aware via multiple virtio-mem devices.
Onlining all hotplugged memory to ZONE_NORMAL means we basically have
no hotunplug guarantees. Onlining all to ZONE_MOVABLE means we can
easily run into zone imbalances when growing a VM. We want a mixture,
and we want as much memory as reasonable/configured in ZONE_MOVABLE.
3) Memory devices consist of 1..X memory block devices, however, the
kernel doesn't really track the relationship. Consequently, also user
space has no idea. We want to make per-device decisions. As one
example, for memory hotunplug it doesn't make sense to use a mixture of
zones within a single DIMM: we want all MOVABLE if possible, otherwise
all !MOVABLE, because any !MOVABLE part will easily block the DIMM from
getting hotunplugged. As another example, virtio-mem operates on
individual units that span 1..X memory blocks. Similar to a DIMM, we
want a unit to either be all MOVABLE or !MOVABLE. Further, we want
as much memory of a virtio-mem device to be MOVABLE as possible.
4) We want memory onlining to be done right from the kernel while adding
memory; for example, this is reqired for fast memory hotplug for
drivers that add individual memory blocks, like virito-mem. We want a
way to configure a policy in the kernel and avoid implementing advanced
policies in user space.
The auto-onlining support we have in the kernel is not sufficient. All we
have is a) online everything movable (online_movable) b) online everything
!movable (online_kernel) c) keep zones contiguous (online). This series
allows configuring c) to mean instead "online movable if possible according
to the coniguration, driven by a maximum MOVABLE:KERNEL ratio" -- a new
onlining policy.
This series does 3 things:
1) Introduces the "auto-movable" online policy that initially operates on
individual memory blocks only. It uses a maximum MOVABLE:KERNEL ratio
to make a decision whether a memory block will be onlined to
ZONE_MOVABLE or not. However, in the basic form, hotplugged KERNEL
memory does not allow for more MOVABLE memory (details in the
patches). CMA memory is treated like MOVABLE memory.
2) Introduces static (e.g., DIMM) and dynamic (e.g., virtio-mem) memory
groups and uses group information to make decisions in the
"auto-movable" online policy accross memory blocks of a single memory
device (modeled as memory group).
3) Maximizes ZONE_MOVABLE memory within dynamic memory groups, by
allowing ZONE_NORMAL memory within a dynamic memory group to allow for
more ZONE_MOVABLE memory within the same memory group. The target use
case is dynamic VM resizing using virtio-mem.
I remember that the basic idea of using a ratio to implement a policy in
the kernel was once mentioned by Vitaly Kuznetsov, but I might be wrong
(I lost the pointer to that discussion).
For me, the main use case is using it along with virtio-mem (and
DIMMs / ppc64 dlpar where necessary) for dynamic resizing of VMs,
increasing the amount of memory we can hotunplug reliably again if we
might eventually hotplug a lot of memory to a VM.
The target usage will be:
1) Linux boots with "mhp_default_online_type=offline"
2) User space (e.g., systemd unit) configures memory onlining (according
to a config file and system properties), for example:
* Setting memory_hotplug.online_policy=auto-movable
* Setting memory_hotplug.auto_movable_ratio=301
* Setting memory_hotplug.auto_movable_numa_aware=true
3) User space enabled auto onlining via "echo online >
/sys/devices/system/memory/auto_online_blocks"
4) User space triggers manual onlining of all already-offline memory
blocks (go over offline memory blocks and set them to "online")
For DIMMs, hotplugging 4 GiB DIMMs to a 4 GiB VM with a configured ratio of
301% results in the following layout:
Memory block 1-15: DMA32 (early)
Memory block 32-47: Normal (early)
Memory block 48-79: Movable (DIMM 0)
Memory block 80-111: Movable (DIMM 1)
Memory block 112-143: Movable (DIMM 2)
Memory block 144-275: Normal (DIMM 3)
Memory block 176-207: Normal (DIMM 4)
... all Normal
(-> hotplugged Normal memory does not allow for more Movable memory)
For virtio-mem, using a simple, single virtio-mem device with a 4 GiB VM
will result in the following layout:
Memory block 1-15: DMA32 (early)
Memory block 32-47: Normal (early)
Memory block 48-143: Movable (virtio-mem, first 12 GiB)
Memory block 144: Normal (virtio-mem, next 128 MiB)
Memory block 145-147: Movable (virtio-mem, next 384 MiB)
Memory block 148: Normal (virtio-mem, next 128 MiB)
Memory block 149-151: Movable (virtio-mem, next 384 MiB)
... Normal/Movable mixture as above
(-> hotplugged Normal memory allows for more Movable memory within
the same device)
Which gives us maximum flexibility when dynamically growing/shrinking a
VM in smaller steps. When shrinking, virtio-mem will prioritize unplug of
MOVABLE memory with [1] sent last week, such that we won't accidentially
trigger zone imbalances in more complicated setups that involve multiple
virtio-mem devices.
I'll update the memory-hotplug.rst documentation separately, once the
overhaul is done. For now I decided to use module parameters that can be
changed at runtime and not add new sysfs files for configuration. Easier
and cleaner IMHO -- especially, temporarily overwritable via the cmdline.
Future work:
- Use a single static memory group for dax/kmem
- Sense upfront for DIMMs (and dax/kmem) which pieces can actually be added
and won't actually contribute to the static memory group size
- Use memory groups for ppc64 dlpar
- More tunables. For example, a way to configure to keep some memory
default offline (s390x standby memory, dax/kmem)
- Indicate to user space that MOVABLE might be a bad idea -- especially
relevant when memory ballooning without support for balloon compaction
is active.
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Vitaly Kuznetsov <vkuznets@redhat.com>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Jason Wang <redacted>
Cc: Marek Kedzierski <redacted>
Cc: Hui Zhu <redacted>
Cc: Pankaj Gupta <pankaj.gupta.linux@gmail.com>
Cc: Wei Yang <redacted>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Dan Williams <redacted>
Cc: Anshuman Khandual <redacted>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Vlastimil Babka <redacted>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: "Rafael J. Wysocki" <redacted>
Cc: Len Brown <lenb@kernel.org>
Cc: Pavel Tatashin <pasha.tatashin@soleen.com>
Cc: virtualization@lists.linux-foundation.org
Cc: linux-mm@kvack.org
Cc: linux-acpi@vger.kernel.org
[1] https://lkml.kernel.org/r/20210602185720.31821-1-david@redhat.com
David Hildenbrand (12):
mm/memory_hotplug: use "unsigned long" for PFN in zone_for_pfn_range()
mm: track present early pages per zone
mm/memory_hotplug: introduce "auto-movable" online policy
mm/memory_hotplug: remove nid parameter from arch_remove_memory()
mm/memory_hotplug: remove nid parameter from remove_memory() and
friends
drivers/base/memory: "memory groups" to logically group memory blocks
mm/memory_hotplug: track present pages in memory groups
ACPI: memhotplug: memory resources cannot be enabled yet
ACPI: memhotplug: use a single static memory group for a single memory
device
virtio-mem: use a single dynamic memory group for a single virtio-mem
device
mm/memory_hotplug: memory group aware "auto-movable" online policy
mm/memory_hotplug: improved dynamic memory group aware "auto-movable"
online policy
arch/arm64/mm/mmu.c | 3 +-
arch/ia64/mm/init.c | 3 +-
arch/powerpc/mm/mem.c | 3 +-
.../platforms/pseries/hotplug-memory.c | 9 +-
arch/s390/mm/init.c | 3 +-
arch/sh/mm/init.c | 3 +-
arch/x86/mm/init_32.c | 3 +-
arch/x86/mm/init_64.c | 3 +-
drivers/acpi/acpi_memhotplug.c | 46 ++-
drivers/base/memory.c | 163 +++++++--
drivers/dax/kmem.c | 3 +-
drivers/virtio/virtio_mem.c | 26 +-
include/linux/memory.h | 53 ++-
include/linux/memory_hotplug.h | 35 +-
include/linux/mmzone.h | 7 +
mm/memory_hotplug.c | 346 +++++++++++++++++-
mm/memremap.c | 5 +-
mm/page_alloc.c | 3 +
18 files changed, 618 insertions(+), 99 deletions(-)
base-commit: 614124bea77e452aa6df7a8714e8bc820b489922
--
2.31.1
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
From: David Hildenbrand <hidden> Date: 2021-06-07 19:54:55
Checkpatch complained on a follow-up patch that we are using "unsigned"
here, which defaults to "unsigned int" and checkpatch is correct.
Use "unsigned long" instead, just as we do in other places when handling
PFNs. This can bite us once we have physical addresses in the range of
multiple TB.
Fixes: e5e689302633 ("mm, memory_hotplug: display allowed zones in the preferred ordering")
Signed-off-by: David Hildenbrand <redacted>
---
include/linux/memory_hotplug.h | 4 ++--
mm/memory_hotplug.c | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
From: David Hildenbrand <hidden> Date: 2021-06-07 19:54:57
For implementing a new memory onlining policy, which determines when to
online memory blocks to ZONE_MOVABLE semi-automatically, we need the number
of present early (boot) pages -- present pages excluding hotplugged pages.
Let's track these pages per zone.
Pass a page instead of the zone to adjust_present_page_count(), similar
as adjust_managed_page_count() and derive the zone from the page.
It's worth noting that a memory block to be offlined/onlined is either
completely "early" or "not early". add_memory() and friends can only add
complete memory blocks and we only online/offline complete (individual)
memory blocks.
Signed-off-by: David Hildenbrand <redacted>
---
drivers/base/memory.c | 14 +++++++-------
include/linux/memory_hotplug.h | 2 +-
include/linux/mmzone.h | 7 +++++++
mm/memory_hotplug.c | 13 ++++++++++---
mm/page_alloc.c | 3 +++
5 files changed, 28 insertions(+), 11 deletions(-)
@@ -107,7 +107,7 @@ static inline void zone_seqlock_init(struct zone *zone)externintzone_grow_free_lists(structzone*zone,unsignedlongnew_nr_pages);externintzone_grow_waitqueues(structzone*zone,unsignedlongnr_pages);externintadd_one_highpage(structpage*page,intpfn,intbad_ppro);-externvoidadjust_present_page_count(structzone*zone,longnr_pages);+externvoidadjust_present_page_count(structpage*page,longnr_pages);/* VM interface that may be used by firmware interface */externintmhp_init_memmap_on_memory(unsignedlongpfn,unsignedlongnr_pages,structzone*zone);
@@ -850,10 +850,17 @@ struct zone *zone_for_pfn_range(int online_type, int nid,*Thisfunctionshouldonlybecalledbymemory_block_{online,offline},*and{online,offline}_pages.*/-voidadjust_present_page_count(structzone*zone,longnr_pages)+voidadjust_present_page_count(structpage*page,longnr_pages){+structzone*zone=page_zone(page);unsignedlongflags;+/*+*Weonlysupportonlining/offlining/adding/removingofcomplete+*memoryblocks;therefore,eitherallisearlyorhotplugged.+*/+if(early_section(__pfn_to_section(page_to_pfn(page))))+zone->present_early_pages+=nr_pages;zone->present_pages+=nr_pages;pgdat_resize_lock(zone->zone_pgdat,&flags);zone->zone_pgdat->node_present_pages+=nr_pages;
@@ -956,7 +963,7 @@ int __ref online_pages(unsigned long pfn, unsigned long nr_pages, struct zone *z}online_pages_range(pfn,nr_pages);-adjust_present_page_count(zone,nr_pages);+adjust_present_page_count(pfn_to_page(pfn),nr_pages);node_states_set_node(nid,&arg);if(need_zonelists_rebuild)
@@ -1827,7 +1834,7 @@ int __ref offline_pages(unsigned long start_pfn, unsigned long nr_pages)/* removal success */adjust_managed_page_count(pfn_to_page(start_pfn),-nr_pages);-adjust_present_page_count(zone,-nr_pages);+adjust_present_page_count(pfn_to_page(start_pfn),-nr_pages);init_per_zone_wmark_min();
From: David Hildenbrand <hidden> Date: 2021-06-07 19:55:15
When onlining without specifying a zone (using "online" instead of
"online_kernel" or "online_movable"), we currently select a zone such that
existing zones are kept contiguous. This online policy made sense in the
past, where contiguous zones where required.
We'd like to implement smarter policies, however:
* User space has little insight. As one example, it has no idea which
memory blocks logically belong together (e.g., to a DIMM or to a
virtio-mem device).
* Drivers that add memory in separate memory blocks, especially
virtio-mem, want memory to get onlined right from the kernel when
adding.
So we really want to have onlining to differing zones managed in the
kernel, configured by user space.
We see more and more cases where we might eventually hotplug a lot of
memory in the future (e.g., eventually grow a 2 GiB VM to 64 GiB), however:
* Resizing happens dynamically, in smaller steps in both directions
(e.g., 2 GiB -> 8 GiB -> 4 GiB -> 16 GiB ...)
* We still want as much flexibility as possible, especially,
hotunplugging as much memory as possible later.
We can really only use "online_movable" if we know that the amount of
memory we are going to hotplug upfront, and we know that it won't result
in a zone imbalance. So in our example, a 2 GiB VM that could grow to 64
GiB could currently not use "online_movable", and instead,
"online_kernel" would have to be used, resulting in worse (no) memory
hotunplug reliability.
Let's add a new "auto-movable" online policy that considers the current
zone ratios (global, per-node) to determine, whether we a memory block
can be onlined to ZONE_MOVABLE:
MOVABLE : KERNEL
However, internally we'll only consider the following ratio for now:
MOVABLE : KERNEL_EARLY
For now, we don't allow for hotplugged KERNEL memory to allow for more
MOVABLE memory, because there is no coordination accross memory devices.
In follow-up patches, we will allow for more KERNEL memory within a memory
device to allow for more MOVABLE memory within the same memory device --
which only makes sense for special memory device types.
We base our calculation on "present pages", see the code comments for
details. Hotplugged memory will get online to ZONE_MOVABLE if the
configured ratio allows for it. Depending on the setup, this can result
in fragmented zones, which can make compaction slower and dynamic
allocation of gigantic pages when not using CMA less reliable
(... which is already pretty unreliable).
The old policy will be the default and called "contig-zones". In follow-up
patches, our new policy will use additional information, such as memory
groups, to make even smarter decisions across memory blocks.
Configuration:
* memory_hotplug.online_policy is used to switch between both polices and
defaults to "contig-zones".
* memory_hotplug.auto_movable_ratio defines the maximum ratio is in percent
and defaults to "301" -- allowing e.g., most 8 GiB machines to grow to 32
GiB and have all hotplugged memory in ZONE_MOVABLE. The additional
percent accounts for a handful of lost present pages (e.g., firmware
allocations). User space is expected to adjust this ratio when enabling
the new "auto-movable" policy, though.
* memory_hotplug.auto_movable_numa_aware considers numa node stats in
addition to global stats, and defaults to "true".
Note: just like the old policy, the new policy won't take things like
unmovable huge pages or memory ballooning that doesn't support balloon
compaction into account. User space has to configure onlining
accordingly.
Signed-off-by: David Hildenbrand <redacted>
---
mm/memory_hotplug.c | 189 ++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 189 insertions(+)
@@ -52,6 +52,73 @@ module_param(memmap_on_memory, bool, 0444);MODULE_PARM_DESC(memmap_on_memory,"Enable memmap on memory for memory hotplug");#endif+enum{+ONLINE_POLICY_CONTIG_ZONES=0,+ONLINE_POLICY_AUTO_MOVABLE,+};++constchar*online_policy_to_str[]={+[ONLINE_POLICY_CONTIG_ZONES]="contig-zones",+[ONLINE_POLICY_AUTO_MOVABLE]="auto-movable",+};++staticintset_online_policy(constchar*val,conststructkernel_param*kp)+{+intret=sysfs_match_string(online_policy_to_str,val);++if(ret<0)+returnret;+*((int*)kp->arg)=ret;+return0;+}++staticintget_online_policy(char*buffer,conststructkernel_param*kp)+{+returnsprintf(buffer,"%s\n",online_policy_to_str[*((int*)kp->arg)]);+}++/*+*memory_hotplug.online_policy:configureonlinebehaviorwhenonliningwithout+*specifyingazone(MMOP_ONLINE)+*+*"contig-zones":keepzonecontiguous+*"auto-movable":onlinememorytoZONE_MOVABLEiftheconfiguration+*(auto_movable_ratio,auto_movable_numa_aware)allowsforit+*/+staticintonline_policy__read_mostly=ONLINE_POLICY_CONTIG_ZONES;+staticconststructkernel_param_opsonline_policy_ops={+.set=set_online_policy,+.get=get_online_policy,+};+module_param_cb(online_policy,&online_policy_ops,&online_policy,0644);+MODULE_PARM_DESC(online_policy,+"Set the online policy (\"contig-zones\", \"auto-movable\") "+"Default: \"contig-zones\"");++/*+*memory_hotplug.auto_movable_ratio:specifymaximumMOVABLE:KERNELratio+*+*Theratiorepresentanupperlimitandthekernelmightdecidetonot+*onlinesomememorytoZONE_MOVABLE--e.g.,becausehotpluggedKERNELmemory+*doesn'tallowformoreMOVABLEmemory.+*/+staticunsignedintauto_movable_ratio__read_mostly=301;+module_param(auto_movable_ratio,uint,0644);+MODULE_PARM_DESC(auto_movable_ratio,+"Set the maximum ratio of MOVABLE:KERNEL memory in the system "+"in percent for \"auto-movable\" online policy. Default: 301");++/*+*memory_hotplug.auto_movable_numa_aware:considernumanodestats+*/+#ifdef CONFIG_NUMA+staticboolauto_movable_numa_aware__read_mostly=true;+module_param(auto_movable_numa_aware,bool,0644);+MODULE_PARM_DESC(auto_movable_numa_aware,+"Consider numa node stats in addition to global stats in "+"\"auto-movable\" online policy. Default: true");+#endif /* CONFIG_NUMA */+/**online_page_callbackcontainspointertocurrentpageonliningfunction.*Initiallyitisgeneric_online_page().Ifitisrequireditcouldbe
@@ -789,6 +856,59 @@ void __ref move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn,set_zone_contiguous(zone);}+structauto_movable_stats{+unsignedlongkernel_early_pages;+unsignedlongmovable_pages;+};++staticvoidauto_movable_stats_account_zone(structauto_movable_stats*stats,+structzone*zone)+{+if(zone_idx(zone)==ZONE_MOVABLE){+stats->movable_pages+=zone->present_pages;+}else{+/*+*CMApages(neveronhotpluggedmemory)behavelike+*ZONE_MOVABLE.+*/+stats->movable_pages+=zone->cma_pages;+stats->kernel_early_pages+=zone->present_early_pages;+stats->kernel_early_pages-=zone->cma_pages;+}+}++staticboolauto_movable_can_online_movable(intnid,unsignedlongnr_pages)+{+structauto_movable_statsstats={};+unsignedlongkernel_early_pages,movable_pages;+pg_data_t*pgdat=NODE_DATA(nid);+structzone*zone;+inti;++/* Walk all relevant zones and collect MOVABLE vs. KERNEL stats. */+if(nid==NUMA_NO_NODE){+/* TODO: cache values */+for_each_populated_zone(zone)+auto_movable_stats_account_zone(&stats,zone);+}else{+for(i=0;i<MAX_NR_ZONES;i++){+zone=pgdat->node_zones+i;+if(populated_zone(zone))+auto_movable_stats_account_zone(&stats,zone);+}+}++kernel_early_pages=stats.kernel_early_pages;+movable_pages=stats.movable_pages;++/*+*TestifwecouldonlinethegivennumberofpagestoZONE_MOVABLE+*andstillstayintheconfiguredratio.+*/+movable_pages+=nr_pages;+returnmovable_pages<=(auto_movable_ratio*kernel_early_pages)/100;+}+/**Returnsadefaultkernelmemoryzoneforthegivenpfnrange.*Ifnokernelzonecoversthispfnrangeitwillautomaticallygo
@@ -843,6 +1029,9 @@ struct zone *zone_for_pfn_range(int online_type, int nid,if(online_type==MMOP_ONLINE_MOVABLE)return&NODE_DATA(nid)->node_zones[ZONE_MOVABLE];+if(online_policy==ONLINE_POLICY_AUTO_MOVABLE)+returnauto_movable_zone_for_pfn(nid,start_pfn,nr_pages);+returndefault_zone_for_pfn(nid,start_pfn,nr_pages);}
--
2.31.1
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
@@ -1254,8 +1254,7 @@ kernel_physical_mapping_remove(unsigned long start, unsigned long end)remove_pagetable(start,end,true,NULL);}-void__refarch_remove_memory(intnid,u64start,u64size,-structvmem_altmap*altmap)+void__refarch_remove_memory(u64start,u64size,structvmem_altmap*altmap){unsignedlongstart_pfn=start>>PAGE_SHIFT;unsignedlongnr_pages=size>>PAGE_SHIFT;
@@ -140,14 +140,11 @@ static void pageunmap_range(struct dev_pagemap *pgmap, int range_id){structrange*range=&pgmap->ranges[range_id];structpage*first_page;-intnid;/* make sure to access a memmap that was actually initialized */first_page=pfn_to_page(pfn_first(pgmap,range_id));/* pages are dead and unused, undo the arch mapping */-nid=page_to_nid(first_page);-mem_hotplug_begin();remove_pfn_range_from_zone(page_zone(first_page),PHYS_PFN(range->start),PHYS_PFN(range_len(range)));
From: David Hildenbrand <hidden> Date: 2021-06-07 19:55:38
There is only a single user remaining. We can simply try to offline all
online nodes - which is fast, because we usually span pages and can skip
such nodes right away.
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <redacted>
Cc: "Rafael J. Wysocki" <redacted>
Cc: Len Brown <lenb@kernel.org>
Cc: Dan Williams <redacted>
Cc: Vishal Verma <vishal.l.verma@intel.com>
Cc: Dave Jiang <dave.jiang@intel.com>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Jason Wang <redacted>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Nathan Lynch <redacted>
Cc: Laurent Dufour <redacted>
Cc: "Aneesh Kumar K.V" <redacted>
Cc: Scott Cheloha <redacted>
Cc: Anton Blanchard <redacted>
Cc: linuxppc-dev@lists.ozlabs.org
Cc: linux-acpi@vger.kernel.org
Cc: nvdimm@lists.linux.dev
Signed-off-by: David Hildenbrand <redacted>
---
.../platforms/pseries/hotplug-memory.c | 9 ++++-----
drivers/acpi/acpi_memhotplug.c | 7 +------
drivers/dax/kmem.c | 3 +--
drivers/virtio/virtio_mem.c | 4 ++--
include/linux/memory_hotplug.h | 10 +++++-----
mm/memory_hotplug.c | 20 +++++++++----------
6 files changed, 23 insertions(+), 30 deletions(-)
@@ -286,7 +286,7 @@ static int pseries_remove_memblock(unsigned long base, unsigned long memblock_si{unsignedlongblock_sz,start_pfn;intsections_per_block;-inti,nid;+inti;start_pfn=base>>PAGE_SHIFT;
@@ -297,10 +297,9 @@ static int pseries_remove_memblock(unsigned long base, unsigned long memblock_siblock_sz=pseries_memory_block_size();sections_per_block=block_sz/MIN_MEMORY_BLOCK_SIZE;-nid=memory_add_physaddr_to_nid(base);for(i=0;i<sections_per_block;i++){-__remove_memory(nid,base,MIN_MEMORY_BLOCK_SIZE);+__remove_memory(base,MIN_MEMORY_BLOCK_SIZE);base+=MIN_MEMORY_BLOCK_SIZE;}
@@ -386,7 +385,7 @@ static int dlpar_remove_lmb(struct drmem_lmb *lmb)block_sz=pseries_memory_block_size();-__remove_memory(mem_block->nid,lmb->base_addr,block_sz);+__remove_memory(lmb->base_addr,block_sz);put_device(&mem_block->dev);/* Update memory regions for memory remove */
@@ -638,7 +637,7 @@ static int dlpar_add_lmb(struct drmem_lmb *lmb)rc=dlpar_online_lmb(lmb);if(rc){-__remove_memory(nid,lmb->base_addr,block_sz);+__remove_memory(lmb->base_addr,block_sz);invalidate_lmb_associativity_index(lmb);}else{lmb->flags|=DRCONF_MEM_ASSIGNED;
From: David Hildenbrand <hidden> Date: 2021-06-07 19:55:44
In our "auto-movable" memory onlining policy, we want to make decisions
across memory blocks of a single memory device. Examples of memory devices
include ACPI memory devices (in the simplest case a single DIMM) a
virtio-mem. For now, we don't have a connection between a single memory
block device and the real memory device. Each memory device consists of
1..X memory block devices.
Let's logically group memory blocks belonging to the same memory device
in "memory groups". Memory groups can span multiple physical ranges and a
memory group itself does not contain any information regarding physical
ranges, only properties (e.g., "max_pages") necessary for improved memory
onlining.
Introduce two memory group types:
1) Static memory group: E.g., a single ACPI memory device, consisting of
1..X memory resources. A memory group consists of 1..Y memory blocks.
The whole group is added/removed in one go. If any part cannot get
offlined, the whole group cannot be removed.
2) Dynamic memory group: E.g., a single virtio-mem device. Memory is
dynamically added/removed in a fixed granularity, called a "unit",
consisting of 1..X memory blocks. A unit is added/removed in one go.
If any part of a unit cannot get offlined, the whole unit cannot be
removed.
In case of 1) we usually want either all memory managed by ZONE_MOVABLE
or none. In case of 2) we usually want to have as many units as possible
managed by ZONE_MOVABLE. We want a single unit to be of the same type.
For now, memory groups are an internal concept that is not exposed to
user space; we might want to change that in the future, though.
add_memory() users can specify a mgid instead of a nid when passing
the MHP_NID_IS_MGID flag.
Signed-off-by: David Hildenbrand <redacted>
---
drivers/base/memory.c | 102 +++++++++++++++++++++++++++++++--
include/linux/memory.h | 46 ++++++++++++++-
include/linux/memory_hotplug.h | 6 +-
mm/memory_hotplug.c | 11 +++-
4 files changed, 158 insertions(+), 7 deletions(-)
@@ -634,7 +639,8 @@ int register_memory(struct memory_block *memory)}staticintinit_memory_block(unsignedlongblock_id,unsignedlongstate,-unsignedlongnr_vmemmap_pages)+unsignedlongnr_vmemmap_pages,+structmemory_group*group){structmemory_block*mem;intret=0;
@@ -653,6 +659,11 @@ static int init_memory_block(unsigned long block_id, unsigned long state,mem->nid=NUMA_NO_NODE;mem->nr_vmemmap_pages=nr_vmemmap_pages;+if(group){+mem->group=group;+refcount_inc(&group->refcount);+}+ret=register_memory(mem);returnret;
@@ -671,7 +682,7 @@ static int add_memory_block(unsigned long base_section_nr)if(section_count==0)return0;returninit_memory_block(memory_block_id(base_section_nr),-MEM_ONLINE,0);+MEM_ONLINE,0,NULL);}staticvoidunregister_memory(structmemory_block*memory)
@@ -681,6 +692,11 @@ static void unregister_memory(struct memory_block *memory)WARN_ON(xa_erase(&memory_blocks,memory->dev.id)==NULL);+if(memory->group){+refcount_dec(&memory->group->refcount);+memory->group=NULL;+}+/* drop the ref. we got via find_memory_block() */put_device(&memory->dev);device_unregister(&memory->dev);
@@ -707,7 +724,8 @@ int create_memory_block_devices(unsigned long start, unsigned long size,return-EINVAL;for(block_id=start_block_id;block_id!=end_block_id;block_id++){-ret=init_memory_block(block_id,MEM_OFFLINE,vmemmap_pages);+ret=init_memory_block(block_id,MEM_OFFLINE,vmemmap_pages,+group);if(ret)break;}
@@ -891,3 +909,79 @@ int for_each_memory_block(void *arg, walk_memory_blocks_func_t func)returnbus_for_each_dev(&memory_subsys,NULL,&cb_data,for_each_memory_block_cb);}++staticintregister_memory_group(structmemory_groupgroup)+{+structmemory_group*new_group;+uint32_tmgid;+intret;++if(!node_possible(group.nid))+return-EINVAL;++new_group=kzalloc(sizeof(group),GFP_KERNEL);+if(!new_group)+return-ENOMEM;+*new_group=group;+refcount_set(&new_group->refcount,1);++ret=xa_alloc(&memory_groups,&mgid,new_group,xa_limit_31b,+GFP_KERNEL);+if(ret)+kfree(new_group);+returnret?ret:mgid;+}++intregister_static_memory_group(intnid,unsignedlongmax_pages)+{+structmemory_groupgroup={+.nid=nid,+.s={+.max_pages=max_pages,+},+};++if(!max_pages)+return-EINVAL;+returnregister_memory_group(group);+}+EXPORT_SYMBOL_GPL(register_static_memory_group);++intregister_dynamic_memory_group(intnid,unsignedlongunit_pages)+{+structmemory_groupgroup={+.nid=nid,+.is_dynamic=true,+.d={+.unit_pages=unit_pages,+},+};++if(!unit_pages||!is_power_of_2(unit_pages)||+unit_pages<PHYS_PFN(memory_block_size_bytes()))+return-EINVAL;+returnregister_memory_group(group);+}+EXPORT_SYMBOL_GPL(register_dynamic_memory_group);++intunregister_memory_group(intmgid)+{+structmemory_group*group;++if(mgid<0)+return-EINVAL;++group=xa_load(&memory_groups,mgid);+if(!group||refcount_read(&group->refcount)>1)+return-EINVAL;++xa_erase(&memory_groups,mgid);+kfree(group);+return0;+}+EXPORT_SYMBOL_GPL(unregister_memory_group);++structmemory_group*get_memory_group(intmgid)+{+returnxa_load(&memory_groups,mgid);+}
@@ -23,6 +23,42 @@#define MIN_MEMORY_BLOCK_SIZE (1UL << SECTION_SIZE_BITS)+structmemory_group{+/* Nid the whole group belongs to. */+intnid;+/* References from memory blocks + 1. */+refcount_trefcount;+/*+*Memorygrouptype:staticvs.dynamic.+*+*Static:Allmemoryinthegroupbelongstoasingleunit,suchas,+*aDIMM.Allmemorybelongingtothegroupwillbeaddedin+*onegoandremovedinonego--it'sstatic.+*+*Dynamic:Memorywithinthegroupisadded/removeddynamicallyin+*unitsofthespecifiedgranularityofatleastonememoryblock.+*/+boolis_dynamic;++union{+struct{+/*+*Maximumnumberofpageswe'llhaveinthisstatic+*memorygroup.+*/+unsignedlongmax_pages;+}s;+struct{+/*+*Unitinpagesinwhichmemoryisadded/removedin+*thisdynamicmemorygroup.Thisgranularitydefines+*thealignmentofaunitinphysicaladdressspace.+*/+unsignedlongunit_pages;+}d;+};+};+structmemory_block{unsignedlongstart_section_nr;unsignedlongstate;/* serialized by the dev->lock */
@@ -34,6 +70,7 @@ struct memory_block {*layatthebeginningofthememoryblock.*/unsignedlongnr_vmemmap_pages;+structmemory_group*group;/* group (if any) for this block */};intarch_get_memory_phys_device(unsignedlongstart_pfn);
@@ -86,7 +123,8 @@ static inline int memory_notify(unsigned long val, void *v)externintregister_memory_notifier(structnotifier_block*nb);externvoidunregister_memory_notifier(structnotifier_block*nb);intcreate_memory_block_devices(unsignedlongstart,unsignedlongsize,-unsignedlongvmemmap_pages);+unsignedlongvmemmap_pages,+structmemory_group*group);voidremove_memory_block_devices(unsignedlongstart,unsignedlongsize);externvoidmemory_dev_init(void);externintmemory_notify(unsignedlongval,void*v);
@@ -1395,6 +1396,13 @@ int __ref add_memory_resource(int nid, struct resource *res, mhp_t mhp_flags)if(ret)returnret;+if(mhp_flags&MHP_NID_IS_MGID){+group=get_memory_group(nid);+if(!group)+return-EINVAL;+nid=group->nid;+}+if(!node_possible(nid)){WARN(1,"node %d was absent from the node_possible_map\n",nid);return-EINVAL;
@@ -1429,7 +1437,8 @@ int __ref add_memory_resource(int nid, struct resource *res, mhp_t mhp_flags)gotoerror;/* create memory block devices after memory was added */-ret=create_memory_block_devices(start,size,mhp_altmap.alloc);+ret=create_memory_block_devices(start,size,mhp_altmap.alloc,+group);if(ret){arch_remove_memory(start,size,NULL);gotoerror;
--
2.31.1
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
From: David Hildenbrand <hidden> Date: 2021-06-07 19:55:46
Let's track all present pages in each memory group. Especially, track
memory present in ZONE_MOVABLE and memory present in one of the kernel
zones (which really only is ZONE_NORMAL right now as memory groups only
apply to hotplugged memory) separate;y within a memory group, to prepare
for making smart auto-online decision for individualmemory blocks within a
memory group based on group statistics.
Signed-off-by: David Hildenbrand <redacted>
---
drivers/base/memory.c | 10 +++++-----
include/linux/memory.h | 4 ++++
include/linux/memory_hotplug.h | 13 +++++++++----
mm/memory_hotplug.c | 19 ++++++++++++++-----
4 files changed, 32 insertions(+), 14 deletions(-)
@@ -111,13 +112,15 @@ static inline void zone_seqlock_init(struct zone *zone)externintzone_grow_free_lists(structzone*zone,unsignedlongnew_nr_pages);externintzone_grow_waitqueues(structzone*zone,unsignedlongnr_pages);externintadd_one_highpage(structpage*page,intpfn,intbad_ppro);-externvoidadjust_present_page_count(structpage*page,longnr_pages);+externvoidadjust_present_page_count(structpage*page,+structmemory_group*group,+longnr_pages);/* VM interface that may be used by firmware interface */externintmhp_init_memmap_on_memory(unsignedlongpfn,unsignedlongnr_pages,structzone*zone);externvoidmhp_deinit_memmap_on_memory(unsignedlongpfn,unsignedlongnr_pages);externintonline_pages(unsignedlongpfn,unsignedlongnr_pages,-structzone*zone);+structzone*zone,structmemory_group*group);externstructzone*test_pages_in_a_zone(unsignedlongstart_pfn,unsignedlongend_pfn);externvoid__offline_isolated_pages(unsignedlongstart_pfn,
@@ -1039,9 +1039,11 @@ struct zone *zone_for_pfn_range(int online_type, int nid,*Thisfunctionshouldonlybecalledbymemory_block_{online,offline},*and{online,offline}_pages.*/-voidadjust_present_page_count(structpage*page,longnr_pages)+voidadjust_present_page_count(structpage*page,structmemory_group*group,+longnr_pages){structzone*zone=page_zone(page);+constboolmovable=zone_idx(zone)==ZONE_MOVABLE;unsignedlongflags;/*
@@ -1054,6 +1056,11 @@ void adjust_present_page_count(struct page *page, long nr_pages)pgdat_resize_lock(zone->zone_pgdat,&flags);zone->zone_pgdat->node_present_pages+=nr_pages;pgdat_resize_unlock(zone->zone_pgdat,&flags);++if(group&&movable)+group->present_movable_pages+=nr_pages;+elseif(group&&!movable)+group->present_kernel_pages+=nr_pages;}intmhp_init_memmap_on_memory(unsignedlongpfn,unsignedlongnr_pages,
@@ -1099,7 +1106,8 @@ void mhp_deinit_memmap_on_memory(unsigned long pfn, unsigned long nr_pages)kasan_remove_zero_shadow(__va(PFN_PHYS(pfn)),PFN_PHYS(nr_pages));}-int__refonline_pages(unsignedlongpfn,unsignedlongnr_pages,structzone*zone)+int__refonline_pages(unsignedlongpfn,unsignedlongnr_pages,+structzone*zone,structmemory_group*group){unsignedlongflags;intneed_zonelists_rebuild=0;
@@ -1152,7 +1160,7 @@ int __ref online_pages(unsigned long pfn, unsigned long nr_pages, struct zone *z}online_pages_range(pfn,nr_pages);-adjust_present_page_count(pfn_to_page(pfn),nr_pages);+adjust_present_page_count(pfn_to_page(pfn),group,nr_pages);node_states_set_node(nid,&arg);if(need_zonelists_rebuild)
@@ -1896,7 +1904,8 @@ static int count_system_ram_pages_cb(unsigned long start_pfn,return0;}-int__refoffline_pages(unsignedlongstart_pfn,unsignedlongnr_pages)+int__refoffline_pages(unsignedlongstart_pfn,unsignedlongnr_pages,+structmemory_group*group){constunsignedlongend_pfn=start_pfn+nr_pages;unsignedlongpfn,system_ram_pages=0;
@@ -2032,7 +2041,7 @@ int __ref offline_pages(unsigned long start_pfn, unsigned long nr_pages)/* removal success */adjust_managed_page_count(pfn_to_page(start_pfn),-nr_pages);-adjust_present_page_count(pfn_to_page(start_pfn),-nr_pages);+adjust_present_page_count(pfn_to_page(start_pfn),group,-nr_pages);init_per_zone_wmark_min();
--
2.31.1
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
From: David Hildenbrand <hidden> Date: 2021-06-07 19:55:54
We allocate + initialize everything from scratch. In case enabling the
device fails, we free all memory resourcs.
Signed-off-by: David Hildenbrand <redacted>
---
drivers/acpi/acpi_memhotplug.c | 4 ----
1 file changed, 4 deletions(-)
From: David Hildenbrand <hidden> Date: 2021-06-07 19:55:57
Let's group all memory we add for a single memory device - we want a
single node for that (which also seems to be the sane thing to do).
We won't care for now about memory that was already added to the system
(e.g., via e820) -- usually *all* memory of a memory device was already
added and we'll fail acpi_memory_enable_device().
Signed-off-by: David Hildenbrand <redacted>
---
drivers/acpi/acpi_memhotplug.c | 35 +++++++++++++++++++++++++++++-----
1 file changed, 30 insertions(+), 5 deletions(-)
@@ -171,10 +172,31 @@ static int acpi_memory_enable_device(struct acpi_memory_device *mem_device)acpi_handlehandle=mem_device->device->handle;intresult,num_enabled=0;structacpi_memory_info*info;-mhp_tmhp_flags=MHP_NONE;-intnode;+mhp_tmhp_flags=MHP_NID_IS_MGID;+u64total_length=0;+intnode,mgid;node=acpi_get_node(handle);++list_for_each_entry(info,&mem_device->res_list,list){+if(!info->length)+continue;+/* We want a single node for the whole memory group */+if(node<0)+node=memory_add_physaddr_to_nid(info->start_addr);+total_length+=info->length;+}++if(!total_length){+dev_err(&mem_device->device->dev,"device is empty\n");+return-EINVAL;+}++mgid=register_static_memory_group(node,PFN_UP(total_length));+if(mgid<0)+returnmgid;+mem_device->mgid=mgid;+/**TelltheVMthereismorememoryhere...*Note:Assumethatthisfunctionreturnszeroonsuccess
@@ -188,12 +210,10 @@ static int acpi_memory_enable_device(struct acpi_memory_device *mem_device)*/if(!info->length)continue;-if(node<0)-node=memory_add_physaddr_to_nid(info->start_addr);if(mhp_supports_memmap_on_memory(info->length))mhp_flags|=MHP_MEMMAP_ON_MEMORY;-result=__add_memory(node,info->start_addr,info->length,+result=__add_memory(mgid,info->start_addr,info->length,mhp_flags);/*
@@ -253,6 +273,10 @@ static void acpi_memory_device_free(struct acpi_memory_device *mem_device)if(!mem_device)return;+/* In case we succeeded adding *some* memory, unregistering fails. */+if(mem_device->mgid>=0)+unregister_memory_group(mem_device->mgid);+acpi_memory_free_device_resources(mem_device);mem_device->device->driver_data=NULL;kfree(mem_device);
@@ -273,6 +297,7 @@ static int acpi_memory_device_add(struct acpi_device *device,INIT_LIST_HEAD(&mem_device->res_list);mem_device->device=device;+mem_device->mgid=-1;sprintf(acpi_device_name(device),"%s",ACPI_MEMORY_DEVICE_NAME);sprintf(acpi_device_class(device),"%s",ACPI_MEMORY_DEVICE_CLASS);device->driver_data=mem_device;
--
2.31.1
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
@@ -2584,6 +2587,16 @@ static int virtio_mem_probe(struct virtio_device *vdev)if(rc)gotoout_del_vq;+/* use a single dynamic memory group to cover the whole memory device */+if(vm->in_sbm)+unit_pages=PHYS_PFN(memory_block_size_bytes());+else+unit_pages=PHYS_PFN(vm->bbm.bb_size);+rc=register_dynamic_memory_group(vm->nid,unit_pages);+if(rc<0)+gotoout_del_resource;+vm->mgid=rc;+/**Ifwestillhavememoryplugged,wehavetounplugallmemoryfirst.*Registeringourparentresourcemakessurethatthismemoryisn't
@@ -2598,7 +2611,7 @@ static int virtio_mem_probe(struct virtio_device *vdev)vm->memory_notifier.notifier_call=virtio_mem_memory_notifier_cb;rc=register_memory_notifier(&vm->memory_notifier);if(rc)-gotoout_del_resource;+gotoout_unreg_group;rc=register_virtio_mem_device(vm);if(rc)gotoout_unreg_mem;
@@ -2612,6 +2625,8 @@ static int virtio_mem_probe(struct virtio_device *vdev)return0;out_unreg_mem:unregister_memory_notifier(&vm->memory_notifier);+out_unreg_group:+unregister_memory_group(vm->mgid);out_del_resource:virtio_mem_delete_resource(vm);out_del_vq:
@@ -2676,6 +2691,7 @@ static void virtio_mem_remove(struct virtio_device *vdev)}else{virtio_mem_delete_resource(vm);kfree_const(vm->resource_name);+unregister_memory_group(vm->mgid);}/* remove all tracking data - no locking needed */
--
2.31.1
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
From: David Hildenbrand <hidden> Date: 2021-06-07 19:56:13
Use memory groups to improve our "auto-movable" onlining policy:
1. For static memory groups (e.g., a DIMM), online a memory block MOVABLE
only if all other memory blocks in the group are either MOVABLE or could
be onlined MOVABLE. A DIMM will either be MOVABLE or not, not a mixture.
2. For dynamic memory groups (e.g., a virtio-mem device), online a
memory block MOVABLE only if all other memory blocks inside the
current unit are either MOVABLE or could be onlined MOVABLE. For a
virtio-mem device with a device block size with 512 MiB, all 128 MiB
memory blocks wihin a 512 MiB unit will either be MOVABLE or not, not
a mixture.
We have to pass the memory group to zone_for_pfn_range() to take the
memory group into account.
Note: for now, there seems to be no compelling reason to make this
behavior configurable.
Signed-off-by: David Hildenbrand <redacted>
---
drivers/base/memory.c | 18 +++++++------
include/linux/memory_hotplug.h | 3 ++-
mm/memory_hotplug.c | 48 +++++++++++++++++++++++++++++++---
3 files changed, 57 insertions(+), 12 deletions(-)
@@ -976,12 +976,53 @@ static struct zone *default_kernel_zone_for_pfn(int nid, unsigned long start_pfn*"present pages"isanupperlimitthatcangetreachedatruntime.As*webaseourcalculationsonKERNEL_EARLY,thisisnotanissue.*/-staticstructzone*auto_movable_zone_for_pfn(intnid,unsignedlongpfn,+staticstructzone*auto_movable_zone_for_pfn(intnid,+structmemory_group*group,+unsignedlongpfn,unsignedlongnr_pages){+unsignedlongonline_pages=0,max_pages,end_pfn;+structpage*page;+if(!auto_movable_ratio)gotokernel_zone;+if(group&&!group->is_dynamic){+max_pages=group->s.max_pages;+online_pages=group->present_movable_pages;++/* If anything is !MOVABLE online the rest !MOVABLE. */+if(group->present_kernel_pages)+gotokernel_zone;+}elseif(!group||group->d.unit_pages==nr_pages){+max_pages=nr_pages;+}else{+max_pages=group->d.unit_pages;+/*+*Takealookatallonlinesectionsinthecurrentunit.+*Wecansafelyassumethatallpageswithinasectionbelong+*tothesamezone,becausedynamicmemorygroupsonlydeal+*withhotpluggedmemory.+*/+pfn=ALIGN_DOWN(pfn,group->d.unit_pages);+end_pfn=pfn+group->d.unit_pages;+for(;pfn<end_pfn;pfn+=PAGES_PER_SECTION){+page=pfn_to_online_page(pfn);+if(!page)+continue;+/* If anything is !MOVABLE online the rest !MOVABLE. */+if(page_zonenum(page)!=ZONE_MOVABLE)+gotokernel_zone;+online_pages+=PAGES_PER_SECTION;+}+}++/*+*OnlineMOVABLEifwecould*currently*onlineallremainingparts+*MOVABLE.Weexpectto(add+)onlinethemimmediatelynext,soif+*nobodyinterferes,allwillbeMOVABLEifpossible.+*/+nr_pages=max_pages-online_pages;if(!auto_movable_can_online_movable(NUMA_NO_NODE,nr_pages))gotokernel_zone;
@@ -1021,7 +1062,8 @@ static inline struct zone *default_zone_for_pfn(int nid, unsigned long start_pfn}structzone*zone_for_pfn_range(intonline_type,intnid,-unsignedlongstart_pfn,unsignedlongnr_pages)+structmemory_group*group,unsignedlongstart_pfn,+unsignedlongnr_pages){if(online_type==MMOP_ONLINE_KERNEL)returndefault_kernel_zone_for_pfn(nid,start_pfn,nr_pages);
@@ -1030,7 +1072,7 @@ struct zone *zone_for_pfn_range(int online_type, int nid,return&NODE_DATA(nid)->node_zones[ZONE_MOVABLE];if(online_policy==ONLINE_POLICY_AUTO_MOVABLE)-returnauto_movable_zone_for_pfn(nid,start_pfn,nr_pages);+returnauto_movable_zone_for_pfn(nid,group,start_pfn,nr_pages);returndefault_zone_for_pfn(nid,start_pfn,nr_pages);}
--
2.31.1
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
From: David Hildenbrand <hidden> Date: 2021-06-07 19:56:25
Currently, the "auto-movable" online policy does not allow for hotplugged
KERNEL (ZONE_NORMAL) memory to increase the amount of MOVABLE memory we can
have, primarily, because there is no coordiantion across memory devices and
we don't want to create zone-imbalances accidentially when unplugging
memory.
However, within a single memory device it's different. Let's allow for
KERNEL memory within a dynamic memory group to allow for more MOVABLE
within the same memory group. The only thing we have to take care of is
that the managing driver avoids zone imbalances by unplugging MOVABLE
memory first, otherwise there can be corner cases where unplug of memory
could result in (accidential) zone imbalances.
virtio-mem is the only user of dynamic memory groups and recently added
support for prioritizing unplug of ZONE_MOVABLE over ZONE_NORMAL, so we
don't need a new toggle to enable it for dynamic memory groups.
We limit this handling to dynamic memory groups, because:
* We want to keep the runtime overhead for collecting stats when onlining
a single memory block small. We tend to have only a handful of dynamic
memory groups, but we can have quite some static memory groups (e.g., 256
DIMMs).
* It doesn't make too much sense for static memory groups, as we try
onlining all applicable memory blocks either completely to ZONE_MOVABLE
or not. In ordinary operation, we won't have a mixture of zones
within a static memory group.
When adding memory to a dynamic memory group, we'll first online memory to
ZONE_MOVABLE as long as early KERNEL memory allows for it. Then, we'll
online the next unit(s) to ZONE_NORMAL, until we can online the next
unit(s) to ZONE_MOVABLE.
For a simple virtio-mem device with a MOVABLE:KERNEL ratio of 3:1, it
will result in a layout like:
[M][M][M][M][M][M][M][M][N][M][M][M][N][M][M][M]...
^ movable memory due to early kernel memory
^ allows for more movable memory ...
^-----^ ... here
^ allows for more movable memory ...
^-----^ ... here
While the created layout is sub-optimal when it comes to contiguous zones,
it gives us the maximum flexibility when dynamically growing/shrinking a
device; we can grow small VMs really big in small steps, and still
shrink reliably to e.g., 1/4 of the maximum VM size in this example,
removing full memory blocks along with meta data more reliably.
Mark dynamic memory groups in the xarray such that we can efficiently
iterate over them when collecting stats. In usual setups, we have one
virtio-mem device per NUMA node, and usually only a small number of NUMA
nodes.
Note: for now, there seems to be no compelling reason to make this
behavior configurable.
Signed-off-by: David Hildenbrand <redacted>
---
drivers/base/memory.c | 25 ++++++++++++++++++
include/linux/memory.h | 3 +++
mm/memory_hotplug.c | 60 +++++++++++++++++++++++++++++++++++++++---
3 files changed, 84 insertions(+), 4 deletions(-)
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2021-06-08 11:12:07
David Hildenbrand [off-list ref] writes:
There is only a single user remaining. We can simply try to offline all
online nodes - which is fast, because we usually span pages and can skip
such nodes right away.
That makes me slightly nervous, because our big powerpc boxes tend to
trip on these scaling issues before others.
But the spanned pages check is just:
void try_offline_node(int nid)
{
pg_data_t *pgdat = NODE_DATA(nid);
...
if (pgdat->node_spanned_pages)
return;
So I guess that's pretty cheap, and it's only O(nodes), which should
never get that big.
quoted hunk
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <redacted>
Cc: "Rafael J. Wysocki" <redacted>
Cc: Len Brown <lenb@kernel.org>
Cc: Dan Williams <redacted>
Cc: Vishal Verma <vishal.l.verma@intel.com>
Cc: Dave Jiang <dave.jiang@intel.com>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Jason Wang <redacted>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Nathan Lynch <redacted>
Cc: Laurent Dufour <redacted>
Cc: "Aneesh Kumar K.V" <redacted>
Cc: Scott Cheloha <redacted>
Cc: Anton Blanchard <redacted>
Cc: linuxppc-dev@lists.ozlabs.org
Cc: linux-acpi@vger.kernel.org
Cc: nvdimm@lists.linux.dev
Signed-off-by: David Hildenbrand <redacted>
---
.../platforms/pseries/hotplug-memory.c | 9 ++++-----
drivers/acpi/acpi_memhotplug.c | 7 +------
drivers/dax/kmem.c | 3 +--
drivers/virtio/virtio_mem.c | 4 ++--
include/linux/memory_hotplug.h | 10 +++++-----
mm/memory_hotplug.c | 20 +++++++++----------
6 files changed, 23 insertions(+), 30 deletions(-)
@@ -286,7 +286,7 @@ static int pseries_remove_memblock(unsigned long base, unsigned long memblock_si{unsignedlongblock_sz,start_pfn;intsections_per_block;-inti,nid;+inti;start_pfn=base>>PAGE_SHIFT;
@@ -297,10 +297,9 @@ static int pseries_remove_memblock(unsigned long base, unsigned long memblock_siblock_sz=pseries_memory_block_size();sections_per_block=block_sz/MIN_MEMORY_BLOCK_SIZE;-nid=memory_add_physaddr_to_nid(base);for(i=0;i<sections_per_block;i++){-__remove_memory(nid,base,MIN_MEMORY_BLOCK_SIZE);+__remove_memory(base,MIN_MEMORY_BLOCK_SIZE);base+=MIN_MEMORY_BLOCK_SIZE;}
@@ -386,7 +385,7 @@ static int dlpar_remove_lmb(struct drmem_lmb *lmb)block_sz=pseries_memory_block_size();-__remove_memory(mem_block->nid,lmb->base_addr,block_sz);+__remove_memory(lmb->base_addr,block_sz);put_device(&mem_block->dev);/* Update memory regions for memory remove */
@@ -638,7 +637,7 @@ static int dlpar_add_lmb(struct drmem_lmb *lmb)rc=dlpar_online_lmb(lmb);if(rc){-__remove_memory(nid,lmb->base_addr,block_sz);+__remove_memory(lmb->base_addr,block_sz);invalidate_lmb_associativity_index(lmb);}else{lmb->flags|=DRCONF_MEM_ASSIGNED;
From: David Hildenbrand <hidden> Date: 2021-06-08 11:19:04
On 08.06.21 13:11, Michael Ellerman wrote:
David Hildenbrand [off-list ref] writes:
quoted
There is only a single user remaining. We can simply try to offline all
online nodes - which is fast, because we usually span pages and can skip
such nodes right away.
That makes me slightly nervous, because our big powerpc boxes tend to
trip on these scaling issues before others.
But the spanned pages check is just:
void try_offline_node(int nid)
{
pg_data_t *pgdat = NODE_DATA(nid);
...
if (pgdat->node_spanned_pages)
return;
So I guess that's pretty cheap, and it's only O(nodes), which should
never get that big.
Exactly. And if it does turn out to be a problem, we can walk all memory
blocks before removing them, collecting the nid(s).
--
Thanks,
David / dhildenb
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
From: "Rafael J. Wysocki" <rafael@kernel.org> Date: 2021-06-08 12:20:39
On Mon, Jun 7, 2021 at 9:55 PM David Hildenbrand [off-list ref] wrote:
We allocate + initialize everything from scratch. In case enabling the
device fails, we free all memory resourcs.
Signed-off-by: David Hildenbrand <redacted>
From: "Rafael J. Wysocki" <rafael@kernel.org> Date: 2021-06-08 12:20:55
On Mon, Jun 7, 2021 at 9:55 PM David Hildenbrand [off-list ref] wrote:
Let's group all memory we add for a single memory device - we want a
single node for that (which also seems to be the sane thing to do).
We won't care for now about memory that was already added to the system
(e.g., via e820) -- usually *all* memory of a memory device was already
added and we'll fail acpi_memory_enable_device().
Signed-off-by: David Hildenbrand <redacted>
@@ -171,10 +172,31 @@ static int acpi_memory_enable_device(struct acpi_memory_device *mem_device)acpi_handlehandle=mem_device->device->handle;intresult,num_enabled=0;structacpi_memory_info*info;-mhp_tmhp_flags=MHP_NONE;-intnode;+mhp_tmhp_flags=MHP_NID_IS_MGID;+u64total_length=0;+intnode,mgid;node=acpi_get_node(handle);++list_for_each_entry(info,&mem_device->res_list,list){+if(!info->length)+continue;+/* We want a single node for the whole memory group */+if(node<0)+node=memory_add_physaddr_to_nid(info->start_addr);+total_length+=info->length;+}++if(!total_length){+dev_err(&mem_device->device->dev,"device is empty\n");+return-EINVAL;+}++mgid=register_static_memory_group(node,PFN_UP(total_length));+if(mgid<0)+returnmgid;+mem_device->mgid=mgid;+/**TelltheVMthereismorememoryhere...*Note:Assumethatthisfunctionreturnszeroonsuccess
@@ -188,12 +210,10 @@ static int acpi_memory_enable_device(struct acpi_memory_device *mem_device)*/if(!info->length)continue;-if(node<0)-node=memory_add_physaddr_to_nid(info->start_addr);if(mhp_supports_memmap_on_memory(info->length))mhp_flags|=MHP_MEMMAP_ON_MEMORY;-result=__add_memory(node,info->start_addr,info->length,+result=__add_memory(mgid,info->start_addr,info->length,mhp_flags);/*
@@ -253,6 +273,10 @@ static void acpi_memory_device_free(struct acpi_memory_device *mem_device)if(!mem_device)return;+/* In case we succeeded adding *some* memory, unregistering fails. */+if(mem_device->mgid>=0)+unregister_memory_group(mem_device->mgid);+acpi_memory_free_device_resources(mem_device);mem_device->device->driver_data=NULL;kfree(mem_device);
@@ -273,6 +297,7 @@ static int acpi_memory_device_add(struct acpi_device *device,INIT_LIST_HEAD(&mem_device->res_list);mem_device->device=device;+mem_device->mgid=-1;sprintf(acpi_device_name(device),"%s",ACPI_MEMORY_DEVICE_NAME);sprintf(acpi_device_class(device),"%s",ACPI_MEMORY_DEVICE_CLASS);device->driver_data=mem_device;--
From: David Hildenbrand <hidden> Date: 2021-06-09 10:05:11
On 08.06.21 13:18, David Hildenbrand wrote:
On 08.06.21 13:11, Michael Ellerman wrote:
quoted
David Hildenbrand [off-list ref] writes:
quoted
There is only a single user remaining. We can simply try to offline all
online nodes - which is fast, because we usually span pages and can skip
such nodes right away.
That makes me slightly nervous, because our big powerpc boxes tend to
trip on these scaling issues before others.
But the spanned pages check is just:
void try_offline_node(int nid)
{
pg_data_t *pgdat = NODE_DATA(nid);
...
if (pgdat->node_spanned_pages)
return;
So I guess that's pretty cheap, and it's only O(nodes), which should
never get that big.
Exactly. And if it does turn out to be a problem, we can walk all memory
blocks before removing them, collecting the nid(s).
@@ -2176,7 +2176,9 @@ int __ref offline_pages(unsigned long start_pfn, unsigned long nr_pages,staticintcheck_memblock_offlined_cb(structmemory_block*mem,void*arg){intret=!is_memblock_offlined(mem);+int*nid=arg;+*nid=mem->nid;if(unlikely(ret)){phys_addr_tbeginpa,endpa;