This patch series aims to support physical memory hot-remove.
[RFC PATCH v3 1/13] memory-hotplug : rename remove_memory to offline_memory
[RFC PATCH v3 2/13] memory-hotplug : add physical memory hotplug code to acpi_memory_device_remove
[RFC PATCH v3 3/13] memory-hotplug : unify argument of firmware_map_add_early/hotplug
[RFC PATCH v3 4/13] memory-hotplug : remove /sys/firmware/memmap/X sysfs
[RFC PATCH v3 5/13] memory-hotplug : does not release memory region in PAGES_PER_SECTION chunks
[RFC PATCH v3 6/13] memory-hotplug : add memory_block_release
[RFC PATCH v3 7/13] memory-hotplug : remove_memory calls __remove_pages
[RFC PATCH v3 8/13] memory-hotplug : check page type in get_page_bootmem
[RFC PATCH v3 9/13] memory-hotplug : move register_page_bootmem_info_node and put_page_bootmem for
sparse-vmemmap
[RFC PATCH v3 10/13] memory-hotplug : implement register_page_bootmem_info_section of sparse-vmemmap
[RFC PATCH v3 11/13] memory-hotplug : free memmap of sparse-vmemmap
[RFC PATCH v3 12/13] memory-hotplug : add node_device_release
[RFC PATCH v3 13/13] memory-hotplug : remove sysfs file of node
Even if you apply these patches, you cannot remove the physical memory
completely since these patches are still under development. I want you to
cooperate to improve the physical memory hot-remove. So please review these
patches and give your comment/idea.
The patches can free/remove following things:
- acpi_memory_info : [RFC PATCH 2/13]
- /sys/firmware/memmap/X/{end, start, type} : [RFC PATCH 4/13]
- iomem_resource : [RFC PATCH 5/13]
- mem_section and related sysfs files : [RFC PATCH 6-11/13]
- node and related sysfs files : [RFC PATCH 12-13/13]
The patches cannot do following things yet:
- page table of removed memory
If you find lack of function for physical memory hot-remove, please let me
know.
change log of v3:
* rebase to 3.5.0-rc6
[RFC PATCH v2 2/13]
* remove extra kobject_put()
* The patch was commented by Wen. Wen's comment is
"acpi_memory_device_remove() should ignore a return value of
remove_memory() since caller does not care the return value".
But I did not change it since I think caller should care the
return value. And I am trying to fix it as follow:
https://lkml.org/lkml/2012/7/5/624
[RFC PATCH v2 4/13]
* remove a firmware_memmap_entry allocated by kzmalloc()
change log of v2:
[RFC PATCH v2 2/13]
* check whether memory block is offline or not before calling offline_memory()
* check whether section is valid or not in is_memblk_offline()
* call kobject_put() for each memory_block in is_memblk_offline()
[RFC PATCH v2 3/13]
* unify the end argument of firmware_map_add_early/hotplug
[RFC PATCH v2 4/13]
* add release_firmware_map_entry() for freeing firmware_map_entry
[RFC PATCH v2 6/13]
* add release_memory_block() for freeing memory_block
[RFC PATCH v2 11/13]
* fix wrong arguments of free_pages()
---
arch/powerpc/platforms/pseries/hotplug-memory.c | 16 +-
arch/x86/mm/init_64.c | 144 ++++++++++++++++++++++++
drivers/acpi/acpi_memhotplug.c | 28 ++++
drivers/base/memory.c | 54 ++++++++-
drivers/base/node.c | 7 +
drivers/firmware/memmap.c | 78 ++++++++++++-
include/linux/firmware-map.h | 6 +
include/linux/memory.h | 5
include/linux/memory_hotplug.h | 17 --
include/linux/mm.h | 5
mm/memory_hotplug.c | 98 ++++++++++++----
mm/sparse.c | 5
12 files changed, 414 insertions(+), 49 deletions(-)
remove_memory() does not remove memory but just offlines memory. The patch
changes name of it to offline_memory().
CC: David Rientjes <rientjes@google.com>
CC: Jiang Liu <redacted>
CC: Len Brown <redacted>
CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
CC: Paul Mackerras <redacted>
CC: Christoph Lameter <redacted>
Cc: Minchan Kim <redacted>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: KOSAKI Motohiro <redacted>
CC: Wen Congyang <redacted>
Signed-off-by: Yasuaki Ishimatsu <redacted>
---
drivers/acpi/acpi_memhotplug.c | 2 +-
drivers/base/memory.c | 4 ++--
include/linux/memory_hotplug.h | 2 +-
mm/memory_hotplug.c | 6 +++---
4 files changed, 7 insertions(+), 7 deletions(-)
Index: linux-3.5-rc4/drivers/acpi/acpi_memhotplug.c
===================================================================
There are two ways to create /sys/firmware/memmap/X sysfs:
- firmware_map_add_early
When the system starts, it is calledd from e820_reserve_resources()
- firmware_map_add_hotplug
When the memory is hot plugged, it is called from add_memory()
But these functions are called without unifying value of end argument as below:
- end argument of firmware_map_add_early() : start + size - 1
- end argument of firmware_map_add_hogplug() : start + size
The patch unifies them to "start + size - 1".
CC: David Rientjes <rientjes@google.com>
CC: Jiang Liu <redacted>
CC: Len Brown <redacted>
CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
CC: Paul Mackerras <redacted>
CC: Christoph Lameter <redacted>
Cc: Minchan Kim <redacted>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: KOSAKI Motohiro <redacted>
CC: Wen Congyang <redacted>
Signed-off-by: Yasuaki Ishimatsu <redacted>
---
mm/memory_hotplug.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
Index: linux-3.5-rc6/mm/memory_hotplug.c
===================================================================
When (hot)adding memory into system, /sys/firmware/memmap/X/{end, start, type}
sysfs files are created. But there is no code to remove these files. The patch
implements the function to remove them.
Note : The code does not free firmware_map_entry since there is no way to free
memory which is allocated by bootmem.
CC: David Rientjes <rientjes@google.com>
CC: Jiang Liu <redacted>
CC: Len Brown <redacted>
CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
CC: Paul Mackerras <redacted>
CC: Christoph Lameter <redacted>
Cc: Minchan Kim <redacted>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: KOSAKI Motohiro <redacted>
CC: Wen Congyang <redacted>
Signed-off-by: Yasuaki Ishimatsu <redacted>
---
drivers/firmware/memmap.c | 78 ++++++++++++++++++++++++++++++++++++++++++-
include/linux/firmware-map.h | 6 +++
mm/memory_hotplug.c | 6 ++-
3 files changed, 88 insertions(+), 2 deletions(-)
Index: linux-3.5-rc6/mm/memory_hotplug.c
===================================================================
@@ -79,7 +80,22 @@ static const struct sysfs_ops memmap_att.show=memmap_attr_show,};+#define to_memmap_entry(obj) container_of(obj, struct firmware_map_entry, kobj)++staticvoidrelease_firmware_map_entry(structkobject*kobj)+{+structfirmware_map_entry*entry=to_memmap_entry(kobj);+structpage*head_page;++head_page=virt_to_head_page(entry);+if(PageSlab(head_page))+kfree(entry);++/* There is no way to free memory allocated from bootmem*/+}+staticstructkobj_typememmap_ktype={+.release=release_firmware_map_entry,.sysfs_ops=&memmap_attr_ops,.default_attrs=def_attrs,};
@@ -123,6 +139,16 @@ static int firmware_map_add_entry(u64 streturn0;}+/**+*firmware_map_remove_entry()-Doestherealworktoremoveafirmware+*memmapentry.+*@entry:removedentry.+**/+staticinlinevoidfirmware_map_remove_entry(structfirmware_map_entry*entry)+{+list_del(&entry->list);+}+/**Addmemmapentryonsysfs*/
@@ -144,6 +170,31 @@ static int add_sysfs_fw_map_entry(structreturn0;}+/*+*Removememmapentryonsysfs+*/+staticinlinevoidremove_sysfs_fw_map_entry(structfirmware_map_entry*entry)+{+kobject_put(&entry->kobj);+}++/*+*Searchmemmapentry+*/++structfirmware_map_entry*__meminit+find_firmware_map_entry(u64start,u64end,constchar*type)+{+structfirmware_map_entry*entry;++list_for_each_entry(entry,&map_entries,list)+if((entry->start==start)&&(entry->end==end)&&+(!strcmp(entry->type,type)))+returnentry;++returnNULL;+}+/***firmware_map_add_hotplug()-Addsafirmwaremappingentrywhenwedo*memoryhotplug.
@@ -196,6 +247,32 @@ int __init firmware_map_add_early(u64 streturnfirmware_map_add_entry(start,end,type,entry);}+/**+*firmware_map_remove()-removeafirmwaremappingentry+*@start:Startofthememoryrange.+*@end:Endofthememoryrange(inclusive).+*@type:Typeofthememoryrange.+*+*removesafirmwaremappingentry.+*+*Returns0onsuccess,or-EINVALifnoentry.+**/+int__meminitfirmware_map_remove(u64start,u64end,constchar*type)+{+structfirmware_map_entry*entry;++entry=find_firmware_map_entry(start,end,type);+if(!entry)+return-EINVAL;++/* remove the memmap entry */+remove_sysfs_fw_map_entry(entry);++firmware_map_remove_entry(entry);++return0;+}+/**Sysfsfunctions-------------------------------------------------------------*/
Since applying a patch(de7f0cba96786c), release_mem_region() has been changed
as called in PAGES_PER_SECTION chunks because register_memory_resource() is
called in PAGES_PER_SECTION chunks by add_memory(). But it seems firmware
dependency. If CRS are written in the PAGES_PER_SECTION chunks in ACPI DSDT
Table, register_memory_resource() is called in PAGES_PER_SECTION chunks.
But if CRS are written in the DIMM unit in ACPI DSDT Table,
register_memory_resource() is called in DIMM unit. So release_mem_region()
should not be called in PAGES_PER_SECTION chunks. The patch fixes it.
CC: David Rientjes <rientjes@google.com>
CC: Jiang Liu <redacted>
CC: Len Brown <redacted>
CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
CC: Paul Mackerras <redacted>
CC: Christoph Lameter <redacted>
Cc: Minchan Kim <redacted>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: KOSAKI Motohiro <redacted>
CC: Wen Congyang <redacted>
Signed-off-by: Yasuaki Ishimatsu <redacted>
---
arch/powerpc/platforms/pseries/hotplug-memory.c | 13 +++++++++----
mm/memory_hotplug.c | 4 ++--
2 files changed, 11 insertions(+), 6 deletions(-)
Index: linux-3.5-rc4/mm/memory_hotplug.c
===================================================================
@@ -358,11 +358,11 @@ int __remove_pages(struct zone *zone, unBUG_ON(phys_start_pfn&~PAGE_SECTION_MASK);BUG_ON(nr_pages%PAGES_PER_SECTION);+release_mem_region(phys_start_pfn<<PAGE_SHIFT,nr_pages*PAGE_SIZE);+sections_to_remove=nr_pages/PAGES_PER_SECTION;for(i=0;i<sections_to_remove;i++){unsignedlongpfn=phys_start_pfn+i*PAGES_PER_SECTION;-release_mem_region(pfn<<PAGE_SHIFT,-PAGES_PER_SECTION<<PAGE_SHIFT);ret=__remove_section(zone,__pfn_to_section(pfn));if(ret)break;
When calling remove_memory_block(), the function shows following message at
device_release().
Device 'memory528' does not have a release() function, it is broken and must
be fixed.
remove_memory_block() calls kfree(mem). I think it shouled be called from
device_release(). So the patch implements memory_block_release()
CC: David Rientjes <rientjes@google.com>
CC: Jiang Liu <redacted>
CC: Len Brown <redacted>
CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
CC: Paul Mackerras <redacted>
CC: Christoph Lameter <redacted>
Cc: Minchan Kim <redacted>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: KOSAKI Motohiro <redacted>
CC: Wen Congyang <redacted>
Signed-off-by: Yasuaki Ishimatsu <redacted>
---
drivers/base/memory.c | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
Index: linux-3.5-rc6/drivers/base/memory.c
===================================================================
@@ -119,6 +128,7 @@ int register_memory(struct memory_blockmemory->dev.bus=&memory_subsys;memory->dev.id=memory->start_section_nr/sections_per_block;+memory->dev.release=release_memory_block;error=device_register(&memory->dev);returnerror;
@@ -669,7 +679,6 @@ int remove_memory_block(unsigned long nomem_remove_simple_file(mem,phys_device);mem_remove_simple_file(mem,removable);unregister_memory(mem);-kfree(mem);}elsekobject_put(&mem->dev.kobj);
The patch adds __remove_pages() to remove_memory(). Then the range of
phys_start_pfn argument and nr_pages argument in __remove_pagse() may
have different zone. So zone argument is removed from __remove_pages()
and __remove_pages() caluculates zone in each section.
When CONFIG_SPARSEMEM_VMEMMAP is defined, there is no way to remove a memmap.
So __remove_section only calls unregister_memory_section().
CC: David Rientjes <rientjes@google.com>
CC: Jiang Liu <redacted>
CC: Len Brown <redacted>
CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
CC: Paul Mackerras <redacted>
CC: Christoph Lameter <redacted>
Cc: Minchan Kim <redacted>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: KOSAKI Motohiro <redacted>
CC: Wen Congyang <redacted>
Signed-off-by: Yasuaki Ishimatsu <redacted>
---
arch/powerpc/platforms/pseries/hotplug-memory.c | 5 +----
include/linux/memory_hotplug.h | 3 +--
mm/memory_hotplug.c | 20 +++++++++++++-------
3 files changed, 15 insertions(+), 13 deletions(-)
Index: linux-3.5-rc4/mm/memory_hotplug.c
===================================================================
@@ -363,6 +366,7 @@ int __remove_pages(struct zone *zone, unsections_to_remove=nr_pages/PAGES_PER_SECTION;for(i=0;i<sections_to_remove;i++){unsignedlongpfn=phys_start_pfn+i*PAGES_PER_SECTION;+zone=page_zone(pfn_to_page(pfn));ret=__remove_section(zone,__pfn_to_section(pfn));if(ret)break;
@@ -89,8 +89,7 @@ extern bool is_pageblock_removable_noloc/* reasonably generic interface to expand the physical pages in a zone */externint__add_pages(intnid,structzone*zone,unsignedlongstart_pfn,unsignedlongnr_pages);-externint__remove_pages(structzone*zone,unsignedlongstart_pfn,-unsignedlongnr_pages);+externint__remove_pages(unsignedlongstart_pfn,unsignedlongnr_pages);#ifdef CONFIG_NUMAexternintmemory_add_physaddr_to_nid(u64start);
There is a possibility that get_page_bootmem() is called to the same page many
times. So when get_page_bootmem is called to the same page, the function only
increments page->_count.
CC: David Rientjes <rientjes@google.com>
CC: Jiang Liu <redacted>
CC: Len Brown <redacted>
CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
CC: Paul Mackerras <redacted>
CC: Christoph Lameter <redacted>
Cc: Minchan Kim <redacted>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: KOSAKI Motohiro <redacted>
CC: Wen Congyang <redacted>
Signed-off-by: Yasuaki Ishimatsu <redacted>
---
mm/memory_hotplug.c | 15 +++++++++++----
1 file changed, 11 insertions(+), 4 deletions(-)
Index: linux-3.5-rc4/mm/memory_hotplug.c
===================================================================
For implementing register_page_bootmem_info_node of sparse-vmemmap,
register_page_bootmem_info_node and put_page_bootmem are moved to
memory_hotplug.c
CC: David Rientjes <rientjes@google.com>
CC: Jiang Liu <redacted>
CC: Len Brown <redacted>
CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
CC: Paul Mackerras <redacted>
CC: Christoph Lameter <redacted>
Cc: Minchan Kim <redacted>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: KOSAKI Motohiro <redacted>
CC: Wen Congyang <redacted>
Signed-off-by: Yasuaki Ishimatsu <redacted>
---
include/linux/memory_hotplug.h | 9 ---------
mm/memory_hotplug.c | 8 ++++++--
2 files changed, 6 insertions(+), 11 deletions(-)
Index: linux-3.5-rc4/include/linux/memory_hotplug.h
===================================================================
For removing memmap region of sparse-vmemmap which is allocated bootmem,
memmap region of sparse-vmemmap needs to be registered by get_page_bootmem().
So the patch searches pages of virtual mapping and registers the pages by
get_page_bootmem().
CC: David Rientjes <rientjes@google.com>
CC: Jiang Liu <redacted>
CC: Len Brown <redacted>
CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
CC: Paul Mackerras <redacted>
CC: Christoph Lameter <redacted>
Cc: Minchan Kim <redacted>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: KOSAKI Motohiro <redacted>
CC: Wen Congyang <redacted>
Signed-off-by: Yasuaki Ishimatsu <redacted>
---
arch/x86/mm/init_64.c | 53 +++++++++++++++++++++++++++++++++++++++++
include/linux/memory_hotplug.h | 2 +
include/linux/mm.h | 3 +-
mm/memory_hotplug.c | 23 +++++++++++++++--
4 files changed, 77 insertions(+), 4 deletions(-)
Index: linux-3.5-rc4/mm/memory_hotplug.c
===================================================================
I don't think that all pages of virtual mapping in removed memory can be
freed, since page which type is MIX_SECTION_INFO is difficult to free.
So, the patch only frees page which type is SECTION_INFO at first.
CC: David Rientjes <rientjes@google.com>
CC: Jiang Liu <redacted>
CC: Len Brown <redacted>
CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
CC: Paul Mackerras <redacted>
CC: Christoph Lameter <redacted>
Cc: Minchan Kim <redacted>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: KOSAKI Motohiro <redacted>
CC: Wen Congyang <redacted>
Signed-off-by: Yasuaki Ishimatsu <redacted>
---
arch/x86/mm/init_64.c | 91 ++++++++++++++++++++++++++++++++++++++++++++++++++
include/linux/mm.h | 2 +
mm/memory_hotplug.c | 5 ++
mm/sparse.c | 5 +-
4 files changed, 101 insertions(+), 2 deletions(-)
Index: linux-3.5-rc4/include/linux/mm.h
===================================================================
@@ -614,12 +614,13 @@ static inline struct page *kmalloc_secti/* This will make the necessary allocations eventually. */returnsparse_mem_map_populate(pnum,nid);}-staticvoid__kfree_section_memmap(structpage*memmap,unsignedlongnr_pages)+staticvoid__kfree_section_memmap(structpage*page,unsignedlongnr_pages){-return;/* XXX: Not implemented yet */+vmemmap_kfree(page,nr_pages);}staticvoidfree_map_bootmem(structpage*page,unsignedlongnr_pages){+vmemmap_free_bootmem(page,nr_pages);}#elsestaticstructpage*__kmalloc_section_memmap(unsignedlongnr_pages)
When calling unregister_node(), the function shows following message at
device_release().
Device 'node2' does not have a release() function, it is broken and must be
fixed.
So the patch implements node_device_release()
CC: David Rientjes <rientjes@google.com>
CC: Jiang Liu <redacted>
CC: Len Brown <redacted>
CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
CC: Paul Mackerras <redacted>
CC: Christoph Lameter <redacted>
Cc: Minchan Kim <redacted>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: KOSAKI Motohiro <redacted>
CC: Wen Congyang <redacted>
Signed-off-by: Yasuaki Ishimatsu <redacted>
---
drivers/base/node.c | 7 +++++++
1 file changed, 7 insertions(+)
Index: linux-3.5-rc4/drivers/base/node.c
===================================================================
From: Christoph Lameter <hidden> Date: 2012-07-09 15:19:00
On Mon, 9 Jul 2012, Yasuaki Ishimatsu wrote:
Even if you apply these patches, you cannot remove the physical memory
completely since these patches are still under development. I want you to
cooperate to improve the physical memory hot-remove. So please review these
patches and give your comment/idea.
Could you at least give a method on how you want to do physical memory
removal? You would have to remove all objects from the range you want to
physically remove. That is only possible under special circumstances and
with a limited set of objects. Even if you exclusively use ZONE_MOVEABLE
you still may get cases where pages are pinned for a long time.
I am not sure that these patches are useful unless we know where you are
going with this. If we end up with a situation where we still cannot
remove physical memory then this patchset is not helpful.
Hi Christoph,
2012/07/10 0:18, Christoph Lameter wrote:
On Mon, 9 Jul 2012, Yasuaki Ishimatsu wrote:
quoted
Even if you apply these patches, you cannot remove the physical memory
completely since these patches are still under development. I want you to
cooperate to improve the physical memory hot-remove. So please review these
patches and give your comment/idea.
Could you at least give a method on how you want to do physical memory
removal?
We plan to release a dynamic hardware partitionable system. It will be
able to hot remove/add a system board which included memory and cpu.
But as you know, Linux does not support memory hot-remove on x86 box.
So I try to develop it.
Current plan to hot remove system board is to use container driver.
Thus I define the system board in ACPI DSDT table as a container device.
It have supported hot-add a container device. And if container device
has _EJ0 ACPI method, "eject" file to remove the container device is
prepared as follow:
# ls -l /sys/bus/acpi/devices/ACPI0004\:01/eject
--w-------. 1 root root 4096 Jul 10 18:19 /sys/bus/acpi/devices/ACPI0004:01/eject
When I hot-remove the container device, I echo 1 to the file as follow:
#echo 1 > /sys/bus/acpi/devices/ACPI0004\:02/eject
Then acpi_bus_trim() is called. And it calls acpi_memory_device_remove()
for removing memory device. But the code does not do nothing.
So I developed the continuation of the function.
You would have to remove all objects from the range you want to
physically remove. That is only possible under special circumstances and
with a limited set of objects. Even if you exclusively use ZONE_MOVEABLE
you still may get cases where pages are pinned for a long time.
I know it. So my memory hot-remove plan is as follows:
1. hot-added a system board
All memory which included the system board is offline.
2. online the memory as removable page
The function has not supported yet. It is being developed by Lai as follow:
http://lkml.indiana.edu/hypermail/linux/kernel/1207.0/01478.html
If it is supported, I will be able to create movable memory.
3. hot-remove the memory by container device's eject file
Thanks,
Yasuaki Ishimatsu
I am not sure that these patches are useful unless we know where you are
going with this. If we end up with a situation where we still cannot
remove physical memory then this patchset is not helpful.
From: Jiang Liu <hidden> Date: 2012-07-10 16:50:29
On 07/10/2012 05:58 PM, Yasuaki Ishimatsu wrote:
Hi Christoph,
2012/07/10 0:18, Christoph Lameter wrote:
quoted
On Mon, 9 Jul 2012, Yasuaki Ishimatsu wrote:
quoted
Even if you apply these patches, you cannot remove the physical memory
completely since these patches are still under development. I want you to
cooperate to improve the physical memory hot-remove. So please review these
patches and give your comment/idea.
Could you at least give a method on how you want to do physical memory
removal?
We plan to release a dynamic hardware partitionable system. It will be
able to hot remove/add a system board which included memory and cpu.
But as you know, Linux does not support memory hot-remove on x86 box.
So I try to develop it.
Current plan to hot remove system board is to use container driver.
Thus I define the system board in ACPI DSDT table as a container device.
It have supported hot-add a container device. And if container device
has _EJ0 ACPI method, "eject" file to remove the container device is
prepared as follow:
# ls -l /sys/bus/acpi/devices/ACPI0004\:01/eject
--w-------. 1 root root 4096 Jul 10 18:19 /sys/bus/acpi/devices/ACPI0004:01/eject
When I hot-remove the container device, I echo 1 to the file as follow:
#echo 1 > /sys/bus/acpi/devices/ACPI0004\:02/eject
Then acpi_bus_trim() is called. And it calls acpi_memory_device_remove()
for removing memory device. But the code does not do nothing.
So I developed the continuation of the function.
quoted
You would have to remove all objects from the range you want to
physically remove. That is only possible under special circumstances and
with a limited set of objects. Even if you exclusively use ZONE_MOVEABLE
you still may get cases where pages are pinned for a long time.
I know it. So my memory hot-remove plan is as follows:
1. hot-added a system board
All memory which included the system board is offline.
2. online the memory as removable page
The function has not supported yet. It is being developed by Lai as follow:
http://lkml.indiana.edu/hypermail/linux/kernel/1207.0/01478.html
If it is supported, I will be able to create movable memory.
3. hot-remove the memory by container device's eject file
We have implemented a prototype to do physical node (mem + CPU + IOH) hotplug
for Itanium and is now porting it to x86. But with currently solution, memory
hotplug functionality may cause 10-20% performance decrease because we concentrate
all DMA/Normal memory to the first NUMA node, and all other NUMA nodes only
hosts ZONE_MOVABLE. We are working on solution to minimize the performance
drop now.
Thanks,
Yasuaki Ishimatsu
quoted
I am not sure that these patches are useful unless we know where you are
going with this. If we end up with a situation where we still cannot
remove physical memory then this patchset is not helpful.
Hi Christoph,
2012/07/10 0:18, Christoph Lameter wrote:
quoted
On Mon, 9 Jul 2012, Yasuaki Ishimatsu wrote:
quoted
Even if you apply these patches, you cannot remove the physical memory
completely since these patches are still under development. I want you to
cooperate to improve the physical memory hot-remove. So please review these
patches and give your comment/idea.
Could you at least give a method on how you want to do physical memory
removal?
We plan to release a dynamic hardware partitionable system. It will be
able to hot remove/add a system board which included memory and cpu.
But as you know, Linux does not support memory hot-remove on x86 box.
So I try to develop it.
Current plan to hot remove system board is to use container driver.
Thus I define the system board in ACPI DSDT table as a container device.
It have supported hot-add a container device. And if container device
has _EJ0 ACPI method, "eject" file to remove the container device is
prepared as follow:
# ls -l /sys/bus/acpi/devices/ACPI0004\:01/eject
--w-------. 1 root root 4096 Jul 10 18:19 /sys/bus/acpi/devices/ACPI0004:01/eject
When I hot-remove the container device, I echo 1 to the file as follow:
#echo 1 > /sys/bus/acpi/devices/ACPI0004\:02/eject
Then acpi_bus_trim() is called. And it calls acpi_memory_device_remove()
for removing memory device. But the code does not do nothing.
So I developed the continuation of the function.
quoted
You would have to remove all objects from the range you want to
physically remove. That is only possible under special circumstances and
with a limited set of objects. Even if you exclusively use ZONE_MOVEABLE
you still may get cases where pages are pinned for a long time.
I know it. So my memory hot-remove plan is as follows:
1. hot-added a system board
All memory which included the system board is offline.
2. online the memory as removable page
The function has not supported yet. It is being developed by Lai as follow:
http://lkml.indiana.edu/hypermail/linux/kernel/1207.0/01478.html
If it is supported, I will be able to create movable memory.
3. hot-remove the memory by container device's eject file
We have implemented a prototype to do physical node (mem + CPU + IOH) hotplug
for Itanium and is now porting it to x86. But with currently solution, memory
hotplug functionality may cause 10-20% performance decrease because we concentrate
all DMA/Normal memory to the first NUMA node, and all other NUMA nodes only
hosts ZONE_MOVABLE. We are working on solution to minimize the performance
drop now.
Thank you for your interesting response.
I have a question. How do you move all other NUMA nodes to ZONE_MOVABLE?
To use ZONE_MOVABLE, we need to use boot options like kernelcore or movablecore.
But it is not enough, since the requested amount is spread evenly throughout
all nodes in the system. So I think we do not have way to move all other NUMA
node to ZONE_MOVABLE.
Thanks,
Yasuaki Ishimatsu
quoted
Thanks,
Yasuaki Ishimatsu
quoted
I am not sure that these patches are useful unless we know where you are
going with this. If we end up with a situation where we still cannot
remove physical memory then this patchset is not helpful.
From: Jiang Liu <hidden> Date: 2012-07-11 00:21:19
On 07/11/2012 08:09 AM, Yasuaki Ishimatsu wrote:
Hi Jiang,
2012/07/11 1:50, Jiang Liu wrote:
quoted
On 07/10/2012 05:58 PM, Yasuaki Ishimatsu wrote:
quoted
Hi Christoph,
2012/07/10 0:18, Christoph Lameter wrote:
quoted
On Mon, 9 Jul 2012, Yasuaki Ishimatsu wrote:
quoted
Even if you apply these patches, you cannot remove the physical memory
completely since these patches are still under development. I want you to
cooperate to improve the physical memory hot-remove. So please review these
patches and give your comment/idea.
Could you at least give a method on how you want to do physical memory
removal?
We plan to release a dynamic hardware partitionable system. It will be
able to hot remove/add a system board which included memory and cpu.
But as you know, Linux does not support memory hot-remove on x86 box.
So I try to develop it.
Current plan to hot remove system board is to use container driver.
Thus I define the system board in ACPI DSDT table as a container device.
It have supported hot-add a container device. And if container device
has _EJ0 ACPI method, "eject" file to remove the container device is
prepared as follow:
# ls -l /sys/bus/acpi/devices/ACPI0004\:01/eject
--w-------. 1 root root 4096 Jul 10 18:19 /sys/bus/acpi/devices/ACPI0004:01/eject
When I hot-remove the container device, I echo 1 to the file as follow:
#echo 1 > /sys/bus/acpi/devices/ACPI0004\:02/eject
Then acpi_bus_trim() is called. And it calls acpi_memory_device_remove()
for removing memory device. But the code does not do nothing.
So I developed the continuation of the function.
quoted
You would have to remove all objects from the range you want to
physically remove. That is only possible under special circumstances and
with a limited set of objects. Even if you exclusively use ZONE_MOVEABLE
you still may get cases where pages are pinned for a long time.
I know it. So my memory hot-remove plan is as follows:
1. hot-added a system board
All memory which included the system board is offline.
2. online the memory as removable page
The function has not supported yet. It is being developed by Lai as follow:
http://lkml.indiana.edu/hypermail/linux/kernel/1207.0/01478.html
If it is supported, I will be able to create movable memory.
3. hot-remove the memory by container device's eject file
We have implemented a prototype to do physical node (mem + CPU + IOH) hotplug
for Itanium and is now porting it to x86. But with currently solution, memory
hotplug functionality may cause 10-20% performance decrease because we concentrate
all DMA/Normal memory to the first NUMA node, and all other NUMA nodes only
hosts ZONE_MOVABLE. We are working on solution to minimize the performance
drop now.
Thank you for your interesting response.
I have a question. How do you move all other NUMA nodes to ZONE_MOVABLE?
To use ZONE_MOVABLE, we need to use boot options like kernelcore or movablecore.
But it is not enough, since the requested amount is spread evenly throughout
all nodes in the system. So I think we do not have way to move all other NUMA
node to ZONE_MOVABLE.
We have modified the ZONE_MOVABLE spreading and bootmem allocation. If the kernelcore
or movablecore kernel parameters are present, we follow current behavior. If those
parameter are absent and the platform supports physical hotplug, we will concentrate
DMA/NORMAL memory to specific nodes.
Thanks,
Yasuaki Ishimatsu
quoted
quoted
Thanks,
Yasuaki Ishimatsu
quoted
I am not sure that these patches are useful unless we know where you are
going with this. If we end up with a situation where we still cannot
remove physical memory then this patchset is not helpful.
Hi Christoph,
2012/07/10 0:18, Christoph Lameter wrote:
quoted
On Mon, 9 Jul 2012, Yasuaki Ishimatsu wrote:
quoted
Even if you apply these patches, you cannot remove the physical memory
completely since these patches are still under development. I want you to
cooperate to improve the physical memory hot-remove. So please review these
patches and give your comment/idea.
Could you at least give a method on how you want to do physical memory
removal?
We plan to release a dynamic hardware partitionable system. It will be
able to hot remove/add a system board which included memory and cpu.
But as you know, Linux does not support memory hot-remove on x86 box.
So I try to develop it.
Current plan to hot remove system board is to use container driver.
Thus I define the system board in ACPI DSDT table as a container device.
It have supported hot-add a container device. And if container device
has _EJ0 ACPI method, "eject" file to remove the container device is
prepared as follow:
# ls -l /sys/bus/acpi/devices/ACPI0004\:01/eject
--w-------. 1 root root 4096 Jul 10 18:19 /sys/bus/acpi/devices/ACPI0004:01/eject
When I hot-remove the container device, I echo 1 to the file as follow:
#echo 1 > /sys/bus/acpi/devices/ACPI0004\:02/eject
Then acpi_bus_trim() is called. And it calls acpi_memory_device_remove()
for removing memory device. But the code does not do nothing.
So I developed the continuation of the function.
quoted
You would have to remove all objects from the range you want to
physically remove. That is only possible under special circumstances and
with a limited set of objects. Even if you exclusively use ZONE_MOVEABLE
you still may get cases where pages are pinned for a long time.
I know it. So my memory hot-remove plan is as follows:
1. hot-added a system board
All memory which included the system board is offline.
2. online the memory as removable page
The function has not supported yet. It is being developed by Lai as follow:
http://lkml.indiana.edu/hypermail/linux/kernel/1207.0/01478.html
If it is supported, I will be able to create movable memory.
3. hot-remove the memory by container device's eject file
We have implemented a prototype to do physical node (mem + CPU + IOH) hotplug
for Itanium and is now porting it to x86. But with currently solution, memory
hotplug functionality may cause 10-20% performance decrease because we concentrate
all DMA/Normal memory to the first NUMA node, and all other NUMA nodes only
hosts ZONE_MOVABLE. We are working on solution to minimize the performance
drop now.
Thank you for your interesting response.
I have a question. How do you move all other NUMA nodes to ZONE_MOVABLE?
To use ZONE_MOVABLE, we need to use boot options like kernelcore or movablecore.
But it is not enough, since the requested amount is spread evenly throughout
all nodes in the system. So I think we do not have way to move all other NUMA
node to ZONE_MOVABLE.
We have modified the ZONE_MOVABLE spreading and bootmem allocation. If the kernelcore
or movablecore kernel parameters are present, we follow current behavior. If those
parameter are absent and the platform supports physical hotplug, we will concentrate
DMA/NORMAL memory to specific nodes.
That's interesting. I want to know more details, if you do not mind.
Current kernel doesn't do the behavior, does it? So I think you have some
patches for changing the behavior. Will you merge these patches into
community kernel?
Thanks,
Yasuaki Ishimatsu
quoted
Thanks,
Yasuaki Ishimatsu
quoted
quoted
Thanks,
Yasuaki Ishimatsu
quoted
I am not sure that these patches are useful unless we know where you are
going with this. If we end up with a situation where we still cannot
remove physical memory then this patchset is not helpful.
This patch series aims to support physical memory hot-remove.
[RFC PATCH v3 1/13] memory-hotplug : rename remove_memory to offline_memory
[RFC PATCH v3 2/13] memory-hotplug : add physical memory hotplug code to acpi_memory_device_remove
[RFC PATCH v3 3/13] memory-hotplug : unify argument of firmware_map_add_early/hotplug
[RFC PATCH v3 4/13] memory-hotplug : remove /sys/firmware/memmap/X sysfs
[RFC PATCH v3 5/13] memory-hotplug : does not release memory region in PAGES_PER_SECTION chunks
[RFC PATCH v3 6/13] memory-hotplug : add memory_block_release
[RFC PATCH v3 7/13] memory-hotplug : remove_memory calls __remove_pages
[RFC PATCH v3 8/13] memory-hotplug : check page type in get_page_bootmem
[RFC PATCH v3 9/13] memory-hotplug : move register_page_bootmem_info_node and put_page_bootmem for
sparse-vmemmap
[RFC PATCH v3 10/13] memory-hotplug : implement register_page_bootmem_info_section of sparse-vmemmap
[RFC PATCH v3 11/13] memory-hotplug : free memmap of sparse-vmemmap
[RFC PATCH v3 12/13] memory-hotplug : add node_device_release
[RFC PATCH v3 13/13] memory-hotplug : remove sysfs file of node
Even if you apply these patches, you cannot remove the physical memory
completely since these patches are still under development. I want you to
cooperate to improve the physical memory hot-remove. So please review these
patches and give your comment/idea.
The patches can free/remove following things:
- acpi_memory_info : [RFC PATCH 2/13]
- /sys/firmware/memmap/X/{end, start, type} : [RFC PATCH 4/13]
- iomem_resource : [RFC PATCH 5/13]
- mem_section and related sysfs files : [RFC PATCH 6-11/13]
- node and related sysfs files : [RFC PATCH 12-13/13]
The patches cannot do following things yet:
- page table of removed memory
If you find lack of function for physical memory hot-remove, please let me
know.
change log of v3:
* rebase to 3.5.0-rc6
[RFC PATCH v2 2/13]
* remove extra kobject_put()
* The patch was commented by Wen. Wen's comment is
"acpi_memory_device_remove() should ignore a return value of
remove_memory() since caller does not care the return value".
But I did not change it since I think caller should care the
return value. And I am trying to fix it as follow:
https://lkml.org/lkml/2012/7/5/624
acpi_memory_device_remove() will be called not only when we write
1 to /sys/bus/acpi/devices/PNP0C80:XX/eject. When we unbind it
from the driver or remove the module acpi_memhotplug, this function
will be called too.
I will check whether your patch can work for these two cases.
Thanks
Wen Congyang
[RFC PATCH v2 4/13]
* remove a firmware_memmap_entry allocated by kzmalloc()
change log of v2:
[RFC PATCH v2 2/13]
* check whether memory block is offline or not before calling offline_memory()
* check whether section is valid or not in is_memblk_offline()
* call kobject_put() for each memory_block in is_memblk_offline()
[RFC PATCH v2 3/13]
* unify the end argument of firmware_map_add_early/hotplug
[RFC PATCH v2 4/13]
* add release_firmware_map_entry() for freeing firmware_map_entry
[RFC PATCH v2 6/13]
* add release_memory_block() for freeing memory_block
[RFC PATCH v2 11/13]
* fix wrong arguments of free_pages()
---
arch/powerpc/platforms/pseries/hotplug-memory.c | 16 +-
arch/x86/mm/init_64.c | 144 ++++++++++++++++++++++++
drivers/acpi/acpi_memhotplug.c | 28 ++++
drivers/base/memory.c | 54 ++++++++-
drivers/base/node.c | 7 +
drivers/firmware/memmap.c | 78 ++++++++++++-
include/linux/firmware-map.h | 6 +
include/linux/memory.h | 5
include/linux/memory_hotplug.h | 17 --
include/linux/mm.h | 5
mm/memory_hotplug.c | 98 ++++++++++++----
mm/sparse.c | 5
12 files changed, 414 insertions(+), 49 deletions(-)
This patch series aims to support physical memory hot-remove.
[RFC PATCH v3 1/13] memory-hotplug : rename remove_memory to offline_memory
[RFC PATCH v3 2/13] memory-hotplug : add physical memory hotplug code to acpi_memory_device_remove
[RFC PATCH v3 3/13] memory-hotplug : unify argument of firmware_map_add_early/hotplug
[RFC PATCH v3 4/13] memory-hotplug : remove /sys/firmware/memmap/X sysfs
[RFC PATCH v3 5/13] memory-hotplug : does not release memory region in PAGES_PER_SECTION chunks
[RFC PATCH v3 6/13] memory-hotplug : add memory_block_release
[RFC PATCH v3 7/13] memory-hotplug : remove_memory calls __remove_pages
[RFC PATCH v3 8/13] memory-hotplug : check page type in get_page_bootmem
[RFC PATCH v3 9/13] memory-hotplug : move register_page_bootmem_info_node and put_page_bootmem for
sparse-vmemmap
[RFC PATCH v3 10/13] memory-hotplug : implement register_page_bootmem_info_section of sparse-vmemmap
[RFC PATCH v3 11/13] memory-hotplug : free memmap of sparse-vmemmap
[RFC PATCH v3 12/13] memory-hotplug : add node_device_release
[RFC PATCH v3 13/13] memory-hotplug : remove sysfs file of node
Even if you apply these patches, you cannot remove the physical memory
completely since these patches are still under development. I want you to
cooperate to improve the physical memory hot-remove. So please review these
patches and give your comment/idea.
The patches can free/remove following things:
- acpi_memory_info : [RFC PATCH 2/13]
- /sys/firmware/memmap/X/{end, start, type} : [RFC PATCH 4/13]
- iomem_resource : [RFC PATCH 5/13]
- mem_section and related sysfs files : [RFC PATCH 6-11/13]
- node and related sysfs files : [RFC PATCH 12-13/13]
The patches cannot do following things yet:
- page table of removed memory
If you find lack of function for physical memory hot-remove, please let me
know.
change log of v3:
* rebase to 3.5.0-rc6
[RFC PATCH v2 2/13]
* remove extra kobject_put()
* The patch was commented by Wen. Wen's comment is
"acpi_memory_device_remove() should ignore a return value of
remove_memory() since caller does not care the return value".
But I did not change it since I think caller should care the
return value. And I am trying to fix it as follow:
https://lkml.org/lkml/2012/7/5/624
acpi_memory_device_remove() will be called not only when we write
1 to /sys/bus/acpi/devices/PNP0C80:XX/eject. When we unbind it
from the driver or remove the module acpi_memhotplug, this function
will be called too.
I will check whether your patch can work for these two cases.
I have checked it, and I think your patch can not work for these 2 cases.
When we unbind the device from the driver(write device name to
/sys/bus/acpi/drivers/acpi_memhotplug/unbind), driver_unbind()
will be called. This function does not care the return value.
When we remove the module acpi_memhotplug, acpi_memory_device_exit()
will be called. This function does not care the return value too.
I don't know whether there are some other cases that acpi_memory_device_remove()
will be called.
Thanks
Wen Congyang
Thanks
Wen Congyang
quoted
[RFC PATCH v2 4/13]
* remove a firmware_memmap_entry allocated by kzmalloc()
change log of v2:
[RFC PATCH v2 2/13]
* check whether memory block is offline or not before calling offline_memory()
* check whether section is valid or not in is_memblk_offline()
* call kobject_put() for each memory_block in is_memblk_offline()
[RFC PATCH v2 3/13]
* unify the end argument of firmware_map_add_early/hotplug
[RFC PATCH v2 4/13]
* add release_firmware_map_entry() for freeing firmware_map_entry
[RFC PATCH v2 6/13]
* add release_memory_block() for freeing memory_block
[RFC PATCH v2 11/13]
* fix wrong arguments of free_pages()
---
arch/powerpc/platforms/pseries/hotplug-memory.c | 16 +-
arch/x86/mm/init_64.c | 144 ++++++++++++++++++++++++
drivers/acpi/acpi_memhotplug.c | 28 ++++
drivers/base/memory.c | 54 ++++++++-
drivers/base/node.c | 7 +
drivers/firmware/memmap.c | 78 ++++++++++++-
include/linux/firmware-map.h | 6 +
include/linux/memory.h | 5
include/linux/memory_hotplug.h | 17 --
include/linux/mm.h | 5
mm/memory_hotplug.c | 98 ++++++++++++----
mm/sparse.c | 5
12 files changed, 414 insertions(+), 49 deletions(-)
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
I don't think that all pages of virtual mapping in removed memory can be
freed, since page which type is MIX_SECTION_INFO is difficult to free.
So, the patch only frees page which type is SECTION_INFO at first.
CC: David Rientjes <rientjes@google.com>
CC: Jiang Liu <redacted>
CC: Len Brown <redacted>
CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
CC: Paul Mackerras <redacted>
CC: Christoph Lameter <redacted>
Cc: Minchan Kim <redacted>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: KOSAKI Motohiro <redacted>
CC: Wen Congyang <redacted>
Signed-off-by: Yasuaki Ishimatsu <redacted>
---
arch/x86/mm/init_64.c | 91 ++++++++++++++++++++++++++++++++++++++++++++++++++
include/linux/mm.h | 2 +
mm/memory_hotplug.c | 5 ++
mm/sparse.c | 5 +-
4 files changed, 101 insertions(+), 2 deletions(-)
Index: linux-3.5-rc4/include/linux/mm.h
===================================================================
@@ -614,12 +614,13 @@ static inline struct page *kmalloc_secti/* This will make the necessary allocations eventually. */returnsparse_mem_map_populate(pnum,nid);}-staticvoid__kfree_section_memmap(structpage*memmap,unsignedlongnr_pages)+staticvoid__kfree_section_memmap(structpage*page,unsignedlongnr_pages){-return;/* XXX: Not implemented yet */+vmemmap_kfree(page,nr_pages);
Hmm, I think you try to free the memory allocated in kmalloc_section_memmap().
+ } else {
+ next = pmd_addr_end(addr, end);
+
+ pmd = pmd_offset(pud, addr);
+ if (pmd_none(*pmd))
+ return next;
+
+ *pp = pmd_page(*pmd);
+ pmd_clear(pmd);
+ }
+
+ return next;
+}
+
+void __meminit
+vmemmap_kfree(struct page *memmap, unsigned long nr_pages)
+{
+ unsigned long addr = (unsigned long)memmap;
+ unsigned long end = (unsigned long)(memmap + nr_pages);
+ unsigned long next;
+ unsigned int order;
+ struct page *page;
+
+ for (; addr < end; addr = next) {
+ page = NULL;
+ next = find_and_clear_pte_page(addr, end, &page);
+ if (!page)
+ continue;
+
+ if (is_vmalloc_addr(page_address(page)))
+ vfree(page_address(page));
Hmm, the memory is allocated in vmemmap_alloc_block(), and the address
can not be vmalloc address.
+ else {
+ order = next - addr;
+ free_pages((unsigned long)page_address(page),
+ get_order(order));
OOPS. I think we cannot free pages here.
sizeof(struct page) is less than PAGE_SIZE. We store more than one struct
page in the same page. If you free it here while the other struct page
is in use, it is very dangerous.
quoted hunk
+ }
+ }
+}
+
+void __meminit
+vmemmap_free_bootmem(struct page *memmap, unsigned long nr_pages)
+{
+ unsigned long addr = (unsigned long)memmap;
+ unsigned long end = (unsigned long)(memmap + nr_pages);
+ unsigned long next;
+ struct page *page;
+ unsigned long magic;
+
+ for (; addr < end; addr = next) {
+ page = NULL;
+ next = find_and_clear_pte_page(addr, end, &page);
+ if (!page)
+ continue;
+
+ magic = (unsigned long) page->lru.next;
+ if (magic == SECTION_INFO)
+ put_page_bootmem(page);
+ }
+}
+
void __meminit
register_page_bootmem_memmap(unsigned long section_nr, struct page *start_page,
unsigned long size)
Index: linux-3.5-rc4/mm/memory_hotplug.c
===================================================================
@@ -303,6 +303,8 @@ static int __meminit __add_section(int n#ifdef CONFIG_SPARSEMEM_VMEMMAP
I think this line can be removed now.
Thanks
Wen Congyang
quoted hunk
static int __remove_section(struct zone *zone, struct mem_section *ms)
{
+ unsigned long flags;
+ struct pglist_data *pgdat = zone->zone_pgdat;
int ret;
if (!valid_section(ms))
@@ -310,6 +312,9 @@ static int __remove_section(struct zone ret = unregister_memory_section(ms);+ pgdat_resize_lock(pgdat, &flags);+ sparse_remove_one_section(zone, ms);+ pgdat_resize_unlock(pgdat, &flags); return ret; } #else
I don't think that all pages of virtual mapping in removed memory can be
freed, since page which type is MIX_SECTION_INFO is difficult to free.
So, the patch only frees page which type is SECTION_INFO at first.
CC: David Rientjes <rientjes@google.com>
CC: Jiang Liu <redacted>
CC: Len Brown <redacted>
CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
CC: Paul Mackerras <redacted>
CC: Christoph Lameter <redacted>
Cc: Minchan Kim <redacted>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: KOSAKI Motohiro <redacted>
CC: Wen Congyang <redacted>
Signed-off-by: Yasuaki Ishimatsu <redacted>
---
arch/x86/mm/init_64.c | 91 ++++++++++++++++++++++++++++++++++++++++++++++++++
include/linux/mm.h | 2 +
mm/memory_hotplug.c | 5 ++
mm/sparse.c | 5 +-
4 files changed, 101 insertions(+), 2 deletions(-)
Index: linux-3.5-rc4/include/linux/mm.h
===================================================================
@@ -614,12 +614,13 @@ static inline struct page *kmalloc_secti/* This will make the necessary allocations eventually. */returnsparse_mem_map_populate(pnum,nid);}-staticvoid__kfree_section_memmap(structpage*memmap,unsignedlongnr_pages)+staticvoid__kfree_section_memmap(structpage*page,unsignedlongnr_pages){-return;/* XXX: Not implemented yet */+vmemmap_kfree(page,nr_pages);
Hmm, I think you try to free the memory allocated in kmalloc_section_memmap().
Hmm, which function is the memory you try to free allocated in?
The function try to free memory allocated from bootmem. The memory has
been registered by get_page_bootmem(). So we can free the memory by
put_page_bootmem().
quoted
#else
static struct page *__kmalloc_section_memmap(unsigned long nr_pages)
Index: linux-3.5-rc4/arch/x86/mm/init_64.c
===================================================================
+ } else {
+ next = pmd_addr_end(addr, end);
+
+ pmd = pmd_offset(pud, addr);
+ if (pmd_none(*pmd))
+ return next;
+
+ *pp = pmd_page(*pmd);
+ pmd_clear(pmd);
+ }
+
+ return next;
+}
+
+void __meminit
+vmemmap_kfree(struct page *memmap, unsigned long nr_pages)
+{
+ unsigned long addr = (unsigned long)memmap;
+ unsigned long end = (unsigned long)(memmap + nr_pages);
+ unsigned long next;
+ unsigned int order;
+ struct page *page;
+
+ for (; addr < end; addr = next) {
+ page = NULL;
+ next = find_and_clear_pte_page(addr, end, &page);
+ if (!page)
+ continue;
+
+ if (is_vmalloc_addr(page_address(page)))
+ vfree(page_address(page));
Hmm, the memory is allocated in vmemmap_alloc_block(), and the address
can not be vmalloc address.
Does it mean the if sentence is unnecessary?
quoted
+ else {
+ order = next - addr;
+ free_pages((unsigned long)page_address(page),
+ get_order(order));
OOPS. I think we cannot free pages here.
sizeof(struct page) is less than PAGE_SIZE. We store more than one struct
page in the same page. If you free it here while the other struct page
is in use, it is very dangerous.
The memory has page structures for hot-removed memory. So nobody is using
these pages, since the hot-removed memory has been offlined.
quoted
+ }
+ }
+}
+
+void __meminit
+vmemmap_free_bootmem(struct page *memmap, unsigned long nr_pages)
+{
+ unsigned long addr = (unsigned long)memmap;
+ unsigned long end = (unsigned long)(memmap + nr_pages);
+ unsigned long next;
+ struct page *page;
+ unsigned long magic;
+
+ for (; addr < end; addr = next) {
+ page = NULL;
+ next = find_and_clear_pte_page(addr, end, &page);
+ if (!page)
+ continue;
+
+ magic = (unsigned long) page->lru.next;
+ if (magic == SECTION_INFO)
+ put_page_bootmem(page);
+ }
+}
+
void __meminit
register_page_bootmem_memmap(unsigned long section_nr, struct page *start_page,
unsigned long size)
Index: linux-3.5-rc4/mm/memory_hotplug.c
===================================================================
@@ -303,6 +303,8 @@ static int __meminit __add_section(int n#ifdef CONFIG_SPARSEMEM_VMEMMAP
I think this line can be removed now.
I'll update it.
Thanks,
Yasuaki Ishimatsu
Thanks
Wen Congyang
quoted
static int __remove_section(struct zone *zone, struct mem_section *ms)
{
+ unsigned long flags;
+ struct pglist_data *pgdat = zone->zone_pgdat;
int ret;
if (!valid_section(ms))
@@ -310,6 +312,9 @@ static int __remove_section(struct zone ret = unregister_memory_section(ms);+ pgdat_resize_lock(pgdat, &flags);+ sparse_remove_one_section(zone, ms);+ pgdat_resize_unlock(pgdat, &flags); return ret; } #else
I don't think that all pages of virtual mapping in removed memory can be
freed, since page which type is MIX_SECTION_INFO is difficult to free.
So, the patch only frees page which type is SECTION_INFO at first.
CC: David Rientjes <rientjes@google.com>
CC: Jiang Liu <redacted>
CC: Len Brown <redacted>
CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
CC: Paul Mackerras <redacted>
CC: Christoph Lameter <redacted>
Cc: Minchan Kim <redacted>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: KOSAKI Motohiro <redacted>
CC: Wen Congyang <redacted>
Signed-off-by: Yasuaki Ishimatsu <redacted>
---
arch/x86/mm/init_64.c | 91 ++++++++++++++++++++++++++++++++++++++++++++++++++
include/linux/mm.h | 2 +
mm/memory_hotplug.c | 5 ++
mm/sparse.c | 5 +-
4 files changed, 101 insertions(+), 2 deletions(-)
Index: linux-3.5-rc4/include/linux/mm.h
===================================================================
@@ -614,12 +614,13 @@ static inline struct page *kmalloc_secti/* This will make the necessary allocations eventually. */returnsparse_mem_map_populate(pnum,nid);}-staticvoid__kfree_section_memmap(structpage*memmap,unsignedlongnr_pages)+staticvoid__kfree_section_memmap(structpage*page,unsignedlongnr_pages){-return;/* XXX: Not implemented yet */+vmemmap_kfree(page,nr_pages);
Hmm, I think you try to free the memory allocated in kmalloc_section_memmap().
Hmm, which function is the memory you try to free allocated in?
The function try to free memory allocated from bootmem. The memory has
been registered by get_page_bootmem(). So we can free the memory by
put_page_bootmem().
OK, I will read these codes, and check it.
quoted
quoted
#else
static struct page *__kmalloc_section_memmap(unsigned long nr_pages)
Index: linux-3.5-rc4/arch/x86/mm/init_64.c
===================================================================
+ } else {
+ next = pmd_addr_end(addr, end);
+
+ pmd = pmd_offset(pud, addr);
+ if (pmd_none(*pmd))
+ return next;
+
+ *pp = pmd_page(*pmd);
+ pmd_clear(pmd);
+ }
+
+ return next;
+}
+
+void __meminit
+vmemmap_kfree(struct page *memmap, unsigned long nr_pages)
+{
+ unsigned long addr = (unsigned long)memmap;
+ unsigned long end = (unsigned long)(memmap + nr_pages);
+ unsigned long next;
+ unsigned int order;
+ struct page *page;
+
+ for (; addr < end; addr = next) {
+ page = NULL;
+ next = find_and_clear_pte_page(addr, end, &page);
+ if (!page)
+ continue;
+
+ if (is_vmalloc_addr(page_address(page)))
+ vfree(page_address(page));
Hmm, the memory is allocated in vmemmap_alloc_block(), and the address
can not be vmalloc address.
Does it mean the if sentence is unnecessary?
quoted
quoted
+ else {
+ order = next - addr;
+ free_pages((unsigned long)page_address(page),
+ get_order(order));
OOPS. I think we cannot free pages here.
sizeof(struct page) is less than PAGE_SIZE. We store more than one struct
page in the same page. If you free it here while the other struct page
is in use, it is very dangerous.
The memory has page structures for hot-removed memory. So nobody is using
these pages, since the hot-removed memory has been offlined.
The memory has page structures for hot-removed memory, but it may contain
page structures for the other hot-added memory.
IIUC, If we use sparse-vmemmap, all page structures is stored here.
Thanks
Wen Congyang
quoted
quoted
+ }
+ }
+}
+
+void __meminit
+vmemmap_free_bootmem(struct page *memmap, unsigned long nr_pages)
+{
+ unsigned long addr = (unsigned long)memmap;
+ unsigned long end = (unsigned long)(memmap + nr_pages);
+ unsigned long next;
+ struct page *page;
+ unsigned long magic;
+
+ for (; addr < end; addr = next) {
+ page = NULL;
+ next = find_and_clear_pte_page(addr, end, &page);
+ if (!page)
+ continue;
+
+ magic = (unsigned long) page->lru.next;
+ if (magic == SECTION_INFO)
+ put_page_bootmem(page);
+ }
+}
+
void __meminit
register_page_bootmem_memmap(unsigned long section_nr, struct page *start_page,
unsigned long size)
Index: linux-3.5-rc4/mm/memory_hotplug.c
===================================================================
@@ -303,6 +303,8 @@ static int __meminit __add_section(int n#ifdef CONFIG_SPARSEMEM_VMEMMAP
I think this line can be removed now.
I'll update it.
Thanks,
Yasuaki Ishimatsu
quoted
Thanks
Wen Congyang
quoted
static int __remove_section(struct zone *zone, struct mem_section *ms)
{
+ unsigned long flags;
+ struct pglist_data *pgdat = zone->zone_pgdat;
int ret;
if (!valid_section(ms))
@@ -310,6 +312,9 @@ static int __remove_section(struct zone ret = unregister_memory_section(ms);+ pgdat_resize_lock(pgdat, &flags);+ sparse_remove_one_section(zone, ms);+ pgdat_resize_unlock(pgdat, &flags); return ret; } #else
I don't think that all pages of virtual mapping in removed memory can be
freed, since page which type is MIX_SECTION_INFO is difficult to free.
So, the patch only frees page which type is SECTION_INFO at first.
CC: David Rientjes <rientjes@google.com>
CC: Jiang Liu <redacted>
CC: Len Brown <redacted>
CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
CC: Paul Mackerras <redacted>
CC: Christoph Lameter <redacted>
Cc: Minchan Kim <redacted>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: KOSAKI Motohiro <redacted>
CC: Wen Congyang <redacted>
Signed-off-by: Yasuaki Ishimatsu <redacted>
---
arch/x86/mm/init_64.c | 91 ++++++++++++++++++++++++++++++++++++++++++++++++++
include/linux/mm.h | 2 +
mm/memory_hotplug.c | 5 ++
mm/sparse.c | 5 +-
4 files changed, 101 insertions(+), 2 deletions(-)
Index: linux-3.5-rc4/include/linux/mm.h
===================================================================
@@ -614,12 +614,13 @@ static inline struct page *kmalloc_secti/* This will make the necessary allocations eventually. */returnsparse_mem_map_populate(pnum,nid);}-staticvoid__kfree_section_memmap(structpage*memmap,unsignedlongnr_pages)+staticvoid__kfree_section_memmap(structpage*page,unsignedlongnr_pages){-return;/* XXX: Not implemented yet */+vmemmap_kfree(page,nr_pages);
Hmm, I think you try to free the memory allocated in kmalloc_section_memmap().
Hmm, which function is the memory you try to free allocated in?
The function try to free memory allocated from bootmem. The memory has
been registered by get_page_bootmem(). So we can free the memory by
put_page_bootmem().
OK, I will read these codes, and check it.
quoted
quoted
quoted
#else
static struct page *__kmalloc_section_memmap(unsigned long nr_pages)
Index: linux-3.5-rc4/arch/x86/mm/init_64.c
===================================================================
+ } else {
+ next = pmd_addr_end(addr, end);
+
+ pmd = pmd_offset(pud, addr);
+ if (pmd_none(*pmd))
+ return next;
+
+ *pp = pmd_page(*pmd);
+ pmd_clear(pmd);
+ }
+
+ return next;
+}
+
+void __meminit
+vmemmap_kfree(struct page *memmap, unsigned long nr_pages)
+{
+ unsigned long addr = (unsigned long)memmap;
+ unsigned long end = (unsigned long)(memmap + nr_pages);
+ unsigned long next;
+ unsigned int order;
+ struct page *page;
+
+ for (; addr < end; addr = next) {
+ page = NULL;
+ next = find_and_clear_pte_page(addr, end, &page);
+ if (!page)
+ continue;
+
+ if (is_vmalloc_addr(page_address(page)))
+ vfree(page_address(page));
Hmm, the memory is allocated in vmemmap_alloc_block(), and the address
can not be vmalloc address.
Does it mean the if sentence is unnecessary?
quoted
quoted
+ else {
+ order = next - addr;
+ free_pages((unsigned long)page_address(page),
+ get_order(order));
OOPS. I think we cannot free pages here.
sizeof(struct page) is less than PAGE_SIZE. We store more than one struct
page in the same page. If you free it here while the other struct page
is in use, it is very dangerous.
The memory has page structures for hot-removed memory. So nobody is using
these pages, since the hot-removed memory has been offlined.
The memory has page structures for hot-removed memory, but it may contain
page structures for the other hot-added memory.
Yes. There may be such corner case. But when does the corner case appear?
When removed memory is not aligned to PMD_SIZE/PAGE_SIZE, does the corner
case appear? Do you know it?
Thank,
Yasuaki Ishimatsu
IIUC, If we use sparse-vmemmap, all page structures is stored here.
Thanks
Wen Congyang
quoted
quoted
quoted
+ }
+ }
+}
+
+void __meminit
+vmemmap_free_bootmem(struct page *memmap, unsigned long nr_pages)
+{
+ unsigned long addr = (unsigned long)memmap;
+ unsigned long end = (unsigned long)(memmap + nr_pages);
+ unsigned long next;
+ struct page *page;
+ unsigned long magic;
+
+ for (; addr < end; addr = next) {
+ page = NULL;
+ next = find_and_clear_pte_page(addr, end, &page);
+ if (!page)
+ continue;
+
+ magic = (unsigned long) page->lru.next;
+ if (magic == SECTION_INFO)
+ put_page_bootmem(page);
+ }
+}
+
void __meminit
register_page_bootmem_memmap(unsigned long section_nr, struct page *start_page,
unsigned long size)
Index: linux-3.5-rc4/mm/memory_hotplug.c
===================================================================
@@ -303,6 +303,8 @@ static int __meminit __add_section(int n#ifdef CONFIG_SPARSEMEM_VMEMMAP
I think this line can be removed now.
I'll update it.
Thanks,
Yasuaki Ishimatsu
quoted
Thanks
Wen Congyang
quoted
static int __remove_section(struct zone *zone, struct mem_section *ms)
{
+ unsigned long flags;
+ struct pglist_data *pgdat = zone->zone_pgdat;
int ret;
if (!valid_section(ms))
@@ -310,6 +312,9 @@ static int __remove_section(struct zone ret = unregister_memory_section(ms);+ pgdat_resize_lock(pgdat, &flags);+ sparse_remove_one_section(zone, ms);+ pgdat_resize_unlock(pgdat, &flags); return ret; } #else
I don't think that all pages of virtual mapping in removed memory can be
freed, since page which type is MIX_SECTION_INFO is difficult to free.
So, the patch only frees page which type is SECTION_INFO at first.
CC: David Rientjes <rientjes@google.com>
CC: Jiang Liu <redacted>
CC: Len Brown <redacted>
CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
CC: Paul Mackerras <redacted>
CC: Christoph Lameter <redacted>
Cc: Minchan Kim <redacted>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: KOSAKI Motohiro <redacted>
CC: Wen Congyang <redacted>
Signed-off-by: Yasuaki Ishimatsu <redacted>
---
arch/x86/mm/init_64.c | 91 ++++++++++++++++++++++++++++++++++++++++++++++++++
include/linux/mm.h | 2 +
mm/memory_hotplug.c | 5 ++
mm/sparse.c | 5 +-
4 files changed, 101 insertions(+), 2 deletions(-)
Index: linux-3.5-rc4/include/linux/mm.h
===================================================================
@@ -614,12 +614,13 @@ static inline struct page *kmalloc_secti/* This will make the necessary allocations eventually. */returnsparse_mem_map_populate(pnum,nid);}-staticvoid__kfree_section_memmap(structpage*memmap,unsignedlongnr_pages)+staticvoid__kfree_section_memmap(structpage*page,unsignedlongnr_pages){-return;/* XXX: Not implemented yet */+vmemmap_kfree(page,nr_pages);
Hmm, I think you try to free the memory allocated in kmalloc_section_memmap().
Hmm, which function is the memory you try to free allocated in?
The function try to free memory allocated from bootmem. The memory has
been registered by get_page_bootmem(). So we can free the memory by
put_page_bootmem().
OK, I will read these codes, and check it.
quoted
quoted
quoted
#else
static struct page *__kmalloc_section_memmap(unsigned long nr_pages)
Index: linux-3.5-rc4/arch/x86/mm/init_64.c
===================================================================
+ } else {
+ next = pmd_addr_end(addr, end);
+
+ pmd = pmd_offset(pud, addr);
+ if (pmd_none(*pmd))
+ return next;
+
+ *pp = pmd_page(*pmd);
+ pmd_clear(pmd);
+ }
+
+ return next;
+}
+
+void __meminit
+vmemmap_kfree(struct page *memmap, unsigned long nr_pages)
+{
+ unsigned long addr = (unsigned long)memmap;
+ unsigned long end = (unsigned long)(memmap + nr_pages);
+ unsigned long next;
+ unsigned int order;
+ struct page *page;
+
+ for (; addr < end; addr = next) {
+ page = NULL;
+ next = find_and_clear_pte_page(addr, end, &page);
+ if (!page)
+ continue;
+
+ if (is_vmalloc_addr(page_address(page)))
+ vfree(page_address(page));
Hmm, the memory is allocated in vmemmap_alloc_block(), and the address
can not be vmalloc address.
Does it mean the if sentence is unnecessary?
quoted
quoted
+ else {
+ order = next - addr;
+ free_pages((unsigned long)page_address(page),
+ get_order(order));
OOPS. I think we cannot free pages here.
sizeof(struct page) is less than PAGE_SIZE. We store more than one struct
page in the same page. If you free it here while the other struct page
is in use, it is very dangerous.
The memory has page structures for hot-removed memory. So nobody is using
these pages, since the hot-removed memory has been offlined.
The memory has page structures for hot-removed memory, but it may contain
page structures for the other hot-added memory.
Yes. There may be such corner case. But when does the corner case appear?
When removed memory is not aligned to PMD_SIZE/PAGE_SIZE, does the corner
case appear? Do you know it?
It does not depend whether the removed memory is aligned to PMD_SIZE/PAGE_SIZE.
If PAGE_SIZE % sizeof(struct page) != 0, this case will happen.
Thanks
Wen Congyang
Thank,
Yasuaki Ishimatsu
quoted
IIUC, If we use sparse-vmemmap, all page structures is stored here.
Thanks
Wen Congyang
quoted
quoted
quoted
+ }
+ }
+}
+
+void __meminit
+vmemmap_free_bootmem(struct page *memmap, unsigned long nr_pages)
+{
+ unsigned long addr = (unsigned long)memmap;
+ unsigned long end = (unsigned long)(memmap + nr_pages);
+ unsigned long next;
+ struct page *page;
+ unsigned long magic;
+
+ for (; addr < end; addr = next) {
+ page = NULL;
+ next = find_and_clear_pte_page(addr, end, &page);
+ if (!page)
+ continue;
+
+ magic = (unsigned long) page->lru.next;
+ if (magic == SECTION_INFO)
+ put_page_bootmem(page);
+ }
+}
+
void __meminit
register_page_bootmem_memmap(unsigned long section_nr, struct page *start_page,
unsigned long size)
Index: linux-3.5-rc4/mm/memory_hotplug.c
===================================================================
@@ -303,6 +303,8 @@ static int __meminit __add_section(int n#ifdef CONFIG_SPARSEMEM_VMEMMAP
I think this line can be removed now.
I'll update it.
Thanks,
Yasuaki Ishimatsu
quoted
Thanks
Wen Congyang
quoted
static int __remove_section(struct zone *zone, struct mem_section *ms)
{
+ unsigned long flags;
+ struct pglist_data *pgdat = zone->zone_pgdat;
int ret;
if (!valid_section(ms))
@@ -310,6 +312,9 @@ static int __remove_section(struct zone ret = unregister_memory_section(ms);+ pgdat_resize_lock(pgdat, &flags);+ sparse_remove_one_section(zone, ms);+ pgdat_resize_unlock(pgdat, &flags); return ret; } #else
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Jiang Liu <hidden> Date: 2012-07-11 14:24:37
On 07/11/2012 08:54 AM, Yasuaki Ishimatsu wrote:
Hi Jiang,
2012/07/11 9:21, Jiang Liu wrote:
quoted
On 07/11/2012 08:09 AM, Yasuaki Ishimatsu wrote:
quoted
Hi Jiang,
2012/07/11 1:50, Jiang Liu wrote:
quoted
On 07/10/2012 05:58 PM, Yasuaki Ishimatsu wrote:
quoted
Hi Christoph,
2012/07/10 0:18, Christoph Lameter wrote:
quoted
On Mon, 9 Jul 2012, Yasuaki Ishimatsu wrote:
quoted
Even if you apply these patches, you cannot remove the physical memory
completely since these patches are still under development. I want you to
cooperate to improve the physical memory hot-remove. So please review these
patches and give your comment/idea.
Could you at least give a method on how you want to do physical memory
removal?
We plan to release a dynamic hardware partitionable system. It will be
able to hot remove/add a system board which included memory and cpu.
But as you know, Linux does not support memory hot-remove on x86 box.
So I try to develop it.
Current plan to hot remove system board is to use container driver.
Thus I define the system board in ACPI DSDT table as a container device.
It have supported hot-add a container device. And if container device
has _EJ0 ACPI method, "eject" file to remove the container device is
prepared as follow:
# ls -l /sys/bus/acpi/devices/ACPI0004\:01/eject
--w-------. 1 root root 4096 Jul 10 18:19 /sys/bus/acpi/devices/ACPI0004:01/eject
When I hot-remove the container device, I echo 1 to the file as follow:
#echo 1 > /sys/bus/acpi/devices/ACPI0004\:02/eject
Then acpi_bus_trim() is called. And it calls acpi_memory_device_remove()
for removing memory device. But the code does not do nothing.
So I developed the continuation of the function.
quoted
You would have to remove all objects from the range you want to
physically remove. That is only possible under special circumstances and
with a limited set of objects. Even if you exclusively use ZONE_MOVEABLE
you still may get cases where pages are pinned for a long time.
I know it. So my memory hot-remove plan is as follows:
1. hot-added a system board
All memory which included the system board is offline.
2. online the memory as removable page
The function has not supported yet. It is being developed by Lai as follow:
http://lkml.indiana.edu/hypermail/linux/kernel/1207.0/01478.html
If it is supported, I will be able to create movable memory.
3. hot-remove the memory by container device's eject file
We have implemented a prototype to do physical node (mem + CPU + IOH) hotplug
for Itanium and is now porting it to x86. But with currently solution, memory
hotplug functionality may cause 10-20% performance decrease because we concentrate
all DMA/Normal memory to the first NUMA node, and all other NUMA nodes only
hosts ZONE_MOVABLE. We are working on solution to minimize the performance
drop now.
Thank you for your interesting response.
I have a question. How do you move all other NUMA nodes to ZONE_MOVABLE?
To use ZONE_MOVABLE, we need to use boot options like kernelcore or movablecore.
But it is not enough, since the requested amount is spread evenly throughout
all nodes in the system. So I think we do not have way to move all other NUMA
node to ZONE_MOVABLE.
We have modified the ZONE_MOVABLE spreading and bootmem allocation. If the kernelcore
or movablecore kernel parameters are present, we follow current behavior. If those
parameter are absent and the platform supports physical hotplug, we will concentrate
DMA/NORMAL memory to specific nodes.
That's interesting. I want to know more details, if you do not mind.
Current kernel doesn't do the behavior, does it? So I think you have some
patches for changing the behavior. Will you merge these patches into
community kernel?
Yeah, we do have patches for that. But it's still prototype, still much work
needed before sending them to the community.
Currently I'm trying to send out patches for an ACPI based system device
hotplug framework, which will support processor, memory, IOH and node hotplug
in a unified way. After that, I will prepare the memory hotplug code.
Thanks!
Gerry
I know the firmware_map_*() calls use inclusive end addresses
internally, but do we really need to expose them? Both of the callers
you mentioned do:
firmware_map_add_hotplug(start, start + size - 1, "System RAM");
or
firmware_map_add_early(entry->addr,
entry->addr + entry->size - 1,
e820_type_to_string(entry->type));
So it seems a _bit_ silly to keep all of the callers doing this size-1
thing. I also noted that the new caller that you added does the same
thing. Could we just change the external calling convention to be
exclusive?
BTW, this patch should probably be first in your series. It's a real
bugfix.
I know the firmware_map_*() calls use inclusive end addresses
internally, but do we really need to expose them? Both of the callers
you mentioned do:
firmware_map_add_hotplug(start, start + size - 1, "System RAM");
or
firmware_map_add_early(entry->addr,
entry->addr + entry->size - 1,
e820_type_to_string(entry->type));
So it seems a _bit_ silly to keep all of the callers doing this size-1
thing. I also noted that the new caller that you added does the same
thing. Could we just change the external calling convention to be
exclusive?
Thank you for your comment.
Does the following patch include your comment? If O.K., I will separate
the patch from the series and send it for bug fix.
---
arch/x86/kernel/e820.c | 2 +-
drivers/firmware/memmap.c | 8 ++++----
2 files changed, 5 insertions(+), 5 deletions(-)
Index: linux-next/arch/x86/kernel/e820.c
===================================================================
From: Dave Hansen <hidden> Date: 2012-07-12 13:41:18
On 07/11/2012 09:52 PM, Yasuaki Ishimatsu wrote:
Does the following patch include your comment? If O.K., I will separate
the patch from the series and send it for bug fix.
Looks sane to me. It does now mean that the calling conventions for
some of the other firmware_map*() functions are different, but I think
that's OK since they're only used internally to memmap.c.
@@ -233,6 +233,7 @@ static inline int is_mem_section_removabexternintmem_online_node(intnid);externintadd_memory(intnid,u64start,u64size);externintarch_add_memory(intnid,u64start,u64size);+externintremove_memory(intnid,u64start,u64size);
Here should be:
#ifdef CONFIG_MEMORY_HOTREMOVE
extern int remove_memory(int nid, u64 start, u64 size);
#else
static int inline remove_memory(int nid, u64 start, u64 size)
{
return -EBUSY;
}
#endif
quoted hunk
extern int offline_memory(u64 start, u64 size);
extern int sparse_add_one_section(struct zone *zone, unsigned long start_pfn,
int nr_pages);
Index: linux-3.5-rc6/mm/memory_hotplug.c
===================================================================
We only need to implement this function when CONFIG_MEMORY_HOTREMOVE
is defined here.
Thanks
Wen Congyang
quoted hunk
+
+
#ifdef CONFIG_MEMORY_HOTREMOVE
/*
* A free page on the buddy free lists (not the per-cpu lists) has PageBuddy
Index: linux-3.5-rc6/drivers/base/memory.c
===================================================================
@@ -452,12 +453,35 @@ static int acpi_memory_device_add(structstaticintacpi_memory_device_remove(structacpi_device*device,inttype){structacpi_memory_device*mem_device=NULL;-+structacpi_memory_info*info,*tmp;+intresult;+intnode;if(!device||!acpi_driver_data(device))return-EINVAL;mem_device=acpi_driver_data(device);++node=acpi_get_node(mem_device->device->handle);++list_for_each_entry_safe(info,tmp,&mem_device->res_list,list){+if(!info->enabled)+continue;++if(!is_memblk_offline(info->start_addr,info->length)){+result=offline_memory(info->start_addr,info->length);+if(result)+returnresult;+}++result=remove_memory(node,info->start_addr,info->length);
The user may online the memory between offline_memory() and remove_memory().
So I think we should lock memory hotplug before check the memory's status
and release it after remove_memory().
Thanks
Wen Congyang
Since applying a patch(de7f0cba96786c), release_mem_region() has been changed
as called in PAGES_PER_SECTION chunks because register_memory_resource() is
called in PAGES_PER_SECTION chunks by add_memory(). But it seems firmware
dependency. If CRS are written in the PAGES_PER_SECTION chunks in ACPI DSDT
Table, register_memory_resource() is called in PAGES_PER_SECTION chunks.
But if CRS are written in the DIMM unit in ACPI DSDT Table,
register_memory_resource() is called in DIMM unit. So release_mem_region()
should not be called in PAGES_PER_SECTION chunks. The patch fixes it.
CC: David Rientjes <rientjes@google.com>
CC: Jiang Liu <redacted>
CC: Len Brown <redacted>
CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
CC: Paul Mackerras <redacted>
CC: Christoph Lameter <redacted>
Cc: Minchan Kim <redacted>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: KOSAKI Motohiro <redacted>
CC: Wen Congyang <redacted>
Signed-off-by: Yasuaki Ishimatsu <redacted>
---
arch/powerpc/platforms/pseries/hotplug-memory.c | 13 +++++++++----
mm/memory_hotplug.c | 4 ++--
2 files changed, 11 insertions(+), 6 deletions(-)
Index: linux-3.5-rc4/mm/memory_hotplug.c
===================================================================
@@ -358,11 +358,11 @@ int __remove_pages(struct zone *zone, unBUG_ON(phys_start_pfn&~PAGE_SECTION_MASK);BUG_ON(nr_pages%PAGES_PER_SECTION);+release_mem_region(phys_start_pfn<<PAGE_SHIFT,nr_pages*PAGE_SIZE);+sections_to_remove=nr_pages/PAGES_PER_SECTION;for(i=0;i<sections_to_remove;i++){unsignedlongpfn=phys_start_pfn+i*PAGES_PER_SECTION;-release_mem_region(pfn<<PAGE_SHIFT,-PAGES_PER_SECTION<<PAGE_SHIFT);ret=__remove_section(zone,__pfn_to_section(pfn));if(ret)break;
Does the following patch include your comment? If O.K., I will separate
the patch from the series and send it for bug fix.
Looks sane to me. It does now mean that the calling conventions for
some of the other firmware_map*() functions are different, but I think
that's OK since they're only used internally to memmap.c.
Thank you for reviewing my patch.
I'll send the patch.
Thanks,
Yasuaki Ishimatsu
Does the following patch include your comment? If O.K., I will separate
the patch from the series and send it for bug fix.
Looks sane to me. It does now mean that the calling conventions for
some of the other firmware_map*() functions are different, but I think
that's OK since they're only used internally to memmap.c.
Can I add "Reviewed-by: Dave Hansen" to the patch?
Thanks,
Yasuaki Ishimatsu
When (hot)adding memory into system, /sys/firmware/memmap/X/{end, start, type}
sysfs files are created. But there is no code to remove these files. The patch
implements the function to remove them.
Note : The code does not free firmware_map_entry since there is no way to free
memory which is allocated by bootmem.
CC: David Rientjes <rientjes@google.com>
CC: Jiang Liu <redacted>
CC: Len Brown <redacted>
CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
CC: Paul Mackerras <redacted>
CC: Christoph Lameter <redacted>
Cc: Minchan Kim <redacted>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: KOSAKI Motohiro <redacted>
CC: Wen Congyang <redacted>
Signed-off-by: Yasuaki Ishimatsu <redacted>
---
drivers/firmware/memmap.c | 78 ++++++++++++++++++++++++++++++++++++++++++-
include/linux/firmware-map.h | 6 +++
mm/memory_hotplug.c | 6 ++-
3 files changed, 88 insertions(+), 2 deletions(-)
Index: linux-3.5-rc6/mm/memory_hotplug.c
===================================================================
@@ -79,7 +80,22 @@ static const struct sysfs_ops memmap_att.show=memmap_attr_show,};+#define to_memmap_entry(obj) container_of(obj, struct firmware_map_entry, kobj)++staticvoidrelease_firmware_map_entry(structkobject*kobj)+{+structfirmware_map_entry*entry=to_memmap_entry(kobj);+structpage*head_page;++head_page=virt_to_head_page(entry);+if(PageSlab(head_page))+kfree(entry);++/* There is no way to free memory allocated from bootmem*/+}+staticstructkobj_typememmap_ktype={+.release=release_firmware_map_entry,.sysfs_ops=&memmap_attr_ops,.default_attrs=def_attrs,};
@@ -123,6 +139,16 @@ static int firmware_map_add_entry(u64 streturn0;}+/**+*firmware_map_remove_entry()-Doestherealworktoremoveafirmware+*memmapentry.+*@entry:removedentry.+**/+staticinlinevoidfirmware_map_remove_entry(structfirmware_map_entry*entry)+{+list_del(&entry->list);+}+/**Addmemmapentryonsysfs*/
@@ -144,6 +170,31 @@ static int add_sysfs_fw_map_entry(structreturn0;}+/*+*Removememmapentryonsysfs+*/+staticinlinevoidremove_sysfs_fw_map_entry(structfirmware_map_entry*entry)+{+kobject_put(&entry->kobj);+}++/*+*Searchmemmapentry+*/++structfirmware_map_entry*__meminit+find_firmware_map_entry(u64start,u64end,constchar*type)+{+structfirmware_map_entry*entry;++list_for_each_entry(entry,&map_entries,list)+if((entry->start==start)&&(entry->end==end)&&+(!strcmp(entry->type,type)))+returnentry;++returnNULL;+}+/***firmware_map_add_hotplug()-Addsafirmwaremappingentrywhenwedo*memoryhotplug.
@@ -196,6 +247,32 @@ int __init firmware_map_add_early(u64 streturnfirmware_map_add_entry(start,end,type,entry);}+/**+*firmware_map_remove()-removeafirmwaremappingentry+*@start:Startofthememoryrange.+*@end:Endofthememoryrange(inclusive).+*@type:Typeofthememoryrange.+*+*removesafirmwaremappingentry.+*+*Returns0onsuccess,or-EINVALifnoentry.+**/+int__meminitfirmware_map_remove(u64start,u64end,constchar*type)+{+structfirmware_map_entry*entry;++entry=find_firmware_map_entry(start,end,type);+if(!entry)+return-EINVAL;++/* remove the memmap entry */+remove_sysfs_fw_map_entry(entry);++firmware_map_remove_entry(entry);++return0;+}+/**Sysfsfunctions-------------------------------------------------------------*/
When (hot)adding memory into system, /sys/firmware/memmap/X/{end, start, type}
sysfs files are created. But there is no code to remove these files. The patch
implements the function to remove them.
Note : The code does not free firmware_map_entry since there is no way to free
memory which is allocated by bootmem.
CC: David Rientjes <rientjes@google.com>
CC: Jiang Liu <redacted>
CC: Len Brown <redacted>
CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
CC: Paul Mackerras <redacted>
CC: Christoph Lameter <redacted>
Cc: Minchan Kim <redacted>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: KOSAKI Motohiro <redacted>
CC: Wen Congyang <redacted>
Signed-off-by: Yasuaki Ishimatsu <redacted>
---
drivers/firmware/memmap.c | 78 ++++++++++++++++++++++++++++++++++++++++++-
include/linux/firmware-map.h | 6 +++
mm/memory_hotplug.c | 6 ++-
3 files changed, 88 insertions(+), 2 deletions(-)
Index: linux-3.5-rc6/mm/memory_hotplug.c
===================================================================
@@ -79,7 +80,22 @@ static const struct sysfs_ops memmap_att.show=memmap_attr_show,};+#define to_memmap_entry(obj) container_of(obj, struct firmware_map_entry, kobj)++staticvoidrelease_firmware_map_entry(structkobject*kobj)+{+structfirmware_map_entry*entry=to_memmap_entry(kobj);+structpage*head_page;++head_page=virt_to_head_page(entry);+if(PageSlab(head_page))+kfree(entry);++/* There is no way to free memory allocated from bootmem*/+}+staticstructkobj_typememmap_ktype={+.release=release_firmware_map_entry,.sysfs_ops=&memmap_attr_ops,.default_attrs=def_attrs,};
@@ -123,6 +139,16 @@ static int firmware_map_add_entry(u64 streturn0;}+/**+*firmware_map_remove_entry()-Doestherealworktoremoveafirmware+*memmapentry.+*@entry:removedentry.+**/+staticinlinevoidfirmware_map_remove_entry(structfirmware_map_entry*entry)+{+list_del(&entry->list);+}+/**Addmemmapentryonsysfs*/
@@ -144,6 +170,31 @@ static int add_sysfs_fw_map_entry(structreturn0;}+/*+*Removememmapentryonsysfs+*/+staticinlinevoidremove_sysfs_fw_map_entry(structfirmware_map_entry*entry)+{+kobject_put(&entry->kobj);+}++/*+*Searchmemmapentry+*/++structfirmware_map_entry*__meminit+find_firmware_map_entry(u64start,u64end,constchar*type)+{+structfirmware_map_entry*entry;++list_for_each_entry(entry,&map_entries,list)+if((entry->start==start)&&(entry->end==end)&&+(!strcmp(entry->type,type)))+returnentry;++returnNULL;+}+/***firmware_map_add_hotplug()-Addsafirmwaremappingentrywhenwedo*memoryhotplug.
@@ -196,6 +247,32 @@ int __init firmware_map_add_early(u64 streturnfirmware_map_add_entry(start,end,type,entry);}+/**+*firmware_map_remove()-removeafirmwaremappingentry+*@start:Startofthememoryrange.+*@end:Endofthememoryrange(inclusive).+*@type:Typeofthememoryrange.+*+*removesafirmwaremappingentry.+*+*Returns0onsuccess,or-EINVALifnoentry.+**/+int__meminitfirmware_map_remove(u64start,u64end,constchar*type)+{+structfirmware_map_entry*entry;++entry=find_firmware_map_entry(start,end,type);+if(!entry)+return-EINVAL;++/* remove the memmap entry */+remove_sysfs_fw_map_entry(entry);++firmware_map_remove_entry(entry);
You should call firmware_map_remove_entry() before remove_sysfs_fw_map_entry(),
otherwise it will cause kernel panicked(entry may be freed in remove_sysfs_fw_map_entry()).
Thanks
Wen Congyang
When (hot)adding memory into system, /sys/firmware/memmap/X/{end, start, type}
sysfs files are created. But there is no code to remove these files. The patch
implements the function to remove them.
Note : The code does not free firmware_map_entry since there is no way to free
memory which is allocated by bootmem.
CC: David Rientjes <rientjes@google.com>
CC: Jiang Liu <redacted>
CC: Len Brown <redacted>
CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
CC: Paul Mackerras <redacted>
CC: Christoph Lameter <redacted>
Cc: Minchan Kim <redacted>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: KOSAKI Motohiro <redacted>
CC: Wen Congyang <redacted>
Signed-off-by: Yasuaki Ishimatsu <redacted>
---
drivers/firmware/memmap.c | 78 ++++++++++++++++++++++++++++++++++++++++++-
include/linux/firmware-map.h | 6 +++
mm/memory_hotplug.c | 6 ++-
3 files changed, 88 insertions(+), 2 deletions(-)
Index: linux-3.5-rc6/mm/memory_hotplug.c
===================================================================
@@ -79,7 +80,22 @@ static const struct sysfs_ops memmap_att.show=memmap_attr_show,};+#define to_memmap_entry(obj) container_of(obj, struct firmware_map_entry, kobj)++staticvoidrelease_firmware_map_entry(structkobject*kobj)+{+structfirmware_map_entry*entry=to_memmap_entry(kobj);+structpage*head_page;++head_page=virt_to_head_page(entry);+if(PageSlab(head_page))+kfree(entry);++/* There is no way to free memory allocated from bootmem*/+}+staticstructkobj_typememmap_ktype={+.release=release_firmware_map_entry,.sysfs_ops=&memmap_attr_ops,.default_attrs=def_attrs,};
@@ -123,6 +139,16 @@ static int firmware_map_add_entry(u64 streturn0;}+/**+*firmware_map_remove_entry()-Doestherealworktoremoveafirmware+*memmapentry.+*@entry:removedentry.+**/+staticinlinevoidfirmware_map_remove_entry(structfirmware_map_entry*entry)+{+list_del(&entry->list);+}+/**Addmemmapentryonsysfs*/
@@ -144,6 +170,31 @@ static int add_sysfs_fw_map_entry(structreturn0;}+/*+*Removememmapentryonsysfs+*/+staticinlinevoidremove_sysfs_fw_map_entry(structfirmware_map_entry*entry)+{+kobject_put(&entry->kobj);+}++/*+*Searchmemmapentry+*/++structfirmware_map_entry*__meminit+find_firmware_map_entry(u64start,u64end,constchar*type)+{+structfirmware_map_entry*entry;++list_for_each_entry(entry,&map_entries,list)+if((entry->start==start)&&(entry->end==end)&&+(!strcmp(entry->type,type)))+returnentry;++returnNULL;+}+/***firmware_map_add_hotplug()-Addsafirmwaremappingentrywhenwedo*memoryhotplug.
@@ -196,6 +247,32 @@ int __init firmware_map_add_early(u64 streturnfirmware_map_add_entry(start,end,type,entry);}+/**+*firmware_map_remove()-removeafirmwaremappingentry+*@start:Startofthememoryrange.+*@end:Endofthememoryrange(inclusive).+*@type:Typeofthememoryrange.+*+*removesafirmwaremappingentry.+*+*Returns0onsuccess,or-EINVALifnoentry.+**/+int__meminitfirmware_map_remove(u64start,u64end,constchar*type)+{+structfirmware_map_entry*entry;++entry=find_firmware_map_entry(start,end,type);+if(!entry)+return-EINVAL;++/* remove the memmap entry */+remove_sysfs_fw_map_entry(entry);++firmware_map_remove_entry(entry);++return0;+}+/**Sysfsfunctions-------------------------------------------------------------*/
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
When (hot)adding memory into system, /sys/firmware/memmap/X/{end, start, type}
sysfs files are created. But there is no code to remove these files. The patch
implements the function to remove them.
Note : The code does not free firmware_map_entry since there is no way to free
memory which is allocated by bootmem.
CC: David Rientjes <rientjes@google.com>
CC: Jiang Liu <redacted>
CC: Len Brown <redacted>
CC: Benjamin Herrenschmidt <benh@kernel.crashing.org>
CC: Paul Mackerras <redacted>
CC: Christoph Lameter <redacted>
Cc: Minchan Kim <redacted>
CC: Andrew Morton <akpm@linux-foundation.org>
CC: KOSAKI Motohiro <redacted>
CC: Wen Congyang <redacted>
Signed-off-by: Yasuaki Ishimatsu <redacted>
---
drivers/firmware/memmap.c | 78 ++++++++++++++++++++++++++++++++++++++++++-
include/linux/firmware-map.h | 6 +++
mm/memory_hotplug.c | 6 ++-
3 files changed, 88 insertions(+), 2 deletions(-)
Index: linux-3.5-rc6/mm/memory_hotplug.c
===================================================================
@@ -79,7 +80,22 @@ static const struct sysfs_ops memmap_att.show=memmap_attr_show,};+#define to_memmap_entry(obj) container_of(obj, struct firmware_map_entry, kobj)++staticvoidrelease_firmware_map_entry(structkobject*kobj)+{+structfirmware_map_entry*entry=to_memmap_entry(kobj);+structpage*head_page;++head_page=virt_to_head_page(entry);+if(PageSlab(head_page))+kfree(entry);++/* There is no way to free memory allocated from bootmem*/+}+staticstructkobj_typememmap_ktype={+.release=release_firmware_map_entry,.sysfs_ops=&memmap_attr_ops,.default_attrs=def_attrs,};
@@ -123,6 +139,16 @@ static int firmware_map_add_entry(u64 streturn0;}+/**+*firmware_map_remove_entry()-Doestherealworktoremoveafirmware+*memmapentry.+*@entry:removedentry.+**/+staticinlinevoidfirmware_map_remove_entry(structfirmware_map_entry*entry)+{+list_del(&entry->list);+}+/**Addmemmapentryonsysfs*/
@@ -144,6 +170,31 @@ static int add_sysfs_fw_map_entry(structreturn0;}+/*+*Removememmapentryonsysfs+*/+staticinlinevoidremove_sysfs_fw_map_entry(structfirmware_map_entry*entry)+{+kobject_put(&entry->kobj);+}++/*+*Searchmemmapentry+*/++structfirmware_map_entry*__meminit+find_firmware_map_entry(u64start,u64end,constchar*type)+{+structfirmware_map_entry*entry;++list_for_each_entry(entry,&map_entries,list)+if((entry->start==start)&&(entry->end==end)&&+(!strcmp(entry->type,type)))+returnentry;++returnNULL;+}+/***firmware_map_add_hotplug()-Addsafirmwaremappingentrywhenwedo*memoryhotplug.
@@ -196,6 +247,32 @@ int __init firmware_map_add_early(u64 streturnfirmware_map_add_entry(start,end,type,entry);}+/**+*firmware_map_remove()-removeafirmwaremappingentry+*@start:Startofthememoryrange.+*@end:Endofthememoryrange(inclusive).+*@type:Typeofthememoryrange.+*+*removesafirmwaremappingentry.+*+*Returns0onsuccess,or-EINVALifnoentry.+**/+int__meminitfirmware_map_remove(u64start,u64end,constchar*type)+{+structfirmware_map_entry*entry;++entry=find_firmware_map_entry(start,end,type);+if(!entry)+return-EINVAL;++/* remove the memmap entry */+remove_sysfs_fw_map_entry(entry);++firmware_map_remove_entry(entry);
You should call firmware_map_remove_entry() before remove_sysfs_fw_map_entry(),
otherwise it will cause kernel panicked(entry may be freed in remove_sysfs_fw_map_entry()).
You are right. I'll update it.
Thanks,
Yasuaki Ishimatsu
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
@@ -233,6 +233,7 @@ static inline int is_mem_section_removabexternintmem_online_node(intnid);externintadd_memory(intnid,u64start,u64size);externintarch_add_memory(intnid,u64start,u64size);+externintremove_memory(intnid,u64start,u64size);
Here should be:
#ifdef CONFIG_MEMORY_HOTREMOVE
extern int remove_memory(int nid, u64 start, u64 size);
#else
static int inline remove_memory(int nid, u64 start, u64 size)
{
return -EBUSY;
}
#endif
O.K. I'll update it.
Thanks,
Yasuaki Ishimatsu
quoted
extern int offline_memory(u64 start, u64 size);
extern int sparse_add_one_section(struct zone *zone, unsigned long start_pfn,
int nr_pages);
Index: linux-3.5-rc6/mm/memory_hotplug.c
===================================================================
We only need to implement this function when CONFIG_MEMORY_HOTREMOVE
is defined here.
Thanks
Wen Congyang
quoted
+
+
#ifdef CONFIG_MEMORY_HOTREMOVE
/*
* A free page on the buddy free lists (not the per-cpu lists) has PageBuddy
Index: linux-3.5-rc6/drivers/base/memory.c
===================================================================
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
@@ -452,12 +453,35 @@ static int acpi_memory_device_add(structstaticintacpi_memory_device_remove(structacpi_device*device,inttype){structacpi_memory_device*mem_device=NULL;-+structacpi_memory_info*info,*tmp;+intresult;+intnode;if(!device||!acpi_driver_data(device))return-EINVAL;mem_device=acpi_driver_data(device);++node=acpi_get_node(mem_device->device->handle);++list_for_each_entry_safe(info,tmp,&mem_device->res_list,list){+if(!info->enabled)+continue;++if(!is_memblk_offline(info->start_addr,info->length)){+result=offline_memory(info->start_addr,info->length);+if(result)+returnresult;+}++result=remove_memory(node,info->start_addr,info->length);
The user may online the memory between offline_memory() and remove_memory().
So I think we should lock memory hotplug before check the memory's status
and release it after remove_memory().
How about get "mem_block->state_mutex" of removed memory? When offlining
memory, we need to change "memory_block->state" into "MEM_OFFLINE".
In this case, we get mem_block->state_mutex. So I think the mutex lock
is beneficial.
Thanks,
Yasuaki Ishimatsu
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
@@ -452,12 +453,35 @@ static int acpi_memory_device_add(structstaticintacpi_memory_device_remove(structacpi_device*device,inttype){structacpi_memory_device*mem_device=NULL;-+structacpi_memory_info*info,*tmp;+intresult;+intnode;if(!device||!acpi_driver_data(device))return-EINVAL;mem_device=acpi_driver_data(device);++node=acpi_get_node(mem_device->device->handle);++list_for_each_entry_safe(info,tmp,&mem_device->res_list,list){+if(!info->enabled)+continue;++if(!is_memblk_offline(info->start_addr,info->length)){+result=offline_memory(info->start_addr,info->length);+if(result)+returnresult;+}++result=remove_memory(node,info->start_addr,info->length);
The user may online the memory between offline_memory() and remove_memory().
So I think we should lock memory hotplug before check the memory's status
and release it after remove_memory().
How about get "mem_block->state_mutex" of removed memory? When offlining
memory, we need to change "memory_block->state" into "MEM_OFFLINE".
In this case, we get mem_block->state_mutex. So I think the mutex lock
is beneficial.
It is not good idea since remove_memory frees mem_block structure...
Do you have any ideas?
Thanks,
Yasuaki Ishimatsu
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
@@ -452,12 +453,35 @@ static int acpi_memory_device_add(structstaticintacpi_memory_device_remove(structacpi_device*device,inttype){structacpi_memory_device*mem_device=NULL;-+structacpi_memory_info*info,*tmp;+intresult;+intnode;if(!device||!acpi_driver_data(device))return-EINVAL;mem_device=acpi_driver_data(device);++node=acpi_get_node(mem_device->device->handle);++list_for_each_entry_safe(info,tmp,&mem_device->res_list,list){+if(!info->enabled)+continue;++if(!is_memblk_offline(info->start_addr,info->length)){+result=offline_memory(info->start_addr,info->length);+if(result)+returnresult;+}++result=remove_memory(node,info->start_addr,info->length);
The user may online the memory between offline_memory() and remove_memory().
So I think we should lock memory hotplug before check the memory's status
and release it after remove_memory().
How about get "mem_block->state_mutex" of removed memory? When offlining
memory, we need to change "memory_block->state" into "MEM_OFFLINE".
In this case, we get mem_block->state_mutex. So I think the mutex lock
is beneficial.
It is not good idea since remove_memory frees mem_block structure...
Do you have any ideas?
Hmm, split offline_memory() to 2 functions: offline_pages() and __offline_pages()
offline_pages()
lock_memory_hotplug();
__offline_pages();
unlock_memory_hotplug();
and implement remove_memory() like this:
remove_memory()
lock_memory_hotplug()
if (!is_memblk_offline()) {
__offline_pages();
}
// cleanup
unlock_memory_hotplug();
What about this?
Thanks
Wen Congyang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
@@ -452,12 +453,35 @@ static int acpi_memory_device_add(structstaticintacpi_memory_device_remove(structacpi_device*device,inttype){structacpi_memory_device*mem_device=NULL;-+structacpi_memory_info*info,*tmp;+intresult;+intnode;if(!device||!acpi_driver_data(device))return-EINVAL;mem_device=acpi_driver_data(device);++node=acpi_get_node(mem_device->device->handle);++list_for_each_entry_safe(info,tmp,&mem_device->res_list,list){+if(!info->enabled)+continue;++if(!is_memblk_offline(info->start_addr,info->length)){+result=offline_memory(info->start_addr,info->length);+if(result)+returnresult;+}++result=remove_memory(node,info->start_addr,info->length);
The user may online the memory between offline_memory() and remove_memory().
So I think we should lock memory hotplug before check the memory's status
and release it after remove_memory().
How about get "mem_block->state_mutex" of removed memory? When offlining
memory, we need to change "memory_block->state" into "MEM_OFFLINE".
In this case, we get mem_block->state_mutex. So I think the mutex lock
is beneficial.
It is not good idea since remove_memory frees mem_block structure...
Do you have any ideas?
Hmm, split offline_memory() to 2 functions: offline_pages() and __offline_pages()
offline_pages()
lock_memory_hotplug();
__offline_pages();
unlock_memory_hotplug();
and implement remove_memory() like this:
remove_memory()
lock_memory_hotplug()
if (!is_memblk_offline()) {
__offline_pages();
}
// cleanup
unlock_memory_hotplug();
What about this?
I also thought about it once. But a problem remains. Current offilne_pages()
cannot realize the memory has been removed by remove_memory(). So even if
protecting the race by lock_memory_hotplug(), offline_pages() can offline
the removed memory. offline_pages() should have the means to know the memory
was removed. But I don't have good idea.
Thanks,
Yasuaki Ishimatsu
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
@@ -452,12 +453,35 @@ static int acpi_memory_device_add(structstaticintacpi_memory_device_remove(structacpi_device*device,inttype){structacpi_memory_device*mem_device=NULL;-+structacpi_memory_info*info,*tmp;+intresult;+intnode;if(!device||!acpi_driver_data(device))return-EINVAL;mem_device=acpi_driver_data(device);++node=acpi_get_node(mem_device->device->handle);++list_for_each_entry_safe(info,tmp,&mem_device->res_list,list){+if(!info->enabled)+continue;++if(!is_memblk_offline(info->start_addr,info->length)){+result=offline_memory(info->start_addr,info->length);+if(result)+returnresult;+}++result=remove_memory(node,info->start_addr,info->length);
The user may online the memory between offline_memory() and remove_memory().
So I think we should lock memory hotplug before check the memory's status
and release it after remove_memory().
How about get "mem_block->state_mutex" of removed memory? When offlining
memory, we need to change "memory_block->state" into "MEM_OFFLINE".
In this case, we get mem_block->state_mutex. So I think the mutex lock
is beneficial.
It is not good idea since remove_memory frees mem_block structure...
Do you have any ideas?
Hmm, split offline_memory() to 2 functions: offline_pages() and __offline_pages()
offline_pages()
lock_memory_hotplug();
__offline_pages();
unlock_memory_hotplug();
and implement remove_memory() like this:
remove_memory()
lock_memory_hotplug()
if (!is_memblk_offline()) {
__offline_pages();
}
// cleanup
unlock_memory_hotplug();
What about this?
I also thought about it once. But a problem remains. Current offilne_pages()
cannot realize the memory has been removed by remove_memory(). So even if
protecting the race by lock_memory_hotplug(), offline_pages() can offline
the removed memory. offline_pages() should have the means to know the memory
was removed. But I don't have good idea.
We can not online/offline part of memory block, so what about this?
remove_memory()
lock_memory_hotplug()
for each memory block:
if (!is_memblk_offline()) {
__offline_pages();
}
// cleanup
unlock_memory_hotplug();
Thanks
Wen Congyang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
@@ -452,12 +453,35 @@ static int acpi_memory_device_add(structstaticintacpi_memory_device_remove(structacpi_device*device,inttype){structacpi_memory_device*mem_device=NULL;-+structacpi_memory_info*info,*tmp;+intresult;+intnode;if(!device||!acpi_driver_data(device))return-EINVAL;mem_device=acpi_driver_data(device);++node=acpi_get_node(mem_device->device->handle);++list_for_each_entry_safe(info,tmp,&mem_device->res_list,list){+if(!info->enabled)+continue;++if(!is_memblk_offline(info->start_addr,info->length)){+result=offline_memory(info->start_addr,info->length);+if(result)+returnresult;+}++result=remove_memory(node,info->start_addr,info->length);
The user may online the memory between offline_memory() and remove_memory().
So I think we should lock memory hotplug before check the memory's status
and release it after remove_memory().
How about get "mem_block->state_mutex" of removed memory? When offlining
memory, we need to change "memory_block->state" into "MEM_OFFLINE".
In this case, we get mem_block->state_mutex. So I think the mutex lock
is beneficial.
It is not good idea since remove_memory frees mem_block structure...
Do you have any ideas?
Hmm, split offline_memory() to 2 functions: offline_pages() and __offline_pages()
offline_pages()
lock_memory_hotplug();
__offline_pages();
unlock_memory_hotplug();
and implement remove_memory() like this:
remove_memory()
lock_memory_hotplug()
if (!is_memblk_offline()) {
__offline_pages();
}
// cleanup
unlock_memory_hotplug();
What about this?
I also thought about it once. But a problem remains. Current offilne_pages()
cannot realize the memory has been removed by remove_memory(). So even if
protecting the race by lock_memory_hotplug(), offline_pages() can offline
the removed memory. offline_pages() should have the means to know the memory
was removed. But I don't have good idea.
We can not online/offline part of memory block, so what about this?
It seems you do not understand my concern.
When memory_remove() and offline_pages() run to same memory simultaneously,
offline_pages runs to removed memory.
memory_remove() | offline_pages()
-----------------------------------------------------------
lock_memory_hotplug() |
| wait at lock_memory_hotplug()
remove memory |
unlock_memory_hotplug() |
| wake up and start offline_pages()
| offline page
| => but the memory has already removed
| by memory_remove()
In this case, offline_page() may access removed memory.
Thanks,
Yasuaki Ishimatsu
remove_memory()
lock_memory_hotplug()
for each memory block:
if (!is_memblk_offline()) {
__offline_pages();
}
// cleanup
unlock_memory_hotplug();
Thanks
Wen Congyang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
@@ -452,12 +453,35 @@ static int acpi_memory_device_add(structstaticintacpi_memory_device_remove(structacpi_device*device,inttype){structacpi_memory_device*mem_device=NULL;-+structacpi_memory_info*info,*tmp;+intresult;+intnode;if(!device||!acpi_driver_data(device))return-EINVAL;mem_device=acpi_driver_data(device);++node=acpi_get_node(mem_device->device->handle);++list_for_each_entry_safe(info,tmp,&mem_device->res_list,list){+if(!info->enabled)+continue;++if(!is_memblk_offline(info->start_addr,info->length)){+result=offline_memory(info->start_addr,info->length);+if(result)+returnresult;+}++result=remove_memory(node,info->start_addr,info->length);
The user may online the memory between offline_memory() and remove_memory().
So I think we should lock memory hotplug before check the memory's status
and release it after remove_memory().
How about get "mem_block->state_mutex" of removed memory? When offlining
memory, we need to change "memory_block->state" into "MEM_OFFLINE".
In this case, we get mem_block->state_mutex. So I think the mutex lock
is beneficial.
It is not good idea since remove_memory frees mem_block structure...
Do you have any ideas?
Hmm, split offline_memory() to 2 functions: offline_pages() and __offline_pages()
offline_pages()
lock_memory_hotplug();
__offline_pages();
unlock_memory_hotplug();
and implement remove_memory() like this:
remove_memory()
lock_memory_hotplug()
if (!is_memblk_offline()) {
__offline_pages();
}
// cleanup
unlock_memory_hotplug();
What about this?
I also thought about it once. But a problem remains. Current offilne_pages()
cannot realize the memory has been removed by remove_memory(). So even if
protecting the race by lock_memory_hotplug(), offline_pages() can offline
the removed memory. offline_pages() should have the means to know the memory
was removed. But I don't have good idea.
We can not online/offline part of memory block, so what about this?
It seems you do not understand my concern.
When memory_remove() and offline_pages() run to same memory simultaneously,
offline_pages runs to removed memory.
memory_remove() | offline_pages()
-----------------------------------------------------------
lock_memory_hotplug() |
| wait at lock_memory_hotplug()
remove memory |
unlock_memory_hotplug() |
| wake up and start offline_pages()
| offline page
| => but the memory has already removed
| by memory_remove()
In this case, offline_page() may access removed memory.
Yes, in this case, the kernel may panic.
I think we can call pfn_present() in online_pages()/offline_pages()
to check whether the memory is removed.
Thanks
Wen Congyang
Thanks,
Yasuaki Ishimatsu
quoted
remove_memory()
lock_memory_hotplug()
for each memory block:
if (!is_memblk_offline()) {
__offline_pages();
}
// cleanup
unlock_memory_hotplug();
Thanks
Wen Congyang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
@@ -452,12 +453,35 @@ static int acpi_memory_device_add(structstaticintacpi_memory_device_remove(structacpi_device*device,inttype){structacpi_memory_device*mem_device=NULL;-+structacpi_memory_info*info,*tmp;+intresult;+intnode;if(!device||!acpi_driver_data(device))return-EINVAL;mem_device=acpi_driver_data(device);++node=acpi_get_node(mem_device->device->handle);++list_for_each_entry_safe(info,tmp,&mem_device->res_list,list){+if(!info->enabled)+continue;++if(!is_memblk_offline(info->start_addr,info->length)){+result=offline_memory(info->start_addr,info->length);+if(result)+returnresult;+}++result=remove_memory(node,info->start_addr,info->length);
The user may online the memory between offline_memory() and remove_memory().
So I think we should lock memory hotplug before check the memory's status
and release it after remove_memory().
How about get "mem_block->state_mutex" of removed memory? When offlining
memory, we need to change "memory_block->state" into "MEM_OFFLINE".
In this case, we get mem_block->state_mutex. So I think the mutex lock
is beneficial.
It is not good idea since remove_memory frees mem_block structure...
Do you have any ideas?
Hmm, split offline_memory() to 2 functions: offline_pages() and __offline_pages()
offline_pages()
lock_memory_hotplug();
__offline_pages();
unlock_memory_hotplug();
and implement remove_memory() like this:
remove_memory()
lock_memory_hotplug()
if (!is_memblk_offline()) {
__offline_pages();
}
// cleanup
unlock_memory_hotplug();
What about this?
I also thought about it once. But a problem remains. Current offilne_pages()
cannot realize the memory has been removed by remove_memory(). So even if
protecting the race by lock_memory_hotplug(), offline_pages() can offline
the removed memory. offline_pages() should have the means to know the memory
was removed. But I don't have good idea.
We can not online/offline part of memory block, so what about this?
It seems you do not understand my concern.
When memory_remove() and offline_pages() run to same memory simultaneously,
offline_pages runs to removed memory.
memory_remove() | offline_pages()
-----------------------------------------------------------
lock_memory_hotplug() |
| wait at lock_memory_hotplug()
remove memory |
unlock_memory_hotplug() |
| wake up and start offline_pages()
| offline page
| => but the memory has already removed
| by memory_remove()
In this case, offline_page() may access removed memory.
Yes, in this case, the kernel may panic.
I think we can call pfn_present() in online_pages()/offline_pages()
to check whether the memory is removed.
Thank you for good idea. I'll add it.
Thanks,
Yasuaki Ishimatsu
Thanks
Wen Congyang
quoted
Thanks,
Yasuaki Ishimatsu
quoted
remove_memory()
lock_memory_hotplug()
for each memory block:
if (!is_memblk_offline()) {
__offline_pages();
}
// cleanup
unlock_memory_hotplug();
Thanks
Wen Congyang
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>