This is a re-send of the entire patch set with updates made from recent
comments received.
The Dynamic Logical Partitioning (DLPAR) capabilities of the powerpc pseries
platform allows for the addition and removal of resources (i.e. cpus,
memory, pci devices) from a partition. The removal of a resource involves
removing the resource's node from the device tree and then returning the
resource to firmware via the rtas set-indicator call. To add a resource, it
is first obtained from firmware via the rtas set-indicator call and then a
new device tree node is created using the ibm,configure-coinnector rtas call
and added to the device tree.
The following set of patches implements the needed infrastructure to have the
kernel handle the DLPAR addition and removal of memory and cpus (other
DLPAR'able items to follow in future patches). The framework for this is
to create a set of probe/release sysfs files that will facilitate
arch-specific call-outs to handle addition and removal of cpus and memory to
the system.
Patches include in this set:
1/6 - DLPAR infracstructure for powerpc/pseries platform.
2/6 - Move the of_drconf_cell struct to prom.h
3/6 - Create memory probe/release files and the powerpc handlers for them
4/6 - Memory DLPAR handling
5/6 - Create sysfs cpu probe/release files and the powerpc handlers for them
6/6 - CPU DLPAR handling
-Nathan Fontenot
This patch provides the kernel DLPAR infrastructure in a new filed named
dlpar.c. The functionality provided is for acquiring and releasing a resource
from firmware and the parsing of information returned from the
ibm,configure-connector rtas call. Additionally this exports the pSeries
reconfiguration notifier chain so that it can be invoked when device tree
updates are made.
Signed-off-by: Nathan Fontenot <nfont at austin.ibm.com>
---
Index: powerpc/arch/powerpc/platforms/pseries/dlpar.c
===================================================================
Move the definition of the of_drconf_cell struct from numa.c to prom.h. This
is needed so that we can parse the ibm,dynamic-memory device-tree property
when DLPAR adding and removing memory.
Signed-off-by: Nathan Fontenot <nfont at austin.ibm.com>
---
Index: powerpc/arch/powerpc/include/asm/prom.h
===================================================================
This patch creates the release sysfs file for memory and updates the
exisiting probe file so both make arch-specific callouts to handle removing
and adding memory to the system. This also creates the powerpc specific stubs
for handling the arch callouts.
The creation and use of these files are governed by the exisitng
CONFIG_ARCH_MEMORY_PROBE and new CONFIG_ARCH_MEMORY_RELEASE config options.
Signed-off-by: Nathan Fontenot <nfont at austin.ibm.com>
---
Index: powerpc/arch/powerpc/mm/mem.c
===================================================================
@@ -414,6 +414,10 @@def_boolydepends onMEMORY_HOTPLUG+configARCH_MEMORY_RELEASE+def_booly+depends onMEMORY_HOTPLUG+# Some NUMA nodes have memory ranges that span# other nodes. Even though a pfn is valid and# between a node's start and end pfns, it may not
This adds the capability to DLPAR add and remove memory from the kernel. The
patch registers handlers for the arch-specific probe and release memory
callouts to handle addition/removal of memory to the system and the associated
device tree updates.
Signed-off-by: Nathan Fontenot <nfont at austin.ibm.com>
---
Index: powerpc/arch/powerpc/platforms/pseries/dlpar.c
===================================================================
@@ -404,11 +408,189 @@return0;}+#ifdef CONFIG_MEMORY_HOTPLUG++staticstructproperty*clone_property(structproperty*old_prop)+{+structproperty*new_prop;++new_prop=kzalloc((sizeof*new_prop),GFP_KERNEL);+if(!new_prop)+returnNULL;++new_prop->name=kstrdup(old_prop->name,GFP_KERNEL);+new_prop->value=kzalloc(old_prop->length+1,GFP_KERNEL);+if(!new_prop->name||!new_prop->value){+free_property(new_prop);+returnNULL;+}++memcpy(new_prop->value,old_prop->value,old_prop->length);+new_prop->length=old_prop->length;++returnnew_prop;+}++#ifdef CONFIG_ARCH_MEMORY_PROBE++intmemory_probe(u64phys_addr)+{+structdevice_node*dn=NULL;+structproperty*new_prop;+structproperty*old_prop;+structof_drconf_cell*drmem;+constu64*lmb_size;+intnum_entries,i;+intrc=-EINVAL;++if(!phys_addr)+gotomemory_probe_exit;++dn=of_find_node_by_path("/ibm,dynamic-reconfiguration-memory");+if(!dn)+gotomemory_probe_exit;++lmb_size=of_get_property(dn,"ibm,lmb-size",NULL);+if(!lmb_size)+gotomemory_probe_exit;++old_prop=of_find_property(dn,"ibm,dynamic-memory",NULL);+if(!old_prop)+gotomemory_probe_exit;++num_entries=*(u32*)old_prop->value;+drmem=(structof_drconf_cell*)+((char*)old_prop->value+sizeof(u32));++for(i=0;i<num_entries;i++){+u64lmb_end_addr=drmem[i].base_addr+*lmb_size;+if(phys_addr>=drmem[i].base_addr+&&phys_addr<lmb_end_addr)+break;+}++if(i>=num_entries)+gotomemory_probe_exit;++if(drmem[i].flags&DRCONF_MEM_ASSIGNED){+/* This lmb is already adssigned to the system, nothing to do */+rc=0;+gotomemory_probe_exit;+}++rc=acquire_drc(drmem[i].drc_index);+if(rc){+rc=-EINVAL;+gotomemory_probe_exit;+}++new_prop=clone_property(old_prop);+drmem=(structof_drconf_cell*)+((char*)new_prop->value+sizeof(u32));++drmem[i].flags|=DRCONF_MEM_ASSIGNED;+rc=prom_update_property(dn,new_prop,old_prop);+if(rc){+free_property(new_prop);+rc=-EINVAL;+gotomemory_probe_exit;+}++rc=blocking_notifier_call_chain(&pSeries_reconfig_chain,+PSERIES_DRCONF_MEM_ADD,+&drmem[i].base_addr);+if(rc==NOTIFY_BAD){+prom_update_property(dn,old_prop,new_prop);+release_drc(drmem[i].drc_index);+rc=-EINVAL;+}else+rc=0;++memory_probe_exit:+of_node_put(dn);+returnrc;+}++#endif /* CONFIG_ARCH_MEMORY_PROBE */++#ifdef CONFIG_ARCH_MEMORY_RELEASE++staticintmemory_release(constchar*buf,size_tcount)+{+unsignedlongdrc_index;+structdevice_node*dn;+structproperty*new_prop,*old_prop;+structof_drconf_cell*drmem;+intnum_entries;+inti;+intrc=-EINVAL;++rc=strict_strtoul(buf,0,&drc_index);+if(rc)+returnrc;++dn=of_find_node_by_path("/ibm,dynamic-reconfiguration-memory");+if(!dn)+returnrc;++old_prop=of_find_property(dn,"ibm,dynamic-memory",NULL);+if(!old_prop)+gotomemory_release_exit;++num_entries=*(u32*)old_prop->value;+drmem=(structof_drconf_cell*)+((char*)old_prop->value+sizeof(u32));++for(i=0;i<num_entries;i++){+if(drmem[i].drc_index==drc_index)+break;+}++if(i>=num_entries)+gotomemory_release_exit;++new_prop=clone_property(old_prop);+drmem=(structof_drconf_cell*)+((char*)new_prop->value+sizeof(u32));++drmem[i].flags&=~DRCONF_MEM_ASSIGNED;+rc=prom_update_property(dn,new_prop,old_prop);+if(rc){+free_property(new_prop);+rc=-EINVAL;+gotomemory_release_exit;+}++rc=blocking_notifier_call_chain(&pSeries_reconfig_chain,+PSERIES_DRCONF_MEM_REMOVE,+&drmem[i].base_addr);+if(rc!=NOTIFY_BAD)+rc=release_drc(drc_index);++if(rc){+prom_update_property(dn,old_prop,new_prop);+rc=-EINVAL;+}++memory_release_exit:+of_node_put(dn);+returnrc?rc:count;+}+#endif /* CONFIG_ARCH_MEMORY_RELEASE */+#endif /* CONFIG_MEMORY_HOTPLUG */+staticintpseries_dlpar_init(void){if(!machine_is(pseries))return0;+#ifdef CONFIG_ARCH_MEMORY_RELEASE+ppc_md.memory_release=memory_release;+#endif+#ifdef CONFIG_ARCH_MEMORY_PROBE+ppc_md.memory_probe=memory_probe;+#endif+return0;}device_initcall(pseries_dlpar_init);
Create new probe and release sysfs files to facilitate adding and removing
cpus from the system. This also creates the powerpc specific stubs to handle
the arch callouts from writes to the sysfs files.
The creation and use of these files is regulated by the
CONFIG_ARCH_CPU_PROBE_RELEASE option so that only architectures that need the
capability will have the files created.
Signed-off-by: Nathan Fontenot <nfont at austin.ibm.com>
---
Index: powerpc/drivers/base/cpu.c
===================================================================
Register the pseries specific handler for the powerpc architecture handlers
for the cpu probe and release files. This also implements the cpu DLPAR
addition and removal of CPUS from the system.
Signed-off-by: Nathan Fontenot <nfont at asutin.ibm.com>
---
Index: powerpc/arch/powerpc/platforms/pseries/dlpar.c
===================================================================
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2009-10-29 03:10:59
On Wed, 2009-10-28 at 15:53 -0500, Nathan Fontenot wrote:
This patch provides the kernel DLPAR infrastructure in a new filed named
dlpar.c. The functionality provided is for acquiring and releasing a resource
from firmware and the parsing of information returned from the
ibm,configure-connector rtas call. Additionally this exports the pSeries
reconfiguration notifier chain so that it can be invoked when device tree
updates are made.
Signed-off-by: Nathan Fontenot <nfont at austin.ibm.com>
---
Hi Nathan !
Finally I get to review this stuff :-)
So I'm not a huge fan of this workarea static. First a static is in
effect a global name (as far as System.map etc... are concerned) so it
would warrant a better name. Then, do we really want that 4K of BSS
taken even on platforms that don't do dlpar ? Any reason why you don't
just pop a free page with __get_free_page() inside of
configure_connector() ?
I'm wondering whether work_area should be a struct cc_workarea * in the
first place with a char data[] at the end, but that would mean probably
tweaking the offsets... no big deal, up to you.
... should probably all go to something like drivers/of/dynamic.c or at
least for now arch/powerpc/kernel/of_dynamic.c along with everything
related to dynamically adding and removing nodes. I see that potentially
useful for more than just DLPAR (though DLPAR is the only user right
now) and should also all be prefixed with of_*
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2009-10-29 03:15:37
On Wed, 2009-10-28 at 15:55 -0500, Nathan Fontenot wrote:
This patch creates the release sysfs file for memory and updates the
exisiting probe file so both make arch-specific callouts to handle removing
and adding memory to the system. This also creates the powerpc specific stubs
for handling the arch callouts.
The creation and use of these files are governed by the exisitng
CONFIG_ARCH_MEMORY_PROBE and new CONFIG_ARCH_MEMORY_RELEASE config options.
Signed-off-by: Nathan Fontenot <nfont at austin.ibm.com>
---
Is there anybody on linux-mm who needs to Ack this patche ?
Cheers,
Ben.
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2009-10-29 03:27:14
On Wed, 2009-10-28 at 15:58 -0500, Nathan Fontenot wrote:
Create new probe and release sysfs files to facilitate adding and removing
cpus from the system. This also creates the powerpc specific stubs to handle
the arch callouts from writes to the sysfs files.
The creation and use of these files is regulated by the
CONFIG_ARCH_CPU_PROBE_RELEASE option so that only architectures that need the
capability will have the files created.
Signed-off-by: Nathan Fontenot <nfont at austin.ibm.com>
---
Same question as before here... need some external acks from others
doing cpu hotplug.
Cheers,
Ben.
Popping a free page with gfp (or just kmalloc'ing 4K) would avoid the
need for the lock too.
Not kmalloc -- the alignment of the buffer isn't guaranteed when
slub/slab debug is on, and iirc the work area needs to be 4K-aligned.
__get_free_page should be fine, I think.
On Wed, 2009-10-28 at 15:55 -0500, Nathan Fontenot wrote:
quoted
This patch creates the release sysfs file for memory and updates the
exisiting probe file so both make arch-specific callouts to handle removing
and adding memory to the system. This also creates the powerpc specific stubs
for handling the arch callouts.
The creation and use of these files are governed by the exisitng
CONFIG_ARCH_MEMORY_PROBE and new CONFIG_ARCH_MEMORY_RELEASE config options.
Signed-off-by: Nathan Fontenot <nfont at austin.ibm.com>
---
Is there anybody on linux-mm who needs to Ack this patche ?
Not sure. I will cc linux-mm on the next set of updated patches I send out.
-Nathan
On Wed, 2009-10-28 at 15:53 -0500, Nathan Fontenot wrote:
quoted
This patch provides the kernel DLPAR infrastructure in a new filed named
dlpar.c. The functionality provided is for acquiring and releasing a resource
from firmware and the parsing of information returned from the
ibm,configure-connector rtas call. Additionally this exports the pSeries
reconfiguration notifier chain so that it can be invoked when device tree
updates are made.
Signed-off-by: Nathan Fontenot <nfont at austin.ibm.com>
---
Hi Nathan !
Finally I get to review this stuff :-)
So I'm not a huge fan of this workarea static. First a static is in
effect a global name (as far as System.map etc... are concerned) so it
would warrant a better name. Then, do we really want that 4K of BSS
taken even on platforms that don't do dlpar ? Any reason why you don't
just pop a free page with __get_free_page() inside of
configure_connector() ?
I'm not either, having a static buffer and a lock feels like overkill
for this. I tried kmalloc, but that didn't work. I'll try using
__get_free_page.
I'm wondering whether work_area should be a struct cc_workarea * in the
first place with a char data[] at the end, but that would mean probably
tweaking the offsets... no big deal, up to you.
I'll look onto that. Anything that makes this easier to understand is good.
... should probably all go to something like drivers/of/dynamic.c or at
least for now arch/powerpc/kernel/of_dynamic.c along with everything
related to dynamically adding and removing nodes. I see that potentially
useful for more than just DLPAR (though DLPAR is the only user right
now) and should also all be prefixed with of_*
I agree, there should be at least a powerpc generic implementation of these
routines. The reason I put them here is that I am doing some oddities with
the next, child, and sibling pointers since they point to items not yet in
the device tree.
I saw that Grant Likely is doing updates to all of the of_* stuff right now,
would it be ok to have these routines here, renamed as dlpar_*, and look
to merge them in with Grant's updates when he finishes?
+static int pseries_dlpar_init(void)
+{
+ if (!machine_is(pseries))
+ return 0;
+
+ return 0;
+}
+device_initcall(pseries_dlpar_init);
What the point ? :-)
Yeah, its a bit odd looking but later patches actually add code to the init routine
to set up memory probe/release and cpu probe/release handlers.
I'll look to add ifdef's around the initcall for cases where no work is to be done.
-Nathan Fontenot
From: Grant Likely <hidden> Date: 2009-11-02 16:41:15
On Mon, Nov 2, 2009 at 9:27 AM, Nathan Fontenot [off-list ref] wrote:
I saw that Grant Likely is doing updates to all of the of_* stuff right now,
would it be ok to have these routines here, renamed as dlpar_*, and look
to merge them in with Grant's updates when he finishes?
No because then we're stuck with renaming the API at a later date.
Name it what it is, and put it where it belongs. I'll deal with any
merge breakage as it occurs.
g.
On Mon, Nov 2, 2009 at 9:27 AM, Nathan Fontenot [off-list ref] wrote:
quoted
I saw that Grant Likely is doing updates to all of the of_* stuff right now,
would it be ok to have these routines here, renamed as dlpar_*, and look
to merge them in with Grant's updates when he finishes?
No because then we're stuck with renaming the API at a later date.
Name it what it is, and put it where it belongs. I'll deal with any
merge breakage as it occurs.
ok. Would this be better off in powerpc code, or should I go ahead and put it
in something like drivers/of/dynamic.c?
-Nathan Fontenot
From: Grant Likely <hidden> Date: 2009-11-02 16:56:40
On Mon, Nov 2, 2009 at 9:47 AM, Nathan Fontenot [off-list ref] wrot=
e:
Grant Likely wrote:
quoted
On Mon, Nov 2, 2009 at 9:27 AM, Nathan Fontenot [off-list ref]
wrote:
quoted
I saw that Grant Likely is doing updates to all of the of_* stuff right
now,
would it be ok to have these routines here, renamed as dlpar_*, and loo=
k
quoted
quoted
to merge them in with Grant's updates when he finishes?
No because then we're stuck with renaming the API at a later date.
Name it what it is, and put it where it belongs. =A0I'll deal with any
merge breakage as it occurs.
ok. =A0Would this be better off in powerpc code, or should I go ahead and=
put
it
in something like drivers/of/dynamic.c?
drivers/of/dynamic.c sounds fine to me. I can always move them if it
find a better place. Send the patch to me and cc: the
devicetree-discuss@lists.ozlabs.org mailing list.
g.
--=20
Grant Likely, B.Sc., P.Eng.
Secret Lab Technologies Ltd.