[RFC 2/3] zImage: SMP hold

Add SMP secondary hold helper routines to the powerpc zImage bootwrapper.  For
platforms which are entered with multiple cpus.  These routines can be used to
hold the secondary cpus until the kernel is ready for entry.

Signed-off-by: Geoff Levand <redacted>

---
 arch/powerpc/boot/main.c |   42 ++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 42 insertions(+)

--- ps3-linux-dev.orig/arch/powerpc/boot/main.c
+++ ps3-linux-dev/arch/powerpc/boot/main.c

@@ -290,6 +290,44 @@ static void set_cmdline(char *buf)
 		setprop(devp, "bootargs", buf, strlen(buf) + 1);
 }
 
+static volatile kernel_entry_t smp_secondary_entry = 0;
+
+/**
+ * smp_secondary_hold - Hold any secondary cpus until kernel is ready to enter.
+ * @cpu_id: Hardware cpu id.
+ *
+ * Called from the early entry code.
+ */
+
+void smp_secondary_hold(unsigned int cpu_id)
+{
+	while(!smp_secondary_entry)
+		(void)0;
+
+	printf("%s:%d: released cpu (%u)\n", __func__, __LINE__, cpu_id);
+
+	smp_secondary_entry(0, 0, NULL);
+
+	printf("Error: secondary cpu (%u) returned to bootwrapper!\n",
+		cpu_id);
+	exit();
+}
+
+/**
+ * smp_secondary_release - Release any secondary cpus.
+ * @kentry: The kernel entry for secondary cpus.
+ *
+ * Typically called by the primary cpu after the kernel is ready for entry.
+ */
+
+static void smp_secondary_release(kernel_entry_t kentry)
+{
+	printf("%s:%d\n", __func__, __LINE__);
+	smp_secondary_entry = kentry;
+
+	/* Do we need to yield to the secondary cpus here??? */
+}
+
 struct platform_ops platform_ops;
 struct dt_ops dt_ops;
 struct console_ops console_ops;

@@ -340,7 +378,10 @@ void start(unsigned long a1, unsigned lo
 	if (console_ops.close)
 		console_ops.close();
 
 	kentry = (kernel_entry_t) vmlinux.addr;
+
+	smp_secondary_release(kentry);
+
 	if (ft_addr)
 		kentry(ft_addr, 0, NULL);
 	else

Geoff Levand writes:

+void smp_secondary_hold(unsigned int cpu_id)
+{
+	while(!smp_secondary_entry)
+		(void)0;

Using a barrier() here rather than making smp_secondary_entry volatile
would be better, I think.

Also, what's the lifetime of this code after smp_secondary_entry is
set, i.e. how do we know the kernel running on the primary cpu will
wait for the secondary cpus to have got through this code into the
kernel before it reuses this memory for something else?

Paul.

Paul Mackerras wrote:

Geoff Levand writes:

quoted

+void smp_secondary_hold(unsigned int cpu_id)
+{
+	while(!smp_secondary_entry)
+		(void)0;

Using a barrier() here rather than making smp_secondary_entry volatile
would be better, I think.

Also, what's the lifetime of this code after smp_secondary_entry is
set, i.e. how do we know the kernel running on the primary cpu will
wait for the secondary cpus to have got through this code into the
kernel before it reuses this memory for something else?

I was thinking the most (only?) reliably way is to do it from inside the kernel.
Once inside, the primary waits for all secondaries to enter before proceeding.
That's why I didn't have anything here.  There is already a mechanism in the
kernel with __secondary_hold_acknowledge, but I don't yet know if it does
exactly what is needed.  Do you see any trouble with this?

-Geoff

On Sat, Feb 17, 2007 at 05:17:04PM -0800, Geoff Levand wrote:

Add SMP secondary hold helper routines to the powerpc zImage
bootwrapper.  For platforms which are entered with multiple cpus.
These routines can be used to hold the secondary cpus until the
kernel is ready for entry.

Could we have these functions in their own library module.  That way
they can be omitted in the wrappers for strictly-UP platforms (many
embedded), or platforms where the secondary CPUs are held in firmware
until the kernel releases.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

David Gibson wrote:

On Sat, Feb 17, 2007 at 05:17:04PM -0800, Geoff Levand wrote:

quoted

Add SMP secondary hold helper routines to the powerpc zImage
bootwrapper.  For platforms which are entered with multiple cpus.
These routines can be used to hold the secondary cpus until the
kernel is ready for entry.

Could we have these functions in their own library module.  That way
they can be omitted in the wrappers for strictly-UP platforms (many
embedded), or platforms where the secondary CPUs are held in firmware
until the kernel releases.

After some feedback, I decided to drop trying to support calling C from
the secondary CPUs.  That will simplify this to just a few bytes.  We can
think about optimization if another platform needs to use either the
exception vectors or the SMP support.  They are independent, but I'll just
put them together in in head.c for convenience.

-Geoff