Re: [PATCH] powerpc/powernv: implement NMI IPI with OPAL_SIGNAL_SYSTEM_RESET
From: Alistair Popple <hidden>
Date: 2017-02-03 06:00:35
On Fri, 3 Feb 2017 03:20:48 PM Nicholas Piggin wrote:
On Fri, 3 Feb 2017 00:25:11 +1000 Nicholas Piggin [off-list ref] wrote:quoted
This goes with the previous NMI IPI series, and a new version of Alistair's opal API I posted to the skiboot list.And here is the incremental bit that is required for Alistair's hardware implementation to work. If the opal broacast call fails with OPAL_PARTIAL, then we designate a bouncer CPU on another core to send NMI IPIs back to our sibling threads. Probably needs more discussion and testing about how to detect and handle failure cases and future compatibility for different types of restrictions, but at least it works in mambo.
Looking at the Mambo implementation you recently posted I couldn't see where OPAL_PARTIAL was returned. So I guess you have other Skiboot patches to test this path using Mambo? I'm wondering because they could be useful for testing the skiboot HW side as well...
Of course the other option rather than doing this in Linux is call into opal in the system reset handler and have it do the bouncing. Something to consider before we finalise the API.
That might not be such a bad idea. OPAL could queue up the threads it couldn't reset and then wait until opal_sreset_complete() is called from an eligible thread to reset the ones it couldn't do in the original call. I might try prototyping something like this when I get some time. One issue would be if there is only a single core in the system, but that's unlikely and I think that's probably something we can't support in any case as cores can't sreset threads on the same core, at least on P8. - Alistair
quoted hunk ↗ jump to hunk
Thanks, Nick --- arch/powerpc/platforms/powernv/smp.c | 88 +++++++++++++++++++++++++++++++++--- 1 file changed, 82 insertions(+), 6 deletions(-)diff --git a/arch/powerpc/platforms/powernv/smp.c b/arch/powerpc/platforms/powernv/smp.c index f90555f75723..4241fda9df9e 100644 --- a/arch/powerpc/platforms/powernv/smp.c +++ b/arch/powerpc/platforms/powernv/smp.c@@ -241,8 +241,32 @@ static int pnv_cpu_bootable(unsigned int nr) return smp_generic_cpu_bootable(nr); } +static int nmi_ipi_bounce_cpu; +static int nmi_ipi_bounce_cpu_done; +static int nmi_ipi_bounce_target_core; +static int nmi_ipi_bounce_target_exclude; + int pnv_system_reset_exception(struct pt_regs *regs) { + smp_rmb(); + if (nmi_ipi_bounce_cpu == smp_processor_id()) { + int64_t rc; + int c; + + for_each_online_cpu(c) { + if (!cpumask_test_cpu(c, cpu_sibling_mask(nmi_ipi_bounce_target_core))) + continue; + if (c == nmi_ipi_bounce_target_exclude) + continue; + rc = opal_signal_system_reset(c); + if (rc != OPAL_SUCCESS) { + nmi_ipi_bounce_cpu_done = -1; + return 1; + } + } + nmi_ipi_bounce_cpu_done = 1; + } + if (smp_handle_nmi_ipi(regs)) return 1; return 0;@@ -252,13 +276,65 @@ static int pnv_cause_nmi_ipi(int cpu) { int64_t rc; - rc = opal_signal_system_reset(cpu); - if (rc == OPAL_SUCCESS) - return 1; + if (cpu >= 0) { + rc = opal_signal_system_reset(cpu); + if (rc == OPAL_SUCCESS) + return 1; + } else { + /* + * Test bounce behavior with broadcast IPI. + */ + rc = OPAL_PARTIAL; + } + if (rc == OPAL_PARTIAL) { + int c; - /* - * Don't cope with OPAL_PARTIAL yet (just punt to regular IPI) - */ + /* + * Some platforms can not send NMI to sibling threads in + * the same core. We can designate one inter-core target + * to bounce NMIs back to our sibling threads. + */ + + if (cpu >= 0) { + /* + * Don't support bouncing unicast NMIs yet (because + * that would have to raise an NMI on an unrelated + * CPU. Revisit this if callers start using unicast. + */ + return 0; + } + + nmi_ipi_bounce_cpu = -1; + nmi_ipi_bounce_cpu_done = 0; + nmi_ipi_bounce_target_core = -1; + nmi_ipi_bounce_target_exclude = -1; + smp_wmb(); + + for_each_online_cpu(c) { + if (cpumask_test_cpu(c, cpu_sibling_mask(smp_processor_id()))) + continue; + + if (nmi_ipi_bounce_cpu == -1) { + nmi_ipi_bounce_cpu = c; + nmi_ipi_bounce_target_core = smp_processor_id(); + if (cpu == NMI_IPI_ALL_OTHERS) + nmi_ipi_bounce_target_exclude = smp_processor_id(); + } + + rc = opal_signal_system_reset(c); + if (rc != OPAL_SUCCESS) + return 0; + } + + if (nmi_ipi_bounce_cpu == -1) + return 0; /* could not find a bouncer */ + + while (!nmi_ipi_bounce_cpu_done) + cpu_relax(); + + if (nmi_ipi_bounce_cpu_done == 1) + return 1; /* bounce worked */ + } return 0; }