Re: [PATCH v2 2/2] powerpc:85xx: fix timebase sync issue when CONFIG_HOTPLUG_CPU=n

Hi there,

I have bisected OpenWrt master, and then the Linux kernel down to this
change, to confirm that this change causes a kernel panic on my
P1020RDB-based, dual-core Aerohive HiveAP 370, at initialization of
the second CPU:

:
[    0.000000] Linux version 5.10.80 (labby@lobon)
(powerpc-openwrt-linux-musl-gcc (OpenWrt GCC 11.2.0
r18111+1-ebb6f9287e) 11.2.0, GNU ld (GNU Binutils) 2.37) #0 SMP Thu
Nov 25 02:49:35 2021
[    0.000000] Using P1020 RDB machine description
:
[    0.627233] smp: Bringing up secondary CPUs ...
[    0.681659] kernel tried to execute user page (0) - exploit attempt? (uid: 0)
[    0.766618] BUG: Unable to handle kernel instruction fetch (NULL pointer?)
[    0.848899] Faulting instruction address: 0x00000000
[    0.908273] Oops: Kernel access of bad area, sig: 11 [#1]
[    0.972851] BE PAGE_SIZE=4K SMP NR_CPUS=2 P1020 RDB
[    1.031179] Modules linked in:
[    1.067640] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.10.80 #0
[    1.139507] NIP:  00000000 LR: c0021d2c CTR: 00000000
[    1.199921] REGS: c1051cf0 TRAP: 0400   Not tainted  (5.10.80)
[    1.269705] MSR:  00021000 <CE,ME>  CR: 84020202  XER: 00000000
[    1.340534]
[    1.340534] GPR00: c0021cb8 c1051da8 c1048000 00000001 00029000
00000000 00000001 00000000
[    1.340534] GPR08: 00000001 00000000 c08b0000 00000040 22000208
00000000 c00032c4 00000000
[    1.340534] GPR16: 00000000 00000000 00000000 00000000 00000000
00000000 00029000 00000001
[    1.340534] GPR24: 1ffff240 20000000 dffff240 c080a1f4 00000001
c08ae0a8 00000001 dffff240
[    1.758220] NIP [00000000] 0x0
[    1.794688] LR [c0021d2c] smp_85xx_kick_cpu+0xe8/0x568
[    1.856126] Call Trace:
[    1.885295] [c1051da8] [c0021cb8] smp_85xx_kick_cpu+0x74/0x568 (unreliable)
[    1.968633] [c1051de8] [c0011460] __cpu_up+0xc0/0x228
[    2.029038] [c1051e18] [c0031bbc] bringup_cpu+0x30/0x224

31 more lines

On 2021/11/25 12:20, Martin Kennedy wrote:

Hi there,

I have bisected OpenWrt master, and then the Linux kernel down to this
change, to confirm that this change causes a kernel panic on my
P1020RDB-based, dual-core Aerohive HiveAP 370, at initialization of
the second CPU:

:
[    0.000000] Linux version 5.10.80 (labby@lobon)
(powerpc-openwrt-linux-musl-gcc (OpenWrt GCC 11.2.0
r18111+1-ebb6f9287e) 11.2.0, GNU ld (GNU Binutils) 2.37) #0 SMP Thu
Nov 25 02:49:35 2021
[    0.000000] Using P1020 RDB machine description
:
[    0.627233] smp: Bringing up secondary CPUs ...
[    0.681659] kernel tried to execute user page (0) - exploit attempt? (uid: 0)
[    0.766618] BUG: Unable to handle kernel instruction fetch (NULL pointer?)
[    0.848899] Faulting instruction address: 0x00000000
[    0.908273] Oops: Kernel access of bad area, sig: 11 [#1]
[    0.972851] BE PAGE_SIZE=4K SMP NR_CPUS=2 P1020 RDB
[    1.031179] Modules linked in:
[    1.067640] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.10.80 #0
[    1.139507] NIP:  00000000 LR: c0021d2c CTR: 00000000
[    1.199921] REGS: c1051cf0 TRAP: 0400   Not tainted  (5.10.80)
[    1.269705] MSR:  00021000 <CE,ME>  CR: 84020202  XER: 00000000
[    1.340534]
[    1.340534] GPR00: c0021cb8 c1051da8 c1048000 00000001 00029000
00000000 00000001 00000000
[    1.340534] GPR08: 00000001 00000000 c08b0000 00000040 22000208
00000000 c00032c4 00000000
[    1.340534] GPR16: 00000000 00000000 00000000 00000000 00000000
00000000 00029000 00000001
[    1.340534] GPR24: 1ffff240 20000000 dffff240 c080a1f4 00000001
c08ae0a8 00000001 dffff240
[    1.758220] NIP [00000000] 0x0
[    1.794688] LR [c0021d2c] smp_85xx_kick_cpu+0xe8/0x568
[    1.856126] Call Trace:
[    1.885295] [c1051da8] [c0021cb8] smp_85xx_kick_cpu+0x74/0x568 (unreliable)
[    1.968633] [c1051de8] [c0011460] __cpu_up+0xc0/0x228
[    2.029038] [c1051e18] [c0031bbc] bringup_cpu+0x30/0x224

33 more quoted lines

When CONFIG_FSL_PMC is set to n, cpu_up_prepare is not assigned to mpc85xx_pm_ops. I suspect that this is the cause of the current null pointer access.
I do not have the corresponding board test environment. Can you help me to test whether the following patch solves the problem?

diff --git a/arch/powerpc/platforms/85xx/smp.c b/arch/powerpc/platforms/85xx/smp.c
index 83f4a6389a28..d7081e9af65c 100644
--- a/arch/powerpc/platforms/85xx/smp.c
+++ b/arch/powerpc/platforms/85xx/smp.c

@@ -220,7 +220,7 @@ static int smp_85xx_start_cpu(int cpu)
        local_irq_save(flags);
        hard_irq_disable();

-   if (qoriq_pm_ops)
+ if (qoriq_pm_ops && qoriq_pm_ops->cpu_up_prepare)
                qoriq_pm_ops->cpu_up_prepare(cpu);

        /* if cpu is not spinning, reset it */

@@ -292,7 +292,7 @@ static int smp_85xx_kick_cpu(int nr)
                booting_thread_hwid = cpu_thread_in_core(nr);
                primary = cpu_first_thread_sibling(nr);

-           if (qoriq_pm_ops)
+         if (qoriq_pm_ops && qoriq_pm_ops->cpu_up_prepare)
                        qoriq_pm_ops->cpu_up_prepare(nr);

                /*

Hi there,

Yes, I can test this patch.

I have added it to my tree and removed the reversion, and can confirm
that the second CPU comes up correctly now.

Martin

On Thu, Nov 25, 2021 at 2:23 AM Xiaoming Ni [off-list ref] wrote:

On 2021/11/25 12:20, Martin Kennedy wrote:

quoted

Hi there,

I have bisected OpenWrt master, and then the Linux kernel down to this
change, to confirm that this change causes a kernel panic on my
P1020RDB-based, dual-core Aerohive HiveAP 370, at initialization of
the second CPU:

:
[    0.000000] Linux version 5.10.80 (labby@lobon)
(powerpc-openwrt-linux-musl-gcc (OpenWrt GCC 11.2.0
r18111+1-ebb6f9287e) 11.2.0, GNU ld (GNU Binutils) 2.37) #0 SMP Thu
Nov 25 02:49:35 2021
[    0.000000] Using P1020 RDB machine description
:
[    0.627233] smp: Bringing up secondary CPUs ...
[    0.681659] kernel tried to execute user page (0) - exploit attempt? (uid: 0)
[    0.766618] BUG: Unable to handle kernel instruction fetch (NULL pointer?)
[    0.848899] Faulting instruction address: 0x00000000
[    0.908273] Oops: Kernel access of bad area, sig: 11 [#1]
[    0.972851] BE PAGE_SIZE=4K SMP NR_CPUS=2 P1020 RDB
[    1.031179] Modules linked in:
[    1.067640] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.10.80 #0
[    1.139507] NIP:  00000000 LR: c0021d2c CTR: 00000000
[    1.199921] REGS: c1051cf0 TRAP: 0400   Not tainted  (5.10.80)
[    1.269705] MSR:  00021000 <CE,ME>  CR: 84020202  XER: 00000000
[    1.340534]
[    1.340534] GPR00: c0021cb8 c1051da8 c1048000 00000001 00029000
00000000 00000001 00000000
[    1.340534] GPR08: 00000001 00000000 c08b0000 00000040 22000208
00000000 c00032c4 00000000
[    1.340534] GPR16: 00000000 00000000 00000000 00000000 00000000
00000000 00029000 00000001
[    1.340534] GPR24: 1ffff240 20000000 dffff240 c080a1f4 00000001
c08ae0a8 00000001 dffff240
[    1.758220] NIP [00000000] 0x0
[    1.794688] LR [c0021d2c] smp_85xx_kick_cpu+0xe8/0x568
[    1.856126] Call Trace:
[    1.885295] [c1051da8] [c0021cb8] smp_85xx_kick_cpu+0x74/0x568 (unreliable)

66 more quoted lines

On 2021/11/25 22:34, Martin Kennedy wrote:

Hi there,

Yes, I can test this patch.

I have added it to my tree and removed the reversion, and can confirm
that the second CPU comes up correctly now.

Martin

Thank you very much for your report and testing, I'll send a patch later

Thanks
Xiaoming Ni

On Thu, Nov 25, 2021 at 2:23 AM Xiaoming Ni [off-list ref] wrote:

quoted

On 2021/11/25 12:20, Martin Kennedy wrote:

quoted

Hi there,

I have bisected OpenWrt master, and then the Linux kernel down to this
change, to confirm that this change causes a kernel panic on my
P1020RDB-based, dual-core Aerohive HiveAP 370, at initialization of
the second CPU:

:
[    0.000000] Linux version 5.10.80 (labby@lobon)
(powerpc-openwrt-linux-musl-gcc (OpenWrt GCC 11.2.0
r18111+1-ebb6f9287e) 11.2.0, GNU ld (GNU Binutils) 2.37) #0 SMP Thu
Nov 25 02:49:35 2021
[    0.000000] Using P1020 RDB machine description
:
[    0.627233] smp: Bringing up secondary CPUs ...
[    0.681659] kernel tried to execute user page (0) - exploit attempt? (uid: 0)
[    0.766618] BUG: Unable to handle kernel instruction fetch (NULL pointer?)
[    0.848899] Faulting instruction address: 0x00000000
[    0.908273] Oops: Kernel access of bad area, sig: 11 [#1]
[    0.972851] BE PAGE_SIZE=4K SMP NR_CPUS=2 P1020 RDB
[    1.031179] Modules linked in:
[    1.067640] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.10.80 #0
[    1.139507] NIP:  00000000 LR: c0021d2c CTR: 00000000
[    1.199921] REGS: c1051cf0 TRAP: 0400   Not tainted  (5.10.80)
[    1.269705] MSR:  00021000 <CE,ME>  CR: 84020202  XER: 00000000
[    1.340534]
[    1.340534] GPR00: c0021cb8 c1051da8 c1048000 00000001 00029000
00000000 00000001 00000000
[    1.340534] GPR08: 00000001 00000000 c08b0000 00000040 22000208
00000000 c00032c4 00000000
[    1.340534] GPR16: 00000000 00000000 00000000 00000000 00000000
00000000 00029000 00000001
[    1.340534] GPR24: 1ffff240 20000000 dffff240 c080a1f4 00000001
c08ae0a8 00000001 dffff240
[    1.758220] NIP [00000000] 0x0
[    1.794688] LR [c0021d2c] smp_85xx_kick_cpu+0xe8/0x568
[    1.856126] Call Trace:

69 more quoted lines

When CONFIG_FSL_PMC is set to n, no value is assigned to cpu_up_prepare
 in the mpc85xx_pm_ops structure. As a result, oops is triggered in
 smp_85xx_start_cpu().

	[    0.627233] smp: Bringing up secondary CPUs ...
	[    0.681659] kernel tried to execute user page (0) - exploit attempt? (uid: 0)
	[    0.766618] BUG: Unable to handle kernel instruction fetch (NULL pointer?)
	[    0.848899] Faulting instruction address: 0x00000000
	[    0.908273] Oops: Kernel access of bad area, sig: 11 [#1]
	...
	[    1.758220] NIP [00000000] 0x0
	[    1.794688] LR [c0021d2c] smp_85xx_kick_cpu+0xe8/0x568
	[    1.856126] Call Trace:
	[    1.885295] [c1051da8] [c0021cb8] smp_85xx_kick_cpu+0x74/0x568 (unreliable)
	[    1.968633] [c1051de8] [c0011460] __cpu_up+0xc0/0x228
	[    2.029038] [c1051e18] [c0031bbc] bringup_cpu+0x30/0x224
	[    2.092572] [c1051e48] [c0031f3c] cpu_up.constprop.0+0x180/0x33c
	[    2.164443] [c1051e88] [c00322e8] bringup_nonboot_cpus+0x88/0xc8
	[    2.236326] [c1051eb8] [c07e67bc] smp_init+0x30/0x78
	[    2.295698] [c1051ed8] [c07d9e28] kernel_init_freeable+0x118/0x2a8
	[    2.369641] [c1051f18] [c00032d8] kernel_init+0x14/0x124
	[    2.433176] [c1051f38] [c0010278] ret_from_kernel_thread+0x14/0x1c

Fixes: c45361abb9185b ("powerpc/85xx: fix timebase sync issue when
 CONFIG_HOTPLUG_CPU=n")
Link: https://lore.kernel.org/lkml/CANA18Uyba4kMJQrbCSZVTFep2Exe5izE45whNJgwwUvNSEcNLg@mail.gmail.com/
Reported-by: Martin Kennedy <redacted>
Signed-off-by: Xiaoming Ni <redacted>
Tested-by: Martin Kennedy <redacted>
Cc: <redacted>
---
 arch/powerpc/platforms/85xx/smp.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/powerpc/platforms/85xx/smp.c b/arch/powerpc/platforms/85xx/smp.c
index 83f4a6389a28..d7081e9af65c 100644
--- a/arch/powerpc/platforms/85xx/smp.c
+++ b/arch/powerpc/platforms/85xx/smp.c

@@ -220,7 +220,7 @@ static int smp_85xx_start_cpu(int cpu)
 	local_irq_save(flags);
 	hard_irq_disable();
 
-	if (qoriq_pm_ops)
+	if (qoriq_pm_ops && qoriq_pm_ops->cpu_up_prepare)
 		qoriq_pm_ops->cpu_up_prepare(cpu);
 
 	/* if cpu is not spinning, reset it */

@@ -292,7 +292,7 @@ static int smp_85xx_kick_cpu(int nr)
 		booting_thread_hwid = cpu_thread_in_core(nr);
 		primary = cpu_first_thread_sibling(nr);
 
-		if (qoriq_pm_ops)
+		if (qoriq_pm_ops && qoriq_pm_ops->cpu_up_prepare)
 			qoriq_pm_ops->cpu_up_prepare(nr);
 
 		/*

-- 
2.27.0

On Fri, 26 Nov 2021 12:11:53 +0800, Xiaoming Ni wrote:

When CONFIG_FSL_PMC is set to n, no value is assigned to cpu_up_prepare
 in the mpc85xx_pm_ops structure. As a result, oops is triggered in
 smp_85xx_start_cpu().

	[    0.627233] smp: Bringing up secondary CPUs ...
	[    0.681659] kernel tried to execute user page (0) - exploit attempt? (uid: 0)
	[    0.766618] BUG: Unable to handle kernel instruction fetch (NULL pointer?)
	[    0.848899] Faulting instruction address: 0x00000000
	[    0.908273] Oops: Kernel access of bad area, sig: 11 [#1]
	...
	[    1.758220] NIP [00000000] 0x0
	[    1.794688] LR [c0021d2c] smp_85xx_kick_cpu+0xe8/0x568
	[    1.856126] Call Trace:
	[    1.885295] [c1051da8] [c0021cb8] smp_85xx_kick_cpu+0x74/0x568 (unreliable)
	[    1.968633] [c1051de8] [c0011460] __cpu_up+0xc0/0x228
	[    2.029038] [c1051e18] [c0031bbc] bringup_cpu+0x30/0x224
	[    2.092572] [c1051e48] [c0031f3c] cpu_up.constprop.0+0x180/0x33c
	[    2.164443] [c1051e88] [c00322e8] bringup_nonboot_cpus+0x88/0xc8
	[    2.236326] [c1051eb8] [c07e67bc] smp_init+0x30/0x78
	[    2.295698] [c1051ed8] [c07d9e28] kernel_init_freeable+0x118/0x2a8
	[    2.369641] [c1051f18] [c00032d8] kernel_init+0x14/0x124
	[    2.433176] [c1051f38] [c0010278] ret_from_kernel_thread+0x14/0x1c

[...]

Applied to powerpc/fixes.

[1/1] powerpc/85xx: fix oops when CONFIG_FSL_PMC=n
      https://git.kernel.org/powerpc/c/3dc709e518b47386e6af937eaec37bb36539edfd

cheers