[PATCH v33 00/14] add kdump support

On 17/03/17 16:59, Marc Zyngier wrote:

On 17/03/17 16:24, Mark Rutland wrote:

quoted

On Fri, Mar 17, 2017 at 03:47:08PM +0000, David Woodhouse wrote:

quoted

On Fri, 2017-03-17 at 15:33 +0000, Mark Rutland wrote:
No, in this case the CPUs *were* offlined correctly, or at least "as
designed", by smp_send_crash_stop(). And if that hadn't worked, as
verified by *its* synchronisation method based on the atomic_t
waiting_for_crash_ipi, then *it* would have complained for itself:

	if (atomic_read(&waiting_for_crash_ipi) > 0)
		pr_warning("SMP: failed to stop secondary CPUs %*pbl\n",
			   cpumask_pr_args(cpu_online_mask));

It's just that smp_send_crash_stop() (or more specifically
ipi_cpu_crash_stop()) doesn't touch the online cpu mask. Unlike the
ARM32 equivalent function machien_crash_nonpanic_core(), which does.

It wasn't clear if that was *intentional*, to allow the original
contents of the online mask before the crash to be seen in the
resulting vmcore... or purely an accident. 

Looking at this, there's a larger mess.

The waiting_for_crash_ipi dance only tells us if CPUs have taken the
IPI, not wether they've been offlined (i.e. actually left the kernel).
We need something closer to the usual cpu_{disable,die,kill} dance,
clearing online as appropriate.

If CPUs haven't left the kernel, we still need to warn about that.

quoted

FWIW if I trigger a crash on CPU 1 my kdump (still 4.9.8+v32) doesn't work.
I end up booting the kdump kernel on CPU#1 and then it gets distinctly unhappy...

[    0.000000] Booting Linux on physical CPU 0x1
...
[    0.017125] Detected PIPT I-cache on CPU1
[    0.017138] GICv3: CPU1: found redistributor 0 region 0:0x00000000f0280000
[    0.017147] CPU1: Booted secondary processor [411fd073]
[    0.017339] Detected PIPT I-cache on CPU2
[    0.017347] GICv3: CPU2: found redistributor 2 region 0:0x00000000f02c0000
[    0.017354] CPU2: Booted secondary processor [411fd073]
[    0.017537] Detected PIPT I-cache on CPU3
[    0.017545] GICv3: CPU3: found redistributor 3 region 0:0x00000000f02e0000
[    0.017551] CPU3: Booted secondary processor [411fd073]
[    0.017576] Brought up 4 CPUs
[    0.017587] SMP: Total of 4 processors activated.
...
[   31.745809] INFO: rcu_sched detected stalls on CPUs/tasks:
[   31.751299] 	1-...: (30 GPs behind) idle=c90/0/0 softirq=0/0 fqs=0 
[   31.757557] 	2-...: (30 GPs behind) idle=608/0/0 softirq=0/0 fqs=0 
[   31.763814] 	3-...: (30 GPs behind) idle=604/0/0 softirq=0/0 fqs=0 
[   31.770069] 	(detected by 0, t=5252 jiffies, g=-270, c=-271, q=0)
[   31.776161] Task dump for CPU 1:
[   31.779381] swapper/1       R  running task        0     0      1 0x00000080
[   31.786446] Task dump for CPU 2:
[   31.789666] swapper/2       R  running task        0     0      1 0x00000080
[   31.796725] Task dump for CPU 3:
[   31.799945] swapper/3       R  running task        0     0      1 0x00000080

Is some of that platform-specific?

That sounds like timer interrupts aren't being taken.

Given that the CPUs have come up, my suspicion would be that the GIC's
been left in some odd state, that the kdump kernel hasn't managed to
recover from.

Marc may have an idea.

I thought kdump was UP only? Anyway, this doesn't look too good.

It would be interesting to find out whether we're still taking
interrupts. Also, being able to reproduce this on mainline would be useful.

I wonder if we don't have a bug when booting on something other than
CPU#0, possibly on a GICv3 platform... I'll give it a go.

Went ahead and tried a couple of kexecs with various CPUs disabled in
order to force kexec not to boot on CPU#0, and the VM did boot just fine.

So I'd really appreciate a mainline reproducer.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...