Re: [Linux PPC] Disable PREEMPT

+ Amit, Gautam - since they recently came across a similar regression 
which is seen with non KVM workload as well.
+ Venkat - for CI improvement suggestion below.

Hi Michal,

On 03/09/26 5:55 pm, Michal Suchánek wrote:

On Thu, Sep 03, 2026 at 05:34:31PM +0530, Shrikanth Hegde wrote:

quoted

Hi Michal.

On 9/3/26 5:10 PM, Michal Suchánek wrote:

quoted

On Thu, Sep 03, 2026 at 05:01:13PM +0530, Shrikanth Hegde wrote:

quoted

quoted

quoted

quoted

This is not the latest kernel so it might have been fixed in the
meantime for the platform in question. Will need to check that.

If possible run against current upstream and share the results.

https://github.com/openSUSE/kernel-source/blob/6824496d1801f73def615dca8794202eeb7b0d86/config/ppc64le/default

[  472.091531][ T6181] Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: kvmhv_run_single_vcpu+0x19d4/0x1b50 [kvm_hv]
[  472.091598][ T6181] CPU: 29 UID: 107 PID: 6181 Comm: CPU 112/KVM Not tainted 7.2.2-5.g6824496-default #1 PREEMPT(full) openSUSE Tumbleweed (unreleased)  61871a5f06863b4006f5ef27cd9f18e8a7a2edad
[  472.091612][ T6181] Hardware name: IBM,9824-42A Power11 (architected) 0x820200 0xf000007 of:IBM,FW1110.20 (OB1110_130) hv:phyp pSeries
[  472.091624][ T6181] Call Trace:
[  472.091630][ T6181] [c00000000fdfb680] [c00000000134ce90] dump_stack_lvl+0x84/0xc0 (unreliable)
[  472.091653][ T6181] [c00000000fdfb6b0] [c00000000022e7c8] vpanic+0x324/0x5e4
[  472.091666][ T6181] [c00000000fdfb760] [c00000000022eac4] do_panic_on_target_cpu+0x0/0x2c
[  472.091677][ T6181] [c00000000fdfb780] [c0000000013c1ff8] __stack_chk_fail+0x48/0x60
[  472.091689][ T6181] [c00000000fdfb7f0] [c00800001aae219c] kvmhv_run_single_vcpu+0x19d4/0x1b50 [kvm_hv]
[  472.091712][ T6181] [c00000000fdfb940] [c00800001aae24b4] kvmppc_vcpu_run_hv+0x19c/0x12f0 [kvm_hv]
[  472.091732][ T6181] [c00000000fdfba10] [c00800001aeeed18] kvmppc_vcpu_run+0x30/0x48 [kvm]
[  472.091779][ T6181] [c00000000fdfba30] [c00800001aee9ef4] kvm_arch_vcpu_ioctl_run+0x35c/0x4a0 [kvm]
[  472.091813][ T6181] [c00000000fdfbac0] [c00800001aedaac4] kvm_vcpu_ioctl+0x1ac/0xad8 [kvm]
[  472.091844][ T6181] [c00000000fdfbca0] [c0000000007f1244] sys_ioctl+0x374/0x1060
[  472.091857][ T6181] [c00000000fdfbdb0] [c00000000002f7f8] system_call_exception+0x188/0x430
[  472.091871][ T6181] [c00000000fdfbe50] [c00000000000cfdc] system_call_vectored_common+0x15c/0x2ec
[  472.091886][ T6181] ---- interrupt: 3000 at 0x7fffb5565fac
[  472.091896][ T6181] NIP:  00007fffb5565fac LR: 00007fffb5565fac CTR: 0000000000000000
[  472.091904][ T6181] REGS: c00000000fdfbe80 TRAP: 3000   Not tainted  (7.2.2-5.g6824496-default)
[  472.091911][ T6181] MSR:  800000000280f033 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE>  CR: 42044402  XER: 00000000
[  472.091938][ T6181] IRQMASK: 0
[  472.091938][ T6181] GPR00: 0000000000000036 00007fbfa77ed7a0 00007fffb5677100 00000000000000fa
[  472.091938][ T6181] GPR04: 000000002000ae80 0000000000000000 0000000000000000 0000000000000000
[  472.091938][ T6181] GPR08: 00000000000000fa 0000000000000000 0000000000000000 0000000000000000
[  472.091938][ T6181] GPR12: 0000000000000000 00007fbfa77f5ec0 000000014676f000 00007fbfa77ee7c0
[  472.091938][ T6181] GPR16: 000000014674e8d0 00007fbfa77eeec0 00007fbfa77eeec0 fffffffffffffff7
[  472.091938][ T6181] GPR20: 00007fffb71210d0 0000000000000001 00007fbfa77eeec0 0000000000000000
[  472.091938][ T6181] GPR24: 00007fbfa77ed8e8 0000000105971428 000000002000ae80 0000000105f77a70
[  472.091938][ T6181] GPR28: 0000000000000000 0000000000000000 000000002000ae80 000000014674f000
[  472.092020][ T6181] NIP [00007fffb5565fac] 0x7fffb5565fac
[  472.092027][ T6181] LR [00007fffb5565fac] 0x7fffb5565fac
[  472.092033][ T6181] ---- interrupt: 3000
[  472.098256][ T6181] pstore: backend (nvram) writing error (-1)

This is the host, cannot run the kernel as guest because it fails to boot most
of the time inside KVM.

Will look into. btw the patch shared i.e below patch fixes the boot hanf issue?

https://lore.kernel.org/all/1d79ec6f-ce7f-4432-80e0-a54fa1668642@linux.ibm.com/ (local)

I see code has changed a bit. So this patch is not correct anymore.
Sorry, i should have checked.

quoted

I would not bet on that. It looks like host side patch, and the problem
is with booting 7.2.2 as guest. Some 6.12 frankenkernel can be booted on
the same host, and used to produce the crash above by running a kernel
build inside the guest.

I didn't get what you are saying. Could you see if it regression only in 7.2?
Does 7.1 works fine? If it is regression could you see if it is bisectable?
Also, could you share the repro steps?

The repro steps are: on a mid-size LPAR (120 CPU threads, 256G RAM)
create a KVM VM that uses all host resources. Run kernel build inside
KVM VM, using all CPUs. In about a minute the host crashes with the
above a or similar trace. I don't have any kernel version with aggressive
preemption enabled that does not suffer from this problem.

Looks like seen under heavy workloads when you say "KVM VM that uses all 
host rsources". Could you please confirm if the issue is seen with any 
other heavy workload also which consumes all host resources? I do not 
think we recently had any PPC KVM specific changes which would cause 
this. Also, if it's a regression, git bisect should point out the real 
culprit.

Venkat,
Can we include such a test case in upstream CI as well?

regards,
Harsh

There is another problem that Linux 7.2.2 often locks up on boot inside
the guest, and as a result a different kernel is needed in the guest to
reproduce the host crash. This is more recent, linux 6.12 does not have
this problem. Will try some different kernel combinations to see if I
can narrow this down. For this the repro is merely booting the kernel in
the VM, it typically fails early before userspace produces any messages.

Thanks

Michal