Re: [Linux PPC] Disable PREEMPT
From: Michal Suchánek <hidden>
Date: 2026-09-04 07:40:18
On Thu, Sep 03, 2026 at 03:28:26PM +0200, Michal Suchánek wrote:
On Thu, Sep 03, 2026 at 06:06:16PM +0530, Harsh Prateek Bora wrote:quoted
+ Amit, Gautam - since they recently came across a similar regression which is seen with non KVM workload as well. + Venkat - for CI improvement suggestion below. Hi Michal, On 03/09/26 5:55 pm, Michal Suchánek wrote:quoted
quoted
There is another problem that Linux 7.2.2 often locks up on boot inside the guest, and as a result a different kernel is needed in the guest to reproduce the host crash. This is more recent, linux 6.12 does not have this problem. Will try some different kernel combinations to see if I can narrow this down. For this the repro is merely booting the kernel in the VM, it typically fails early before userspace produces any messages.This one regressed between 6.19.12 https://github.com/openSUSE/kernel-source/blob/9f6830f/config/ppc64le/default and 7.0.12 https://github.com/openSUSE/kernel-source/blob/2ebf0bc/config/ppc64le/default Thanks Michal SLOF ********************************************************************** QEMU Starting Build Date = Aug 13 2026 14:47:36 FW Version = abuild@OBS release 20230918 Press "s" to enter Open Firmware. Populating /vdevice methods Populating /vdevice/vty@30000000 Populating /vdevice/nvram@71000000 Populating /pci@800000020000000 Loading Linux 7.0.12-1.g2ebf0bc-default ... Loading initial ramdisk ... OF stdout device is: /vdevice/vty@30000000 Preparing to boot Linux version 7.0.12-1.g2ebf0bc-default (geeko@buildhost) (gcc (SUSE Linux) 15.3.0, GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.45.0.20251103-4) #1 SMP PREEMPT_DYNAMIC Mon Jun 15 08:39:32 UTC 2026 (2ebf0bc) Detected machine type: 0000000000000101 command line: BOOT_IMAGE=/boot/vmlinux-7.0.12-1.g2ebf0bc-default root=UUID=304c7f07-efbb-1070-2f27-accd935ec088 rw quiet systemd.show_status=1 security=selinux selinux=1 Max number of cores passed to firmware: 8192 (NR_CPUS = 8192) Calling ibm,client-architecture-support... done memory layout at init: memory_limit : 0000000000000000 (16 MB aligned) alloc_bottom : 00000000066b0000 alloc_top : 0000000030000000 alloc_top_hi : 0000003e00000000 rmo_top : 0000000030000000 ram_top : 0000003e00000000 instantiating rtas at 0x000000002fff0000... done prom_hold_cpus: skipped copying OF device tree... Building dt strings... Building dt structure... Device tree strings 0x00000000066c0000 -> 0x00000000066c0bec Device tree struct 0x00000000066d0000 -> 0x00000000066f0000 Quiescing Open Firmware ... Booting Linux via __start() @ 0x0000000000250000 ... [ 0.000000][ T0] ERROR: Failed to allocate trace buffer [ 0.000000][ T0] ERROR: tracer: failed to allocate ring buffer! Linux ppc64le #1 SMP PREEMPT_D[ 0.285758][ T673] BUG: Kernel NULL pointer dereference on read at 0x00000010 [ 0.285877][ T673] Faulting instruction address: 0xc000000000485bd0 [ 0.285903][ T1] VFS: Dquot-cache hash table entries: 8192 (order 0, 65536 bytes) [ 0.285954][ T673] Oops: Kernel access of bad area, sig: 7 [#1] [ 0.286133][ T673] LE PAGE_SIZE=64K MMU=Radix SMP NR_CPUS=8192 NUMA pSeries [ 0.286240][ T673] Modules linked in: [ 0.286288][ T673] CPU: 15 UID: 0 PID: 673 Comm: kworker/u531:0 Not tainted 7.0.12-1.g2ebf0bc-default #1 PREEMPT(full) openSUSE Tumbleweed (unreleased) cf0846124d7853fab338aeae87203b36cac18419 [ 0.286527][ T673] Hardware name: IBM pSeries (emulated by qemu) Power11 (architected) 0x820200 0xf000007 of:SLOF,HEAD hv:linux,kvm pSeries [ 0.286739][ T673] Workqueue: trace_init_wq tracer_init_tracefs_work_func [ 0.286818][ T673] NIP: c000000000485bd0 LR: c000000000448864 CTR: c0000000005a3420 [ 0.286906][ T673] REGS: c00000000a7a7960 TRAP: 0300 Not tainted (7.0.12-1.g2ebf0bc-default) [ 0.287014][ T673] MSR: 8000000000009033 <SF,EE,ME,IR,DR,RI,LE> CR: 44088404 XER: 00000000 [ 0.287122][ T673] CFAR: c000000000448860 DAR: 0000000000000010 DSISR: 00080000 IRQMASK: 0 [ 0.287122][ T673] GPR00: c000000000448864 c00000000a7a7c00 c000000001f38100 c000000002a5c098 [ 0.287122][ T673] GPR04: c0000000017dc5f0 c0000000017dc5e8 000000000000001f 0000000000000064 [ 0.287122][ T673] GPR08: c0000000017dc628 00000000000005f0 00000000000005e8 0000000084000404 [ 0.287122][ T673] GPR12: c0000000005a3420 c000003dfff43d80 c0000000018e5600 c0000000018e54f0 [ 0.287122][ T673] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 0.287122][ T673] GPR20: c0000000013ec418 c0000000013ef6d8 c0000000017dc5a0 c0000000017dc590 [ 0.287122][ T673] GPR24: c000000001825330 c0000000017dc628 c000000014b52a05 0000000000000000 [ 0.287122][ T673] GPR28: c0000000017dc5e8 0000000000000000 c0000000017dc5f0 c000000002a5e008 [ 0.287970][ T673] NIP [c000000000485bd0] __find_event_file+0x70/0x3c0 [ 0.288046][ T673] LR [c000000000448864] init_tracer_tracefs+0x274/0xc80 [ 0.288122][ T673] Call Trace: [ 0.288167][ T673] [c00000000a7a7c00] [c00000000a7a7c60] 0xc00000000a7a7c60 (unreliable) [ 0.288258][ T673] [c00000000a7a7c60] [c000000000448864] init_tracer_tracefs+0x274/0xc80 [ 0.288348][ T673] [c00000000a7a7dc0] [c00000000203fcf0] tracer_init_tracefs_work_func+0x50/0x320 [ 0.288452][ T673] [c00000000a7a7e50] [c0000000002620e8] process_one_work+0x1e8/0x5c0 [ 0.288541][ T673] [c00000000a7a7f10] [c00000000026309c] worker_thread+0x1dc/0x3d0 [ 0.288630][ T673] [c00000000a7a7f90] [c00000000026fa34] kthread+0x194/0x1b0 [ 0.288721][ T673] [c00000000a7a7fe0] [c00000000000de58] start_kernel_thread+0x14/0x18 [ 0.288810][ T673] Code: fb410030 fb810040 fba10048 7cbc2b78 3ba00000 fbc10050 7d194378 7c9e2378 2e2a0fc0 2da90fc0 f8010070 60420000 <e93b0010> 81490058 e8890018 714a0208 [ 0.289001][ T673] ---[ end trace 0000000000000000 ]--- [ 0.290718][ T673] pstore: backend (nvram) writing error (-1)
This crash might be relevant to the problem, and while qemu can dump the memory content crash refuses to open it: crash: incompatible arguments: /usr/lib/modules/7.2.2-5.g6824496-default/vmlinux is not SMP -- /scratch/vmcore is SMP The problem is not 100% reproducible, bisecting it may be tricky. Thanks Michal