Hi Peter,Thomas,
On Thu, May 19, 2016 at 04:27:49PM +0530, abdhalee wrote:
Hi
Today's mainline stable 4.6 on ppc64le bare metal booted with the following
warning.
I am able to see this on the mainline 4.7-rc1 version as well.
[ 0.080615] EEH: PowerNV platform initialized
[ 0.080709] POWER8 performance monitor hardware support registered
[ 0.080791] power8-pmu: PMAO restore workaround active.
[ 0.100780] ------------[ cut here ]------------
[ 0.100869] WARNING: CPU: 40 PID: 248 at kernel/sched/core.c:1166
__set_cpus_allowed_ptr+0x21c/0x290
[ 0.100958] Modules linked in:
[ 0.101013] CPU: 40 PID: 248 Comm: cpuhp/40 Not tainted 4.6.0-autotest #1
[ 0.101076] task: c000000f272bde00 ti: c000000f2737c000 task.ti:
c000000f2737c000
[ 0.101151] NIP: c0000000000fbacc LR: c0000000000fbab4 CTR:
0000000000000000
[ 0.101226] REGS: c000000f2737f7d0 TRAP: 0700 Not tainted
(4.6.0-autotest)
[ 0.101300] MSR: 9000000100029033 <SF,HV,EE,ME,IR,DR,RI,LE,TM[E]> CR:
28002804 XER: 20000000
[ 0.101528] CFAR: c00000000059c648 SOFTE: 0
[ 0.101528] GPR00: c0000000000fb9cc c000000f2737fa50 c0000000013ae000
0000000000000000
[ 0.101528] GPR04: c0000000013ded98 0000000000000800 c0000007f64d9408
0000000000000000
[ 0.101528] GPR08: 0000000000000000 0000000000000028 c0000000013dee90
0000000000000020
[ 0.101528] GPR12: 0000000000002200 c00000000fb96800 c0000000000eb6f8
c0000007fa158480
[ 0.101528] GPR16: c0000007fc61fa70 c000000f2721f800 0000000000000000
0000000000000001
[ 0.101528] GPR20: c000000001539d70 0000000000000000 c00000000132841e
c00000000129c510
[ 0.101528] GPR24: 0000010000000000 0000000000000000 c0000000013dea98
c0000007f64d9408
[ 0.101528] GPR28: c0000007fbc1fc00 ffffffffffffffea 0000000000000000
c000000f27280000
[ 0.102489] NIP [c0000000000fbacc] __set_cpus_allowed_ptr+0x21c/0x290
[ 0.102553] LR [c0000000000fbab4] __set_cpus_allowed_ptr+0x204/0x290
[ 0.102615] Call Trace:
[ 0.102643] [c000000f2737fa50] [c0000000000fb9cc]
__set_cpus_allowed_ptr+0x11c/0x290 (unreliable)
[ 0.102745] [c000000f2737fac0] [c0000000000e4710]
9 more quoted lines
Suspecting a race between CPU-ONLINE during boot-up and workqueue
initialization I added some debug to print the mask of the kthread if
it happens to be a worker thread. I am able to hit this WARN_ON on my
2-node 80CPUs box once every five times but only during boot-up.
============================================================================
[ 0.093671] Worker kworker/u162:0 on non-active CPU: new_mask = 40-79
[ 0.093690] ------------[ cut here ]------------
[ 0.093700] WARNING: CPU: 40 PID: 248 at kernel/sched/core.c:1177 __set_cpus_allowed_ptr+0x228/0x2e0
[ 0.093706] Modules linked in:
[ 0.093713] CPU: 40 PID: 248 Comm: cpuhp/40 Not tainted 4.6.0-autotest+ #4
[ 0.093719] task: c000000f27284200 ti: c000000f273fc000 task.ti: c000000f273fc000
============================================================================
In this case, kworker/u162:0 happens to be an WQ_UNBOUND worker thread
associated with node1 having cpumask 40-79.
During the boot up while onlining CPU40, the
workqueue_cpu_up_callback() is called which in-turn calls
restore_unbound_workers_cpumask().
Since at this point only CPU40 from the worker thread's mask is
online, restore_unbound_workers_cpumask() ends up calling
set_cpus_allowed_ptr().
Again, since CPU40 is not yet in the active_mask, we hit the WARN_ON
added in __set_cpus_allowed_ptr() via commit e9d867a67fd03ccc (sched:
Allow per-cpu kernel threads to run on online && !active).
I am not able to reproduce this issue once the system came up by
offlining CPUs 40-90 and then bringing them back online in the same order.
Could you please take a look at this ?
[ 0.103248] [c000000f2737fce0] [c0000000000bd6b4]
cpuhp_thread_fun+0x184/0x1c0
[ 0.103336] [c000000f2737fd20] [c0000000000f1290]
smpboot_thread_fn+0x290/0x2a0
[ 0.103424] [c000000f2737fd80] [c0000000000eb800] kthread+0x110/0x130
[ 0.103501] [c000000f2737fe30] [c000000000009570]
ret_from_kernel_thread+0x5c/0x6c
[ 0.103632] Instruction dump:
[ 0.103708] 419eff3c 3d420003 38a00800 388a0d98 7f63db78 484a0b21
60000000 2fa30000
[ 0.103963] 409eff1c 813f0378 2f890001 419eff10 <0fe00000> 4bffff08
60000000 60000000
[ 0.104230] ---[ end trace 7934a27c43ef63a5 ]---
[ 0.130901] Brought up 80 CPUs
[ 0.130955] Node 0 CPUs: 0-39
[ 0.130958] Node 1 CPUs: 40-79
[ 0.133075] devtmpfs: initialized
[ 0.150580] evm: security.selinux
[ 0.150583] evm: security.SMACK64
[ 0.150584] evm: security.SMACK64EXEC
[ 0.150586] evm: security.SMACK64TRANSMUTE
[ 0.150588] evm: security.SMACK64MMAP
[ 0.150590] evm: security.ima
[ 0.150591] evm: security.capability
[ 0.150761] EEH: devices created
[ 0.150977] clocksource: jiffies: mask: 0xffffffff max_cycles:
0xffffffff, max_idle_ns: 7645041785100000 ns
config and dmesg logs are attached.
Thanks
- Abdul
#
# Automatically generated file; DO NOT EDIT.
# Linux/powerpc 4.5.0-rc3 Kernel Configuration
#
CONFIG_PPC64=y
#
# Processor support
#
CONFIG_PPC_BOOK3S_64=y
# CONFIG_PPC_BOOK3E_64 is not set
CONFIG_POWER7_CPU=y
# CONFIG_POWER8_CPU is not set
CONFIG_PPC_BOOK3S=y
CONFIG_PPC_FPU=y
CONFIG_ALTIVEC=y
CONFIG_VSX=y
# CONFIG_PPC_ICSWX is not set
CONFIG_PPC_STD_MMU=y
CONFIG_PPC_STD_MMU_64=y
CONFIG_PPC_RADIX_MMU=y
CONFIG_PPC_MM_SLICES=y
CONFIG_PPC_HAVE_PMU_SUPPORT=y
CONFIG_PPC_PERF_CTRS=y
CONFIG_SMP=y
CONFIG_NR_CPUS=2048
CONFIG_PPC_DOORBELL=y
# CONFIG_CPU_BIG_ENDIAN is not set
CONFIG_CPU_LITTLE_ENDIAN=y
CONFIG_PPC64_BOOT_WRAPPER=y
CONFIG_64BIT=y
CONFIG_WORD_SIZE=64
CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
CONFIG_ARCH_DMA_ADDR_T_64BIT=y
CONFIG_MMU=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
CONFIG_NR_IRQS=512
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
7867 more quoted lines
[ 0.000000] opal: OPAL detected !
[ 0.000000] Allocated 4718592 bytes for 2048 pacas at c00000000fb80000
[ 0.000000] Using PowerNV machine description
[ 0.000000] Page sizes from device-tree:
[ 0.000000] base_shift=12: shift=12, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=0
[ 0.000000] base_shift=12: shift=16, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=7
[ 0.000000] base_shift=12: shift=24, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=56
[ 0.000000] base_shift=16: shift=16, sllp=0x0110, avpnm=0x00000000, tlbiel=1, penc=1
[ 0.000000] base_shift=16: shift=24, sllp=0x0110, avpnm=0x00000000, tlbiel=1, penc=8
[ 0.000000] base_shift=24: shift=24, sllp=0x0100, avpnm=0x00000001, tlbiel=0, penc=0
[ 0.000000] base_shift=34: shift=34, sllp=0x0120, avpnm=0x000007ff, tlbiel=0, penc=3
[ 0.000000] Page orders: linear mapping = 24, virtual = 16, io = 16, vmemmap = 24
[ 0.000000] Using 1TB segments
[ 0.000000] cma: Reserved 3280 MiB at 0x0000000f2b000000
[ 0.000000] Found initrd at 0xc000000002fa0000:0xc000000004dfe626
[ 0.000000] bootconsole [udbg0] enabled
[ 0.000000] CPU maps initialized for 8 threads per core
[ 0.000000] (thread shift is 3)
[ 0.000000] Freed 4521984 bytes for unused pacas
[ 0.000000] Starting Linux ppc64le #1 SMP Thu May 19 02:36:13 EDT 2016
[ 0.000000] -----------------------------------------------------
[ 0.000000] ppc64_pft_size = 0x0
[ 0.000000] phys_mem_size = 0x1000000000
[ 0.000000] cpu_features = 0x37fc7aed18500249
[ 0.000000] possible = 0x3fffffff18500649
[ 0.000000] always = 0x0000000018100040
[ 0.000000] cpu_user_features = 0xdc0065c2 0xef000000
[ 0.000000] mmu_features = 0x7c000001
[ 0.000000] firmware_features = 0x0000000010000000
[ 0.000000] htab_address = 0xc000000ff8000000
[ 0.000000] htab_hash_mask = 0x7ffff
[ 0.000000] -----------------------------------------------------
[ 0.000000] Linux version 4.6.0-autotest (root@ltc-test-ci3) (gcc version 5.3.1 20160413 (Ubuntu/IBM 5.3.1-14ubuntu2) ) #1 SMP Thu May 19 02:36:13 EDT 2016
[ 0.000000] Node 0 Memory: 0x0-0x800000000
[ 0.000000] Node 1 Memory: 0x800000000-0x1000000000
[ 0.000000] numa: Initmem setup node 0 [mem 0x00000000-0x7ffffffff]
[ 0.000000] numa: NODE_DATA [mem 0x7ffff6100-0x7ffffffff]
[ 0.000000] numa: Initmem setup node 1 [mem 0x800000000-0xfffffffff]
[ 0.000000] numa: NODE_DATA [mem 0xfffd26100-0xfffd2ffff]
[ 0.000000] Initializing IODA1 OPAL PHB /pciex@3fffe40000000
666 more quoted lines
_______________________________________________
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev