WARNING at kernel/sched/core.c:1166 while booting 4.6.0 mainline on ppc64le bare metal

Started by abdhalee on May 19, 2016 · 30 messages from 6 authors · Last reply Jul 15, 2016

open the first message on its own page

  1. [PATCH 1/2] workqueue: Move wq_update_unbound_numa() to the beginning of CPU_ONLINE
    Related patches
  2. [PATCH 2/2] workqueue:Fix affinity of an unbound worker of a node with 1 online CPU
    Related patches
Message details and related content
Hi

Today's mainline stable 4.6 on ppc64le bare metal booted with the following warning.

[    0.080615] EEH: PowerNV platform initialized
[    0.080709] POWER8 performance monitor hardware support registered
[    0.080791] power8-pmu: PMAO restore workaround active.
[    0.100780] ------------[ cut here ]------------
[    0.100869] WARNING: CPU: 40 PID: 248 at kernel/sched/core.c:1166 __set_cpus_allowed_ptr+0x21c/0x290
[    0.100958] Modules linked in:
[    0.101013] CPU: 40 PID: 248 Comm: cpuhp/40 Not tainted 4.6.0-autotest #1
[    0.101076] task: c000000f272bde00 ti: c000000f2737c000 task.ti: c000000f2737c000
[    0.101151] NIP: c0000000000fbacc LR: c0000000000fbab4 CTR: 0000000000000000
[    0.101226] REGS: c000000f2737f7d0 TRAP: 0700   Not tainted (4.6.0-autotest)
[    0.101300] MSR: 9000000100029033 <SF,HV,EE,ME,IR,DR,RI,LE,TM[E]>  CR: 28002804  XER: 20000000
[    0.101528] CFAR: c00000000059c648 SOFTE: 0
[    0.101528] GPR00: c0000000000fb9cc c000000f2737fa50 c0000000013ae000 0000000000000000
[    0.101528] GPR04: c0000000013ded98 0000000000000800 c0000007f64d9408 0000000000000000
[    0.101528] GPR08: 0000000000000000 0000000000000028 c0000000013dee90 0000000000000020
[    0.101528] GPR12: 0000000000002200 c00000000fb96800 c0000000000eb6f8 c0000007fa158480
[    0.101528] GPR16: c0000007fc61fa70 c000000f2721f800 0000000000000000 0000000000000001
[    0.101528] GPR20: c000000001539d70 0000000000000000 c00000000132841e c00000000129c510
[    0.101528] GPR24: 0000010000000000 0000000000000000 c0000000013dea98 c0000007f64d9408
[    0.101528] GPR28: c0000007fbc1fc00 ffffffffffffffea 0000000000000000 c000000f27280000
[    0.102489] NIP [c0000000000fbacc] __set_cpus_allowed_ptr+0x21c/0x290
[    0.102553] LR [c0000000000fbab4] __set_cpus_allowed_ptr+0x204/0x290
[    0.102615] Call Trace:
[    0.102643] [c000000f2737fa50] [c0000000000fb9cc] __set_cpus_allowed_ptr+0x11c/0x290 (unreliable)
[    0.102745] [c000000f2737fac0] [c0000000000e4710] workqueue_cpu_up_callback+0x2c0/0x470
[    0.102833] [c000000f2737fb70] [c0000000000ecea8] notifier_call_chain+0x98/0x100
[    0.102922] [c000000f2737fbc0] [c0000000000bd0b0] __cpu_notify+0x70/0xe0
[    0.102997] [c000000f2737fc00] [c0000000000bd208] notify_online+0x38/0x50
[    0.103073] [c000000f2737fc30] [c0000000000bc3f4] cpuhp_invoke_callback+0x84/0x250
[    0.103161] [c000000f2737fc90] [c0000000000bc80c] cpuhp_up_callbacks+0x5c/0x120
[    0.103248] [c000000f2737fce0] [c0000000000bd6b4] cpuhp_thread_fun+0x184/0x1c0
[    0.103336] [c000000f2737fd20] [c0000000000f1290] smpboot_thread_fn+0x290/0x2a0
[    0.103424] [c000000f2737fd80] [c0000000000eb800] kthread+0x110/0x130
[    0.103501] [c000000f2737fe30] [c000000000009570] ret_from_kernel_thread+0x5c/0x6c
[    0.103632] Instruction dump:
[    0.103708] 419eff3c 3d420003 38a00800 388a0d98 7f63db78 484a0b21 60000000 2fa30000

20 more lines

Message details and related content
On Thu, May 19, 2016 at 04:27:49PM +0530, abdhalee wrote:
Hi

Today's mainline stable 4.6 on ppc64le bare metal booted with the following
warning.

[    0.080615] EEH: PowerNV platform initialized
[    0.080709] POWER8 performance monitor hardware support registered
[    0.080791] power8-pmu: PMAO restore workaround active.
[    0.100780] ------------[ cut here ]------------
[    0.100869] WARNING: CPU: 40 PID: 248 at kernel/sched/core.c:1166
__set_cpus_allowed_ptr+0x21c/0x290
I ran into same issue on yesterday's linux-next. Also, I added some logs
and it seems the CPU isn't marked as active in time. The stack trace is
poped up under the circumstance: CPU#80 is online, but not active yet.

======> cpuhp_thread_fun: CPU=80
cpuhp_thread_fun: state=10 target=45
cpuhp_ap_online: CPU=80, state=10 target=45
smpboot_unpark_threads: CPU=80
notify_online: CPU=80                        <<<< CPU#80 isn't active yet.
------------[ cut here ]------------
WARNING: CPU: 80 PID: 408 at kernel/sched/core.c:1166 __set_cpus_allowed_ptr+0x22c/0x290
Modules linked in:
CPU: 80 PID: 408 Comm: cpuhp/80 Not tainted 4.6.0-next-20160517-gavin-00020-g176bf86-dirty #35
task: c000001e5243de00 ti: c000001ffc10c000 task.ti: c000001ffc10c000
NIP: c0000000000d923c LR: c0000000000d9224 CTR: 0000000000000000
REGS: c000001ffc10f730 TRAP: 0700   Not tainted  (4.6.0-next-20160517-gavin-00020-g176bf86-dirty)
MSR: 9000000000029033 <SF,HV,EE,ME,IR,DR,RI,LE>  CR: 28002044  XER: 20000000
CFAR: c00000000047135c SOFTE: 0 
GPR00: c0000000000d9138 c000001ffc10f9b0 c000000001321300 0000000000000000 
GPR04: c00000000135aa18 0000000000000400 0000000000000010 0000000000000000 
GPR08: 0000000000000000 0000000000000050 c00000000135aa90 0000000000000000 
GPR12: 0000000000002200 c00000000ff14000 c000000ffa60c5d0 c000000001292800 
GPR16: 0000000000000001 c0000000012780a8 c00000000139b678 0000000000010000 
GPR20: c000001e523b0000 c000000001278048 0000000000000008 c0000000012cfa8e 
GPR24: c0000000012780c8 c000001ffc10fa40 c000000001278048 c00000000135a898 
GPR28: c000000ff133ff08 c000000ff9c0c780 c000001e52400000 0000000000000000 
NIP [c0000000000d923c] __set_cpus_allowed_ptr+0x22c/0x290
LR [c0000000000d9224] __set_cpus_allowed_ptr+0x214/0x290
Call Trace:
[c000001ffc10f9b0] [c0000000000d9138] __set_cpus_allowed_ptr+0x128/0x290 (unreliable)
[c000001ffc10fa20] [c0000000000c65e0] workqueue_cpu_up_callback+0x460/0x5d0
[c000001ffc10faf0] [c0000000000cee6c] notifier_call_chain+0xac/0x110
[c000001ffc10fb40] [c00000000009fc64] __cpu_notify+0x54/0xa0
[c000001ffc10fb60] [c00000000009fd9c] notify_online+0x4c/0x70
[c000001ffc10fbd0] [c00000000009f5b4] cpuhp_up_callbacks+0x74/0x1a0
[c000001ffc10fc20] [c0000000000a0100] cpuhp_thread_fun+0x1e0/0x2a0
[c000001ffc10fcc0] [c0000000000d2ac0] smpboot_thread_fn+0x290/0x2a0
[c000001ffc10fd20] [c0000000000cd578] kthread+0x108/0x130
[c000001ffc10fe30] [c000000000009578] ret_from_kernel_thread+0x5c/0x64

8 more lines

Message details and related content
Hi Peter,Thomas,

On Thu, May 19, 2016 at 04:27:49PM +0530, abdhalee wrote:
Hi

Today's mainline stable 4.6 on ppc64le bare metal booted with the following
warning.
I am able to see this on the mainline 4.7-rc1 version as well.
[    0.080615] EEH: PowerNV platform initialized
[    0.080709] POWER8 performance monitor hardware support registered
[    0.080791] power8-pmu: PMAO restore workaround active.
[    0.100780] ------------[ cut here ]------------
[    0.100869] WARNING: CPU: 40 PID: 248 at kernel/sched/core.c:1166
__set_cpus_allowed_ptr+0x21c/0x290
[    0.100958] Modules linked in:
[    0.101013] CPU: 40 PID: 248 Comm: cpuhp/40 Not tainted 4.6.0-autotest #1
[    0.101076] task: c000000f272bde00 ti: c000000f2737c000 task.ti:
c000000f2737c000
[    0.101151] NIP: c0000000000fbacc LR: c0000000000fbab4 CTR:
0000000000000000
[    0.101226] REGS: c000000f2737f7d0 TRAP: 0700   Not tainted
(4.6.0-autotest)
[    0.101300] MSR: 9000000100029033 <SF,HV,EE,ME,IR,DR,RI,LE,TM[E]>  CR:
28002804  XER: 20000000
[    0.101528] CFAR: c00000000059c648 SOFTE: 0
[    0.101528] GPR00: c0000000000fb9cc c000000f2737fa50 c0000000013ae000
0000000000000000
[    0.101528] GPR04: c0000000013ded98 0000000000000800 c0000007f64d9408
0000000000000000
[    0.101528] GPR08: 0000000000000000 0000000000000028 c0000000013dee90
0000000000000020
[    0.101528] GPR12: 0000000000002200 c00000000fb96800 c0000000000eb6f8
c0000007fa158480
[    0.101528] GPR16: c0000007fc61fa70 c000000f2721f800 0000000000000000
0000000000000001
[    0.101528] GPR20: c000000001539d70 0000000000000000 c00000000132841e
c00000000129c510
[    0.101528] GPR24: 0000010000000000 0000000000000000 c0000000013dea98
c0000007f64d9408
[    0.101528] GPR28: c0000007fbc1fc00 ffffffffffffffea 0000000000000000
c000000f27280000
[    0.102489] NIP [c0000000000fbacc] __set_cpus_allowed_ptr+0x21c/0x290
[    0.102553] LR [c0000000000fbab4] __set_cpus_allowed_ptr+0x204/0x290
[    0.102615] Call Trace:
[    0.102643] [c000000f2737fa50] [c0000000000fb9cc]
__set_cpus_allowed_ptr+0x11c/0x290 (unreliable)
[    0.102745] [c000000f2737fac0] [c0000000000e4710]

9 more quoted lines

Suspecting a race between CPU-ONLINE during boot-up and workqueue
initialization I added some debug to print the mask of the kthread if
it happens to be a worker thread. I am able to hit this WARN_ON on my
2-node 80CPUs box once every five times but only during boot-up.

============================================================================
[    0.093671] Worker kworker/u162:0 on non-active CPU: new_mask = 40-79
[    0.093690] ------------[ cut here ]------------
[    0.093700] WARNING: CPU: 40 PID: 248 at kernel/sched/core.c:1177 __set_cpus_allowed_ptr+0x228/0x2e0
[    0.093706] Modules linked in:
[    0.093713] CPU: 40 PID: 248 Comm: cpuhp/40 Not tainted 4.6.0-autotest+ #4
[    0.093719] task: c000000f27284200 ti: c000000f273fc000 task.ti: c000000f273fc000
============================================================================

In this case, kworker/u162:0 happens to be an WQ_UNBOUND worker thread
associated with node1 having cpumask 40-79.

During the boot up while onlining CPU40, the
workqueue_cpu_up_callback() is called which in-turn calls
restore_unbound_workers_cpumask().

Since at this point only CPU40 from the worker thread's mask is
online, restore_unbound_workers_cpumask() ends up calling
set_cpus_allowed_ptr().

Again, since CPU40 is not yet in the active_mask, we hit the WARN_ON
 added in __set_cpus_allowed_ptr() via commit e9d867a67fd03ccc (sched:
 Allow per-cpu kernel threads to run on online && !active).

I am not able to reproduce this issue once the system came up by
offlining CPUs 40-90 and then bringing them back online in the same order.

Could you please take a look at this ?
[    0.103248] [c000000f2737fce0] [c0000000000bd6b4]
cpuhp_thread_fun+0x184/0x1c0
[    0.103336] [c000000f2737fd20] [c0000000000f1290]
smpboot_thread_fn+0x290/0x2a0
[    0.103424] [c000000f2737fd80] [c0000000000eb800] kthread+0x110/0x130
[    0.103501] [c000000f2737fe30] [c000000000009570]
ret_from_kernel_thread+0x5c/0x6c
[    0.103632] Instruction dump:
[    0.103708] 419eff3c 3d420003 38a00800 388a0d98 7f63db78 484a0b21
60000000 2fa30000
[    0.103963] 409eff1c 813f0378 2f890001 419eff10 <0fe00000> 4bffff08
60000000 60000000
[    0.104230] ---[ end trace 7934a27c43ef63a5 ]---
[    0.130901] Brought up 80 CPUs
[    0.130955] Node 0 CPUs: 0-39
[    0.130958] Node 1 CPUs: 40-79
[    0.133075] devtmpfs: initialized
[    0.150580] evm: security.selinux
[    0.150583] evm: security.SMACK64
[    0.150584] evm: security.SMACK64EXEC
[    0.150586] evm: security.SMACK64TRANSMUTE
[    0.150588] evm: security.SMACK64MMAP
[    0.150590] evm: security.ima
[    0.150591] evm: security.capability
[    0.150761] EEH: devices created
[    0.150977] clocksource: jiffies: mask: 0xffffffff max_cycles:
0xffffffff, max_idle_ns: 7645041785100000 ns

config and dmesg logs are attached.

Thanks
- Abdul
#
# Automatically generated file; DO NOT EDIT.
# Linux/powerpc 4.5.0-rc3 Kernel Configuration
#
CONFIG_PPC64=y

#
# Processor support
#
CONFIG_PPC_BOOK3S_64=y
# CONFIG_PPC_BOOK3E_64 is not set
CONFIG_POWER7_CPU=y
# CONFIG_POWER8_CPU is not set
CONFIG_PPC_BOOK3S=y
CONFIG_PPC_FPU=y
CONFIG_ALTIVEC=y
CONFIG_VSX=y
# CONFIG_PPC_ICSWX is not set
CONFIG_PPC_STD_MMU=y
CONFIG_PPC_STD_MMU_64=y
CONFIG_PPC_RADIX_MMU=y
CONFIG_PPC_MM_SLICES=y
CONFIG_PPC_HAVE_PMU_SUPPORT=y
CONFIG_PPC_PERF_CTRS=y
CONFIG_SMP=y
CONFIG_NR_CPUS=2048
CONFIG_PPC_DOORBELL=y
# CONFIG_CPU_BIG_ENDIAN is not set
CONFIG_CPU_LITTLE_ENDIAN=y
CONFIG_PPC64_BOOT_WRAPPER=y
CONFIG_64BIT=y
CONFIG_WORD_SIZE=64
CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
CONFIG_ARCH_DMA_ADDR_T_64BIT=y
CONFIG_MMU=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
CONFIG_NR_IRQS=512
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_TRACE_IRQFLAGS_SUPPORT=y

7867 more quoted lines

[    0.000000] opal: OPAL detected !
[    0.000000] Allocated 4718592 bytes for 2048 pacas at c00000000fb80000
[    0.000000] Using PowerNV machine description
[    0.000000] Page sizes from device-tree:
[    0.000000] base_shift=12: shift=12, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=0
[    0.000000] base_shift=12: shift=16, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=7
[    0.000000] base_shift=12: shift=24, sllp=0x0000, avpnm=0x00000000, tlbiel=1, penc=56
[    0.000000] base_shift=16: shift=16, sllp=0x0110, avpnm=0x00000000, tlbiel=1, penc=1
[    0.000000] base_shift=16: shift=24, sllp=0x0110, avpnm=0x00000000, tlbiel=1, penc=8
[    0.000000] base_shift=24: shift=24, sllp=0x0100, avpnm=0x00000001, tlbiel=0, penc=0
[    0.000000] base_shift=34: shift=34, sllp=0x0120, avpnm=0x000007ff, tlbiel=0, penc=3
[    0.000000] Page orders: linear mapping = 24, virtual = 16, io = 16, vmemmap = 24
[    0.000000] Using 1TB segments
[    0.000000] cma: Reserved 3280 MiB at 0x0000000f2b000000
[    0.000000] Found initrd at 0xc000000002fa0000:0xc000000004dfe626
[    0.000000] bootconsole [udbg0] enabled
[    0.000000] CPU maps initialized for 8 threads per core
[    0.000000]  (thread shift is 3)
[    0.000000] Freed 4521984 bytes for unused pacas
[    0.000000] Starting Linux ppc64le #1 SMP Thu May 19 02:36:13 EDT 2016
[    0.000000] -----------------------------------------------------
[    0.000000] ppc64_pft_size    = 0x0
[    0.000000] phys_mem_size     = 0x1000000000
[    0.000000] cpu_features      = 0x37fc7aed18500249
[    0.000000]   possible        = 0x3fffffff18500649
[    0.000000]   always          = 0x0000000018100040
[    0.000000] cpu_user_features = 0xdc0065c2 0xef000000
[    0.000000] mmu_features      = 0x7c000001
[    0.000000] firmware_features = 0x0000000010000000
[    0.000000] htab_address      = 0xc000000ff8000000
[    0.000000] htab_hash_mask    = 0x7ffff
[    0.000000] -----------------------------------------------------
[    0.000000] Linux version 4.6.0-autotest (root@ltc-test-ci3) (gcc version 5.3.1 20160413 (Ubuntu/IBM 5.3.1-14ubuntu2) ) #1 SMP Thu May 19 02:36:13 EDT 2016
[    0.000000] Node 0 Memory: 0x0-0x800000000
[    0.000000] Node 1 Memory: 0x800000000-0x1000000000
[    0.000000] numa: Initmem setup node 0 [mem 0x00000000-0x7ffffffff]
[    0.000000] numa:   NODE_DATA [mem 0x7ffff6100-0x7ffffffff]
[    0.000000] numa: Initmem setup node 1 [mem 0x800000000-0xfffffffff]
[    0.000000] numa:   NODE_DATA [mem 0xfffd26100-0xfffd2ffff]
[    0.000000] Initializing IODA1 OPAL PHB /pciex@3fffe40000000

666 more quoted lines

_______________________________________________
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev
Message details and related content
Hi Ranjal,

Today's rafael's next build on ppc64le hit with the same problem.

On Thursday 26 May 2016 08:41 PM, Gautham R Shenoy wrote:
Hi Peter,Thomas,

On Thu, May 19, 2016 at 04:27:49PM +0530, abdhalee wrote:
quoted
Hi

Today's mainline stable 4.6 on ppc64le bare metal booted with the following
warning.
I am able to see this on the mainline 4.7-rc1 version as well.
quoted
[    0.080615] EEH: PowerNV platform initialized
[    0.080709] POWER8 performance monitor hardware support registered
[    0.080791] power8-pmu: PMAO restore workaround active.
[    0.100780] ------------[ cut here ]------------
[    0.100869] WARNING: CPU: 40 PID: 248 at kernel/sched/core.c:1166
__set_cpus_allowed_ptr+0x21c/0x290
[    0.100958] Modules linked in:
[    0.101013] CPU: 40 PID: 248 Comm: cpuhp/40 Not tainted 4.6.0-autotest #1
[    0.101076] task: c000000f272bde00 ti: c000000f2737c000 task.ti:
c000000f2737c000
[    0.101151] NIP: c0000000000fbacc LR: c0000000000fbab4 CTR:
0000000000000000
[    0.101226] REGS: c000000f2737f7d0 TRAP: 0700   Not tainted
(4.6.0-autotest)
[    0.101300] MSR: 9000000100029033 <SF,HV,EE,ME,IR,DR,RI,LE,TM[E]>  CR:
28002804  XER: 20000000
[    0.101528] CFAR: c00000000059c648 SOFTE: 0
[    0.101528] GPR00: c0000000000fb9cc c000000f2737fa50 c0000000013ae000
0000000000000000
[    0.101528] GPR04: c0000000013ded98 0000000000000800 c0000007f64d9408
0000000000000000
[    0.101528] GPR08: 0000000000000000 0000000000000028 c0000000013dee90
0000000000000020
[    0.101528] GPR12: 0000000000002200 c00000000fb96800 c0000000000eb6f8
c0000007fa158480
[    0.101528] GPR16: c0000007fc61fa70 c000000f2721f800 0000000000000000
0000000000000001
[    0.101528] GPR20: c000000001539d70 0000000000000000 c00000000132841e
c00000000129c510
[    0.101528] GPR24: 0000010000000000 0000000000000000 c0000000013dea98
c0000007f64d9408

8700 more quoted lines

[PATCH 0/2] Fix CPU Online handling for unbounded worker threads

Message details and related content
Hi,

This patchset fixes a couple of issues in the CPU_ONLINE notification
handling for the workqueues with respect to unbounded worker threads.

Patch 1 ensures that the affinity of a unbound worker thread
associated with a node whose very first CPU has come online is set
correctly. In the existing code path we will never call
set_cpus_allowed_ptr() for unbound worker threads that have been
created on a CPU Online operation after boot.

Patch 2 fixes the following WARN_ON() reported by Abdul when
set_cpus_allowed_ptr() for an unbound worker thread is invoked when
only one of the CPUs in its cpumask is online but not yet active.

 ------------[ cut here ]------------
 WARNING: CPU: 40 PID: 248 at kernel/sched/core.c:1166 __set_cpus_allowed_ptr+0x21c/0x290
 Modules linked in:
CPU: 40 PID: 248 Comm: cpuhp/40 Not tainted 4.6.0-autotest #1
task: c000000f27284200 ti: c000000f273fc000 task.ti: c000000f273fc000
NIP: c00000000010488c LR: c000000000104874 CTR: 0000000000000000
REGS: c000000f273ff7d0 TRAP: 0700   Not tainted  (4.6.0-autotest)
MSR: 9000000100029033 <SF,HV,EE,ME,IR,DR,RI,LE,TM[E]>  CR: 28002804  XER: 20000000
CFAR: c0000000005b0888 SOFTE: 0
GPR00: c00000000010478c c000000f273ffa50 c0000000013ce400 0000000000000000
GPR04: c00000000140ed98 0000000000000800 c0000007f64d9408 0000000000000000
GPR08: 0000000000000000 0000000000000028 c00000000140ee90 0000000000000020
GPR12: 0000000000002200 c00000000fb96800 c0000000000f44a8 c0000007fa158480
GPR16: c0000007fc621a70 c000000f2721f800 0000000000000000 0000000000000001
GPR20: c000000001571ef0 0000000000000000 c00000000134879f c0000000012bc510
GPR24: 0000010000000000 0000000000000000 c00000000140ea98 c0000007f64d9408
GPR28: c0000007fbc21c00 ffffffffffffffea 0000000000000000 c000000f27280000
NIP [c00000000010488c] __set_cpus_allowed_ptr+0x21c/0x290
LR [c000000000104874] __set_cpus_allowed_ptr+0x204/0x290
Call Trace:
[c000000f273ffa50] [c00000000010478c] __set_cpus_allowed_ptr+0x11c/0x290 (unreliable)
[c000000f273ffac0] [c0000000000ed4b0] workqueue_cpu_up_callback+0x2c0/0x470
[c000000f273ffb70] [c0000000000f5c58] notifier_call_chain+0x98/0x100
[c000000f273ffbc0] [c0000000000c5ed0] __cpu_notify+0x70/0xe0
[c000000f273ffc00] [c0000000000c6028] notify_online+0x38/0x50

25 more lines

[PATCH 1/2] workqueue: Move wq_update_unbound_numa() to the beginning of CPU_ONLINE

Message details and related content
Currently in the CPU_ONLINE workqueue handler, the
restore_unbound_workers_cpumask() will never call
set_cpus_allowed_ptr() for a newly created unbound worker thread.

This is because the function which creates a new unbound worker thread
when the first CPU in the node comes online [wq_update_unbound_numa()]
is invoked after the call to restore_unbound_workers_cpumask(). Thus
the affinity is never set for this worker thread when the first CPU in
the node comes online.

Furthermore, due to an optimization in
restore_unbound_workers_cpumask(), set_cpus_allowed_ptr() is not
called when subsequent CPUs in the node come online since it assumes
that the affinity would have been set when the first CPU has come
online.

This patch fixes this issue by invoking wq_update_unbound_numa()
before the calling restore_unbound_workers_cpumask().

Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <redacted>
Cc: Tejun Heo <redacted>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Signed-off-by: Gautham R. Shenoy <redacted>
---
 kernel/workqueue.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index e1c0e99..e412794 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -4638,6 +4638,10 @@ static int workqueue_cpu_up_callback(struct notifier_block *nfb,
 	case CPU_ONLINE:
 		mutex_lock(&wq_pool_mutex);
 
+		/* update NUMA affinity of unbound workqueues */
+		list_for_each_entry(wq, &workqueues, list)
+			wq_update_unbound_numa(wq, cpu, true);
+
 		for_each_pool(pool, pi) {
 			mutex_lock(&pool->attach_mutex);
 
@@ -4649,10 +4653,6 @@ static int workqueue_cpu_up_callback(struct notifier_block *nfb,
 			mutex_unlock(&pool->attach_mutex);
 		}
 
-		/* update NUMA affinity of unbound workqueues */
-		list_for_each_entry(wq, &workqueues, list)
-			wq_update_unbound_numa(wq, cpu, true);
-
 		mutex_unlock(&wq_pool_mutex);
 		break;
 	}
-- 
1.9.3

[PATCH 2/2] workqueue:Fix affinity of an unbound worker of a node with 1 online CPU

Message details and related content
With commit e9d867a67fd03ccc ("sched: Allow per-cpu kernel threads to
run on online && !active"), __set_cpus_allowed_ptr() expects that only
strict per-cpu kernel threads can have affinity to an online CPU which
is not yet active.

This assumption is currently broken in the CPU_ONLINE notification
handler for the workqueues where restore_unbound_workers_cpumask()
calls set_cpus_allowed_ptr() when the first cpu in the unbound
worker's pool->attr->cpumask comes online. Since
set_cpus_allowed_ptr() is called with pool->attr->cpumask in which
only one CPU is online which is not yet active, we get the following
WARN_ON during an CPU online operation.

------------[ cut here ]------------
WARNING: CPU: 40 PID: 248 at kernel/sched/core.c:1166
__set_cpus_allowed_ptr+0x228/0x2e0
Modules linked in:
CPU: 40 PID: 248 Comm: cpuhp/40 Not tainted 4.6.0-autotest+ #4
<..snip..>
Call Trace:
[c000000f273ff920] [c00000000010493c] __set_cpus_allowed_ptr+0x2cc/0x2e0 (unreliable)
[c000000f273ffac0] [c0000000000ed4b0] workqueue_cpu_up_callback+0x2c0/0x470
[c000000f273ffb70] [c0000000000f5c58] notifier_call_chain+0x98/0x100
[c000000f273ffbc0] [c0000000000c5ed0] __cpu_notify+0x70/0xe0
[c000000f273ffc00] [c0000000000c6028] notify_online+0x38/0x50
[c000000f273ffc30] [c0000000000c5214] cpuhp_invoke_callback+0x84/0x250
[c000000f273ffc90] [c0000000000c562c] cpuhp_up_callbacks+0x5c/0x120
[c000000f273ffce0] [c0000000000c64d4] cpuhp_thread_fun+0x184/0x1c0
[c000000f273ffd20] [c0000000000fa050] smpboot_thread_fn+0x290/0x2a0
[c000000f273ffd80] [c0000000000f45b0] kthread+0x110/0x130
[c000000f273ffe30] [c000000000009570] ret_from_kernel_thread+0x5c/0x6c
---[ end trace 00f1456578b2a3b2 ]---

This patch sets the affinity of the worker to
a) the only online CPU in the cpumask of the worker pool when it comes
   online.
b) the cpumask of the worker pool when the second CPU in the pool's
   cpumask comes online.

Reported-by: Abdul Haleem <redacted>

9 more lines

diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index e412794..1199f73 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -4586,7 +4586,7 @@ static void rebind_workers(struct worker_pool *pool)
  *
  * An unbound pool may end up with a cpumask which doesn't have any online
  * CPUs.  When a worker of such pool get scheduled, the scheduler resets
- * its cpus_allowed.  If @cpu is in @pool's cpumask which didn't have any
+ * its cpus_allowed.  If @cpu is in @pool's cpumask which had at most one
  * online CPU before, cpus_allowed of all its workers should be restored.
  */
 static void restore_unbound_workers_cpumask(struct worker_pool *pool, int cpu)
@@ -4600,15 +4600,26 @@ static void restore_unbound_workers_cpumask(struct worker_pool *pool, int cpu)
 	if (!cpumask_test_cpu(cpu, pool->attrs->cpumask))
 		return;
 
-	/* is @cpu the only online CPU? */
 	cpumask_and(&cpumask, pool->attrs->cpumask, cpu_online_mask);
-	if (cpumask_weight(&cpumask) != 1)
+
+	/*
+	 * The affinity needs to be set
+	 * a) to @cpu when that is the only online CPU in
+	 *    pool->attrs->cpumask.
+	 * b) to pool->attrs->cpumask when exactly two CPUs in
+	 *    pool->attrs->cpumask are online. This affinity will be
+	 *    retained when subsequent CPUs come online.
+	 */
+	if (cpumask_weight(&cpumask) > 2)
 		return;
 
+	if (cpumask_weight(&cpumask) == 2)
+		cpumask_copy(&cpumask, pool->attrs->cpumask);
+
 	/* as we're called from CPU_ONLINE, the following shouldn't fail */
 	for_each_pool_worker(worker, pool)
 		WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task,
-						  pool->attrs->cpumask) < 0);
+						  &cpumask) < 0);
 }
 
 /*
-- 
1.9.3

Re: [PATCH 2/2] workqueue:Fix affinity of an unbound worker of a node with 1 online CPU

Message details and related content
Hi Gautham,

Thanks a lot for the fix.

With your patches applied, 4.7.0-rc2 builds fine on ppc64le bare metal.
Boot was successful with No call traces.

Thanks for all your support !

Regard's
Abdul

On Tuesday 07 June 2016 08:44 PM, Gautham R. Shenoy wrote:
With commit e9d867a67fd03ccc ("sched: Allow per-cpu kernel threads to
run on online && !active"), __set_cpus_allowed_ptr() expects that only
strict per-cpu kernel threads can have affinity to an online CPU which
is not yet active.

This assumption is currently broken in the CPU_ONLINE notification
handler for the workqueues where restore_unbound_workers_cpumask()
calls set_cpus_allowed_ptr() when the first cpu in the unbound
worker's pool->attr->cpumask comes online. Since
set_cpus_allowed_ptr() is called with pool->attr->cpumask in which
only one CPU is online which is not yet active, we get the following
WARN_ON during an CPU online operation.

------------[ cut here ]------------
WARNING: CPU: 40 PID: 248 at kernel/sched/core.c:1166
__set_cpus_allowed_ptr+0x228/0x2e0
Modules linked in:
CPU: 40 PID: 248 Comm: cpuhp/40 Not tainted 4.6.0-autotest+ #4
<..snip..>
Call Trace:
[c000000f273ff920] [c00000000010493c] __set_cpus_allowed_ptr+0x2cc/0x2e0 (unreliable)
[c000000f273ffac0] [c0000000000ed4b0] workqueue_cpu_up_callback+0x2c0/0x470
[c000000f273ffb70] [c0000000000f5c58] notifier_call_chain+0x98/0x100
[c000000f273ffbc0] [c0000000000c5ed0] __cpu_notify+0x70/0xe0
[c000000f273ffc00] [c0000000000c6028] notify_online+0x38/0x50
[c000000f273ffc30] [c0000000000c5214] cpuhp_invoke_callback+0x84/0x250
[c000000f273ffc90] [c0000000000c562c] cpuhp_up_callbacks+0x5c/0x120
[c000000f273ffce0] [c0000000000c64d4] cpuhp_thread_fun+0x184/0x1c0
[c000000f273ffd20] [c0000000000fa050] smpboot_thread_fn+0x290/0x2a0
[c000000f273ffd80] [c0000000000f45b0] kthread+0x110/0x130
[c000000f273ffe30] [c000000000009570] ret_from_kernel_thread+0x5c/0x6c
---[ end trace 00f1456578b2a3b2 ]---

This patch sets the affinity of the worker to
a) the only online CPU in the cpumask of the worker pool when it comes
    online.
b) the cpumask of the worker pool when the second CPU in the pool's
    cpumask comes online.

Reported-by: Abdul Haleem <redacted>

6 more quoted lines

Re: [PATCH 0/2] Fix CPU Online handling for unbounded worker threads

Message details and related content
Hi Peter, Thomas,
On Tue, Jun 07, 2016 at 08:44:01PM +0530, Gautham R. Shenoy wrote:
Hi,

This patchset fixes a couple of issues in the CPU_ONLINE notification
handling for the workqueues with respect to unbounded worker
threads.

Any thoughts on these patches ? They fix a race which
was causing WARN_ON() to be consistently reproduced on POWER machines
since 4.6.

Could you please review these patches ?

--
Thanks and Regard
gautham.
Patch 1 ensures that the affinity of a unbound worker thread
associated with a node whose very first CPU has come online is set
correctly. In the existing code path we will never call
set_cpus_allowed_ptr() for unbound worker threads that have been
created on a CPU Online operation after boot.

Patch 2 fixes the following WARN_ON() reported by Abdul when
set_cpus_allowed_ptr() for an unbound worker thread is invoked when
only one of the CPUs in its cpumask is online but not yet active.

 ------------[ cut here ]------------
 WARNING: CPU: 40 PID: 248 at kernel/sched/core.c:1166 __set_cpus_allowed_ptr+0x21c/0x290
 Modules linked in:
CPU: 40 PID: 248 Comm: cpuhp/40 Not tainted 4.6.0-autotest #1
task: c000000f27284200 ti: c000000f273fc000 task.ti: c000000f273fc000
NIP: c00000000010488c LR: c000000000104874 CTR: 0000000000000000
REGS: c000000f273ff7d0 TRAP: 0700   Not tainted  (4.6.0-autotest)
MSR: 9000000100029033 <SF,HV,EE,ME,IR,DR,RI,LE,TM[E]>  CR: 28002804  XER: 20000000
CFAR: c0000000005b0888 SOFTE: 0
GPR00: c00000000010478c c000000f273ffa50 c0000000013ce400 0000000000000000
GPR04: c00000000140ed98 0000000000000800 c0000007f64d9408 0000000000000000
GPR08: 0000000000000000 0000000000000028 c00000000140ee90 0000000000000020
GPR12: 0000000000002200 c00000000fb96800 c0000000000f44a8 c0000007fa158480
GPR16: c0000007fc621a70 c000000f2721f800 0000000000000000 0000000000000001
GPR20: c000000001571ef0 0000000000000000 c00000000134879f c0000000012bc510
GPR24: 0000010000000000 0000000000000000 c00000000140ea98 c0000007f64d9408
GPR28: c0000007fbc21c00 ffffffffffffffea 0000000000000000 c000000f27280000
NIP [c00000000010488c] __set_cpus_allowed_ptr+0x21c/0x290
LR [c000000000104874] __set_cpus_allowed_ptr+0x204/0x290
Call Trace:
[c000000f273ffa50] [c00000000010478c] __set_cpus_allowed_ptr+0x11c/0x290 (unreliable)
[c000000f273ffac0] [c0000000000ed4b0] workqueue_cpu_up_callback+0x2c0/0x470
[c000000f273ffb70] [c0000000000f5c58] notifier_call_chain+0x98/0x100
[c000000f273ffbc0] [c0000000000c5ed0] __cpu_notify+0x70/0xe0
[c000000f273ffc00] [c0000000000c6028] notify_online+0x38/0x50
[c000000f273ffc30] [c0000000000c5214] cpuhp_invoke_callback+0x84/0x250
[c000000f273ffc90] [c0000000000c562c] cpuhp_up_callbacks+0x5c/0x120
[c000000f273ffce0] [c0000000000c64d4] cpuhp_thread_fun+0x184/0x1c0
[c000000f273ffd20] [c0000000000fa050] smpboot_thread_fn+0x290/0x2a0

22 more quoted lines

Re: [PATCH 2/2] workqueue:Fix affinity of an unbound worker of a node with 1 online CPU

Message details and related content
On Tue, Jun 07, 2016 at 08:44:03PM +0530, Gautham R. Shenoy wrote:

I'm still puzzled why we don't see this on x86. Afaict there's nothing
PPC specific about this.
This patch sets the affinity of the worker to
a) the only online CPU in the cpumask of the worker pool when it comes
   online.
b) the cpumask of the worker pool when the second CPU in the pool's
   cpumask comes online.
This basically works around the WARN conditions, which I suppose is fair
enough, but I would like a note here to revisit this once the whole cpu
hotplug rework has settled.

The real problem is that workqueues seem to want to create worker
threads before there's anybody who would use them or something like
that.

Or is that what PPC does funny? Use an unbound workqueue this early in
cpu bringup?
quoted hunk
@@ -4600,15 +4600,26 @@ static void restore_unbound_workers_cpumask(struct worker_pool *pool, int cpu)
 	if (!cpumask_test_cpu(cpu, pool->attrs->cpumask))
 		return;
 
-	/* is @cpu the only online CPU? */
 	cpumask_and(&cpumask, pool->attrs->cpumask, cpu_online_mask);
-	if (cpumask_weight(&cpumask) != 1)
+
+	/*
+	 * The affinity needs to be set
+	 * a) to @cpu when that is the only online CPU in
+	 *    pool->attrs->cpumask.
+	 * b) to pool->attrs->cpumask when exactly two CPUs in
+	 *    pool->attrs->cpumask are online. This affinity will be
+	 *    retained when subsequent CPUs come online.
+	 */
+	if (cpumask_weight(&cpumask) > 2)
 		return;
 
+	if (cpumask_weight(&cpumask) == 2)
+		cpumask_copy(&cpumask, pool->attrs->cpumask);
+
 	/* as we're called from CPU_ONLINE, the following shouldn't fail */
 	for_each_pool_worker(worker, pool)
 		WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task,
-						  pool->attrs->cpumask) < 0);
+						  &cpumask) < 0);
 }

Re: [PATCH 2/2] workqueue:Fix affinity of an unbound worker of a node with 1 online CPU

Message details and related content
Hi Peter,

On Tue, Jun 14, 2016 at 01:22:34PM +0200, Peter Zijlstra wrote:
On Tue, Jun 07, 2016 at 08:44:03PM +0530, Gautham R. Shenoy wrote:

I'm still puzzled why we don't see this on x86. Afaict there's nothing
PPC specific about this.
You are right. On PPC, at boot time we hit the WARN_ON like once in 5
times. Using some debug prints, I have verified that these are
instances when the workqueue subsystem gets initialized before all the
CPUs come online. On x86, I have never been able to hit this since it
appears that every time the workqueues get initialized only after all
the CPUs have come online.

PPC doesn't uses any specific unbound workqueue early in the boot. The
unbound workqueues causing the WARN_ON() were the
"events_unbound" workqueue which was created by workqueue_init().

=================================================================================
[WQ] Creating Unbound workers for WQ events_unbound,cpumask 0-127.
     online mask 0
[WQ] Creating Unbound workers for WQ events_unbound,cpumask 0-31. 
     online mask 0
[WQ] Creating Unbound workers for WQ events_unbound,cpumask 32-63.
      online mask 0
[WQ] Creating Unbound workers for WQ events_unbound,cpumask 64-95.
      online mask 0
[WQ] Creating Unbound workers for WQ events_unbound,cpumask 96-127.
      online mask 0
=================================================================================

Also, with the first patch in the series (which ensures that
restore_unbound_workers are called *after* the new workers for the
newly onlined CPUs are created) and without this one, you can
reproduce this WARN_ON on both x86 and PPC by offlining all the CPUs
of a node and bringing just one of them online. So essentially the BUG
fixed by the previous patch is currently hiding this BUG which is why
we are not able to reproduce this WARN_ON() with CPU-hotplug once the
system has booted.
quoted
This patch sets the affinity of the worker to
a) the only online CPU in the cpumask of the worker pool when it comes
   online.
b) the cpumask of the worker pool when the second CPU in the pool's
   cpumask comes online.
This basically works around the WARN conditions, which I suppose is fair
enough, but I would like a note here to revisit this once the whole cpu
hotplug rework has settled.
Sure.
The real problem is that workqueues seem to want to create worker
threads before there's anybody who would use them or something like
that.
I am not sure about that. The workqueue creates unbound workers for a
node via wq_update_unbound_numa() whenever the first CPU of every node
comes online. So that seems legitimate. It then tries to affine these
workers to the cpumask of that node. Again this seems right. As an
optimization, it does this only when the first CPU of the node comes
online. Since this online CPU is not yet active, and since
nr_cpus_allowed > 1, we will hit the WARN_ON().

However, I agree with you that during boot-up, the workqueue subsystem
needs to create unbound worker threads for only the online CPUs
(instead of all possible CPUs as it currently does!) and let the
CPU_ONLINE notification take care of creating the remaining workers
when they are really required.
Or is that what PPC does funny? Use an unbound workqueue this early in
cpu bringup?
Like I pointed out above, PPC doesn't use an unbound workqueue early
in the CPU bring up.

--
Thanks and Regards
gautham.

Re: [PATCH 2/2] workqueue:Fix affinity of an unbound worker of a node with 1 online CPU

Message details and related content
On Wed, Jun 15, 2016 at 03:49:36PM +0530, Gautham R Shenoy wrote:
Also, with the first patch in the series (which ensures that
restore_unbound_workers are called *after* the new workers for the
newly onlined CPUs are created) and without this one, you can
reproduce this WARN_ON on both x86 and PPC by offlining all the CPUs
of a node and bringing just one of them online.
Ah good.
I am not sure about that. The workqueue creates unbound workers for a
node via wq_update_unbound_numa() whenever the first CPU of every node
comes online. So that seems legitimate. It then tries to affine these
workers to the cpumask of that node. Again this seems right. As an
optimization, it does this only when the first CPU of the node comes
online. Since this online CPU is not yet active, and since
nr_cpus_allowed > 1, we will hit the WARN_ON().
So I had another look and isn't the below a much simpler solution?

It seems to work on my x86 with:

  for i in /sys/devices/system/cpu/cpu*/online ; do echo 0 > $i ; done
  for i in /sys/devices/system/cpu/cpu*/online ; do echo 1 > $i ; done

without complaint.

---
 kernel/workqueue.c | 14 +++++---------
 1 file changed, 5 insertions(+), 9 deletions(-)
diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index e1c0e99..09c9160 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -4600,15 +4600,11 @@ static void restore_unbound_workers_cpumask(struct worker_pool *pool, int cpu)
 	if (!cpumask_test_cpu(cpu, pool->attrs->cpumask))
 		return;
 
-	/* is @cpu the only online CPU? */
 	cpumask_and(&cpumask, pool->attrs->cpumask, cpu_online_mask);
-	if (cpumask_weight(&cpumask) != 1)
-		return;
 
 	/* as we're called from CPU_ONLINE, the following shouldn't fail */
 	for_each_pool_worker(worker, pool)
-		WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task,
-						  pool->attrs->cpumask) < 0);
+		WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task, &cpumask) < 0);
 }
 
 /*
@@ -4638,6 +4634,10 @@ static int workqueue_cpu_up_callback(struct notifier_block *nfb,
 	case CPU_ONLINE:
 		mutex_lock(&wq_pool_mutex);
 
+		/* update NUMA affinity of unbound workqueues */
+		list_for_each_entry(wq, &workqueues, list)
+			wq_update_unbound_numa(wq, cpu, true);
+
 		for_each_pool(pool, pi) {
 			mutex_lock(&pool->attach_mutex);
 
@@ -4649,10 +4649,6 @@ static int workqueue_cpu_up_callback(struct notifier_block *nfb,
 			mutex_unlock(&pool->attach_mutex);
 		}
 
-		/* update NUMA affinity of unbound workqueues */
-		list_for_each_entry(wq, &workqueues, list)
-			wq_update_unbound_numa(wq, cpu, true);
-
 		mutex_unlock(&wq_pool_mutex);
 		break;
 	}

Re: [PATCH 2/2] workqueue:Fix affinity of an unbound worker of a node with 1 online CPU

Message details and related content
On Wed, Jun 15, 2016 at 01:32:49PM +0200, Peter Zijlstra wrote:
On Wed, Jun 15, 2016 at 03:49:36PM +0530, Gautham R Shenoy wrote:
quoted
Also, with the first patch in the series (which ensures that
restore_unbound_workers are called *after* the new workers for the
newly onlined CPUs are created) and without this one, you can
reproduce this WARN_ON on both x86 and PPC by offlining all the CPUs
of a node and bringing just one of them online.
Ah good.
quoted
I am not sure about that. The workqueue creates unbound workers for a
node via wq_update_unbound_numa() whenever the first CPU of every node
comes online. So that seems legitimate. It then tries to affine these
workers to the cpumask of that node. Again this seems right. As an
optimization, it does this only when the first CPU of the node comes
online. Since this online CPU is not yet active, and since
nr_cpus_allowed > 1, we will hit the WARN_ON().
So I had another look and isn't the below a much simpler solution?

It seems to work on my x86 with:

  for i in /sys/devices/system/cpu/cpu*/online ; do echo 0 > $i ; done
  for i in /sys/devices/system/cpu/cpu*/online ; do echo 1 > $i ; done

without complaint.
Yup. This will work on PPC as well. 

We will no longer have the optimization in
restore_unbound_workers_cpumask() but I suppose we don't lose much by
resetting the affinity every time a CPU in the pool->attr->cpumask
comes online.

--
Thanks and Regards
gautham.

Re: [PATCH 2/2] workqueue:Fix affinity of an unbound worker of a node with 1 online CPU

Message details and related content
On Wed, Jun 15, 2016 at 06:20:33PM +0530, Gautham R Shenoy wrote:
We will no longer have the optimization in
restore_unbound_workers_cpumask() but I suppose we don't lose much by
resetting the affinity every time a CPU in the pool->attr->cpumask
comes online.
Right; optimizing hotplug really isn't worth it. The code needs to be
simple and robust (ha! funny).

In any case, Tejun, does this work for you?

Re: [PATCH 1/2] workqueue: Move wq_update_unbound_numa() to the beginning of CPU_ONLINE

Message details and related content
Hello,

On Tue, Jun 07, 2016 at 08:44:02PM +0530, Gautham R. Shenoy wrote:
Currently in the CPU_ONLINE workqueue handler, the
restore_unbound_workers_cpumask() will never call
set_cpus_allowed_ptr() for a newly created unbound worker thread.
Hmmm... did you actually verify that this happens?  A new kworker
always gets bound to the cpumask that it's assigned to in
create_worker().

Thanks.

-- 
tejun

Re: [PATCH 2/2] workqueue:Fix affinity of an unbound worker of a node with 1 online CPU

Message details and related content
Hello,

On Wed, Jun 15, 2016 at 03:14:15PM +0200, Peter Zijlstra wrote:
On Wed, Jun 15, 2016 at 06:20:33PM +0530, Gautham R Shenoy wrote:
quoted
We will no longer have the optimization in
restore_unbound_workers_cpumask() but I suppose we don't lose much by
resetting the affinity every time a CPU in the pool->attr->cpumask
comes online.
Right; optimizing hotplug really isn't worth it. The code needs to be
simple and robust (ha! funny).
The only case it might matter is CPU hotplug being used aggressively
for power saving.  No idea how popular that is now tho.
set_cpus_allowed isn't that expensive and phones don't tend to have
massive number of kworkers, so hopefully it won't show up.
In any case, Tejun, does this work for you?
I'm not sure about the reordering part but for setting affinity on
each onlining, no objection.  If it ever shows up as performance /
power regression, we can revisit it later.

Thanks.

-- 
tejun

Re: [PATCH 1/2] workqueue: Move wq_update_unbound_numa() to the beginning of CPU_ONLINE

Message details and related content
Hello Tejun,

On Wed, Jun 15, 2016 at 11:53:50AM -0400, Tejun Heo wrote:
Hello,

On Tue, Jun 07, 2016 at 08:44:02PM +0530, Gautham R. Shenoy wrote:
quoted
Currently in the CPU_ONLINE workqueue handler, the
restore_unbound_workers_cpumask() will never call
set_cpus_allowed_ptr() for a newly created unbound worker thread.
Hmmm... did you actually verify that this happens?  A new kworker
always gets bound to the cpumask that it's assigned to in
create_worker().
Yes I have verified that this happens despite the fact that
create_worker() calls kthread_bind_mask() to bind the worker thread to
attrs->cpumask. However, this doesn't seem to be sufficient.

Consider the following case of a 2-node POWER machine running 4.7-rc3.

CPUs 0-79 belong to the 1st node and CPUs 80-159 belong to the second.

======================================================================
root@fir01:~# uname -r
4.7.0-rc3-vanilla
root@fir01:~# numactl -H
available: 2 nodes (0,8)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79
node 0 size: 65246 MB
node 0 free: 64025 MB
node 8 cpus: 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159
node 8 size: 65304 MB
node 8 free: 64985 MB
node distances:
node   0   8 
  0:  10  40 
  8:  40  10 
======================================================================

If we inspect the unbound worker threads affinity we will observe that
the ordered unbound worker threads (pids 6, 1088, 1122) are affined to
the online CPUs while the remaining unbound worker threads are affined
to the nodemask.
======================================================================
root@fir01:/home/ego# ./pr_unbound_workers_affinity.sh #See [1] below
pid 6's current affinity list: 0-159
pid 7's current affinity list: 0-79
pid 1018's current affinity list: 80-159
pid 1054's current affinity list: 80-159
pid 1088's current affinity list: 0-159
pid 1089's current affinity list: 0-79
pid 1090's current affinity list: 80-159
pid 1122's current affinity list: 0-159

86 more lines

Re: [PATCH 2/2] workqueue:Fix affinity of an unbound worker of a node with 1 online CPU

Message details and related content
On Wed, 2016-06-15 at 12:01 -0400, Tejun Heo wrote:
On Wed, Jun 15, 2016 at 03:14:15PM +0200, Peter Zijlstra wrote:
quoted
On Wed, Jun 15, 2016 at 06:20:33PM +0530, Gautham R Shenoy wrote:
quoted
We will no longer have the optimization in
restore_unbound_workers_cpumask() but I suppose we don't lose much by
resetting the affinity every time a CPU in the pool->attr->cpumask
comes online.
Right; optimizing hotplug really isn't worth it. The code needs to be
simple and robust (ha! funny).
The only case it might matter is CPU hotplug being used aggressively
for power saving.  No idea how popular that is now tho.
set_cpus_allowed isn't that expensive and phones don't tend to have
massive number of kworkers, so hopefully it won't show up.
quoted
In any case, Tejun, does this work for you?
I'm not sure about the reordering part but for setting affinity on
each onlining, no objection.  If it ever shows up as performance /
power regression, we can revisit it later.
Peterz do you want to send a SOB'ed patch, or can we take what you posted and
add your SOB?

And Tejun are you happy to merge this for rc4?

cheers

Re: [PATCH 2/2] workqueue:Fix affinity of an unbound worker of a node with 1 online CPU

Message details and related content
On Thu, Jun 16, 2016 at 10:11:24PM +1000, Michael Ellerman wrote:
Peterz do you want to send a SOB'ed patch, or can we take what you posted and
add your SOB?
So I took Ego's first patch, so as to not steal his credits take that
one and then see below.

---
Subject: workqueue: Fix setting affinity of unbound worker threads
From: Peter Zijlstra <peterz@infradead.org>
Date: Thu Jun 16 14:38:42 CEST 2016

With commit e9d867a67fd03ccc ("sched: Allow per-cpu kernel threads to
run on online && !active"), __set_cpus_allowed_ptr() expects that only
strict per-cpu kernel threads can have affinity to an online CPU which
is not yet active.

This assumption is currently broken in the CPU_ONLINE notification
handler for the workqueues where restore_unbound_workers_cpumask()
calls set_cpus_allowed_ptr() when the first cpu in the unbound
worker's pool->attr->cpumask comes online. Since
set_cpus_allowed_ptr() is called with pool->attr->cpumask in which
only one CPU is online which is not yet active, we get the following
WARN_ON during an CPU online operation.

------------[ cut here ]------------
WARNING: CPU: 40 PID: 248 at kernel/sched/core.c:1166
__set_cpus_allowed_ptr+0x228/0x2e0
Modules linked in:
CPU: 40 PID: 248 Comm: cpuhp/40 Not tainted 4.6.0-autotest+ #4
<..snip..>
Call Trace:
[c000000f273ff920] [c00000000010493c] __set_cpus_allowed_ptr+0x2cc/0x2e0 (unreliable)
[c000000f273ffac0] [c0000000000ed4b0] workqueue_cpu_up_callback+0x2c0/0x470
[c000000f273ffb70] [c0000000000f5c58] notifier_call_chain+0x98/0x100
[c000000f273ffbc0] [c0000000000c5ed0] __cpu_notify+0x70/0xe0
[c000000f273ffc00] [c0000000000c6028] notify_online+0x38/0x50
[c000000f273ffc30] [c0000000000c5214] cpuhp_invoke_callback+0x84/0x250
[c000000f273ffc90] [c0000000000c562c] cpuhp_up_callbacks+0x5c/0x120
[c000000f273ffce0] [c0000000000c64d4] cpuhp_thread_fun+0x184/0x1c0
[c000000f273ffd20] [c0000000000fa050] smpboot_thread_fn+0x290/0x2a0
[c000000f273ffd80] [c0000000000f45b0] kthread+0x110/0x130
[c000000f273ffe30] [c000000000009570] ret_from_kernel_thread+0x5c/0x6c

12 more lines

--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -4600,15 +4600,11 @@ static void restore_unbound_workers_cpum
 	if (!cpumask_test_cpu(cpu, pool->attrs->cpumask))
 		return;
 
-	/* is @cpu the only online CPU? */
 	cpumask_and(&cpumask, pool->attrs->cpumask, cpu_online_mask);
-	if (cpumask_weight(&cpumask) != 1)
-		return;
 
 	/* as we're called from CPU_ONLINE, the following shouldn't fail */
 	for_each_pool_worker(worker, pool)
-		WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task,
-						  pool->attrs->cpumask) < 0);
+		WARN_ON_ONCE(set_cpus_allowed_ptr(worker->task, &cpumask) < 0);
 }
 
 /*

Re: [PATCH 1/2] workqueue: Move wq_update_unbound_numa() to the beginning of CPU_ONLINE

Message details and related content
Hello,

So, the issue of the initial worker not having its affinity set
correctly wasn't caused by the order of the operations.  Reordering
just made set_cpus_allowed tried one more time late enough so that it
hides the race condition most of the time.  The problem is that
CPU_ONLINE callbacks are called while the cpu being onlined is online
but not active and select_fallback_rq() only considers active cpus, so
if a kthread gets scheduled in the meantime and it doesn't have any
cpu which is active in its allowed mask, it's allowed mask gets reset
to cpu_possible_mask.

Would something like the following make sense?

Thanks.
------ 8< ------
Subject: [PATCH] sched: allow kthreads to fallback to online && !active cpus

During CPU hotplug, CPU_ONLINE callbacks are run while the CPU is
online but not active.  A CPU_ONLINE callback may create or bind a
kthread so that its cpus_allowed mask only allows the CPU which is
being brought online.  The kthread may start executing before the CPU
is made active and can end up in select_fallback_rq().

In such cases, the expected behavior is selecting the CPU which is
coming online; however, because select_fallback_rq() only chooses from
active CPUs, it determines that the task doesn't have any viable CPU
in its allowed mask and ends up overriding it to cpu_possible_mask.

CPU_ONLINE callbacks should be able to put kthreads on the CPU which
is coming online.  Update select_fallback_rq() so that it follows
cpu_online() rather than cpu_active() for kthreads.

Signed-off-by: Tejun Heo <tj@kernel.org>
Reported-by: Gautham R Shenoy <redacted>
---
 kernel/sched/core.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 017d539..a12e3db 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -1536,7 +1536,9 @@ static int select_fallback_rq(int cpu, struct task_struct *p)
 	for (;;) {
 		/* Any allowed, online CPU? */
 		for_each_cpu(dest_cpu, tsk_cpus_allowed(p)) {
-			if (!cpu_active(dest_cpu))
+			if (!(p->flags & PF_KTHREAD) && !cpu_active(dest_cpu))
+				continue;
+			if (!cpu_online(dest_cpu))
 				continue;
 			goto out;
 		}

Re: [PATCH 2/2] workqueue:Fix affinity of an unbound worker of a node with 1 online CPU

Message details and related content
On Thu, Jun 16, 2016 at 02:45:48PM +0200, Peter Zijlstra wrote:
Subject: workqueue: Fix setting affinity of unbound worker threads
From: Peter Zijlstra <peterz@infradead.org>
Date: Thu Jun 16 14:38:42 CEST 2016

With commit e9d867a67fd03ccc ("sched: Allow per-cpu kernel threads to
run on online && !active"), __set_cpus_allowed_ptr() expects that only
strict per-cpu kernel threads can have affinity to an online CPU which
is not yet active.

This assumption is currently broken in the CPU_ONLINE notification
handler for the workqueues where restore_unbound_workers_cpumask()
calls set_cpus_allowed_ptr() when the first cpu in the unbound
worker's pool->attr->cpumask comes online. Since
set_cpus_allowed_ptr() is called with pool->attr->cpumask in which
only one CPU is online which is not yet active, we get the following
WARN_ON during an CPU online operation.
Applied to wq/for-4.7-fixes.

Thanks.

-- 
tejun

Re: [PATCH 2/2] workqueue:Fix affinity of an unbound worker of a node with 1 online CPU

Message details and related content
On Thu, 2016-06-16 at 15:39 -0400, Tejun Heo wrote:
On Thu, Jun 16, 2016 at 02:45:48PM +0200, Peter Zijlstra wrote:
quoted
Subject: workqueue: Fix setting affinity of unbound worker threads
From: Peter Zijlstra <peterz@infradead.org>
Date: Thu Jun 16 14:38:42 CEST 2016

With commit e9d867a67fd03ccc ("sched: Allow per-cpu kernel threads to
run on online && !active"), __set_cpus_allowed_ptr() expects that only
strict per-cpu kernel threads can have affinity to an online CPU which
is not yet active.

This assumption is currently broken in the CPU_ONLINE notification
handler for the workqueues where restore_unbound_workers_cpumask()
calls set_cpus_allowed_ptr() when the first cpu in the unbound
worker's pool->attr->cpumask comes online. Since
set_cpus_allowed_ptr() is called with pool->attr->cpumask in which
only one CPU is online which is not yet active, we get the following
WARN_ON during an CPU online operation.
Applied to wq/for-4.7-fixes.
Thanks all.

cheers

Re: [PATCH 1/2] workqueue: Move wq_update_unbound_numa() to the beginning of CPU_ONLINE

Message details and related content
Hi Tejun,
On Thu, Jun 16, 2016 at 03:35:04PM -0400, Tejun Heo wrote:
Hello,

So, the issue of the initial worker not having its affinity set
correctly wasn't caused by the order of the operations.  Reordering
just made set_cpus_allowed tried one more time late enough so that it
hides the race condition most of the time.  The problem is that
CPU_ONLINE callbacks are called while the cpu being onlined is online
but not active and select_fallback_rq() only considers active cpus, so
if a kthread gets scheduled in the meantime and it doesn't have any
cpu which is active in its allowed mask, it's allowed mask gets reset
to cpu_possible_mask.

Would something like the following make sense?

Thanks.
------ 8< ------
Subject: [PATCH] sched: allow kthreads to fallback to online && !active cpus

During CPU hotplug, CPU_ONLINE callbacks are run while the CPU is
online but not active.  A CPU_ONLINE callback may create or bind a
kthread so that its cpus_allowed mask only allows the CPU which is
being brought online.  The kthread may start executing before the CPU
is made active and can end up in select_fallback_rq().

In such cases, the expected behavior is selecting the CPU which is
coming online; however, because select_fallback_rq() only chooses from
active CPUs, it determines that the task doesn't have any viable CPU
in its allowed mask and ends up overriding it to cpu_possible_mask.

CPU_ONLINE callbacks should be able to put kthreads on the CPU which
is coming online.  Update select_fallback_rq() so that it follows
cpu_online() rather than cpu_active() for kthreads.

Signed-off-by: Tejun Heo <tj@kernel.org>
Reported-by: Gautham R Shenoy <redacted>
Hi Tejun,

This patch fixes the issue on POWER. I am able to see the worker
threads of the unbound workqueues of the newly onlined node with this.

Tested-by: Gautham R. Shenoy <redacted>
quoted hunk
---
 kernel/sched/core.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 017d539..a12e3db 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -1536,7 +1536,9 @@ static int select_fallback_rq(int cpu, struct task_struct *p)
 	for (;;) {
 		/* Any allowed, online CPU? */
 		for_each_cpu(dest_cpu, tsk_cpus_allowed(p)) {
-			if (!cpu_active(dest_cpu))
+			if (!(p->flags & PF_KTHREAD) && !cpu_active(dest_cpu))
+				continue;
+			if (!cpu_online(dest_cpu))
 				continue;
 			goto out;
 		}

Re: [PATCH 1/2] workqueue: Move wq_update_unbound_numa() to the beginning of CPU_ONLINE

Message details and related content
On Tue, Jun 21, 2016 at 07:42:31PM +0530, Gautham R Shenoy wrote:
quoted
Subject: [PATCH] sched: allow kthreads to fallback to online && !active cpus

During CPU hotplug, CPU_ONLINE callbacks are run while the CPU is
online but not active.  A CPU_ONLINE callback may create or bind a
kthread so that its cpus_allowed mask only allows the CPU which is
being brought online.  The kthread may start executing before the CPU
is made active and can end up in select_fallback_rq().

In such cases, the expected behavior is selecting the CPU which is
coming online; however, because select_fallback_rq() only chooses from
active CPUs, it determines that the task doesn't have any viable CPU
in its allowed mask and ends up overriding it to cpu_possible_mask.

CPU_ONLINE callbacks should be able to put kthreads on the CPU which
is coming online.  Update select_fallback_rq() so that it follows
cpu_online() rather than cpu_active() for kthreads.

Signed-off-by: Tejun Heo <tj@kernel.org>
Reported-by: Gautham R Shenoy <redacted>
Hi Tejun,

This patch fixes the issue on POWER. I am able to see the worker
threads of the unbound workqueues of the newly onlined node with this.

Tested-by: Gautham R. Shenoy <redacted>
Peter?

-- 
tejun

Re: [PATCH 1/2] workqueue: Move wq_update_unbound_numa() to the beginning of CPU_ONLINE

Message details and related content
On Tue, Jun 21, 2016 at 09:37:09PM +0200, Peter Zijlstra wrote:
Hurm.. So I've applied it, just to get this issue sorted, but I'm not
entirely sure I like it.

I think I prefer ego's version because that makes it harder to get stuff
to run on !active,online cpus. I think we really want to be careful what
gets to run during that state.
The original patch just did set_cpus_allowed one more time late enough
so that the target kthread (in most cases) doesn't have to go through
fallback rq selection afterwards.  I don't know what the long term
solution is but CPU_ONLINE callbacks should be able to bind kthreads
to the new CPU one way or the other.

Thanks.

-- 
tejun

Re: [PATCH 1/2] workqueue: Move wq_update_unbound_numa() to the beginning of CPU_ONLINE

Message details and related content
On Tue, Jun 21, 2016 at 03:43:56PM -0400, Tejun Heo wrote:
On Tue, Jun 21, 2016 at 09:37:09PM +0200, Peter Zijlstra wrote:
quoted
Hurm.. So I've applied it, just to get this issue sorted, but I'm not
entirely sure I like it.

I think I prefer ego's version because that makes it harder to get stuff
to run on !active,online cpus. I think we really want to be careful what
gets to run during that state.
The original patch just did set_cpus_allowed one more time late enough
so that the target kthread (in most cases) doesn't have to go through
fallback rq selection afterwards.  I don't know what the long term
solution is but CPU_ONLINE callbacks should be able to bind kthreads
to the new CPU one way or the other.
Fair enough; clearly I need to stare harder. In any case, patch is on
its way to sched/urgent.

Re: [PATCH 1/2] workqueue: Move wq_update_unbound_numa() to the beginning of CPU_ONLINE

Message details and related content
On Tue, Jun 21, 2016 at 11:36:51AM -0400, Tejun Heo wrote:
On Tue, Jun 21, 2016 at 07:42:31PM +0530, Gautham R Shenoy wrote:
quoted
quoted
Subject: [PATCH] sched: allow kthreads to fallback to online && !active cpus

During CPU hotplug, CPU_ONLINE callbacks are run while the CPU is
online but not active.  A CPU_ONLINE callback may create or bind a
kthread so that its cpus_allowed mask only allows the CPU which is
being brought online.  The kthread may start executing before the CPU
is made active and can end up in select_fallback_rq().

In such cases, the expected behavior is selecting the CPU which is
coming online; however, because select_fallback_rq() only chooses from
active CPUs, it determines that the task doesn't have any viable CPU
in its allowed mask and ends up overriding it to cpu_possible_mask.

CPU_ONLINE callbacks should be able to put kthreads on the CPU which
is coming online.  Update select_fallback_rq() so that it follows
cpu_online() rather than cpu_active() for kthreads.

Signed-off-by: Tejun Heo <tj@kernel.org>
Reported-by: Gautham R Shenoy <redacted>
Hi Tejun,

This patch fixes the issue on POWER. I am able to see the worker
threads of the unbound workqueues of the newly onlined node with this.

Tested-by: Gautham R. Shenoy <redacted>
Peter?
Hurm.. So I've applied it, just to get this issue sorted, but I'm not
entirely sure I like it.

I think I prefer ego's version because that makes it harder to get stuff
to run on !active,online cpus. I think we really want to be careful what
gets to run during that state.

Re: [PATCH 1/2] workqueue: Move wq_update_unbound_numa() to the beginning of CPU_ONLINE

Message details and related content
On Tue, Jun 21, 2016 at 09:47:19PM +0200, Peter Zijlstra wrote:
On Tue, Jun 21, 2016 at 03:43:56PM -0400, Tejun Heo wrote:
quoted
On Tue, Jun 21, 2016 at 09:37:09PM +0200, Peter Zijlstra wrote:
quoted
Hurm.. So I've applied it, just to get this issue sorted, but I'm not
entirely sure I like it.

I think I prefer ego's version because that makes it harder to get stuff
to run on !active,online cpus. I think we really want to be careful what
gets to run during that state.
The original patch just did set_cpus_allowed one more time late enough
so that the target kthread (in most cases) doesn't have to go through
fallback rq selection afterwards.  I don't know what the long term
solution is but CPU_ONLINE callbacks should be able to bind kthreads
to the new CPU one way or the other.
Fair enough; clearly I need to stare harder. In any case, patch is on
its way to sched/urgent.
Thanks Tejun, Peter!
--
Regards
gautham.

Re: [PATCH 2/2] workqueue:Fix affinity of an unbound worker of a node with 1 online CPU

Message details and related content
Hi Tejun,

On Thu, Jun 16, 2016 at 03:39:05PM -0400, Tejun Heo wrote:
On Thu, Jun 16, 2016 at 02:45:48PM +0200, Peter Zijlstra wrote:
quoted
Subject: workqueue: Fix setting affinity of unbound worker threads
From: Peter Zijlstra <peterz@infradead.org>
Date: Thu Jun 16 14:38:42 CEST 2016

With commit e9d867a67fd03ccc ("sched: Allow per-cpu kernel threads to
run on online && !active"), __set_cpus_allowed_ptr() expects that only
strict per-cpu kernel threads can have affinity to an online CPU which
is not yet active.

This assumption is currently broken in the CPU_ONLINE notification
handler for the workqueues where restore_unbound_workers_cpumask()
calls set_cpus_allowed_ptr() when the first cpu in the unbound
worker's pool->attr->cpumask comes online. Since
set_cpus_allowed_ptr() is called with pool->attr->cpumask in which
only one CPU is online which is not yet active, we get the following
WARN_ON during an CPU online operation.
Applied to wq/for-4.7-fixes.
Did this patch get missed by any chance? It is not in
the master branch of
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git yet.

We're still hitting the WARN_ON() during boot-up on the mainline kernel.
Thanks.

-- 
tejun
--
Thanks and Regards
gautham.

Re: [PATCH 2/2] workqueue:Fix affinity of an unbound worker of a node with 1 online CPU

Message details and related content
Tejun Heo [off-list ref] writes:
On Thu, Jun 16, 2016 at 02:45:48PM +0200, Peter Zijlstra wrote:
quoted
Subject: workqueue: Fix setting affinity of unbound worker threads
From: Peter Zijlstra <peterz@infradead.org>
Date: Thu Jun 16 14:38:42 CEST 2016

With commit e9d867a67fd03ccc ("sched: Allow per-cpu kernel threads to
run on online && !active"), __set_cpus_allowed_ptr() expects that only
strict per-cpu kernel threads can have affinity to an online CPU which
is not yet active.

This assumption is currently broken in the CPU_ONLINE notification
handler for the workqueues where restore_unbound_workers_cpumask()
calls set_cpus_allowed_ptr() when the first cpu in the unbound
worker's pool->attr->cpumask comes online. Since
set_cpus_allowed_ptr() is called with pool->attr->cpumask in which
only one CPU is online which is not yet active, we get the following
WARN_ON during an CPU online operation.
Applied to wq/for-4.7-fixes.
It looks like this still hasn't gone to Linus for 4.7?

Could it please, it's a pretty nasty regression on our boxes.

cheers
Keyboard shortcuts
hback out one level
jnext message in thread / next series-diff position
kprevious message in thread / previous series-diff position
ldrill in
Escclose help / fold thread tree
?toggle this help