Hi,
The kthread code was enhanced lately to provide an infrastructure which
manages the preferred affinity of unbound kthreads (node or custom
cpumask) against housekeeping constraints and CPU hotplug events.
One crucial missing piece is cpuset: when an isolated partition is
created, deleted, or its CPUs updated, all the unbound kthreads in the
top cpuset are affine to _all_ the non-isolated CPUs, possibly breaking
their preferred affinity along the way
Solve this with performing the kthreads affinity update from cpuset to
the kthreads consolidated relevant code instead so that preferred
affinities are honoured.
The dispatch of the new cpumasks to workqueues and kthreads is performed
by housekeeping, as per the nice Tejun's suggestion.
As a welcome side effect, HK_TYPE_DOMAIN then integrates both the set
from isolcpus= and cpuset isolated partitions. Housekeeping cpumasks are
now modifyable with specific synchronization. A big step toward making
nohz_full= also mutable through cpuset in the future.
Changes since v5:
* Add more tags
* Fix leaked destroy_work_on_stack() (Zhang Qiao, Waiman Long)
* Comment schedule_drain_work() synchronization requirement (Tejun)
* s/Revert of/Inverse of (Waiman Long)
* Remove housekeeping_update() needless (for now) parameter (Chen Ridong)
* Don't propagate housekeeping_update() failures beyond allocations (Waiman Long)
* Whitespace cleanup (Waiman Long)
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
kthread/core-v6
HEAD: 811e87ca8a0a1e54eb5f23e71896cb97436cccdc
Happy new year,
Frederic
---
Frederic Weisbecker (33):
PCI: Prepare to protect against concurrent isolated cpuset change
cpu: Revert "cpu/hotplug: Prevent self deadlock on CPU hot-unplug"
memcg: Prepare to protect against concurrent isolated cpuset change
mm: vmstat: Prepare to protect against concurrent isolated cpuset change
sched/isolation: Save boot defined domain flags
cpuset: Convert boot_hk_cpus to use HK_TYPE_DOMAIN_BOOT
driver core: cpu: Convert /sys/devices/system/cpu/isolated to use HK_TYPE_DOMAIN_BOOT
net: Keep ignoring isolated cpuset change
block: Protect against concurrent isolated cpuset change
timers/migration: Prevent from lockdep false positive warning
cpu: Provide lockdep check for CPU hotplug lock write-held
cpuset: Provide lockdep check for cpuset lock held
sched/isolation: Convert housekeeping cpumasks to rcu pointers
cpuset: Update HK_TYPE_DOMAIN cpumask from cpuset
sched/isolation: Flush memcg workqueues on cpuset isolated partition change
sched/isolation: Flush vmstat workqueues on cpuset isolated partition change
PCI: Flush PCI probe workqueue on cpuset isolated partition change
cpuset: Propagate cpuset isolation update to workqueue through housekeeping
cpuset: Propagate cpuset isolation update to timers through housekeeping
timers/migration: Remove superfluous cpuset isolation test
cpuset: Remove cpuset_cpu_is_isolated()
sched/isolation: Remove HK_TYPE_TICK test from cpu_is_isolated()
PCI: Remove superfluous HK_TYPE_WQ check
kthread: Refine naming of affinity related fields
kthread: Include unbound kthreads in the managed affinity list
kthread: Include kthreadd to the managed affinity list
kthread: Rely on HK_TYPE_DOMAIN for preferred affinity management
sched: Switch the fallback task allowed cpumask to HK_TYPE_DOMAIN
sched/arm64: Move fallback task cpumask to HK_TYPE_DOMAIN
kthread: Honour kthreads preferred affinity after cpuset changes
kthread: Comment on the purpose and placement of kthread_affine_node() call
kthread: Document kthread_affine_preferred()
doc: Add housekeeping documentation
Documentation/core-api/housekeeping.rst | 111 ++++++++++++++++++++++
Documentation/core-api/index.rst | 1 +
arch/arm64/kernel/cpufeature.c | 18 +++-
block/blk-mq.c | 6 +-
drivers/base/cpu.c | 2 +-
drivers/pci/pci-driver.c | 71 ++++++++++----
include/linux/cpu.h | 4 +
include/linux/cpuhplock.h | 1 +
include/linux/cpuset.h | 8 +-
include/linux/kthread.h | 1 +
include/linux/memcontrol.h | 4 +
include/linux/mmu_context.h | 2 +-
include/linux/pci.h | 3 +
include/linux/percpu-rwsem.h | 1 +
include/linux/sched/isolation.h | 16 +++-
include/linux/vmstat.h | 2 +
include/linux/workqueue.h | 2 +-
init/Kconfig | 1 +
kernel/cgroup/cpuset.c | 68 +++++++-------
kernel/cpu.c | 42 ++++-----
kernel/kthread.c | 160 +++++++++++++++++++++-----------
kernel/sched/isolation.c | 141 +++++++++++++++++++++++-----
kernel/sched/sched.h | 4 +
kernel/time/timer_migration.c | 25 +++--
kernel/workqueue.c | 17 ++--
mm/memcontrol.c | 31 ++++++-
mm/vmstat.c | 15 ++-
net/core/net-sysfs.c | 2 +-
28 files changed, 557 insertions(+), 202 deletions(-)
HK_TYPE_DOMAIN will soon integrate cpuset isolated partitions and
therefore be made modifiable at runtime. Synchronize against the cpumask
update using RCU.
The RCU locked section includes both the housekeeping CPU target
election for the PCI probe work and the work enqueue.
This way the housekeeping update side will simply need to flush the
pending related works after updating the housekeeping mask in order to
make sure that no PCI work ever executes on an isolated CPU. This part
will be handled in a subsequent patch.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
---
drivers/pci/pci-driver.c | 47 ++++++++++++++++++++++++++++++++--------
1 file changed, 38 insertions(+), 9 deletions(-)
1) The commit:
2b8272ff4a70 ("cpu/hotplug: Prevent self deadlock on CPU hot-unplug")
was added to fix an issue where the hotplug control task (BP) was
throttled between CPUHP_AP_IDLE_DEAD and CPUHP_HRTIMERS_PREPARE waiting
in the hrtimer blindspot for the bandwidth callback queued in the dead
CPU.
2) Later on, the commit:
38685e2a0476 ("cpu/hotplug: Don't offline the last non-isolated CPU")
plugged on the target selection for the workqueue offloaded CPU down
process to prevent from destroying the last CPU domain.
3) Finally:
5c0930ccaad5 ("hrtimers: Push pending hrtimers away from outgoing CPU earlier")
removed entirely the conditions for the race exposed and partially fixed
in 1). The offloading of the CPU down process to a workqueue on another
CPU then becomes unnecessary. But the last CPU belonging to scheduler
domains must still remain online.
Therefore revert the now obsolete commit
2b8272ff4a70b866106ae13c36be7ecbef5d5da2 and move the housekeeping check
under the cpu_hotplug_lock write held. Since HK_TYPE_DOMAIN will include
both isolcpus and cpuset isolated partition, the hotplug lock will
synchronize against concurrent cpuset partition updates.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
---
kernel/cpu.c | 37 +++++++++++--------------------------
1 file changed, 11 insertions(+), 26 deletions(-)
@@ -1410,6 +1410,16 @@ static int __ref _cpu_down(unsigned int cpu, int tasks_frozen,cpus_write_lock();+/*+*Keepatleastonehousekeepingcpuonlinedtoavoidgenerating+*anemptysched_domainspan.+*/+if(cpumask_any_and(cpu_online_mask,+housekeeping_cpumask(HK_TYPE_DOMAIN))>=nr_cpu_ids){+ret=-EBUSY;+gotoout;+}+cpuhp_tasks_frozen=tasks_frozen;prev_state=cpuhp_set_state(cpu,st,target);
@@ -1456,22 +1466,8 @@ static int __ref _cpu_down(unsigned int cpu, int tasks_frozen,returnret;}-structcpu_down_work{-unsignedintcpu;-enumcpuhp_statetarget;-};--staticlong__cpu_down_maps_locked(void*arg)-{-structcpu_down_work*work=arg;--return_cpu_down(work->cpu,0,work->target);-}-staticintcpu_down_maps_locked(unsignedintcpu,enumcpuhp_statetarget){-structcpu_down_workwork={.cpu=cpu,.target=target,};-/**Iftheplatformdoesnotsupporthotplug,reportitexplicitlyto*differentiateitfromatransientoffliningfailure.
@@ -1480,18 +1476,7 @@ static int cpu_down_maps_locked(unsigned int cpu, enum cpuhp_state target)return-EOPNOTSUPP;if(cpu_hotplug_disabled)return-EBUSY;--/*-*Ensurethatthecontroltaskdoesnotrunonthetobeofflined-*CPUtopreventadeadlockagainstcfs_b->period_timer.-*Alsokeepatleastonehousekeepingcpuonlinedtoavoidgenerating-*anemptysched_domainspan.-*/-for_each_cpu_and(cpu,cpu_online_mask,housekeeping_cpumask(HK_TYPE_DOMAIN)){-if(cpu!=work.cpu)-returnwork_on_cpu(cpu,__cpu_down_maps_locked,&work);-}-return-EBUSY;+return_cpu_down(cpu,0,target);}staticintcpu_down(unsignedintcpu,enumcpuhp_statetarget)
The HK_TYPE_DOMAIN housekeeping cpumask will soon be made modifiable at
runtime. In order to synchronize against memcg workqueue to make sure
that no asynchronous draining is pending or executing on a newly made
isolated CPU, target and queue a drain work under the same RCU critical
section.
Whenever housekeeping will update the HK_TYPE_DOMAIN cpumask, a memcg
workqueue flush will also be issued in a further change to make sure
that no work remains pending after a CPU has been made isolated.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
---
mm/memcontrol.c | 21 +++++++++++++++++----
1 file changed, 17 insertions(+), 4 deletions(-)
The HK_TYPE_DOMAIN housekeeping cpumask will soon be made modifiable at
runtime. In order to synchronize against vmstat workqueue to make sure
that no asynchronous vmstat work is pending or executing on a newly made
isolated CPU, target and queue a vmstat work under the same RCU read
side critical section.
Whenever housekeeping will update the HK_TYPE_DOMAIN cpumask, a vmstat
workqueue flush will also be issued in a further change to make sure
that no work remains pending after a CPU has been made isolated.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
---
mm/vmstat.c | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
HK_TYPE_DOMAIN will soon integrate not only boot defined isolcpus= CPUs
but also cpuset isolated partitions.
Housekeeping still needs a way to record what was initially passed
to isolcpus= in order to keep these CPUs isolated after a cpuset
isolated partition is modified or destroyed while containing some of
them.
Create a new HK_TYPE_DOMAIN_BOOT to keep track of those.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Reviewed-by: Phil Auld <redacted>
---
include/linux/sched/isolation.h | 4 ++++
kernel/sched/isolation.c | 5 +++--
2 files changed, 7 insertions(+), 2 deletions(-)
boot_hk_cpus is an ad-hoc copy of HK_TYPE_DOMAIN_BOOT. Remove it and use
the official version.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Reviewed-by: Phil Auld <redacted>
Reviewed-by: Chen Ridong <redacted>
Reviewed-by: Waiman Long <longman@redhat.com>
---
kernel/cgroup/cpuset.c | 22 +++++++---------------
1 file changed, 7 insertions(+), 15 deletions(-)
Make sure /sys/devices/system/cpu/isolated only prints what was passed
through the isolcpus= parameter before HK_TYPE_DOMAIN will also
integrate cpuset isolated partitions.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
---
drivers/base/cpu.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
RPS cpumask can be overriden through sysfs/syctl. The boot defined
isolated CPUs are then excluded from that cpumask.
However HK_TYPE_DOMAIN will soon integrate cpuset isolated
CPUs updates and the RPS infrastructure needs more thoughts to be able
to propagate such changes and synchronize against them.
Keep handling only what was passed through "isolcpus=" for now.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
---
net/core/net-sysfs.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
The block subsystem prevents running the workqueue to isolated CPUs,
including those defined by cpuset isolated partitions. Since
HK_TYPE_DOMAIN will soon contain both and be subject to runtime
modifications, synchronize against housekeeping using the relevant lock.
For full support of cpuset changes, the block subsystem may need to
propagate changes to isolated cpumask through the workqueue in the
future.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Acked-by: Jens Axboe <axboe@kernel.dk>
---
block/blk-mq.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
Testing housekeeping_cpu() will soon require that either the RCU "lock"
is held or the cpuset mutex.
When CPUs get isolated through cpuset, the change is propagated to
timer migration such that isolation is also performed from the migration
tree. However that propagation is done using workqueue which tests if
the target is actually isolated before proceeding.
Lockdep doesn't know that the workqueue caller holds cpuset mutex and
that it waits for the work, making the housekeeping cpumask read safe.
Shut down the future warning by removing this test. It is unecessary
beyond hotplug, the workqueue is already targeted towards isolated CPUs.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
---
kernel/time/timer_migration.c | 20 +++++++++++++++-----
1 file changed, 15 insertions(+), 5 deletions(-)
@@ -1497,7 +1497,7 @@ static int tmigr_clear_cpu_available(unsigned int cpu)return0;}-staticinttmigr_set_cpu_available(unsignedintcpu)+staticint__tmigr_set_cpu_available(unsignedintcpu){structtmigr_cpu*tmc=this_cpu_ptr(&tmigr_cpu);
@@ -1505,9 +1505,6 @@ static int tmigr_set_cpu_available(unsigned int cpu)if(WARN_ON_ONCE(!tmc->tmgroup))return-EINVAL;-if(tmigr_is_isolated(cpu))-return0;-guard(mutex)(&tmigr_available_mutex);cpumask_set_cpu(cpu,tmigr_available_cpumask);
@@ -1523,6 +1520,14 @@ static int tmigr_set_cpu_available(unsigned int cpu)return0;}+staticinttmigr_set_cpu_available(unsignedintcpu)+{+if(tmigr_is_isolated(cpu))+return0;++return__tmigr_set_cpu_available(cpu);+}+staticvoidtmigr_cpu_isolate(structwork_struct*ignored){tmigr_clear_cpu_available(smp_processor_id());
cpuset modifies partitions, including isolated, while holding the cpu
hotplug lock read-held.
This means that write-holding the CPU hotplug lock is safe to
synchronize against housekeeping cpumask changes.
Provide a lockdep check to validate that.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
---
include/linux/cpuhplock.h | 1 +
include/linux/percpu-rwsem.h | 1 +
kernel/cpu.c | 5 +++++
3 files changed, 7 insertions(+)
cpuset modifies partitions, including isolated, while holding the cpuset
mutex.
This means that holding the cpuset mutex is safe to synchronize against
housekeeping cpumask changes.
Provide a lockdep check to validate that.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
---
include/linux/cpuset.h | 2 ++
kernel/cgroup/cpuset.c | 7 +++++++
2 files changed, 9 insertions(+)
HK_TYPE_DOMAIN's cpumask will soon be made modifiable by cpuset.
A synchronization mechanism is then needed to synchronize the updates
with the housekeeping cpumask readers.
Turn the housekeeping cpumasks into RCU pointers. Once a housekeeping
cpumask will be modified, the update side will wait for an RCU grace
period and propagate the change to interested subsystem when deemed
necessary.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
---
kernel/sched/isolation.c | 58 +++++++++++++++++++++++++---------------
kernel/sched/sched.h | 1 +
2 files changed, 37 insertions(+), 22 deletions(-)
@@ -59,28 +70,18 @@ int housekeeping_any_cpu(enum hk_type type)}EXPORT_SYMBOL_GPL(housekeeping_any_cpu);-conststructcpumask*housekeeping_cpumask(enumhk_typetype)-{-if(static_branch_unlikely(&housekeeping_overridden))-if(housekeeping.flags&BIT(type))-returnhousekeeping.cpumasks[type];-returncpu_possible_mask;-}-EXPORT_SYMBOL_GPL(housekeeping_cpumask);-voidhousekeeping_affine(structtask_struct*t,enumhk_typetype){if(static_branch_unlikely(&housekeeping_overridden))if(housekeeping.flags&BIT(type))-set_cpus_allowed_ptr(t,housekeeping.cpumasks[type]);+set_cpus_allowed_ptr(t,housekeeping_cpumask(type));}EXPORT_SYMBOL_GPL(housekeeping_affine);boolhousekeeping_test_cpu(intcpu,enumhk_typetype){-if(static_branch_unlikely(&housekeeping_overridden))-if(housekeeping.flags&BIT(type))-returncpumask_test_cpu(cpu,housekeeping.cpumasks[type]);+if(static_branch_unlikely(&housekeeping_overridden)&&housekeeping.flags&BIT(type))+returncpumask_test_cpu(cpu,housekeeping_cpumask(type));returntrue;}EXPORT_SYMBOL_GPL(housekeeping_test_cpu);
@@ -96,20 +97,33 @@ void __init housekeeping_init(void)if(housekeeping.flags&HK_FLAG_KERNEL_NOISE)sched_tick_offload_init();-+/*+*Reallocwithaproperallocatorsothatanycpumaskupdate+*canindifferentlyfreetheoldversionwithkfree().+*/for_each_set_bit(type,&housekeeping.flags,HK_TYPE_MAX){+structcpumask*omask,*nmask=kmalloc(cpumask_size(),GFP_KERNEL);++if(WARN_ON_ONCE(!nmask))+return;++omask=rcu_dereference(housekeeping.cpumasks[type]);+/* We need at least one CPU to handle housekeeping work */-WARN_ON_ONCE(cpumask_empty(housekeeping.cpumasks[type]));+WARN_ON_ONCE(cpumask_empty(omask));+cpumask_copy(nmask,omask);+RCU_INIT_POINTER(housekeeping.cpumasks[type],nmask);+memblock_free(omask,cpumask_size());}}staticvoid__inithousekeeping_setup_type(enumhk_typetype,cpumask_var_thousekeeping_staging){+structcpumask*mask=memblock_alloc_or_panic(cpumask_size(),SMP_CACHE_BYTES);-alloc_bootmem_cpumask_var(&housekeeping.cpumasks[type]);-cpumask_copy(housekeeping.cpumasks[type],-housekeeping_staging);+cpumask_copy(mask,housekeeping_staging);+RCU_INIT_POINTER(housekeeping.cpumasks[type],mask);}staticint__inithousekeeping_setup(char*str,unsignedlongflags)
@@ -162,7 +176,7 @@ static int __init housekeeping_setup(char *str, unsigned long flags)for_each_set_bit(type,&iter_flags,HK_TYPE_MAX){if(!cpumask_equal(housekeeping_staging,-housekeeping.cpumasks[type])){+housekeeping_cpumask(type))){pr_warn("Housekeeping: nohz_full= must match isolcpus=\n");gotofree_housekeeping_staging;}
Until now, HK_TYPE_DOMAIN used to only include boot defined isolated
CPUs passed through isolcpus= boot option. Users interested in also
knowing the runtime defined isolated CPUs through cpuset must use
different APIs: cpuset_cpu_is_isolated(), cpu_is_isolated(), etc...
There are many drawbacks to that approach:
1) Most interested subsystems want to know about all isolated CPUs, not
just those defined on boot time.
2) cpuset_cpu_is_isolated() / cpu_is_isolated() are not synchronized with
concurrent cpuset changes.
3) Further cpuset modifications are not propagated to subsystems
Solve 1) and 2) and centralize all isolated CPUs within the
HK_TYPE_DOMAIN housekeeping cpumask.
Subsystems can rely on RCU to synchronize against concurrent changes.
The propagation mentioned in 3) will be handled in further patches.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Reviewed-by: Waiman Long <longman@redhat.com>
Reviewed-by: Chen Ridong <redacted>
---
include/linux/sched/isolation.h | 7 ++++
kernel/cgroup/cpuset.c | 3 ++
kernel/sched/isolation.c | 73 ++++++++++++++++++++++++++++++---
kernel/sched/sched.h | 1 +
4 files changed, 78 insertions(+), 6 deletions(-)
The HK_TYPE_DOMAIN housekeeping cpumask is now modifiable at runtime. In
order to synchronize against memcg workqueue to make sure that no
asynchronous draining is still pending or executing on a newly made
isolated CPU, the housekeeping susbsystem must flush the memcg
workqueues.
However the memcg workqueues can't be flushed easily since they are
queued to the main per-CPU workqueue pool.
Solve this with creating a memcg specific pool and provide and use the
appropriate flushing API.
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
---
include/linux/memcontrol.h | 4 ++++
kernel/sched/isolation.c | 2 ++
kernel/sched/sched.h | 1 +
mm/memcontrol.c | 12 +++++++++++-
4 files changed, 18 insertions(+), 1 deletion(-)
The HK_TYPE_DOMAIN housekeeping cpumask is now modifiable at runtime.
In order to synchronize against vmstat workqueue to make sure
that no asynchronous vmstat work is still pending or executing on a
newly made isolated CPU, the housekeeping susbsystem must flush the
vmstat workqueues.
This involves flushing the whole mm_percpu_wq workqueue, shared with
LRU drain, introducing here a welcome side effect.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
---
include/linux/vmstat.h | 2 ++
kernel/sched/isolation.c | 1 +
kernel/sched/sched.h | 1 +
mm/vmstat.c | 5 +++++
4 files changed, 9 insertions(+)
The HK_TYPE_DOMAIN housekeeping cpumask is now modifiable at runtime. In
order to synchronize against PCI probe works and make sure that no
asynchronous probing is still pending or executing on a newly isolated
CPU, the housekeeping subsystem must flush the PCI probe works.
However the PCI probe works can't be flushed easily since they are
queued to the main per-CPU workqueue pool.
Solve this with creating a PCI probe-specific pool and provide and use
the appropriate flushing API.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
---
drivers/pci/pci-driver.c | 17 ++++++++++++++++-
include/linux/pci.h | 3 +++
kernel/sched/isolation.c | 2 ++
3 files changed, 21 insertions(+), 1 deletion(-)
Until now, cpuset would propagate isolated partition changes to
workqueues so that unbound workers get properly reaffined.
Since housekeeping now centralizes, synchronize and propagates isolation
cpumask changes, perform the work from that subsystem for consolidation
and consistency purposes.
For simplification purpose, the target function is adapted to take the
new housekeeping mask instead of the isolated mask.
Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Reviewed-by: Waiman Long <longman@redhat.com>
Acked-by: Tejun Heo <tj@kernel.org>
---
include/linux/workqueue.h | 2 +-
init/Kconfig | 1 +
kernel/cgroup/cpuset.c | 9 +++------
kernel/sched/isolation.c | 3 +++
kernel/workqueue.c | 17 ++++++++++-------
5 files changed, 18 insertions(+), 14 deletions(-)
@@ -6959,13 +6959,16 @@ static int workqueue_apply_unbound_cpumask(const cpumask_var_t unbound_cpumask)}/**-*workqueue_unbound_exclude_cpumask-ExcludegivenCPUsfromunboundcpumask-*@exclude_cpumask:thecpumasktobeexcludedfromwq_unbound_cpumask+*workqueue_unbound_housekeeping_update-Propagatehousekeepingcpumaskupdate+*@hk:thenewhousekeepingcpumask*-*Thisfunctioncanbecalledfromcpusetcodetoprovideasetofisolated-*CPUsthatshouldbeexcludedfromwq_unbound_cpumask.+*Updatetheunboundworkqueuecpumaskontopofthenewhousekeepingcpumasksuch+*thattheeffectiveunboundaffinityistheintersectionofthenewhousekeeping+*withtherequestedaffinitysetvianohz_full=/isolcpus=orsysfs.+*+*Return:0onsuccessand-errnoonfailure.*/-intworkqueue_unbound_exclude_cpumask(cpumask_var_texclude_cpumask)+intworkqueue_unbound_housekeeping_update(conststructcpumask*hk){cpumask_var_tcpumask;intret=0;
@@ -6981,14 +6984,14 @@ int workqueue_unbound_exclude_cpumask(cpumask_var_t exclude_cpumask)*(HK_TYPE_WQ∩HK_TYPE_DOMAIN)housekeepingmaskandrewritten*byanysubsequentwritetoworkqueue/cpumasksysfsfile.*/-if(!cpumask_andnot(cpumask,wq_requested_unbound_cpumask,exclude_cpumask))+if(!cpumask_and(cpumask,wq_requested_unbound_cpumask,hk))cpumask_copy(cpumask,wq_requested_unbound_cpumask);if(!cpumask_equal(cpumask,wq_unbound_cpumask))ret=workqueue_apply_unbound_cpumask(cpumask);/* Save the current isolated cpumask & export it via sysfs */if(!ret)-cpumask_copy(wq_isolated_cpumask,exclude_cpumask);+cpumask_andnot(wq_isolated_cpumask,cpu_possible_mask,hk);mutex_unlock(&wq_pool_mutex);free_cpumask_var(cpumask);
Until now, cpuset would propagate isolated partition changes to
timer migration so that unbound timers don't get migrated to isolated
CPUs.
Since housekeeping now centralizes, synchronize and propagates isolation
cpumask changes, perform the work from that subsystem for consolidation
and consistency purposes.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
---
kernel/cgroup/cpuset.c | 3 ---
kernel/sched/isolation.c | 4 ++++
2 files changed, 4 insertions(+), 3 deletions(-)
Cpuset isolated partitions are now included in HK_TYPE_DOMAIN. Testing
if a CPU is part of an isolated partition alone is now useless.
Remove the superflous test.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Reviewed-by: Waiman Long <longman@redhat.com>
---
kernel/time/timer_migration.c | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
The set of cpuset isolated CPUs is now included in HK_TYPE_DOMAIN
housekeeping cpumask. There is no usecase left interested in just
checking what is isolated by cpuset and not by the isolcpus= kernel
boot parameter.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Reviewed-by: Waiman Long <longman@redhat.com>
---
include/linux/cpuset.h | 6 ------
include/linux/sched/isolation.h | 4 +---
kernel/cgroup/cpuset.c | 12 ------------
3 files changed, 1 insertion(+), 21 deletions(-)
It doesn't make sense to use nohz_full without also isolating the
related CPUs from the domain topology, either through the use of
isolcpus= or cpuset isolated partitions.
And now HK_TYPE_DOMAIN includes all kinds of domain isolated CPUs.
This means that HK_TYPE_KERNEL_NOISE (of which HK_TYPE_TICK is only an
alias) should always be a subset of HK_TYPE_DOMAIN.
Therefore if a CPU is not HK_TYPE_DOMAIN, it shouldn't be
HK_TYPE_KERNEL_NOISE either. Testing the former is then enough.
Simplify cpu_is_isolated() accordingly.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Acked-by: Waiman Long <longman@redhat.com>
---
include/linux/sched/isolation.h | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
It doesn't make sense to use nohz_full without also isolating the
related CPUs from the domain topology, either through the use of
isolcpus= or cpuset isolated partitions.
And now HK_TYPE_DOMAIN includes all kinds of domain isolated CPUs.
This means that HK_TYPE_KERNEL_NOISE (of which HK_TYPE_WQ is only an
alias) should always be a subset of HK_TYPE_DOMAIN.
Therefore sane configurations verify:
HK_TYPE_KERNEL_NOISE | HK_TYPE_DOMAIN == HK_TYPE_DOMAIN
Simplify the PCI probe target election accordingly.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
---
drivers/pci/pci-driver.c | 17 +++--------------
1 file changed, 3 insertions(+), 14 deletions(-)
The kthreads preferred affinity related fields use "hotplug" as the base
of their naming because the affinity management was initially deemed to
deal with CPU hotplug.
The scope of this role is going to broaden now and also deal with
cpuset isolated partition updates.
Switch the naming accordingly.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Acked-by: Waiman Long <longman@redhat.com>
---
kernel/kthread.c | 38 +++++++++++++++++++-------------------
1 file changed, 19 insertions(+), 19 deletions(-)
@@ -69,7 +69,7 @@ struct kthread {/* To store the full name if task comm is truncated. */char*full_name;structtask_struct*task;-structlist_headhotplug_node;+structlist_headaffinity_node;structcpumask*preferred_affinity;};
@@ -903,9 +903,9 @@ static int kthreads_online_cpu(unsigned int cpu)structkthread*k;intret;-guard(mutex)(&kthreads_hotplug_lock);+guard(mutex)(&kthread_affinity_lock);-if(list_empty(&kthreads_hotplug))+if(list_empty(&kthread_affinity_list))return0;if(!zalloc_cpumask_var(&affinity,GFP_KERNEL))
@@ -913,7 +913,7 @@ static int kthreads_online_cpu(unsigned int cpu)ret=0;-list_for_each_entry(k,&kthreads_hotplug,hotplug_node){+list_for_each_entry(k,&kthread_affinity_list,affinity_node){if(WARN_ON_ONCE((k->task->flags&PF_NO_SETAFFINITY)||kthread_is_per_cpu(k->task))){ret=-EINVAL;
The managed affinity list currently contains only unbound kthreads that
have affinity preferences. Unbound kthreads globally affine by default
are outside of the list because their affinity is automatically managed
by the scheduler (through the fallback housekeeping mask) and by cpuset.
However in order to preserve the preferred affinity of kthreads, cpuset
will delegate the isolated partition update propagation to the
housekeeping and kthread code.
Prepare for that with including all unbound kthreads in the managed
affinity list.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Reviewed-by: Waiman Long <longman@redhat.com>
---
kernel/kthread.c | 70 ++++++++++++++++++++++++++++--------------------
1 file changed, 41 insertions(+), 29 deletions(-)
@@ -919,8 +917,22 @@ static int kthreads_online_cpu(unsigned int cpu)ret=-EINVAL;continue;}-kthread_fetch_affinity(k,affinity);-set_cpus_allowed_ptr(k->task,affinity);++/*+*Unboundkthreadswithoutpreferredaffinityarealreadyaffine+*tohousekeeping,whetherthoseCPUsareonlineornot.Sononeed+*tohandlenewlyonlineCPUsforthem.+*+*Butkthreadswithapreferredaffinityornodearedifferent:+*ifnoneoftheirpreferredCPUsareonlineandpartof+*housekeepingatthesametime,theymustbeaffinetohousekeeping.+*ButassoonasoneoftheirpreferredCPUbecomesonline,theymust+*beaffinetothem.+*/+if(k->preferred_affinity||k->node!=NUMA_NO_NODE){+kthread_fetch_affinity(k,affinity);+set_cpus_allowed_ptr(k->task,affinity);+}}free_cpumask_var(affinity);
The unbound kthreads affinity management performed by cpuset is going to
be imported to the kthread core code for consolidation purposes.
Treat kthreadd just like any other kthread.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Reviewed-by: Waiman Long <longman@redhat.com>
---
kernel/kthread.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
@@ -818,12 +818,13 @@ int kthreadd(void *unused)/* Setup a clean context for our children to inherit. */set_task_comm(tsk,comm);ignore_signals(tsk);-set_cpus_allowed_ptr(tsk,housekeeping_cpumask(HK_TYPE_KTHREAD));set_mems_allowed(node_states[N_MEMORY]);current->flags|=PF_NOFREEZE;cgroup_init_kthreadd();+kthread_affine_node();+for(;;){set_current_state(TASK_INTERRUPTIBLE);if(list_empty(&kthread_create_list))
Unbound kthreads want to run neither on nohz_full CPUs nor on domain
isolated CPUs. And since nohz_full implies domain isolation, checking
the latter is enough to verify both.
Therefore exclude kthreads from domain isolation.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Reviewed-by: Waiman Long <longman@redhat.com>
---
kernel/kthread.c | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
Tasks that have all their allowed CPUs offline don't want their affinity
to fallback on either nohz_full CPUs or on domain isolated CPUs. And
since nohz_full implies domain isolation, checking the latter is enough
to verify both.
Therefore exclude domain isolation from fallback task affinity.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Acked-by: Waiman Long <longman@redhat.com>
---
include/linux/mmu_context.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
When none of the allowed CPUs of a task are online, it gets migrated
to the fallback cpumask which is all the non nohz_full CPUs.
However just like nohz_full CPUs, domain isolated CPUs don't want to be
disturbed by tasks that have lost their CPU affinities.
And since nohz_full rely on domain isolation to work correctly, the
housekeeping mask of domain isolated CPUs should always be a superset of
the housekeeping mask of nohz_full CPUs (there can be CPUs that are
domain isolated but not nohz_full, OTOH there shouldn't be nohz_full
CPUs that are not domain isolated):
HK_TYPE_DOMAIN | HK_TYPE_KERNEL_NOISE == HK_TYPE_DOMAIN
Therefore use HK_TYPE_DOMAIN as the appropriate fallback target for
tasks and since this cpumask can be modified at runtime, make sure
that 32 bits support CPUs on ARM64 mismatched systems are not isolated
by cpusets.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Reviewed-by: Waiman Long <longman@redhat.com>
---
arch/arm64/kernel/cpufeature.c | 18 +++++++++++++++---
include/linux/cpu.h | 4 ++++
kernel/cgroup/cpuset.c | 17 ++++++++++++++---
3 files changed, 33 insertions(+), 6 deletions(-)
@@ -3987,8 +3999,8 @@ static int enable_mismatched_32bit_el0(unsigned int cpu)boolcpu_32bit=false;if(id_aa64pfr0_32bit_el0(info->reg_id_aa64pfr0)){-if(!housekeeping_cpu(cpu,HK_TYPE_TICK))-pr_info("Treating adaptive-ticks CPU %u as 64-bit only\n",cpu);+if(!housekeeping_cpu(cpu,HK_TYPE_DOMAIN))+pr_info("Treating domain isolated CPU %u as 64-bit only\n",cpu);elsecpu_32bit=true;}
When cpuset isolated partitions get updated, unbound kthreads get
indifferently affine to all non isolated CPUs, regardless of their
individual affinity preferences.
For example kswapd is a per-node kthread that prefers to be affine to
the node it refers to. Whenever an isolated partition is created,
updated or deleted, kswapd's node affinity is going to be broken if any
CPU in the related node is not isolated because kswapd will be affine
globally.
Fix this with letting the consolidated kthread managed affinity code do
the affinity update on behalf of cpuset.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Reviewed-by: Waiman Long <longman@redhat.com>
---
include/linux/kthread.h | 1 +
kernel/cgroup/cpuset.c | 5 ++---
kernel/kthread.c | 41 ++++++++++++++++++++++++++++++----------
kernel/sched/isolation.c | 3 +++
4 files changed, 37 insertions(+), 13 deletions(-)
@@ -924,7 +917,8 @@ static int kthreads_online_cpu(unsigned int cpu)/**Unboundkthreadswithoutpreferredaffinityarealreadyaffine*tohousekeeping,whetherthoseCPUsareonlineornot.Sononeed-*tohandlenewlyonlineCPUsforthem.+*tohandlenewlyonlineCPUsforthem.Howeverhousekeepingchanges+*havetobeapplied.**Butkthreadswithapreferredaffinityornodearedifferent:*ifnoneoftheirpreferredCPUsareonlineandpartof
@@ -932,7 +926,7 @@ static int kthreads_online_cpu(unsigned int cpu)*ButassoonasoneoftheirpreferredCPUbecomesonline,theymust*beaffinetothem.*/-if(k->preferred_affinity||k->node!=NUMA_NO_NODE){+if(force||k->preferred_affinity||k->node!=NUMA_NO_NODE){kthread_fetch_affinity(k,affinity);set_cpus_allowed_ptr(k->task,affinity);}
@@ -943,6 +937,33 @@ static int kthreads_online_cpu(unsigned int cpu)returnret;}+/**+*kthreads_update_housekeeping-Updatekthreadsaffinityoncpusetchange+*+*Whencpusetchangesapartitiontypeto/from"isolated"orupdatesrelated+*cpumasks,propagatethehousekeepingcpumaskchangetopreferredkthreads+*affinity.+*+*Returns0ifsuccessful,-ENOMEMiftemporarymaskcouldn't+*beallocatedor-EINVALincaseofinternalerror.+*/+intkthreads_update_housekeeping(void)+{+returnkthreads_update_affinity(true);+}++/*+*Re-affinekthreadsaccordingtotheirpreferences+*andthenewlyonlineCPU.TheCPUdownpartishandled+*byselect_fallback_rq()whichdefaultre-affinesto+*housekeepersfromothernodesincasethepreferred+*affinitydoesn'tapplyanymore.+*/+staticintkthreads_online_cpu(unsignedintcpu)+{+returnkthreads_update_affinity(false);+}+staticintkthreads_init(void){returncpuhp_setup_state(CPUHP_AP_KTHREADS_ONLINE,"kthreads:online",
It may not appear obvious why kthread_affine_node() is not called before
the kthread creation completion instead of after the first wake-up.
The reason is that kthread_affine_node() applies a default affinity
behaviour that only takes place if no affinity preference have already
been passed by the kthread creation call site.
Add a comment to clarify that.
Reported-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
---
kernel/kthread.c | 4 ++++
1 file changed, 4 insertions(+)
The documentation of this new API has been overlooked during its
introduction. Fill the gap.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
---
kernel/kthread.c | 12 ++++++++++++
1 file changed, 12 insertions(+)
@@ -0,0 +1,111 @@+======================================+Housekeeping+======================================+++CPU Isolation moves away kernel work that may otherwise run on any CPU.+The purpose of its related features is to reduce the OS jitter that some+extreme workloads can't stand, such as in some DPDK usecases.++The kernel work moved away by CPU isolation is commonly described as+"housekeeping" because it includes ground work that performs cleanups,+statistics maintainance and actions relying on them, memory release,+various deferrals etc...++Sometimes housekeeping is just some unbound work (unbound workqueues,+unbound timers, ...) that gets easily assigned to non-isolated CPUs.+But sometimes housekeeping is tied to a specific CPU and requires+elaborated tricks to be offloaded to non-isolated CPUs (RCU_NOCB, remote+scheduler tick, etc...).++Thus, a housekeeping CPU can be considered as the reverse of an isolated+CPU. It is simply a CPU that can execute housekeeping work. There must+always be at least one online housekeeping CPU at any time. The CPUs that+are not isolated are automatically assigned as housekeeping.++Housekeeping is currently divided in four features described+by the ``enum hk_type type``:++1. HK_TYPE_DOMAIN matches the work moved away by scheduler domain+ isolation performed through ``isolcpus=domain`` boot parameter or+ isolated cpuset partitions in cgroup v2. This includes scheduler+ load balancing, unbound workqueues and timers.++2. HK_TYPE_KERNEL_NOISE matches the work moved away by tick isolation+ performed through ``nohz_full=`` or ``isolcpus=nohz`` boot+ parameters. This includes remote scheduler tick, vmstat and lockup+ watchdog.++3. HK_TYPE_MANAGED_IRQ matches the IRQ handlers moved away by managed+ IRQ isolation performed through ``isolcpus=managed_irq``.++4. HK_TYPE_DOMAIN_BOOT matches the work moved away by scheduler domain+ isolation performed through ``isolcpus=domain`` only. It is similar+ to HK_TYPE_DOMAIN except it ignores the isolation performed by+ cpusets.+++Housekeeping cpumasks+=================================++Housekeeping cpumasks include the CPUs that can execute the work moved+away by the matching isolation feature. These cpumasks are returned by+the following function::++ const struct cpumask *housekeeping_cpumask(enum hk_type type)++By default, if neither ``nohz_full=``, nor ``isolcpus``, nor cpuset's+isolated partitions are used, which covers most usecases, this function+returns the cpu_possible_mask.++Otherwise the function returns the cpumask complement of the isolation+feature. For example:++With isolcpus=domain,7 the following will return a mask with all possible+CPUs except 7::++ housekeeping_cpumask(HK_TYPE_DOMAIN)++Similarly with nohz_full=5,6 the following will return a mask with all+possible CPUs except 5,6::++ housekeeping_cpumask(HK_TYPE_KERNEL_NOISE)+++Synchronization against cpusets+=================================++Cpuset can modify the HK_TYPE_DOMAIN housekeeping cpumask while creating,+modifying or deleting an isolated partition.++The users of HK_TYPE_DOMAIN cpumask must then make sure to synchronize+properly against cpuset in order to make sure that:++1. The cpumask snapshot stays coherent.++2. No housekeeping work is queued on a newly made isolated CPU.++3. Pending housekeeping work that was queued to a non isolated+ CPU which just turned isolated through cpuset must be flushed+ before the related created/modified isolated partition is made+ available to userspace.++This synchronization is maintained by an RCU based scheme. The cpuset update+side waits for an RCU grace period after updating the HK_TYPE_DOMAIN+cpumask and before flushing pending works. On the read side, care must be+taken to gather the housekeeping target election and the work enqueue within+the same RCU read side critical section.++A typical layout example would look like this on the update side+(``housekeeping_update()``)::++ rcu_assign_pointer(housekeeping_cpumasks[type], trial);+ synchronize_rcu();+ flush_workqueue(example_workqueue);++And then on the read side::++ rcu_read_lock();+ cpu = housekeeping_any_cpu(HK_TYPE_DOMAIN);+ queue_work_on(cpu, example_workqueue, work);+ rcu_read_unlock();
@@ -25,6 +25,7 @@ it. symbol-namespaces asm-annotations real-time/index+ housekeeping.rst Data structures and low-level utilities =======================================
From: Simon Horman <horms@kernel.org> Date: 2026-01-07 11:57:03
On Thu, Jan 01, 2026 at 11:13:38PM +0100, Frederic Weisbecker wrote:
quoted hunk
HK_TYPE_DOMAIN's cpumask will soon be made modifiable by cpuset.
A synchronization mechanism is then needed to synchronize the updates
with the housekeeping cpumask readers.
Turn the housekeeping cpumasks into RCU pointers. Once a housekeeping
cpumask will be modified, the update side will wait for an RCU grace
period and propagate the change to interested subsystem when deemed
necessary.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
---
kernel/sched/isolation.c | 58 +++++++++++++++++++++++++---------------
kernel/sched/sched.h | 1 +
2 files changed, 37 insertions(+), 22 deletions(-)
Hi Frederic,
I think this patch should also update the access to housekeeping.cpumasks
in housekeeping_setup(), on line 200, to use housekeeping_cpumask().
As is, sparse flags __rcu a annotation miss match there.
kernel/sched/isolation.c:200:80: warning: incorrect type in argument 3 (different address spaces)
kernel/sched/isolation.c:200:80: expected struct cpumask const *srcp3
kernel/sched/isolation.c:200:80: got struct cpumask [noderef] __rcu *
...
[+cc Jinhui]
On Thu, Jan 01, 2026 at 11:13:26PM +0100, Frederic Weisbecker wrote:
HK_TYPE_DOMAIN will soon integrate cpuset isolated partitions and
therefore be made modifiable at runtime. Synchronize against the cpumask
update using RCU.
The RCU locked section includes both the housekeeping CPU target
election for the PCI probe work and the work enqueue.
This way the housekeeping update side will simply need to flush the
pending related works after updating the housekeeping mask in order to
make sure that no PCI work ever executes on an isolated CPU. This part
will be handled in a subsequent patch.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Just FYI, Jinhui posted a series that touches this same code and might
need some coordination:
https://lore.kernel.org/r/20260107175548.1792-1-guojinhui.liam@bytedance.com
IIUC, Jinhui's series adds some more NUMA smarts in the driver core
sync probing path and removes corresponding NUMA code from the PCI
core probe path.
Bjorn
Le Wed, Jan 07, 2026 at 01:05:34PM -0600, Bjorn Helgaas a écrit :
[+cc Jinhui]
On Thu, Jan 01, 2026 at 11:13:26PM +0100, Frederic Weisbecker wrote:
quoted
HK_TYPE_DOMAIN will soon integrate cpuset isolated partitions and
therefore be made modifiable at runtime. Synchronize against the cpumask
update using RCU.
The RCU locked section includes both the housekeeping CPU target
election for the PCI probe work and the work enqueue.
This way the housekeeping update side will simply need to flush the
pending related works after updating the housekeeping mask in order to
make sure that no PCI work ever executes on an isolated CPU. This part
will be handled in a subsequent patch.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Just FYI, Jinhui posted a series that touches this same code and might
need some coordination:
https://lore.kernel.org/r/20260107175548.1792-1-guojinhui.liam@bytedance.com
IIUC, Jinhui's series adds some more NUMA smarts in the driver core
sync probing path and removes corresponding NUMA code from the PCI
core probe path.
I see. I can't drop my change, otherwise my series alone could crash
dereferencing garbage. But Jinhui's series removes the need for my changes.
So an unpleasant conflict will happen in -next (and if everything goes well,
further in next merge window) and it should be resolved with simply ignoring
my changes and only apply those of Jinhui.
Should we inform Linux Next people ahead?
Thanks for making me notice!
--
Frederic Weisbecker
SUSE Labs
On Thu, Jan 08, 2026 at 12:30:13AM +0100, Frederic Weisbecker wrote:
Le Wed, Jan 07, 2026 at 01:05:34PM -0600, Bjorn Helgaas a écrit :
quoted
On Thu, Jan 01, 2026 at 11:13:26PM +0100, Frederic Weisbecker wrote:
quoted
HK_TYPE_DOMAIN will soon integrate cpuset isolated partitions and
therefore be made modifiable at runtime. Synchronize against the cpumask
update using RCU.
The RCU locked section includes both the housekeeping CPU target
election for the PCI probe work and the work enqueue.
This way the housekeeping update side will simply need to flush the
pending related works after updating the housekeeping mask in order to
make sure that no PCI work ever executes on an isolated CPU. This part
will be handled in a subsequent patch.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Just FYI, Jinhui posted a series that touches this same code and might
need some coordination:
https://lore.kernel.org/r/20260107175548.1792-1-guojinhui.liam@bytedance.com
IIUC, Jinhui's series adds some more NUMA smarts in the driver core
sync probing path and removes corresponding NUMA code from the PCI
core probe path.
I see. I can't drop my change, otherwise my series alone could crash
dereferencing garbage. But Jinhui's series removes the need for my
changes.
So an unpleasant conflict will happen in -next (and if everything
goes well, further in next merge window) and it should be resolved
with simply ignoring my changes and only apply those of Jinhui.
I don't want to derail your series, and I don't think you need to
change anything right now. Jinhui's series is early and might not be
ready to merge until after yours, which should be fine.
Bjorn
On Wed Jan 7, 2026 at 13:05:34 -0600, Bjorn Helgaas worte:
[+cc Jinhui]
On Thu, Jan 01, 2026 at 11:13:26PM +0100, Frederic Weisbecker wrote:
quoted
HK_TYPE_DOMAIN will soon integrate cpuset isolated partitions and
therefore be made modifiable at runtime. Synchronize against the cpumask
update using RCU.
The RCU locked section includes both the housekeeping CPU target
election for the PCI probe work and the work enqueue.
This way the housekeeping update side will simply need to flush the
pending related works after updating the housekeeping mask in order to
make sure that no PCI work ever executes on an isolated CPU. This part
will be handled in a subsequent patch.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Just FYI, Jinhui posted a series that touches this same code and might
need some coordination:
https://lore.kernel.org/r/20260107175548.1792-1-guojinhui.liam@bytedance.com
IIUC, Jinhui's series adds some more NUMA smarts in the driver core
sync probing path and removes corresponding NUMA code from the PCI
core probe path.
Hi Bjorn,
Thanks for pointing out the series.
I’ll resolve the conflicts and send a new patchset once this one is merged.
Best Regards,
Jinhui
From: Waiman Long <hidden> Date: 2026-01-12 01:43:33
On 1/1/26 5:13 PM, Frederic Weisbecker wrote:
quoted hunk
cpuset modifies partitions, including isolated, while holding the cpuset
mutex.
This means that holding the cpuset mutex is safe to synchronize against
housekeeping cpumask changes.
Provide a lockdep check to validate that.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
---
include/linux/cpuset.h | 2 ++
kernel/cgroup/cpuset.c | 7 +++++++
2 files changed, 9 insertions(+)
The cgroup/for-next tree already have a similar
lockdep_assert_cpuset_lock_held() defined. So you can drop this patch if
this series won't land in the next merge window.
Cheers,
Longman
From: Waiman Long <hidden> Date: 2026-01-12 02:45:55
On 1/7/26 6:56 AM, Simon Horman wrote:
On Thu, Jan 01, 2026 at 11:13:38PM +0100, Frederic Weisbecker wrote:
quoted
HK_TYPE_DOMAIN's cpumask will soon be made modifiable by cpuset.
A synchronization mechanism is then needed to synchronize the updates
with the housekeeping cpumask readers.
Turn the housekeeping cpumasks into RCU pointers. Once a housekeeping
cpumask will be modified, the update side will wait for an RCU grace
period and propagate the change to interested subsystem when deemed
necessary.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
---
kernel/sched/isolation.c | 58 +++++++++++++++++++++++++---------------
kernel/sched/sched.h | 1 +
2 files changed, 37 insertions(+), 22 deletions(-)
Hi Frederic,
I think this patch should also update the access to housekeeping.cpumasks
in housekeeping_setup(), on line 200, to use housekeeping_cpumask().
As is, sparse flags __rcu a annotation miss match there.
kernel/sched/isolation.c:200:80: warning: incorrect type in argument 3 (different address spaces)
kernel/sched/isolation.c:200:80: expected struct cpumask const *srcp3
kernel/sched/isolation.c:200:80: got struct cpumask [noderef] __rcu *
...
The direct housekeeping.cpumasks[type] reference is in the newly merged
check after Federic's initial patch series.
iter_flags = housekeeping.flags & (HK_FLAG_KERNEL_NOISE
| HK_FLAG_DOMAIN);
type = find_first_bit(&iter_flags, HK_TYPE_MAX);
/*
* Pass the check if none of these flags were
previously set or
* are not in the current selection.
*/
iter_flags = flags & (HK_FLAG_KERNEL_NOISE |
HK_FLAG_DOMAIN);
first_cpu = (type == HK_TYPE_MAX || !iter_flags) ? 0 :
cpumask_first_and_and(cpu_present_mask,
housekeeping_staging,
housekeeping.cpumasks[type]);
Maybe that is why it is missed.
Cheers,
Longman
From: Waiman Long <hidden> Date: 2026-01-12 17:54:07
On 1/11/26 8:43 PM, Waiman Long wrote:
On 1/1/26 5:13 PM, Frederic Weisbecker wrote:
quoted
cpuset modifies partitions, including isolated, while holding the cpuset
mutex.
This means that holding the cpuset mutex is safe to synchronize against
housekeeping cpumask changes.
Provide a lockdep check to validate that.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
---
include/linux/cpuset.h | 2 ++
kernel/cgroup/cpuset.c | 7 +++++++
2 files changed, 9 insertions(+)
The cgroup/for-next tree already have a similar
lockdep_assert_cpuset_lock_held() defined. So you can drop this patch
if this series won't land in the next merge window.
Sorry, the other new lockdep API isn't exactly the same as what you
propose here. So it is not a replacement for your use case. Sorry for
the noise.
Cheers,
Longman
From: Waiman Long <hidden> Date: 2026-01-12 18:04:07
On 1/1/26 5:13 PM, Frederic Weisbecker wrote:
quoted hunk
HK_TYPE_DOMAIN will soon integrate not only boot defined isolcpus= CPUs
but also cpuset isolated partitions.
Housekeeping still needs a way to record what was initially passed
to isolcpus= in order to keep these CPUs isolated after a cpuset
isolated partition is modified or destroyed while containing some of
them.
Create a new HK_TYPE_DOMAIN_BOOT to keep track of those.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Reviewed-by: Phil Auld <redacted>
---
include/linux/sched/isolation.h | 4 ++++
kernel/sched/isolation.c | 5 +++--
2 files changed, 7 insertions(+), 2 deletions(-)
From: Waiman Long <hidden> Date: 2026-01-12 18:24:01
On 1/1/26 5:13 PM, Frederic Weisbecker wrote:
Hi,
The kthread code was enhanced lately to provide an infrastructure which
manages the preferred affinity of unbound kthreads (node or custom
cpumask) against housekeeping constraints and CPU hotplug events.
One crucial missing piece is cpuset: when an isolated partition is
created, deleted, or its CPUs updated, all the unbound kthreads in the
top cpuset are affine to _all_ the non-isolated CPUs, possibly breaking
their preferred affinity along the way
Solve this with performing the kthreads affinity update from cpuset to
the kthreads consolidated relevant code instead so that preferred
affinities are honoured.
The dispatch of the new cpumasks to workqueues and kthreads is performed
by housekeeping, as per the nice Tejun's suggestion.
As a welcome side effect, HK_TYPE_DOMAIN then integrates both the set
from isolcpus= and cpuset isolated partitions. Housekeeping cpumasks are
now modifyable with specific synchronization. A big step toward making
nohz_full= also mutable through cpuset in the future.
Changes since v5:
* Add more tags
* Fix leaked destroy_work_on_stack() (Zhang Qiao, Waiman Long)
* Comment schedule_drain_work() synchronization requirement (Tejun)
* s/Revert of/Inverse of (Waiman Long)
* Remove housekeeping_update() needless (for now) parameter (Chen Ridong)
* Don't propagate housekeeping_update() failures beyond allocations (Waiman Long)
* Whitespace cleanup (Waiman Long)
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
kthread/core-v6
HEAD: 811e87ca8a0a1e54eb5f23e71896cb97436cccdc
Happy new year,
Frederic
I don't see any major issue with this v6 version. There may be some
minor issues that can be cleaned up later. Now the issue is which tree
should this series go to as it touches a number of different subsystems
with different maintainers.
Cheers,
Longman
Le Sun, Jan 11, 2026 at 09:45:36PM -0500, Waiman Long a écrit :
On 1/7/26 6:56 AM, Simon Horman wrote:
quoted
On Thu, Jan 01, 2026 at 11:13:38PM +0100, Frederic Weisbecker wrote:
quoted
HK_TYPE_DOMAIN's cpumask will soon be made modifiable by cpuset.
A synchronization mechanism is then needed to synchronize the updates
with the housekeeping cpumask readers.
Turn the housekeeping cpumasks into RCU pointers. Once a housekeeping
cpumask will be modified, the update side will wait for an RCU grace
period and propagate the change to interested subsystem when deemed
necessary.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
---
kernel/sched/isolation.c | 58 +++++++++++++++++++++++++---------------
kernel/sched/sched.h | 1 +
2 files changed, 37 insertions(+), 22 deletions(-)
Hi Frederic,
I think this patch should also update the access to housekeeping.cpumasks
in housekeeping_setup(), on line 200, to use housekeeping_cpumask().
As is, sparse flags __rcu a annotation miss match there.
kernel/sched/isolation.c:200:80: warning: incorrect type in argument 3 (different address spaces)
kernel/sched/isolation.c:200:80: expected struct cpumask const *srcp3
kernel/sched/isolation.c:200:80: got struct cpumask [noderef] __rcu *
...
The direct housekeeping.cpumasks[type] reference is in the newly merged
check after Federic's initial patch series.
iter_flags = housekeeping.flags & (HK_FLAG_KERNEL_NOISE |
HK_FLAG_DOMAIN);
type = find_first_bit(&iter_flags, HK_TYPE_MAX);
/*
* Pass the check if none of these flags were previously set
or
* are not in the current selection.
*/
iter_flags = flags & (HK_FLAG_KERNEL_NOISE |
HK_FLAG_DOMAIN);
first_cpu = (type == HK_TYPE_MAX || !iter_flags) ? 0 :
cpumask_first_and_and(cpu_present_mask,
housekeeping_staging,
housekeeping.cpumasks[type]);
Maybe that is why it is missed.
Le Mon, Jan 12, 2026 at 01:23:40PM -0500, Waiman Long a écrit :
On 1/1/26 5:13 PM, Frederic Weisbecker wrote:
quoted
Hi,
The kthread code was enhanced lately to provide an infrastructure which
manages the preferred affinity of unbound kthreads (node or custom
cpumask) against housekeeping constraints and CPU hotplug events.
One crucial missing piece is cpuset: when an isolated partition is
created, deleted, or its CPUs updated, all the unbound kthreads in the
top cpuset are affine to _all_ the non-isolated CPUs, possibly breaking
their preferred affinity along the way
Solve this with performing the kthreads affinity update from cpuset to
the kthreads consolidated relevant code instead so that preferred
affinities are honoured.
The dispatch of the new cpumasks to workqueues and kthreads is performed
by housekeeping, as per the nice Tejun's suggestion.
As a welcome side effect, HK_TYPE_DOMAIN then integrates both the set
from isolcpus= and cpuset isolated partitions. Housekeeping cpumasks are
now modifyable with specific synchronization. A big step toward making
nohz_full= also mutable through cpuset in the future.
Changes since v5:
* Add more tags
* Fix leaked destroy_work_on_stack() (Zhang Qiao, Waiman Long)
* Comment schedule_drain_work() synchronization requirement (Tejun)
* s/Revert of/Inverse of (Waiman Long)
* Remove housekeeping_update() needless (for now) parameter (Chen Ridong)
* Don't propagate housekeeping_update() failures beyond allocations (Waiman Long)
* Whitespace cleanup (Waiman Long)
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
kthread/core-v6
HEAD: 811e87ca8a0a1e54eb5f23e71896cb97436cccdc
Happy new year,
Frederic
I don't see any major issue with this v6 version. There may be some minor
issues that can be cleaned up later. Now the issue is which tree should this
series go to as it touches a number of different subsystems with different
maintainers.
It indeed crosses many subsystems. I would be fine if anybody takes it but
nobody volunteered so far.
The main purpose is to fix kthreads affinity (HK_TYPE_DOMAIN handling cpuset is
a bonus). And since I made the pull request myself to Linus when I introduced
kthreads managed affinity, I guess I could reiterate with this patchset. I
already pushed it to linux-next.
But if anybody wants to pull that to another tree, that's fine, just tell me
so that we synchronize to avoid duplication on linux-next.
Thanks.
--
Frederic Weisbecker
SUSE Labs
From: Waiman Long <hidden> Date: 2026-01-13 01:49:30
On 1/12/26 5:09 PM, Frederic Weisbecker wrote:
Le Mon, Jan 12, 2026 at 01:23:40PM -0500, Waiman Long a écrit :
quoted
On 1/1/26 5:13 PM, Frederic Weisbecker wrote:
quoted
Hi,
The kthread code was enhanced lately to provide an infrastructure which
manages the preferred affinity of unbound kthreads (node or custom
cpumask) against housekeeping constraints and CPU hotplug events.
One crucial missing piece is cpuset: when an isolated partition is
created, deleted, or its CPUs updated, all the unbound kthreads in the
top cpuset are affine to _all_ the non-isolated CPUs, possibly breaking
their preferred affinity along the way
Solve this with performing the kthreads affinity update from cpuset to
the kthreads consolidated relevant code instead so that preferred
affinities are honoured.
The dispatch of the new cpumasks to workqueues and kthreads is performed
by housekeeping, as per the nice Tejun's suggestion.
As a welcome side effect, HK_TYPE_DOMAIN then integrates both the set
from isolcpus= and cpuset isolated partitions. Housekeeping cpumasks are
now modifyable with specific synchronization. A big step toward making
nohz_full= also mutable through cpuset in the future.
Changes since v5:
* Add more tags
* Fix leaked destroy_work_on_stack() (Zhang Qiao, Waiman Long)
* Comment schedule_drain_work() synchronization requirement (Tejun)
* s/Revert of/Inverse of (Waiman Long)
* Remove housekeeping_update() needless (for now) parameter (Chen Ridong)
* Don't propagate housekeeping_update() failures beyond allocations (Waiman Long)
* Whitespace cleanup (Waiman Long)
git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
kthread/core-v6
HEAD: 811e87ca8a0a1e54eb5f23e71896cb97436cccdc
Happy new year,
Frederic
I don't see any major issue with this v6 version. There may be some minor
issues that can be cleaned up later. Now the issue is which tree should this
series go to as it touches a number of different subsystems with different
maintainers.
It indeed crosses many subsystems. I would be fine if anybody takes it but
nobody volunteered so far.
The main purpose is to fix kthreads affinity (HK_TYPE_DOMAIN handling cpuset is
a bonus). And since I made the pull request myself to Linus when I introduced
kthreads managed affinity, I guess I could reiterate with this patchset. I
already pushed it to linux-next.
But if anybody wants to pull that to another tree, that's fine, just tell me
so that we synchronize to avoid duplication on linux-next.
Thanks.
Good to know as I am wondering where it will go. So you are going to
push that directly to Linus.
Cheers,
Longman
From: Will Deacon <will@kernel.org> Date: 2026-01-20 15:15:24
Hi Frederic,
On Thu, Jan 01, 2026 at 11:13:54PM +0100, Frederic Weisbecker wrote:
When none of the allowed CPUs of a task are online, it gets migrated
to the fallback cpumask which is all the non nohz_full CPUs.
However just like nohz_full CPUs, domain isolated CPUs don't want to be
disturbed by tasks that have lost their CPU affinities.
And since nohz_full rely on domain isolation to work correctly, the
housekeeping mask of domain isolated CPUs should always be a superset of
the housekeeping mask of nohz_full CPUs (there can be CPUs that are
domain isolated but not nohz_full, OTOH there shouldn't be nohz_full
CPUs that are not domain isolated):
HK_TYPE_DOMAIN | HK_TYPE_KERNEL_NOISE == HK_TYPE_DOMAIN
Therefore use HK_TYPE_DOMAIN as the appropriate fallback target for
tasks and since this cpumask can be modified at runtime, make sure
that 32 bits support CPUs on ARM64 mismatched systems are not isolated
by cpusets.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Reviewed-by: Waiman Long <longman@redhat.com>
---
arch/arm64/kernel/cpufeature.c | 18 +++++++++++++++---
include/linux/cpu.h | 4 ++++
kernel/cgroup/cpuset.c | 17 ++++++++++++++---
3 files changed, 33 insertions(+), 6 deletions(-)
tbh, I'd also be fine just saying that isolation isn't reliable on these
systems and then you don't need to add the extra arch hook.
Whatever you prefer, but please can you update the text in
Documentation/arch/arm64/asymmetric-32bit.rst to cover the interaction
between the asymmetric stuff and cpu isolation?
Cheers,
Will
Le Tue, Jan 20, 2026 at 03:15:14PM +0000, Will Deacon a écrit :
Hi Frederic,
On Thu, Jan 01, 2026 at 11:13:54PM +0100, Frederic Weisbecker wrote:
quoted
When none of the allowed CPUs of a task are online, it gets migrated
to the fallback cpumask which is all the non nohz_full CPUs.
However just like nohz_full CPUs, domain isolated CPUs don't want to be
disturbed by tasks that have lost their CPU affinities.
And since nohz_full rely on domain isolation to work correctly, the
housekeeping mask of domain isolated CPUs should always be a superset of
the housekeeping mask of nohz_full CPUs (there can be CPUs that are
domain isolated but not nohz_full, OTOH there shouldn't be nohz_full
CPUs that are not domain isolated):
HK_TYPE_DOMAIN | HK_TYPE_KERNEL_NOISE == HK_TYPE_DOMAIN
Therefore use HK_TYPE_DOMAIN as the appropriate fallback target for
tasks and since this cpumask can be modified at runtime, make sure
that 32 bits support CPUs on ARM64 mismatched systems are not isolated
by cpusets.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Reviewed-by: Waiman Long <longman@redhat.com>
---
arch/arm64/kernel/cpufeature.c | 18 +++++++++++++++---
include/linux/cpu.h | 4 ++++
kernel/cgroup/cpuset.c | 17 ++++++++++++++---
3 files changed, 33 insertions(+), 6 deletions(-)
tbh, I'd also be fine just saying that isolation isn't reliable on these
systems and then you don't need to add the extra arch hook.
Hmm, I think I heard about nohz_full usage on arm64 but I'm not sure.
And I usually expect isolcpus or cpuset isolated partitions to be even
more broadly used, it's lighter isolation with less constraints.
Anyway you're probably right that we could remove isolation support here
but I don't want to break any existing user.
Whatever you prefer, but please can you update the text in
Documentation/arch/arm64/asymmetric-32bit.rst to cover the interaction
between the asymmetric stuff and cpu isolation?
I'll keep that path and update the documentation. I guess we can still
consider removing that support afterward. If we do so anyway, it would
deserve its own patchset and shouldn't be hidden in this pile.
Thanks.
--
Frederic Weisbecker
SUSE Labs
From: Will Deacon <will@kernel.org> Date: 2026-01-22 09:56:40
On Wed, Jan 21, 2026 at 06:06:07PM +0100, Frederic Weisbecker wrote:
Le Tue, Jan 20, 2026 at 03:15:14PM +0000, Will Deacon a écrit :
quoted
Hi Frederic,
On Thu, Jan 01, 2026 at 11:13:54PM +0100, Frederic Weisbecker wrote:
quoted
When none of the allowed CPUs of a task are online, it gets migrated
to the fallback cpumask which is all the non nohz_full CPUs.
However just like nohz_full CPUs, domain isolated CPUs don't want to be
disturbed by tasks that have lost their CPU affinities.
And since nohz_full rely on domain isolation to work correctly, the
housekeeping mask of domain isolated CPUs should always be a superset of
the housekeeping mask of nohz_full CPUs (there can be CPUs that are
domain isolated but not nohz_full, OTOH there shouldn't be nohz_full
CPUs that are not domain isolated):
HK_TYPE_DOMAIN | HK_TYPE_KERNEL_NOISE == HK_TYPE_DOMAIN
Therefore use HK_TYPE_DOMAIN as the appropriate fallback target for
tasks and since this cpumask can be modified at runtime, make sure
that 32 bits support CPUs on ARM64 mismatched systems are not isolated
by cpusets.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Reviewed-by: Waiman Long <longman@redhat.com>
---
arch/arm64/kernel/cpufeature.c | 18 +++++++++++++++---
include/linux/cpu.h | 4 ++++
kernel/cgroup/cpuset.c | 17 ++++++++++++++---
3 files changed, 33 insertions(+), 6 deletions(-)
tbh, I'd also be fine just saying that isolation isn't reliable on these
systems and then you don't need to add the extra arch hook.
Hmm, I think I heard about nohz_full usage on arm64 but I'm not sure.
And I usually expect isolcpus or cpuset isolated partitions to be even
more broadly used, it's lighter isolation with less constraints.
Anyway you're probably right that we could remove isolation support here
but I don't want to break any existing user.
fwiw, I think it's only some Android markets using the mismatched 32-bit
support and we're definitely not using nohz_full there.
Will
Until now, HK_TYPE_DOMAIN used to only include boot defined isolated
CPUs passed through isolcpus= boot option. Users interested in also
knowing the runtime defined isolated CPUs through cpuset must use
different APIs: cpuset_cpu_is_isolated(), cpu_is_isolated(), etc...
There are many drawbacks to that approach:
1) Most interested subsystems want to know about all isolated CPUs, not
just those defined on boot time.
2) cpuset_cpu_is_isolated() / cpu_is_isolated() are not synchronized with
concurrent cpuset changes.
3) Further cpuset modifications are not propagated to subsystems
Solve 1) and 2) and centralize all isolated CPUs within the
HK_TYPE_DOMAIN housekeeping cpumask.
Subsystems can rely on RCU to synchronize against concurrent changes.
The propagation mentioned in 3) will be handled in further patches.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Reviewed-by: Waiman Long <longman@redhat.com>
Reviewed-by: Chen Ridong <redacted>
---
include/linux/sched/isolation.h | 7 ++++
kernel/cgroup/cpuset.c | 3 ++
kernel/sched/isolation.c | 73 ++++++++++++++++++++++++++++++---
kernel/sched/sched.h | 1 +
4 files changed, 78 insertions(+), 6 deletions(-)
Le Thu, Jan 22, 2026 at 09:56:29AM +0000, Will Deacon a écrit :
On Wed, Jan 21, 2026 at 06:06:07PM +0100, Frederic Weisbecker wrote:
quoted
Le Tue, Jan 20, 2026 at 03:15:14PM +0000, Will Deacon a écrit :
quoted
Hi Frederic,
On Thu, Jan 01, 2026 at 11:13:54PM +0100, Frederic Weisbecker wrote:
quoted
When none of the allowed CPUs of a task are online, it gets migrated
to the fallback cpumask which is all the non nohz_full CPUs.
However just like nohz_full CPUs, domain isolated CPUs don't want to be
disturbed by tasks that have lost their CPU affinities.
And since nohz_full rely on domain isolation to work correctly, the
housekeeping mask of domain isolated CPUs should always be a superset of
the housekeeping mask of nohz_full CPUs (there can be CPUs that are
domain isolated but not nohz_full, OTOH there shouldn't be nohz_full
CPUs that are not domain isolated):
HK_TYPE_DOMAIN | HK_TYPE_KERNEL_NOISE == HK_TYPE_DOMAIN
Therefore use HK_TYPE_DOMAIN as the appropriate fallback target for
tasks and since this cpumask can be modified at runtime, make sure
that 32 bits support CPUs on ARM64 mismatched systems are not isolated
by cpusets.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Reviewed-by: Waiman Long <longman@redhat.com>
---
arch/arm64/kernel/cpufeature.c | 18 +++++++++++++++---
include/linux/cpu.h | 4 ++++
kernel/cgroup/cpuset.c | 17 ++++++++++++++---
3 files changed, 33 insertions(+), 6 deletions(-)
tbh, I'd also be fine just saying that isolation isn't reliable on these
systems and then you don't need to add the extra arch hook.
Hmm, I think I heard about nohz_full usage on arm64 but I'm not sure.
And I usually expect isolcpus or cpuset isolated partitions to be even
more broadly used, it's lighter isolation with less constraints.
Anyway you're probably right that we could remove isolation support here
but I don't want to break any existing user.
fwiw, I think it's only some Android markets using the mismatched 32-bit
support and we're definitely not using nohz_full there.
Now that removal becomes appealing. And what about isolcpus= / isolated cpuset
which only consist in scheduler domain isolation? Probably not used by android
either.
Ok but is there a way to detect on early boot that the system has mismatched
32 bits support? Because I need to fail nohz_full= and isolcpus= boot parameters
early on top of this information without waiting for secondary CPUs boot.
Thanks.
--
Frederic Weisbecker
SUSE Labs
From: Will Deacon <will@kernel.org> Date: 2026-01-22 11:36:31
On Thu, Jan 22, 2026 at 12:29:17PM +0100, Frederic Weisbecker wrote:
Le Thu, Jan 22, 2026 at 09:56:29AM +0000, Will Deacon a écrit :
quoted
On Wed, Jan 21, 2026 at 06:06:07PM +0100, Frederic Weisbecker wrote:
quoted
Le Tue, Jan 20, 2026 at 03:15:14PM +0000, Will Deacon a écrit :
quoted
Hi Frederic,
On Thu, Jan 01, 2026 at 11:13:54PM +0100, Frederic Weisbecker wrote:
quoted
When none of the allowed CPUs of a task are online, it gets migrated
to the fallback cpumask which is all the non nohz_full CPUs.
However just like nohz_full CPUs, domain isolated CPUs don't want to be
disturbed by tasks that have lost their CPU affinities.
And since nohz_full rely on domain isolation to work correctly, the
housekeeping mask of domain isolated CPUs should always be a superset of
the housekeeping mask of nohz_full CPUs (there can be CPUs that are
domain isolated but not nohz_full, OTOH there shouldn't be nohz_full
CPUs that are not domain isolated):
HK_TYPE_DOMAIN | HK_TYPE_KERNEL_NOISE == HK_TYPE_DOMAIN
Therefore use HK_TYPE_DOMAIN as the appropriate fallback target for
tasks and since this cpumask can be modified at runtime, make sure
that 32 bits support CPUs on ARM64 mismatched systems are not isolated
by cpusets.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Reviewed-by: Waiman Long <longman@redhat.com>
---
arch/arm64/kernel/cpufeature.c | 18 +++++++++++++++---
include/linux/cpu.h | 4 ++++
kernel/cgroup/cpuset.c | 17 ++++++++++++++---
3 files changed, 33 insertions(+), 6 deletions(-)
tbh, I'd also be fine just saying that isolation isn't reliable on these
systems and then you don't need to add the extra arch hook.
Hmm, I think I heard about nohz_full usage on arm64 but I'm not sure.
And I usually expect isolcpus or cpuset isolated partitions to be even
more broadly used, it's lighter isolation with less constraints.
Anyway you're probably right that we could remove isolation support here
but I don't want to break any existing user.
fwiw, I think it's only some Android markets using the mismatched 32-bit
support and we're definitely not using nohz_full there.
Now that removal becomes appealing. And what about isolcpus= / isolated cpuset
which only consist in scheduler domain isolation? Probably not used by android
either.
Ok but is there a way to detect on early boot that the system has mismatched
32 bits support? Because I need to fail nohz_full= and isolcpus= boot parameters
early on top of this information without waiting for secondary CPUs boot.
Honestly, I'm not sure I'd bother trying to be smart here. Even if the
system has enabled support for mismatched 32-bit CPUs, things should
still work properly with nohz_full/isolcpus if all the tasks are 64-bit,
right?
In which case, I'd just document whatever weird behaviour you get if
somebody throws 32-bit tasks into the mix. Adding hooks to the generic
code for this use-case just seems like a waste, as they're not going to
be used in practice and it increases the maintenance burden.
Will
Le Thu, Jan 22, 2026 at 11:36:21AM +0000, Will Deacon a écrit :
On Thu, Jan 22, 2026 at 12:29:17PM +0100, Frederic Weisbecker wrote:
quoted
Le Thu, Jan 22, 2026 at 09:56:29AM +0000, Will Deacon a écrit :
quoted
On Wed, Jan 21, 2026 at 06:06:07PM +0100, Frederic Weisbecker wrote:
quoted
Le Tue, Jan 20, 2026 at 03:15:14PM +0000, Will Deacon a écrit :
quoted
Hi Frederic,
On Thu, Jan 01, 2026 at 11:13:54PM +0100, Frederic Weisbecker wrote:
quoted
When none of the allowed CPUs of a task are online, it gets migrated
to the fallback cpumask which is all the non nohz_full CPUs.
However just like nohz_full CPUs, domain isolated CPUs don't want to be
disturbed by tasks that have lost their CPU affinities.
And since nohz_full rely on domain isolation to work correctly, the
housekeeping mask of domain isolated CPUs should always be a superset of
the housekeeping mask of nohz_full CPUs (there can be CPUs that are
domain isolated but not nohz_full, OTOH there shouldn't be nohz_full
CPUs that are not domain isolated):
HK_TYPE_DOMAIN | HK_TYPE_KERNEL_NOISE == HK_TYPE_DOMAIN
Therefore use HK_TYPE_DOMAIN as the appropriate fallback target for
tasks and since this cpumask can be modified at runtime, make sure
that 32 bits support CPUs on ARM64 mismatched systems are not isolated
by cpusets.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Reviewed-by: Waiman Long <longman@redhat.com>
---
arch/arm64/kernel/cpufeature.c | 18 +++++++++++++++---
include/linux/cpu.h | 4 ++++
kernel/cgroup/cpuset.c | 17 ++++++++++++++---
3 files changed, 33 insertions(+), 6 deletions(-)
tbh, I'd also be fine just saying that isolation isn't reliable on these
systems and then you don't need to add the extra arch hook.
Hmm, I think I heard about nohz_full usage on arm64 but I'm not sure.
And I usually expect isolcpus or cpuset isolated partitions to be even
more broadly used, it's lighter isolation with less constraints.
Anyway you're probably right that we could remove isolation support here
but I don't want to break any existing user.
fwiw, I think it's only some Android markets using the mismatched 32-bit
support and we're definitely not using nohz_full there.
Now that removal becomes appealing. And what about isolcpus= / isolated cpuset
which only consist in scheduler domain isolation? Probably not used by android
either.
Ok but is there a way to detect on early boot that the system has mismatched
32 bits support? Because I need to fail nohz_full= and isolcpus= boot parameters
early on top of this information without waiting for secondary CPUs boot.
Honestly, I'm not sure I'd bother trying to be smart here. Even if the
system has enabled support for mismatched 32-bit CPUs, things should
still work properly with nohz_full/isolcpus if all the tasks are 64-bit,
right?
In which case, I'd just document whatever weird behaviour you get if
somebody throws 32-bit tasks into the mix. Adding hooks to the generic
code for this use-case just seems like a waste, as they're not going to
be used in practice and it increases the maintenance burden.
Ok, how does this updated version look now?
---
From: Frederic Weisbecker <frederic@kernel.org>
Date: Thu, 24 Jul 2025 23:38:48 +0200
Subject: [PATCH] sched/arm64: Move fallback task cpumask to HK_TYPE_DOMAIN
When none of the allowed CPUs of a task are online, it gets migrated
to the fallback cpumask which is all the non nohz_full CPUs.
However just like nohz_full CPUs, domain isolated CPUs don't want to be
disturbed by tasks that have lost their CPU affinities.
And since nohz_full rely on domain isolation to work correctly, the
housekeeping mask of domain isolated CPUs should always be a subset of
the housekeeping mask of nohz_full CPUs (there can be CPUs that are
domain isolated but not nohz_full, OTOH there shouldn't be nohz_full
CPUs that are not domain isolated):
HK_TYPE_DOMAIN & HK_TYPE_KERNEL_NOISE == HK_TYPE_DOMAIN
Therefore use HK_TYPE_DOMAIN as the appropriate fallback target for
tasks. Note that cpuset isolated partitions are not supported on those
systems and may result in undefined behaviour.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Marco Crivellari <redacted>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Tejun Heo <tj@kernel.org>
Cc: Thomas Gleixner <redacted>
Cc: Waiman Long <longman@redhat.com>
Cc: Will Deacon <will@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: linux-arm-kernel@lists.infradead.org
---
Documentation/arch/arm64/asymmetric-32bit.rst | 12 ++++++++----
arch/arm64/kernel/cpufeature.c | 6 +++---
2 files changed, 11 insertions(+), 7 deletions(-)
@@ -154,10 +154,14 @@ mode will return to host userspace with an ``exit_reason`` of``KVM_EXIT_FAIL_ENTRY`` and will remain non-runnable until successfully re-initialised by a subsequent ``KVM_ARM_VCPU_INIT`` operation.-NOHZ FULL----------+SCHEDULER DOMAIN ISOLATION+---------------------------To avoid perturbing an adaptive-ticks CPU (specified using-``nohz_full=``) when a 32-bit task is forcefully migrated, these CPUs+To avoid perturbing a boot-defined domain isolated CPU (specified using+``isolcpus=[domain]``) when a 32-bit task is forcefully migrated, these CPUs are treated as 64-bit-only when support for asymmetric 32-bit systems is enabled.++However as opposed to boot-defined domain isolation, runtime-defined domain+isolation using cpuset isolated partition is not advised on asymmetric+32-bit systems and will result in undefined behaviour.
@@ -3987,8 +3987,8 @@ static int enable_mismatched_32bit_el0(unsigned int cpu)boolcpu_32bit=false;if(id_aa64pfr0_32bit_el0(info->reg_id_aa64pfr0)){-if(!housekeeping_cpu(cpu,HK_TYPE_TICK))-pr_info("Treating adaptive-ticks CPU %u as 64-bit only\n",cpu);+if(!housekeeping_cpu(cpu,HK_TYPE_DOMAIN))+pr_info("Treating domain isolated CPU %u as 64-bit only\n",cpu);elsecpu_32bit=true;}
On Thu, Jan 01, 2026 at 11:13:26PM +0100, Frederic Weisbecker wrote:
HK_TYPE_DOMAIN will soon integrate cpuset isolated partitions and
therefore be made modifiable at runtime. Synchronize against the cpumask
update using RCU.
The RCU locked section includes both the housekeeping CPU target
election for the PCI probe work and the work enqueue.
This way the housekeeping update side will simply need to flush the
pending related works after updating the housekeeping mask in order to
make sure that no PCI work ever executes on an isolated CPU. This part
will be handled in a subsequent patch.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
On Thu, Jan 01, 2026 at 11:13:42PM +0100, Frederic Weisbecker wrote:
The HK_TYPE_DOMAIN housekeeping cpumask is now modifiable at runtime. In
order to synchronize against PCI probe works and make sure that no
asynchronous probing is still pending or executing on a newly isolated
CPU, the housekeeping subsystem must flush the PCI probe works.
However the PCI probe works can't be flushed easily since they are
queued to the main per-CPU workqueue pool.
Solve this with creating a PCI probe-specific pool and provide and use
the appropriate flushing API.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
On Thu, Jan 01, 2026 at 11:13:48PM +0100, Frederic Weisbecker wrote:
It doesn't make sense to use nohz_full without also isolating the
related CPUs from the domain topology, either through the use of
isolcpus= or cpuset isolated partitions.
And now HK_TYPE_DOMAIN includes all kinds of domain isolated CPUs.
This means that HK_TYPE_KERNEL_NOISE (of which HK_TYPE_WQ is only an
alias) should always be a subset of HK_TYPE_DOMAIN.
Therefore sane configurations verify:
HK_TYPE_KERNEL_NOISE | HK_TYPE_DOMAIN == HK_TYPE_DOMAIN
Simplify the PCI probe target election accordingly.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
From: David Lechner <dlechner@baylibre.com> Date: 2026-02-23 16:01:06
On 1/1/26 4:13 PM, Frederic Weisbecker wrote:
quoted hunk
The HK_TYPE_DOMAIN housekeeping cpumask will soon be made modifiable at
runtime. In order to synchronize against vmstat workqueue to make sure
that no asynchronous vmstat work is pending or executing on a newly made
isolated CPU, target and queue a vmstat work under the same RCU read
side critical section.
Whenever housekeeping will update the HK_TYPE_DOMAIN cpumask, a vmstat
workqueue flush will also be issued in a further change to make sure
that no work remains pending after a CPU has been made isolated.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
---
mm/vmstat.c | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
I think this might have introduced a bug - or at least an unintentional change
in the program flow.
scoped_guard() is implemented using a for loop. Now this continue statement will
only exit the scoped_guard() scope rather than continuing the outer for loop. This
means that cond_resched() will be called when it previously was not.
From: Marc Zyngier <maz@kernel.org> Date: 2026-02-23 16:09:04
On Mon, 23 Feb 2026 16:01:03 +0000,
David Lechner [off-list ref] wrote:
On 1/1/26 4:13 PM, Frederic Weisbecker wrote:
quoted
The HK_TYPE_DOMAIN housekeeping cpumask will soon be made modifiable at
runtime. In order to synchronize against vmstat workqueue to make sure
that no asynchronous vmstat work is pending or executing on a newly made
isolated CPU, target and queue a vmstat work under the same RCU read
side critical section.
Whenever housekeeping will update the HK_TYPE_DOMAIN cpumask, a vmstat
workqueue flush will also be issued in a further change to make sure
that no work remains pending after a CPU has been made isolated.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
---
mm/vmstat.c | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
I think this might have introduced a bug - or at least an
unintentional change in the program flow.
scoped_guard() is implemented using a for loop. Now this continue
statement will only exit the scoped_guard() scope rather than
continuing the outer for loop. This means that cond_resched() will
be called when it previously was not.
Yup, I've been bitten by that once before. It lead to very subtle
breakage that took me a while to figure out. I've now mentally
banished the use of scoped_guard() inside any form of loop, but that's
a pretty brittle strategy...
M.
--
Without deviation from the norm, progress is not possible.