[PATCH v11 05/12] sched/core: Try to use a preferred CPU in is_cpu_allowed
From: Shrikanth Hegde <sshegde@linux.ibm.com>
Date: 2026-08-25 10:40:43
Also in:
lkml, virtualization
Subsystem:
scheduler, the rest · Maintainers:
Ingo Molnar, Peter Zijlstra, Juri Lelli, Vincent Guittot, Linus Torvalds
When possible, try to choose a preferred CPU. This is essential to maintain user affinities when preferred CPUs change. A task pinned on a non-preferred CPU should continue to run there, since this is a non-user triggered event. If a CPU is non-preferred and the task can run on other CPUs which are currently preferred, then choose a preferred CPU instead. This is decided by checking if cpus_ptr and cpu_preferred_mask intersect or not. If yes, then the task has other preferred CPUs. The push task mechanism uses a stopper thread which calls select_fallback_rq() and uses this mechanism to pick a preferred CPU. This takes care of the wakeup path for FAIR tasks too. is_cpu_allowed() is called to ensure wakeups happen on preferred CPUs. With that, additional checks in available_idle_cpu() are not necessary. Ignore the preferred CPU state if a task's affinity is changing and its new mask no longer includes the CPU it is currently running on. This ensures migration_cpu_stop() does not abort, preventing the task from being stranded outside its allowed affinity. Account for tasks with architecture-specific CPU masks (e.g., 32-bit tasks on arm64). For such tasks, explicitly check against the arch-allowed CPUs to determine if any of the preferred CPUs are actually valid. For the majority of cases, this would still keep select_fallback_rq() as O(N). cpumask_intersects(), which is O(N), is called only if !cpu_preferred. The task running there is expected to move out. Subsequently, it should run on a preferred CPU. This becomes O(N**2) only for tasks pinned solely to non-preferred CPUs. That is a rare case. Overhead is minimal when the CPU is preferred. Signed-off-by: Shrikanth Hegde <sshegde@linux.ibm.com> --- kernel/sched/core.c | 41 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index a45f7c308329..f71317fe281d 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c@@ -2494,6 +2494,35 @@ static inline bool rq_has_pinned_tasks(struct rq *rq) return rq->nr_pinned; } +static inline bool task_can_sched_on_preferred(int cpu, struct task_struct *p) +{ + const struct cpumask *valid_mask; + int i; + + if (cpu_preferred(cpu)) + return false; + + /* Only FAIR tasks honor preferred CPU state */ + if (unlikely(p->sched_class != &fair_sched_class)) + return false; + + /* Ignore preferred state if task affinity is changing */ + if (unlikely(!cpumask_test_cpu(task_cpu(p), p->cpus_ptr))) + return false; + + valid_mask = task_cpu_possible_mask(p); + if (likely(valid_mask == cpu_possible_mask)) + return cpumask_intersects(p->cpus_ptr, cpu_preferred_mask); + + /* Tasks with arch-specific CPU masks. e.g. 32-bit tasks on arm64. */ + for_each_cpu_and(i, p->cpus_ptr, cpu_preferred_mask) { + if (cpumask_test_cpu(i, valid_mask)) + return true; + } + + return false; +} + /* * Per-CPU kthreads are allowed to run on !active && online CPUs, see * __set_cpus_allowed_ptr() and select_fallback_rq().
@@ -2509,8 +2538,12 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu) return cpu_online(cpu); /* Non kernel threads are not allowed during either online or offline. */ - if (!(p->flags & PF_KTHREAD)) + if (!(p->flags & PF_KTHREAD)) { + /* Try to use preferred CPU if task's affinity allows */ + if (task_can_sched_on_preferred(cpu, p)) + return false; return cpu_active(cpu); + } /* KTHREAD_IS_PER_CPU is always allowed. */ if (kthread_is_per_cpu(p))
@@ -2520,7 +2553,11 @@ static inline bool is_cpu_allowed(struct task_struct *p, int cpu) if (cpu_dying(cpu)) return false; - /* But are allowed during online. */ + /* Try to keep unbound kthreads on a preferred CPU if possible. */ + if (task_can_sched_on_preferred(cpu, p)) + return false; + + /* Otherwise, they are allowed to run on online CPU. */ return cpu_online(cpu); }
--
2.47.3