Re: [bisected][mainline]Kernel warnings at kernel/sched/cpudeadline.c:219
From: Peter Zijlstra <peterz@infradead.org>
Date: 2025-10-08 11:13:24
Also in:
lkml
On Wed, Oct 08, 2025 at 03:47:16PM +0530, Shrikanth Hegde wrote:
On 10/8/25 3:20 PM, Peter Zijlstra wrote:quoted
On Wed, Oct 08, 2025 at 07:41:10AM +0530, Venkat Rao Bagalkote wrote:quoted
Greetings!!! IBM CI has reported a kernel warnings while running CPU hot plug operation on IBM Power9 system. Command to reproduce the issue: drmgr -c cpu -r -q 1 Git Bisect is pointing to below commit as the first bad commit.Does something like this help? (also, for future reference, please don't line wrap logs, it makes them very hard to read)diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 198d2dd45f59..65f37bfcd661 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c@@ -8328,6 +8328,7 @@ static inline void sched_set_rq_offline(struct rq *rq, int cpu) BUG_ON(!cpumask_test_cpu(cpu, rq->rd->span)); set_rq_offline(rq); } + dl_server_stop(&rq->fair_server); rq_unlock_irqrestore(rq, &rf); }Hi Peter. Thanks for looking into it. I was able to repro this issue on my system. This above diff didn't help. I still see the warning. I have to understand this dl server stuff still. So not sure if my understanding is completely correct. Looks like the hrtimer is firing after the cpu was removed. The warn on hit only with drmgr. Regular hotplug with chcpu doesn;t hit. That's because drmgr changes the cpu_present mask. and warning is hit with it.
I do not know what drmgr is. I am not familiar with PowerPC tools. AFAICT x86 never modifies cpu_present_mask after boot.
maybe during drmgr, the dl server gets started again? Maybe that's why above patch it didn't work. Will see and understand this bit more.
dl_server is per cpu and is started on enqueue of a fair task when: - the runqueue was empty; and - the dl_server wasn't already active Once the dl_server is active it has this timer (you already found this), this timer is set for the 0-laxity moment (the last possible moment in time where it can still run its budget and not be late), during this time any fair runtime is accounted against it budget (subtracted from). Once the timer fires and it still has budget left; it will enqueue the deadline entity. However the more common case is that its budget will be depleted, in which case the timer is reset to its period end for replenish (where it gets new runtime budget), after which its back to the 0-laxity. If the deadline entity gets scheduled, it will try and pick a fair task and run that. In the case where there is no fair task, it will deactivate itself. The patch I sent earlier would force stop the deadline timer on CPU offline.
quoted hunk ↗ jump to hunk
Also, i tried this below diff which fixes it. Just ignore the hrtimer if the cpu is offline. Does this makes sense? ---diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index 615411a0a881..a342cf5e4624 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c@@ -1160,6 +1160,9 @@ static enum hrtimer_restart dl_server_timer(struct hrtimer *timer, struct sched_ scoped_guard (rq_lock, rq) { struct rq_flags *rf = &scope.rf; + if (!cpu_online(rq->cpu)) + return HRTIMER_NORESTART; + if (!dl_se->dl_throttled || !dl_se->dl_runtime) return HRTIMER_NORESTART;
This could leave the dl_server in inconsistent state. It would have to call dl_server_stop() or something along those lines. Also, this really should not happen; per my previous patch we should be stopping the timer when we go offline. Since you can readily reproduce this; perhaps you could stick something like this in dl_server_start(): WARN_ON_ONCE(!cpu_online(rq->cpu)) See if anybody is (re)starting the thing ?