On an idle system with large amount of cpus it might happen that
klp_update_patch_state() is not reached in do_idle() for a long periods
of time. With debug messages enabled log is filled with:
[ 499.442643] livepatch: klp_try_switch_task: swapper/63:0 is running
without any signs of progress. Ending up with "failed to complete
transition".
On s390 LPAR with 128 cpus not a single transition is able to complete
and livepatch kselftests fail.
To deal with that, make sure we break out of do_idle() inner loop to
reach klp_update_patch_state() by marking idle tasks as NEED_RESCHED
as well as kick cpus out of idle state.
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
---
kernel/livepatch/transition.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
From: Petr Mladek <pmladek@suse.com> Date: 2021-07-08 10:12:23
On Wed 2021-07-07 14:49:38, Vasily Gorbik wrote:
On an idle system with large amount of cpus it might happen that
klp_update_patch_state() is not reached in do_idle() for a long periods
of time. With debug messages enabled log is filled with:
[ 499.442643] livepatch: klp_try_switch_task: swapper/63:0 is running
I see. I guess that the problem is only when CONFIG_NO_HZ is enabled.
Do I get it correctly, please?
quoted hunk
without any signs of progress. Ending up with "failed to complete
transition".
On s390 LPAR with 128 cpus not a single transition is able to complete
and livepatch kselftests fail.
To deal with that, make sure we break out of do_idle() inner loop to
reach klp_update_patch_state() by marking idle tasks as NEED_RESCHED
as well as kick cpus out of idle state.
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
---
kernel/livepatch/transition.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
First, we should kick the idle threads in klp_send_signals().
It already solves similar problem when normal threads and kthreads
stay in the incorruptible sleep for too long.
Second, the way looks a bit hacky to me. need_resched() depends on
the currect implementation of the idle loop. kick_process() has
a completely different purpose and does checks that do not fit well
this use-case.
I wonder if wake_up_nohz_cpu() would fit better here. Please, add
scheduler people into CC, namely:
Ingo Molnar [off-list ref]
Peter Zijlstra [off-list ref]
and NOHZ guys:
Frederic Weisbecker [off-list ref]
Thomas Gleixner [off-list ref]
Best Regards,
Petr
On Wed, 2021-07-07 at 14:49 +0200, Vasily Gorbik wrote:
On an idle system with large amount of cpus it might happen that
klp_update_patch_state() is not reached in do_idle() for a long
periods
of time. With debug messages enabled log is filled with:
[ 499.442643] livepatch: klp_try_switch_task: swapper/63:0 is
running
without any signs of progress. Ending up with "failed to complete
transition".
On s390 LPAR with 128 cpus not a single transition is able to
complete
and livepatch kselftests fail.
This also speeds up completion of the transition on high cpu count arm
instances.
Interested in seeing the correct way to address this.
Suraj.
quoted hunk
To deal with that, make sure we break out of do_idle() inner loop to
reach klp_update_patch_state() by marking idle tasks as NEED_RESCHED
as well as kick cpus out of idle state.
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
---
kernel/livepatch/transition.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/kernel/livepatch/transition.c
b/kernel/livepatch/transition.c
index 3a4beb9395c4..793eba46e970 100644
From: Petr Mladek <pmladek@suse.com> Date: 2021-08-27 12:54:43
On Wed 2021-07-07 14:49:38, Vasily Gorbik wrote:
On an idle system with large amount of cpus it might happen that
klp_update_patch_state() is not reached in do_idle() for a long periods
of time. With debug messages enabled log is filled with:
[ 499.442643] livepatch: klp_try_switch_task: swapper/63:0 is running
without any signs of progress. Ending up with "failed to complete
transition".
On s390 LPAR with 128 cpus not a single transition is able to complete
and livepatch kselftests fail.
To deal with that, make sure we break out of do_idle() inner loop to
reach klp_update_patch_state() by marking idle tasks as NEED_RESCHED
as well as kick cpus out of idle state.
This would probably do the job. Well, I wonder if the following is
a bit cleaner.
wake_up_if_idle(cpu);
Also, please do this in klp_send_signals(). We kick there all other
tasks that block the transition for too long.
Best Regards,
Petr
Yes, otherwise the inner idle loop is not left and
klp_update_patch_state() is not reached. Only waking up idle
cpus is not enough.
quoted
+ kick_process(task);
This would probably do the job. Well, I wonder if the following is
a bit cleaner.
wake_up_if_idle(cpu);
wake_up_if_idle() is nice way to identify idle cpus, but it does not
force idle task rescheduling in our case.
Also, please do this in klp_send_signals(). We kick there all other
tasks that block the transition for too long.
#define SIGNALS_TIMEOUT 15
Hm, kicking the idle threads in klp_send_signals() means extra 15 seconds
delay for every transition in our case and failing kselftests:
# --- expected
# +++ result
...
# livepatch: 'test_klp_livepatch': starting patching transition
# +livepatch: signaling remaining tasks
# livepatch: 'test_klp_livepatch': completing patching transition
BTW, for x86 I made a lousy tests with 128 cpus in kvm. With default
CONFIG_NO_HZ_IDLE=y kselftests are failing the same way. Sometimes
transition times out as well, despite NO_HZ options configurations.
Jul 09 11:43:33 q.q kernel: livepatch: 'test_klp_livepatch': starting patching transition
Jul 09 11:44:37 q.q kernel: livepatch: 'test_klp_livepatch': starting unpatching transition
Jul 09 11:45:40 q.q ERROR: failed to disable livepatch test_klp_livepatch
I understand this 15 seconds delay for loaded system and tasks doing real
work is good, but those lazy idle "running" tasks could be kicked right
away with no harm done, right? Given we are able to identify them reliably.
I'll send another patch version.
Yes, otherwise the inner idle loop is not left and
klp_update_patch_state() is not reached. Only waking up idle
cpus is not enough.
I see.
quoted
Also, please do this in klp_send_signals(). We kick there all other
tasks that block the transition for too long.
#define SIGNALS_TIMEOUT 15
Hm, kicking the idle threads in klp_send_signals() means extra 15 seconds
delay for every transition in our case and failing kselftests:
I understand this 15 seconds delay for loaded system and tasks doing real
work is good,
Yup. Also normal processes should not stay in the running state
for this long. They are typically migrated quickly. But the idle task is
special.
but those lazy idle "running" tasks could be kicked right
away with no harm done, right?