From: Josef Bacik <josef@toxicpanda.com> Date: 2026-09-10 18:50:50
Tasks RCU only treats a voluntary context switch, usermode or idle as a
quiescent state, because a preempted task may be sitting in a trampoline
that is about to be freed. That was a fine trade when PREEMPT_NONE
servers compiled Tasks RCU away and PREEMPT desktops rarely ran
long-lived in-kernel loops. PREEMPT_LAZY changes both halves at once:
Tasks RCU is now real on server configs, and cond_resched() is a no-op,
so a CPU-bound kthread or kworker only ever loses the CPU by being
preempted, which is exactly the event Tasks RCU refuses to count.
The way this showed up for us was a cgroup writeback worker draining a
very large cgwb for around eleven minutes on an arm64 box. Nothing wrong
with that on its own, but a BPF program detach on another CPU went
bpf_trampoline_update() -> ftrace_shutdown() -> synchronize_rcu_tasks()
while holding trampoline_mutex, forty-odd tasks piled up behind the
mutex, and the hung task detector panicked the machine. The kprobe jump
optimizer is worse in principle: it does synchronize_rcu_tasks() under
kprobe_mutex, text_mutex and cpus_read_lock(), so one long-running
kthread can stall static key updates and CPU hotplug for its whole run.
The current answer is to find each such loop and add
cond_resched_tasks_rcu_qs() to it, which is the kind of annotation
PREEMPT_LAZY was supposed to let us stop writing.
This series tries the other direction: have the trampolines say when a
task is inside them, so that a preemption anywhere else can be a
quiescent state.
- task_struct grows an int, rcu_tramp_nesting. Every trampoline whose
lifetime Tasks RCU guards increments it before calling out and
decrements it before returning: ftrace_caller and its dynamic copies,
the BPF trampoline (which drops it again around the call to the
original function, since im->pcref covers that), the x86 optprobe
template, and out-of-line register_ftrace_direct() trampolines. Only
current writes it and nested users are balanced, so it is a plain
non-atomic inc/dec, one load of current plus one RMW per entry/exit.
- The inc/dec are inside the trampoline, so there is a window of a few
instructions on each side where the count is zero but the task is in
(or on its way into) trampoline text. Nothing there can be preempted
synchronously, only from an interrupt, so the irq-exit preemption path
looks at regs->ip and holds the count across preempt_schedule_irq()
when the IP is somewhere the counter cannot cover: outside core and
module text (all the dynamically allocated trampolines and slots), in
the static ftrace stubs or the x86 return thunks that still hold a
direct-call target, in a module that hosts its own direct trampoline,
or inside the bytes after a kprobe that the jump optimizer may be
about to rewrite (the one synchronize_rcu_tasks() user that is not
about trampolines at all).
- With those in place, rcu_tasks_classic_qs() also clears the holdout
flag on a preemption when the count is zero, on architectures that
opt in. x86-64 and arm64 do so here. Everyone else keeps the
voluntary-only rule and is untouched apart from the (unused) field.
A running holdout already gets poked via rcu_request_urgent_qs_task(),
which makes the next tick set NEED_RESCHED, so with this the resulting
preemption retires it and a Tasks RCU grace period is bounded by roughly
a tick plus the longest preempt-off section rather than by the longest
stretch without a voluntary schedule().
Patches 1-12 are scaffolding and change no behaviour on their own; patch
13 flips the rule and selects the option for the two architectures.
Testing so far is QEMU only: x86-64, PREEMPT_LAZY with PREEMPT_RCU=n,
PROVE_RCU and lockdep, with and without PREEMPT_DYNAMIC. A kthread
spinning in-kernel for 30s with the function tracer, an ftrace kprobe,
an optimized kprobe and fentry/fexit programs attached:
synchronize_rcu_tasks() goes from 29.7s to 0.1-0.3s, tearing down a
DYNAMIC ftrace_ops (tracefs instance function -> nop) from 27s to
0.2-0.8s, and the ftrace-direct sample modules load, fire and unload in
about 2.5s each while the spinner runs, with no warnings and the new
return-to-user assertion quiet. arm64 is build-tested only at this
point; real hardware numbers for both are the obvious next step and I
did not want to sit on the idea waiting for them.
Things I would particularly like opinions on:
- Whether hooking rcu_tasks_classic_qs() is the right place, or whether
Paul would rather see this expressed differently inside Tasks RCU.
- return_to_handler and the rethook/kretprobe trampolines are not
instrumented. Their C callees take the ftrace recursion lock before
touching any ops and the trampolines themselves are static text, so I
believe they do not need it, but I would like Steven and Masami to
confirm.
- The register_ftrace_direct() contract change: out-of-line direct
trampolines now have to maintain the count themselves (the samples
are converted). I do not know of out-of-tree users beyond BPF, but
this is the one place an existing user could be silently weakened.
- Whether arm64 folks are comfortable with the ldr/add/str in
ftrace_caller and the BPF trampoline, and with treating all of
ftrace_caller as trampoline text for the IP check.
- If this holds up, cond_resched_tasks_rcu_qs() and
rcu_softirq_qs_periodic() become unnecessary on the opted-in
architectures; I have not touched them here.
Based on v7.3-rc2+ (893e11787f78).
---
Josef Bacik (13):
rcu-tasks: Add per-task trampoline nesting count
entry: Pass pt_regs to irqentry_exit_cond_resched()
rcu-tasks: Hold trampoline nesting across irq-exit preemption in trampoline text
kprobes: Let Tasks RCU recognise tasks preempted in an optprobe jump window
ftrace: Mark modules hosting direct-call trampolines for Tasks RCU
x86/ftrace: Maintain Tasks RCU trampoline nesting in ftrace_caller
x86/kprobes: Maintain Tasks RCU trampoline nesting in the optprobe template
bpf, x86: Maintain Tasks RCU trampoline nesting in the BPF trampoline
arm64: ftrace: Maintain Tasks RCU trampoline nesting in ftrace_caller
bpf, arm64: Maintain Tasks RCU trampoline nesting in the BPF trampoline
samples: ftrace: Maintain Tasks RCU trampoline nesting in direct-call trampolines
rcutorture: Bracket Tasks RCU readers with trampoline nesting
rcu-tasks: Treat preemption outside trampolines as a quiescent state
arch/arm64/Kconfig | 1 +
arch/arm64/kernel/asm-offsets.c | 3 ++
arch/arm64/kernel/entry-ftrace.S | 35 +++++++++++++
arch/arm64/kernel/ftrace.c | 16 ++++++
arch/arm64/net/bpf_jit_comp.c | 46 ++++++++++++++++
arch/x86/Kconfig | 1 +
arch/x86/kernel/asm-offsets.c | 3 ++
arch/x86/kernel/ftrace.c | 37 +++++++++++++
arch/x86/kernel/ftrace_64.S | 43 +++++++++++++++
arch/x86/kernel/kprobes/opt.c | 20 +++++++
arch/x86/kernel/vmlinux.lds.S | 4 ++
arch/x86/net/bpf_jit_comp.c | 43 +++++++++++++++
arch/x86/xen/enlighten_pv.c | 2 +-
include/linux/irq-entry-common.h | 14 ++---
include/linux/kprobes.h | 8 ++-
include/linux/module.h | 7 +++
include/linux/rcupdate.h | 70 ++++++++++++++++++++++++-
include/linux/sched.h | 1 +
kernel/entry/common.c | 29 +++++++++--
kernel/fork.c | 1 +
kernel/kprobes.c | 24 +++++++++
kernel/rcu/Kconfig | 17 ++++--
kernel/rcu/rcutorture.c | 6 +++
kernel/rcu/tasks.h | 81 +++++++++++++++++++++++++++--
kernel/rcu/update.c | 2 +
kernel/trace/ftrace.c | 39 ++++++++++++++
samples/ftrace/ftrace-direct-modify.c | 9 ++++
samples/ftrace/ftrace-direct-multi-modify.c | 9 ++++
samples/ftrace/ftrace-direct-multi.c | 5 ++
samples/ftrace/ftrace-direct-too.c | 5 ++
samples/ftrace/ftrace-direct.c | 5 ++
samples/ftrace/ftrace-direct.h | 64 +++++++++++++++++++++++
32 files changed, 629 insertions(+), 21 deletions(-)
---
base-commit: 893e11787f78e43b534e252249ac3fff4d1333f8
change-id: 20260910-b4-rcu-tasks-preempt-qs-401ff45465c7
Best regards,
--
Josef Bacik [off-list ref]
From: Josef Bacik <josef@toxicpanda.com> Date: 2026-09-10 18:50:58
The irq-exit preemption path is about to need the interrupted context's
registers to decide whether the preemption may be reported to Tasks RCU
as a quiescent state. irqentry_exit_to_kernel_mode_preempt() already
has them; hand them down through irqentry_exit_cond_resched(), its
PREEMPT_DYNAMIC static-call and static-key variants, and
raw_irqentry_exit_cond_resched(). The only caller outside the generic
entry code is Xen PV's upcall handler, which has regs as well.
No functional change.
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
arch/x86/xen/enlighten_pv.c | 2 +-
include/linux/irq-entry-common.h | 12 ++++++------
kernel/entry/common.c | 6 +++---
3 files changed, 10 insertions(+), 10 deletions(-)
From: Josef Bacik <josef@toxicpanda.com> Date: 2026-09-10 18:51:00
Tasks RCU exists so that ftrace, BPF and kprobes can free trampoline
text once no task can still be executing in it. Today the only way a
task tells Tasks RCU "I am not in a trampoline" is a voluntary context
switch, so a preempted task is always assumed to be inside one.
Add task_struct::rcu_tramp_nesting so that trampolines can say so
directly: a trampoline increments it before calling out and decrements
it before returning, and while it is non-zero the task must not be
treated as Tasks-RCU quiescent. Provide rcu_tasks_trampoline_enter()
and rcu_tasks_trampoline_exit() for C users, report the count in the
Tasks RCU stall output, and, under CONFIG_PROVE_RCU, assert that it is
zero on every return to userspace since no task can legitimately reach
userspace with a trampoline on its stack.
Only current ever writes the count and every nested user (interrupts
running their own trampolines) is balanced, so plain accesses suffice.
Nothing increments the count and nothing consults it for quiescent-state
decisions yet; both come in later patches.
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
include/linux/irq-entry-common.h | 2 ++
include/linux/rcupdate.h | 37 +++++++++++++++++++++++++++++++++++++
include/linux/sched.h | 1 +
kernel/fork.c | 1 +
kernel/rcu/tasks.h | 3 ++-
5 files changed, 43 insertions(+), 1 deletion(-)
@@ -214,6 +215,7 @@ static __always_inline void __exit_to_user_mode_validate(void){/* Ensure that kernel state is sane for a return to userspace */kmap_assert_nomap();+rcu_tasks_trampoline_assert_none();lockdep_assert_irqs_disabled();lockdep_sys_exit();}
@@ -180,6 +180,37 @@ static inline void rcu_nocb_flush_deferred_wakeup(void) { }#ifdef CONFIG_TASKS_RCU_GENERIC# ifdef CONFIG_TASKS_RCU++/*+*Trampolinenesting:dynamicallyallocatedtext(ftracetrampolines,BPF+*trampolineimages,kprobeoptinsnslots)thatreliesonTasksRCUforits+*lifetimebracketsitselfwithanincrement/decrementof+*current->rcu_tramp_nesting.Whilethecountisnon-zerothetaskisinside,+*orwascalledfrom,suchtextandaninvoluntarycontextswitchmustnotbe+*treatedasaTasksRCUquiescentstate.+*+*Onlycurrentwritesthecountandonlycurrent(oraninterruptonthesame+*CPU)readsit,soplainaccessessuffice.+*/+static__always_inlinevoidrcu_tasks_trampoline_enter(void)+{+current->rcu_tramp_nesting++;+barrier();+}++static__always_inlinevoidrcu_tasks_trampoline_exit(void)+{+barrier();+current->rcu_tramp_nesting--;+}++/* A task must never reach userspace with a trampoline on its stack. */+static__always_inlinevoidrcu_tasks_trampoline_assert_none(void)+{+if(IS_ENABLED(CONFIG_PROVE_RCU))+WARN_ON_ONCE(current->rcu_tramp_nesting);+}+# define rcu_tasks_classic_qs(t, preempt) \do{\if(!(preempt)&&READ_ONCE((t)->rcu_tasks_holdout))\
@@ -208,6 +242,9 @@ void exit_tasks_rcu_finish(void);#define rcu_tasks_classic_qs(t, preempt) do { } while (0)#define rcu_tasks_qs(t, preempt) do { } while (0)#define rcu_note_voluntary_context_switch(t) do { } while (0)+staticinlinevoidrcu_tasks_trampoline_enter(void){}+staticinlinevoidrcu_tasks_trampoline_exit(void){}+staticinlinevoidrcu_tasks_trampoline_assert_none(void){}#define call_rcu_tasks call_rcu#define synchronize_rcu_tasks synchronize_rcustaticinlinevoidexit_tasks_rcu_start(void){}
From: Josef Bacik <josef@toxicpanda.com> Date: 2026-09-10 18:51:01
A trampoline's own rcu_tramp_nesting increment and decrement live inside
the trampoline, so there is a window of a few instructions on entry and
exit where the count is zero while the CPU is executing trampoline text
(or text on the way into one, such as a static ftrace stub holding a
direct-call target). In that window the task has not called out, so it
can only be preempted from an interrupt, and the interrupted instruction
pointer identifies where it is.
Add rcu_tasks_ip_in_trampoline(), which treats any IP outside core
kernel and module text as potentially Tasks-RCU-protected (ftrace
trampolines, BPF images and programs, kprobe slots are all dynamically
allocated text; is_ftrace_trampoline() and friends are deliberately not
used because text being torn down may already be unregistered from them
while a task still stands on it), plus a __weak
arch_rcu_tasks_ip_in_trampoline() for core text an architecture needs
to flag. On irq-exit preemption, if the IP matches, hold the count
elevated across preempt_schedule_irq().
Introduce ARCH_HAS_RCU_TASKS_PREEMPT_QS / RCU_TASKS_PREEMPT_QS to gate
this; no architecture selects it yet, so the check compiles away and
there is no functional change.
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
include/linux/rcupdate.h | 17 +++++++++++++++++
kernel/entry/common.c | 23 ++++++++++++++++++++++-
kernel/rcu/Kconfig | 10 ++++++++++
kernel/rcu/tasks.h | 38 ++++++++++++++++++++++++++++++++++++++
kernel/rcu/update.c | 2 ++
5 files changed, 89 insertions(+), 1 deletion(-)
@@ -107,6 +107,16 @@ config TASKS_RCUdefaultNEED_TASKS_RCU&&PREEMPTIONselectIRQ_WORK+# Selected by architectures whose ftrace, BPF and kprobe trampolines maintain+# current->rcu_tramp_nesting and which use the generic irqentry code, so that+# a preemption outside any trampoline can be treated as a Tasks RCU+# quiescent state. See rcu_tasks_trampoline_enter().+configARCH_HAS_RCU_TASKS_PREEMPT_QS+bool++configRCU_TASKS_PREEMPT_QS+def_boolTASKS_RCU&&ARCH_HAS_RCU_TASKS_PREEMPT_QS&&GENERIC_IRQ_ENTRY+configFORCE_TASKS_RUDE_RCUbool"Force selection of Tasks Rude RCU"depends onRCU_EXPERT
@@ -1089,6 +1089,44 @@ static void rcu_tasks_postscan(struct list_head *hop)timer_delete_sync(&tasks_rcu_exit_stall_timer);}+/*+*ArchitecturesselectingARCH_HAS_RCU_TASKS_PREEMPT_QSoverridethistoflag+*corekerneltextthatmustbetreatedlikeatrampoline,e.g.staticftrace+*entrystubsandreturnthunksthatrunwithatrampolineaddressinhand.+*/+bool__weakarch_rcu_tasks_ip_in_trampoline(unsignedlongip)+{+returnfalse;+}++/**+*rcu_tasks_ip_in_trampoline-Couldataskinterruptedat@ipbeaTasksRCUreader?+*@ip:interruptedinstructionpointer+*+*Calledfromtheirq-exitpreemptionpathwithinterruptsdisabled,todecide+*whethertheimminentpreemptionmaybereportedasaTasksRCUquiescent+*statewhencurrent->rcu_tramp_nestingiszero.Returnstrue,meaning"do+*notreport", when @ip is:+*+*-outsidestatickernelandmoduletext,i.e.possiblyinanftrace+*trampoline,BPFtrampolineimageorprogram,kprobeinsn/optinsnslotor+*otherdynamicallyallocatedtextwhoselifetimeTasksRCUguards.This+*deliberatelydoesnotconsultis_ftrace_trampoline()andfriends:text+*beingtorndownmayalreadybeunregisteredtherewhileataskstill+*standsonit;+*-incoretextthearchitectureflagsviaarch_rcu_tasks_ip_in_trampoline().+*+*Afalsepositiveonlydefersthequiescentstatetothetask'snext+*contextswitch.+*/+boolrcu_tasks_ip_in_trampoline(unsignedlongip)+{+if(core_kernel_text(ip))+returnarch_rcu_tasks_ip_in_trampoline(ip);+return!is_module_text_address(ip);+}+NOKPROBE_SYMBOL(rcu_tasks_ip_in_trampoline);+/* See if tasks are still holding out, complain if so. */staticvoidcheck_holdout_task(structtask_struct*t,boolneedreport,bool*firstreport)
From: Josef Bacik <josef@toxicpanda.com> Date: 2026-09-10 18:51:05
An out-of-line direct trampoline registered with register_ftrace_direct()
is kept alive only by Tasks RCU while a task executes it or is preempted
in something it called; ftrace_shutdown()'s synchronize_rcu_tasks() is
what stops rmmod freeing it under such a task. Once preemption becomes a
Tasks RCU quiescent state, such a trampoline must hold
current->rcu_tramp_nesting across its call-out like the ftrace and BPF
trampolines do, so document that in register_ftrace_direct().
That still leaves the few instructions before the increment and after
the decrement. For BPF images those are in dynamically allocated text
that rcu_tasks_ip_in_trampoline() already treats as protected, but the
in-tree samples (and any similar user) place their trampolines in module
.text. Add a sticky module::ftrace_direct_tramp flag, set by every
register/modify path when the direct address is module text, and have
rcu_tasks_ip_in_trampoline() treat a task interrupted anywhere in such a
module as a potential reader. Other modules' text is unaffected.
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
include/linux/module.h | 7 +++++++
kernel/rcu/tasks.h | 23 +++++++++++++++++++++--
kernel/trace/ftrace.c | 39 +++++++++++++++++++++++++++++++++++++++
3 files changed, 67 insertions(+), 2 deletions(-)
@@ -6169,6 +6203,7 @@ int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)ops->flags|=MULTI_FLAGS;ops->trampoline=FTRACE_REGS_ADDR;ops->direct_call=addr;+ftrace_direct_mark_module(addr);err=register_ftrace_function_nolock(ops);if(err)
@@ -6237,6 +6272,8 @@ __modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr)lockdep_assert_held_once(&direct_mutex);+ftrace_direct_mark_module(addr);+/* Enable the tmp_ops to have the same functions as the direct ops */ftrace_ops_init(&tmp_ops);tmp_ops.func_hash=ops->func_hash;
From: Josef Bacik <josef@toxicpanda.com> Date: 2026-09-10 18:51:06
Bracket the call out to the ftrace_ops callback in ftrace_caller and
ftrace_regs_caller with an increment/decrement of
current->rcu_tramp_nesting. The instructions sit inside the region that
create_trampoline() copies for per-ops dynamic trampolines, so those
inherit them; the %rip-relative per-CPU reference to current_task is
fixed up by text_poke_apply_relocation() like CALL_DEPTH_ACCOUNT's.
%rdx is dead at both points (about to be loaded with the ops pointer on
entry, restored by restore_mcount_regs on exit).
Two pieces of core text still run with the count at zero while holding
the address of a Tasks-RCU-protected trampoline they are about to
enter: the static stubs themselves, whose direct-call tails keep a BPF
trampoline address on the stack until the final RET, and, under
CONFIG_MITIGATION_RETHUNK, the return thunk that RET expands to. Add an
ftrace_static_tramp_end marker after ftrace_stub_direct_tramp and linker
symbols around .text..__x86.return_thunk and .text..__x86.rethunk_safe,
and provide arch_rcu_tasks_ip_in_trampoline() covering
[ftrace_caller, ftrace_static_tramp_end) and both thunk ranges so the
irq-exit check treats a task interrupted there as still inside a
trampoline.
The hook is built only under CONFIG_RCU_TASKS_PREEMPT_QS, which x86 does
not select until a later patch.
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
arch/x86/kernel/asm-offsets.c | 3 +++
arch/x86/kernel/ftrace.c | 37 +++++++++++++++++++++++++++++++++++++
arch/x86/kernel/ftrace_64.S | 43 +++++++++++++++++++++++++++++++++++++++++++
arch/x86/kernel/vmlinux.lds.S | 4 ++++
4 files changed, 87 insertions(+)
@@ -275,6 +275,43 @@ static inline void tramp_free(void *tramp)execmem_free(tramp);}+#ifdef CONFIG_RCU_TASKS_PREEMPT_QS+externvoidftrace_static_tramp_end(void);+externchar__return_thunk_start[],__return_thunk_end[];+externchar__rethunk_safe_start[],__rethunk_safe_end[];++/*+*Seercu_tasks_ip_in_trampoline().Somecorekerneltextbehaveslikea+*trampolineforTasksRCUpurposesbecauseataskexecutingtherewith+*rcu_tramp_nesting==0maystillbeabouttoenteraTasks-RCU-protected+*trampolinewhoseaddressitalreadyholds:+*+*-thestaticftrace_caller/ftrace_regs_caller/ftrace_stub_direct_tramp+*stubs,whichcarryadirect-calltargetonthestackuntiltheirfinal+*RET,and+*-thereturnthunksthatRETexpandstounderCONFIG_MITIGATION_RETHUNK,+*whichrunafterleavingthestubsaboveandbeforelandinginthat+*target.+*/+boolarch_rcu_tasks_ip_in_trampoline(unsignedlongip)+{+if(ip>=(unsignedlong)ftrace_caller&&+ip<(unsignedlong)ftrace_static_tramp_end)+returntrue;+#ifdef CONFIG_MITIGATION_RETPOLINE+if(ip>=(unsignedlong)__return_thunk_start&&+ip<(unsignedlong)__return_thunk_end)+returntrue;+#endif+#ifdef CONFIG_MITIGATION_SRSO+if(ip>=(unsignedlong)__rethunk_safe_start&&+ip<(unsignedlong)__rethunk_safe_end)+returntrue;+#endif+returnfalse;+}+#endif /* CONFIG_RCU_TASKS_PREEMPT_QS */+/* Defined as markers to the end of the ftrace default trampolines */externvoidftrace_regs_caller_end(void);externvoidftrace_caller_end(void);
From: Josef Bacik <josef@toxicpanda.com> Date: 2026-09-10 18:51:08
kprobe_optimizer() is the one synchronize_rcu_tasks() user that is not
about trampoline text: it waits for tasks that were preempted on an
instruction boundary inside the bytes it is about to overwrite with the
optimized jump, so that none of them resumes into the middle of the new
instruction. Such a task sits in ordinary kernel or module text with
rcu_tramp_nesting == 0, and can only have got there via an irq-exit
preemption.
Add kprobe_in_optimized_region(), a lockless and conservative form of
get_optimized_kprobe() that reports whether any registered kprobe lies
within MAX_OPTIMIZED_LENGTH before the given address regardless of its
optimization state, and have rcu_tasks_ip_in_trampoline() consult it so
that a task interrupted there keeps holding off the Tasks RCU grace
period once preemption becomes a quiescent state.
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
include/linux/kprobes.h | 8 +++++++-
include/linux/rcupdate.h | 4 +++-
kernel/kprobes.c | 24 ++++++++++++++++++++++++
kernel/rcu/tasks.h | 6 ++++++
4 files changed, 40 insertions(+), 2 deletions(-)
From: Josef Bacik <josef@toxicpanda.com> Date: 2026-09-10 18:51:12
Emit an increment of current->rcu_tramp_nesting once the trampoline's
frame is set up and a decrement before the final register restore, so
that a task preempted while running fentry/fexit/fmod_ret/LSM programs
or the __bpf_tramp_enter()/__bpf_tramp_exit() glue is not treated as
Tasks-RCU quiescent. Drop the count around the call to the original
function: that may run arbitrarily long without sleeping and must not pin
a Tasks RCU grace period, and the trampoline frame above it is held by
im->pcref rather than by Tasks RCU (see bpf_tramp_image_put()). The
fmod_ret early-exit branch and the ip_after_call -> ip_epilogue poke both
skip the decrement/increment pair around the original call, so the count
stays balanced on every path.
The sequence is "mov r11, gs:[current_task]; inc/dec dword [r11 + off]";
r11 is scratch at every emission point and (u32)¤t_task is a valid
sign-extended %gs-absolute with the current per-CPU layout, the same form
the JIT already uses for this_cpu_off. The image is dynamically
allocated text, so the instructions outside the bracketed region are
covered by the irq-exit IP check.
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
arch/x86/net/bpf_jit_comp.c | 43 +++++++++++++++++++++++++++++++++++++++++++
1 file changed, 43 insertions(+)
@@ -3610,6 +3635,13 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im/* mov QWORD PTR [rbp - rbx_off], rbx */emit_stx(&prog,BPF_DW,BPF_REG_FP,BPF_REG_6,-rbx_off);+/*+*Fromhereuntilthematchingdecrementbeforethefinalreturn,a+*preemptionofthistaskisnotaTasksRCUquiescentstate.The+*instructionsabovethispointarecoveredbytheirq-exitIPcheck.+*/+emit_rcu_tasks_tramp_nesting(&prog,true);+func_meta=nr_regs;/* Store number of argument registers of the traced function */emit_store_stack_imm64(&prog,BPF_REG_0,-func_meta_off,func_meta);
@@ -3680,6 +3719,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_imgotocleanup;}}+emit_rcu_tasks_tramp_nesting(&prog,true);/* remember return value in a stack for bpf prog to access */emit_stx(&prog,BPF_DW,BPF_REG_FP,BPF_REG_0,-8);im->ip_after_call=image+(prog-(u8*)rw_image);
@@ -3741,6 +3781,9 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_imif(save_ret)emit_ldx(&prog,BPF_DW,BPF_REG_0,BPF_REG_FP,-8);+/* Remaining instructions are covered by the irq-exit IP check. */+emit_rcu_tasks_tramp_nesting(&prog,false);+emit_ldx(&prog,BPF_DW,BPF_REG_6,BPF_REG_FP,-rbx_off);EMIT1(0xC9);/* leave */
From: Josef Bacik <josef@toxicpanda.com> Date: 2026-09-10 18:51:16
Same scheme as x86: emit "mrs x10, sp_el0; ldr/add|sub/str w11" to bump
current->rcu_tramp_nesting after the callee-saved registers are stored
and to drop it before they are restored, and release it around the call
to the original function, which im->pcref protects and which must not
pin a Tasks RCU grace period. x10/x11 are scratch at every emission
point; the fmod_ret cbnz target lies after the decrement/increment pair
around the original call, and the ip_after_call nop follows the
re-increment, so the count is balanced on every path. BUILD_BUG_ON
guards the LDR/STR immediate range for the task_struct offset.
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
arch/arm64/net/bpf_jit_comp.c | 46 +++++++++++++++++++++++++++++++++++++++++++
1 file changed, 46 insertions(+)
@@ -2854,6 +2882,13 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,emit(A64_STR64I(A64_R(19),A64_SP,regs_off),ctx);emit(A64_STR64I(A64_R(20),A64_SP,regs_off+8),ctx);+/*+*Fromhereuntilthematchingdecrementintheepilogue,apreemption+*ofthistaskisnotaTasksRCUquiescentstate.Theinstructions+*abovethispointarecoveredbytheirq-exitIPcheck.+*/+emit_rcu_tasks_tramp_nesting(ctx,true);+if(flags&BPF_TRAMP_F_CALL_ORIG){/* for the first pass, assume the worst case */if(!ctx->image)
@@ -2898,12 +2933,20 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,if(flags&BPF_TRAMP_F_CALL_ORIG){/* the original func takes kernel addresses, never converted ones */save_args(ctx,bargs_off,oargs_off,m,a,true,is_struct_ops,0);+/*+*Theoriginalfunctionmayrunforalongtimewithout+*sleeping;donotletitpinaTasksRCUgraceperiod.The+*trampolineframeaboveitisheldbyim->pcref+*(__bpf_tramp_enter()),notbyTasksRCU,acrossthecall.+*/+emit_rcu_tasks_tramp_nesting(ctx,false);/* call original func */emit(A64_LDR64I(A64_R(10),A64_SP,retaddr_off),ctx);emit(A64_ADR(A64_LR,AARCH64_INSN_SIZE*2),ctx);emit(A64_RET(A64_R(10)),ctx);/* store return value */emit(A64_STR64I(A64_R(0),A64_SP,retval_off),ctx);+emit_rcu_tasks_tramp_nesting(ctx,true);/* reserve a nop for bpf_tramp_image_put */im->ip_after_call=ctx->ro_image+ctx->idx;emit(A64_NOP,ctx);
@@ -2945,6 +2988,9 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,if(flags&BPF_TRAMP_F_RESTORE_REGS)restore_args(ctx,bargs_off,a->regs_for_args);+/* Remaining instructions are covered by the irq-exit IP check. */+emit_rcu_tasks_tramp_nesting(ctx,false);+/* restore callee saved register x19 and x20 */emit(A64_LDR64I(A64_R(19),A64_SP,regs_off),ctx);emit(A64_LDR64I(A64_R(20),A64_SP,regs_off+8),ctx);
From: Josef Bacik <josef@toxicpanda.com> Date: 2026-09-10 18:51:16
The jump-optimized kprobe template calls optimized_callback() with
preemption still enabled for its first few instructions, so bracket the
call with an increment/decrement of current->rcu_tramp_nesting. The
template lives in .rodata and is memcpy()d into each optinsn slot without
relocation processing, so the per-CPU reference to current_task must be
an absolute %gs: address (R_X86_64_32S, relocated for KASLR like any
other) rather than %rip-relative. %rax has already been saved by
SAVE_REGS_STRING and is dead after the call.
The slot itself is dynamically allocated text, so the instructions before
the increment and after the decrement are covered by the irq-exit IP
check. 64-bit only; 32-bit x86 does not take part.
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
arch/x86/kernel/kprobes/opt.c | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)
From: Josef Bacik <josef@toxicpanda.com> Date: 2026-09-10 18:51:17
rcutorture's tasks flavor models a Tasks RCU reader as "any stretch of
kernel code", and rcu_read_delay() deliberately preempts inside it to
check that a preemption does not end the read-side critical section.
Once preemption outside a trampoline becomes a quiescent state that
model no longer matches what Tasks RCU protects, and the readers would
report false too-short grace periods.
Have tasks_torture_read_lock()/unlock() raise and drop
current->rcu_tramp_nesting so the reader models a trampoline, which is
the thing Tasks RCU actually guards; the deliberate preemption inside it
then continues to be, correctly, not a quiescent state.
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
kernel/rcu/rcutorture.c | 6 ++++++
1 file changed, 6 insertions(+)
From: Josef Bacik <josef@toxicpanda.com> Date: 2026-09-10 18:51:17
Follow the register_ftrace_direct() contract in the sample modules: on
x86-64 and arm64, have each hand-written trampoline increment
current->rcu_tramp_nesting before calling its C handler and decrement it
before returning, via a small shared samples/ftrace/ftrace-direct.h.
%r11 and x12/w13 are used as scratch; both are caller-saved, non-argument
registers and therefore dead on entry to and exit from an fentry
trampoline.
The header pulls in the generated asm-offsets.h only on those two
architectures, since it is not generally safe to include from C (PPC32's
TASK_SIZE and arm64's TRAMP_VALIAS clash with the C definitions; the
latter is worked around locally with push_macro/pop_macro). Other
architectures get empty macros and are unchanged.
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
samples/ftrace/ftrace-direct-modify.c | 9 ++++
samples/ftrace/ftrace-direct-multi-modify.c | 9 ++++
samples/ftrace/ftrace-direct-multi.c | 5 +++
samples/ftrace/ftrace-direct-too.c | 5 +++
samples/ftrace/ftrace-direct.c | 5 +++
samples/ftrace/ftrace-direct.h | 64 +++++++++++++++++++++++++++++
6 files changed, 97 insertions(+)
From: Josef Bacik <josef@toxicpanda.com> Date: 2026-09-10 18:51:18
Bracket the call out to the ftrace_ops callback in ftrace_caller with an
increment/decrement of current->rcu_tramp_nesting, using x12/w13 which
are scratch there. The read-modify-write is not atomic, but only current
modifies the count and every interrupting user is balanced, so nothing is
lost.
ftrace_caller itself, including the early CALL_OPS direct path and the
late direct tail that carry a BPF trampoline address in x17 with the
count at zero, is static kernel text: add an ftrace_static_tramp_end
marker after ftrace_stub_direct_tramp and provide
arch_rcu_tasks_ip_in_trampoline() covering
[ftrace_caller, ftrace_static_tramp_end) so the irq-exit check treats a
task interrupted anywhere in it as inside a trampoline.
The hook is built only under CONFIG_RCU_TASKS_PREEMPT_QS, which arm64
does not select until a later patch.
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
arch/arm64/kernel/asm-offsets.c | 3 +++
arch/arm64/kernel/entry-ftrace.S | 35 +++++++++++++++++++++++++++++++++++
arch/arm64/kernel/ftrace.c | 16 ++++++++++++++++
3 files changed, 54 insertions(+)
From: Josef Bacik <josef@toxicpanda.com> Date: 2026-09-10 18:51:20
Tasks RCU only accepts a voluntary context switch, usermode or idle as a
quiescent state, because a task that was preempted may be sitting in a
trampoline whose text is about to be freed. On PREEMPT_LAZY kernels,
where cond_resched() is a no-op and CPU-bound kernel threads only ever
lose the CPU through preemption, that means any long-running kthread or
kworker stalls every synchronize_rcu_tasks() caller -- BPF and LSM
program detach and DYNAMIC ftrace_ops teardown via ftrace_shutdown(),
and kprobe (un)registration via the jump optimizer, which waits under
kprobe_mutex, text_mutex and cpus_read_lock() -- for its entire run,
unless someone sprinkles cond_resched_tasks_rcu_qs() into it. A cgroup
writeback worker draining a large cgwb for eleven minutes was enough to
back 40+ tasks up behind trampoline_mutex and trip the hung-task panic.
With the previous patches, every Tasks-RCU-protected trampoline on
x86-64 and arm64 (ftrace_caller and its dynamic copies, BPF trampoline
images, the optprobe template, out-of-line direct trampolines) holds
current->rcu_tramp_nesting across its call-out, and the irq-exit
preemption path holds it across preempt_schedule_irq() whenever the
interrupted IP is somewhere the counter cannot cover: trampoline
entry/exit instructions and other dynamically allocated text, the static
ftrace stubs and x86 return thunks on the way into a direct-call target,
modules hosting their own direct trampolines, and the kprobe
jump-optimization window. A task that is context-switched with the
count at zero therefore cannot be inside, called from, or about to
resume into anything Tasks RCU protects.
So let rcu_tasks_classic_qs() clear the holdout flag on a preemption
too when rcu_tramp_nesting is zero, on architectures that select
ARCH_HAS_RCU_TASKS_PREEMPT_QS, and select it for x86-64 and for arm64
with DYNAMIC_FTRACE_WITH_ARGS. A running holdout is already poked via
rcu_request_urgent_qs_task(), which makes the next tick set
NEED_RESCHED; the resulting preemption -- from irq exit, or synchronously
at the next preempt_enable() -- now retires it, so a Tasks RCU grace
period is bounded by roughly a tick plus the longest preempt-disabled
section instead of by the longest stretch without a voluntary schedule().
Other architectures keep the voluntary-only rule. Update the Tasks RCU
documentation comments and the FORCE_TASKS_RCU help text to match.
Cost: one load of current plus an inc/dec per trampoline entry and exit,
and on irq-exit preemption one core_kernel_text() check plus, with
OPTPROBES, MAX_OPTIMIZED_LENGTH-1 lockless kprobe hash lookups.
Not covered: x86-32 and the other GENERIC_IRQ_ENTRY architectures, and
return_to_handler / the rethook trampoline, whose C callees take the
ftrace recursion lock before touching any ops.
Tested under QEMU (x86-64, PREEMPT_LAZY, PREEMPT_RCU=n, PROVE_RCU, with
and without PREEMPT_DYNAMIC) against a kthread spinning in-kernel for
30s with the function tracer, an ftrace kprobe, an optimized kprobe and
fentry/fexit programs live: synchronize_rcu_tasks() 29.7s -> 0.1-0.3s,
ftrace_shutdown() of a DYNAMIC ops 27s -> 0.2-0.8s, the ftrace-direct
sample modules load/fire/unload in ~2.5s each during the spin, no
warnings. arm64 is build-tested only.
Assisted-by: LLM
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
arch/arm64/Kconfig | 1 +
arch/x86/Kconfig | 1 +
include/linux/rcupdate.h | 14 +++++++++++++-
kernel/rcu/Kconfig | 7 ++++---
kernel/rcu/tasks.h | 15 +++++++++++----
5 files changed, 30 insertions(+), 8 deletions(-)
@@ -905,7 +905,10 @@ static void rcu_tasks_wait_gp(struct rcu_tasks *rtp)//// Simple variant of RCU whose quiescent states are voluntary context// switch, cond_resched_tasks_rcu_qs(), user-space execution, and idle.-// As such, grace periods can take one good long time. There are no+// With CONFIG_RCU_TASKS_PREEMPT_QS, a preemption taken while the task is+// not inside a trampoline (current->rcu_tramp_nesting == 0, see+// rcu_tasks_trampoline_enter()) is a quiescent state as well; without it,+// grace periods can take one good long time. There are no// read-side primitives similar to rcu_read_lock() and rcu_read_unlock()// because this implementation is intended to get the system into a safe// state for some of the manipulations involved in tracing and the like.
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Build failure due to missing `ftrace_static_tramp_end` symbol when `CONFIG_DYNAMIC_FTRACE_WITH_ARGS` is disabled.
--
commit 459a25e6b720c08167da420eb83180b2856610e3
Author: Josef Bacik [off-list ref]
arm64: ftrace: Maintain Tasks RCU trampoline nesting in ftrace_caller
Bracket the call out to the ftrace_ops callback in ftrace_caller with an
increment/decrement of current->rcu_tramp_nesting, using x12/w13 which
are scratch there. The read-modify-write is not atomic, but only current
modifies the count and every interrupting user is balanced, so nothing is
lost.
@@ -178,6 +209,10 @@ SYM_CODE_START(ftrace_stub_direct_tramp) SYM_CODE_END(ftrace_stub_direct_tramp) #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */+/* End of [ftrace_caller, ...) for arch_rcu_tasks_ip_in_trampoline(). */+SYM_CODE_START(ftrace_static_tramp_end)+SYM_CODE_END(ftrace_static_tramp_end)+ #else /* CONFIG_DYNAMIC_FTRACE_WITH_ARGS */ /*
[Severity: Low]
Will this cause a linker error when compiling with
CONFIG_RCU_TASKS_PREEMPT_QS=y but CONFIG_DYNAMIC_FTRACE_WITH_ARGS=n?
The ftrace_static_tramp_end label is defined here exclusively inside the
CONFIG_DYNAMIC_FTRACE_WITH_ARGS block. The legacy mcount-based ftrace path
in the #else block lacks this definition.
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Massive performance overhead in the kernel preemption fast-path due to unbounded hash table lookups.
- [High] kprobe_optimizer() frees kprobes without synchronize_rcu(), causing a Use-After-Free when get_kprobe() is called from irqentry_preempt() on idle CPUs.
--
commit c6c38cd5a975e6f9c2037bf63fa6d732d6cd1fab
Author: Josef Bacik [off-list ref]
kprobes: Let Tasks RCU recognise tasks preempted in an optprobe jump window
This patch adds kprobe_in_optimized_region() to determine if an instruction
pointer falls within the bytes of an optimized kprobe. It then updates
rcu_tasks_ip_in_trampoline() to use this check, allowing Tasks RCU to
recognize and hold off grace periods for tasks preempted within these jump
windows.
+bool kprobe_in_optimized_region(unsigned long addr)
+{
+ int i;
+
+ for (i = 1; i < MAX_OPTIMIZED_LENGTH / sizeof(kprobe_opcode_t); i++)
+ if (get_kprobe((kprobe_opcode_t *)addr - i))
[Severity: High]
Could this loop introduce significant overhead in the preemption fast-path?
On architectures like x86, this loop executes up to 18 times, querying the
kprobe hash table via get_kprobe() on each iteration. Since the hash table
can have long chains on systems with many active kprobes, this could result
in thousands of pointer dereferences during a preemption event.
Also, does this introduce a use-after-free risk for interrupted idle tasks?
kprobe_optimizer() unlinks kprobes and frees them after waiting only for
synchronize_rcu_tasks(). Because synchronize_rcu_tasks() explicitly ignores
idle tasks, an idle CPU that is interrupted could end up traversing the
kprobe_table here via get_kprobe() while the kprobe is concurrently freed,
as Tasks RCU will not wait for the idle task's traversal to finish.
bool rcu_tasks_ip_in_trampoline(unsigned long ip)
{
+ if (kprobe_in_optimized_region(ip))
[Severity: High]
Is it safe to place this expensive check unconditionally at the start of this
function?
When CONFIG_RCU_TASKS_PREEMPT_QS is enabled, rcu_tasks_ip_in_trampoline()
is called on every kernel preemption via irqentry_preempt(). By placing
kprobe_in_optimized_region() before cheaper checks like
core_kernel_text(ip), every preemption event incurs the cost of the loop
and hash table lookups, which could severely degrade system performance.
+ return true;
+
if (core_kernel_text(ip))
return arch_rcu_tasks_ip_in_trampoline(ip);
return !is_module_text_address(ip);
From: Steven Rostedt <rostedt@goodmis.org> Date: 2026-09-10 19:43:50
On Thu, 10 Sep 2026 18:50:23 +0000
Josef Bacik [off-list ref] wrote:
Tasks RCU only treats a voluntary context switch, usermode or idle as a
quiescent state, because a preempted task may be sitting in a trampoline
that is about to be freed. That was a fine trade when PREEMPT_NONE
servers compiled Tasks RCU away and PREEMPT desktops rarely ran
long-lived in-kernel loops. PREEMPT_LAZY changes both halves at once:
Tasks RCU is now real on server configs, and cond_resched() is a no-op,
so a CPU-bound kthread or kworker only ever loses the CPU by being
preempted, which is exactly the event Tasks RCU refuses to count.
The way this showed up for us was a cgroup writeback worker draining a
very large cgwb for around eleven minutes on an arm64 box. Nothing wrong
So you have a kernel thread running for 11 minutes without a schedule?
You could still put in a cond_resched_tasks_rcu_qs() in that loop. But I
guess you are trying to get rid of doing that too.
with that on its own, but a BPF program detach on another CPU went
bpf_trampoline_update() -> ftrace_shutdown() -> synchronize_rcu_tasks()
while holding trampoline_mutex, forty-odd tasks piled up behind the
mutex, and the hung task detector panicked the machine. The kprobe jump
optimizer is worse in principle: it does synchronize_rcu_tasks() under
kprobe_mutex, text_mutex and cpus_read_lock(), so one long-running
kthread can stall static key updates and CPU hotplug for its whole run.
The current answer is to find each such loop and add
cond_resched_tasks_rcu_qs() to it, which is the kind of annotation
PREEMPT_LAZY was supposed to let us stop writing.
This series tries the other direction: have the trampolines say when a
task is inside them, so that a preemption anywhere else can be a
quiescent state.
- task_struct grows an int, rcu_tramp_nesting. Every trampoline whose
lifetime Tasks RCU guards increments it before calling out and
decrements it before returning: ftrace_caller and its dynamic copies,
the BPF trampoline (which drops it again around the call to the
original function, since im->pcref covers that), the x86 optprobe
template, and out-of-line register_ftrace_direct() trampolines. Only
current writes it and nested users are balanced, so it is a plain
non-atomic inc/dec, one load of current plus one RMW per entry/exit.
- The inc/dec are inside the trampoline, so there is a window of a few
instructions on each side where the count is zero but the task is in
(or on its way into) trampoline text. Nothing there can be preempted
synchronously, only from an interrupt, so the irq-exit preemption path
looks at regs->ip and holds the count across preempt_schedule_irq()
when the IP is somewhere the counter cannot cover: outside core and
module text (all the dynamically allocated trampolines and slots), in
the static ftrace stubs or the x86 return thunks that still hold a
direct-call target, in a module that hosts its own direct trampoline,
or inside the bytes after a kprobe that the jump optimizer may be
about to rewrite (the one synchronize_rcu_tasks() user that is not
about trampolines at all).
So basically if the preemption happens outside of core or module text
(which should be the case of any dynamically allocated trampoline), the
task is marked to be in the grace period across its schedule, so that the
RCU_TASK cannot move forward?
- With those in place, rcu_tasks_classic_qs() also clears the holdout
flag on a preemption when the count is zero, on architectures that
opt in. x86-64 and arm64 do so here. Everyone else keeps the
voluntary-only rule and is untouched apart from the (unused) field.
A running holdout already gets poked via rcu_request_urgent_qs_task(),
which makes the next tick set NEED_RESCHED, so with this the resulting
preemption retires it and a Tasks RCU grace period is bounded by roughly
a tick plus the longest preempt-off section rather than by the longest
stretch without a voluntary schedule().
Patches 1-12 are scaffolding and change no behaviour on their own; patch
13 flips the rule and selects the option for the two architectures.
Testing so far is QEMU only: x86-64, PREEMPT_LAZY with PREEMPT_RCU=n,
PROVE_RCU and lockdep, with and without PREEMPT_DYNAMIC. A kthread
spinning in-kernel for 30s with the function tracer, an ftrace kprobe,
an optimized kprobe and fentry/fexit programs attached:
synchronize_rcu_tasks() goes from 29.7s to 0.1-0.3s, tearing down a
DYNAMIC ftrace_ops (tracefs instance function -> nop) from 27s to
0.2-0.8s, and the ftrace-direct sample modules load, fire and unload in
about 2.5s each while the spinner runs, with no warnings and the new
return-to-user assertion quiet. arm64 is build-tested only at this
point; real hardware numbers for both are the obvious next step and I
did not want to sit on the idea waiting for them.
Things I would particularly like opinions on:
- Whether hooking rcu_tasks_classic_qs() is the right place, or whether
Paul would rather see this expressed differently inside Tasks RCU.
- return_to_handler and the rethook/kretprobe trampolines are not
instrumented. Their C callees take the ftrace recursion lock before
touching any ops and the trampolines themselves are static text, so I
believe they do not need it, but I would like Steven and Masami to
confirm.
Note, there has been some work in the past (and may happen again in the
future) that will remove the preempt_disable() from the trace_recursion
locking. If that happens, then I believe the trace_recursion would need to
increment (and decrement) your counter. Probably need a comment there to
let whomever know about it if they decide to remove the preempt_disable().
- The register_ftrace_direct() contract change: out-of-line direct
trampolines now have to maintain the count themselves (the samples
are converted). I do not know of out-of-tree users beyond BPF, but
this is the one place an existing user could be silently weakened.
- Whether arm64 folks are comfortable with the ldr/add/str in
ftrace_caller and the BPF trampoline, and with treating all of
ftrace_caller as trampoline text for the IP check.
- If this holds up, cond_resched_tasks_rcu_qs() and
rcu_softirq_qs_periodic() become unnecessary on the opted-in
architectures; I have not touched them here.
I don't know. It may work, but I have a feeling there's a devil in the
details here that is waiting to bite us in the underside when we are not
(RCU) watching.
-- Steve
From: Josef Bacik <josef@toxicpanda.com> Date: 2026-09-10 22:46:36
On Thu, 10 Sep 2026 19:16:33 +0000, sashiko-bot@kernel.org wrote:
quoted
+bool kprobe_in_optimized_region(unsigned long addr)
+{
+ int i;
+
+ for (i = 1; i < MAX_OPTIMIZED_LENGTH / sizeof(kprobe_opcode_t); i++)
+ if (get_kprobe((kprobe_opcode_t *)addr - i))
[Severity: High]
Could this loop introduce significant overhead in the preemption fast-path?
On architectures like x86, this loop executes up to 18 times, querying the
kprobe hash table via get_kprobe() on each iteration. Since the hash table
can have long chains on systems with many active kprobes, this could result
in thousands of pointer dereferences during a preemption event.
This one is fair. It is not the scheduler fast path in general, only the
irq-exit preemption path (need_resched set on return to kernel with
preempt_count() == 0), but 18 hash lookups there is still more than it
needs to be. For v2 the walk only runs while kprobe_optimizer() is
actually sitting in its synchronize_rcu_tasks() -- a flag set and cleared
around that call -- and rcu_tasks_ip_in_trampoline() only asks for core
and module text. A preemption that does not see the flag predates the
grace period; the task is then just an ordinary preempted holdout and the
jump is not written until it has run again and left the window, so
skipping the walk there is safe. Common-case cost becomes one load.
Also, does this introduce a use-after-free risk for interrupted idle tasks?
kprobe_optimizer() unlinks kprobes and frees them after waiting only for
synchronize_rcu_tasks(). Because synchronize_rcu_tasks() explicitly ignores
idle tasks, an idle CPU that is interrupted could end up traversing the
kprobe_table here via get_kprobe() while the kprobe is concurrently freed,
as Tasks RCU will not wait for the idle task's traversal to finish.
This one is not right, for two independent reasons:
- kprobe_table is an RCU hlist and nothing frees a kprobe on the
strength of Tasks RCU alone. Every unregistration path does
hlist_del_rcu() and then synchronize_rcu() before the object goes
away, and the optimizer's own free step runs after a Tasks RCU grace
period, which begins and ends with synchronize_rcu(). The caller here
runs with interrupts disabled, which is a normal RCU read-side section,
so the walk cannot outlive the object regardless of what Tasks RCU
thinks of the task.
- The idle case cannot reach this code. irqentry_exit_to_kernel_mode_preempt()
returns early when state.exit_rcu is set, i.e. when the interrupt was
taken with RCU not watching, so the irq-exit preemption path (and this
check with it) only ever runs with RCU watching. And the idle task is
not preempted through this path in the first place.
So: overhead finding valid and addressed in v2, UAF finding invalid. The
commit message in v2 spells out the RCU-safety argument so the next
reader does not have to reconstruct it.
Thanks,
Josef
From: Josef Bacik <josef@toxicpanda.com> Date: 2026-09-10 22:46:42
On Thu, 10 Sep 2026 19:04:36 +0000, sashiko-bot@kernel.org wrote:
quoted
+/* End of [ftrace_caller, ...) for arch_rcu_tasks_ip_in_trampoline(). */
+SYM_CODE_START(ftrace_static_tramp_end)
+SYM_CODE_END(ftrace_static_tramp_end)
+
#else /* CONFIG_DYNAMIC_FTRACE_WITH_ARGS */
[Severity: Low]
Will this cause a linker error when compiling with
CONFIG_RCU_TASKS_PREEMPT_QS=y but CONFIG_DYNAMIC_FTRACE_WITH_ARGS=n?
No, that configuration cannot exist. CONFIG_RCU_TASKS_PREEMPT_QS is
def_bool on ARCH_HAS_RCU_TASKS_PREEMPT_QS, and the only place arm64 sets
that is in patch 13:
select ARCH_HAS_RCU_TASKS_PREEMPT_QS if DYNAMIC_FTRACE_WITH_ARGS
so on arm64 RCU_TASKS_PREEMPT_QS=y implies DYNAMIC_FTRACE_WITH_ARGS=y and
both the marker in entry-ftrace.S and the arch_rcu_tasks_ip_in_trampoline()
that references it are built together or not at all. At this point in the
series (patch 9) nothing selects the option yet, so the reference is not
built either.
For the tool: the two symbols are tied by a Kconfig dependency introduced
later in the same series; checking the select conditions across the
series would have avoided this one. No change for v2 beyond a note in the
changelog.
Thanks,
Josef
From: "Paul E. McKenney" <paulmck@kernel.org> Date: 2026-09-10 22:59:57
On Thu, Sep 10, 2026 at 03:44:50PM -0400, Steven Rostedt wrote:
On Thu, 10 Sep 2026 18:50:23 +0000
Josef Bacik [off-list ref] wrote:
quoted
Tasks RCU only treats a voluntary context switch, usermode or idle as a
quiescent state, because a preempted task may be sitting in a trampoline
that is about to be freed. That was a fine trade when PREEMPT_NONE
servers compiled Tasks RCU away and PREEMPT desktops rarely ran
long-lived in-kernel loops. PREEMPT_LAZY changes both halves at once:
Tasks RCU is now real on server configs, and cond_resched() is a no-op,
so a CPU-bound kthread or kworker only ever loses the CPU by being
preempted, which is exactly the event Tasks RCU refuses to count.
The way this showed up for us was a cgroup writeback worker draining a
very large cgwb for around eleven minutes on an arm64 box. Nothing wrong
So you have a kernel thread running for 11 minutes without a schedule?
We have seen this from time to time here as well.
You could still put in a cond_resched_tasks_rcu_qs() in that loop. But I
guess you are trying to get rid of doing that too.
And we have done this a few times, but if this proves to be an acceptable
alternative, that would be wonderful. ;-)
quoted
with that on its own, but a BPF program detach on another CPU went
bpf_trampoline_update() -> ftrace_shutdown() -> synchronize_rcu_tasks()
while holding trampoline_mutex, forty-odd tasks piled up behind the
mutex, and the hung task detector panicked the machine. The kprobe jump
optimizer is worse in principle: it does synchronize_rcu_tasks() under
kprobe_mutex, text_mutex and cpus_read_lock(), so one long-running
kthread can stall static key updates and CPU hotplug for its whole run.
The current answer is to find each such loop and add
cond_resched_tasks_rcu_qs() to it, which is the kind of annotation
PREEMPT_LAZY was supposed to let us stop writing.
This series tries the other direction: have the trampolines say when a
task is inside them, so that a preemption anywhere else can be a
quiescent state.
- task_struct grows an int, rcu_tramp_nesting. Every trampoline whose
lifetime Tasks RCU guards increments it before calling out and
decrements it before returning: ftrace_caller and its dynamic copies,
the BPF trampoline (which drops it again around the call to the
original function, since im->pcref covers that), the x86 optprobe
template, and out-of-line register_ftrace_direct() trampolines. Only
current writes it and nested users are balanced, so it is a plain
non-atomic inc/dec, one load of current plus one RMW per entry/exit.
- The inc/dec are inside the trampoline, so there is a window of a few
instructions on each side where the count is zero but the task is in
(or on its way into) trampoline text. Nothing there can be preempted
synchronously, only from an interrupt, so the irq-exit preemption path
looks at regs->ip and holds the count across preempt_schedule_irq()
when the IP is somewhere the counter cannot cover: outside core and
module text (all the dynamically allocated trampolines and slots), in
the static ftrace stubs or the x86 return thunks that still hold a
direct-call target, in a module that hosts its own direct trampoline,
or inside the bytes after a kprobe that the jump optimizer may be
about to rewrite (the one synchronize_rcu_tasks() user that is not
about trampolines at all).
So basically if the preemption happens outside of core or module text
(which should be the case of any dynamically allocated trampoline), the
task is marked to be in the grace period across its schedule, so that the
RCU_TASK cannot move forward?
If I understand correctly, the difference with Josef's patch is
that rcu_tasks_classic_qs(current, true), when called without the
direct or indirect aid of a trampoline, will provide an RCU Tasks
quiescent state. In contrast, without Josef's patch, no call to
rcu_tasks_classic_qs(current, true) will provide such a quiescent state.
More to the point, because rcu_tasks_classic_qs(current, false) is
invoked from rcu_note_context_switch(), any preemption to kernel code not
within or called from a trampoline will now provide a quiescent state.
Keeping in mind that cond_resched() is treated as a preemption, this
change could potentially greatly reduce the need for sprinkling calls
to cond_resched_tasks_rcu_qs() throughout the kernel. (Except that
the call to cond_resched() has to actually invoke the scheduler for
anything to happen.)
Which, if it works, would of course be a good thing. ;-)
quoted
- With those in place, rcu_tasks_classic_qs() also clears the holdout
flag on a preemption when the count is zero, on architectures that
opt in. x86-64 and arm64 do so here. Everyone else keeps the
voluntary-only rule and is untouched apart from the (unused) field.
A running holdout already gets poked via rcu_request_urgent_qs_task(),
which makes the next tick set NEED_RESCHED, so with this the resulting
preemption retires it and a Tasks RCU grace period is bounded by roughly
a tick plus the longest preempt-off section rather than by the longest
stretch without a voluntary schedule().
Patches 1-12 are scaffolding and change no behaviour on their own; patch
13 flips the rule and selects the option for the two architectures.
Testing so far is QEMU only: x86-64, PREEMPT_LAZY with PREEMPT_RCU=n,
PROVE_RCU and lockdep, with and without PREEMPT_DYNAMIC. A kthread
spinning in-kernel for 30s with the function tracer, an ftrace kprobe,
an optimized kprobe and fentry/fexit programs attached:
synchronize_rcu_tasks() goes from 29.7s to 0.1-0.3s, tearing down a
DYNAMIC ftrace_ops (tracefs instance function -> nop) from 27s to
0.2-0.8s, and the ftrace-direct sample modules load, fire and unload in
about 2.5s each while the spinner runs, with no warnings and the new
return-to-user assertion quiet. arm64 is build-tested only at this
point; real hardware numbers for both are the obvious next step and I
did not want to sit on the idea waiting for them.
Things I would particularly like opinions on:
- Whether hooking rcu_tasks_classic_qs() is the right place, or whether
Paul would rather see this expressed differently inside Tasks RCU.
We might well need more:
o The rcu_tasks_pertask() might need to check to see if task "t"
is in a quiescent state. The task_call_func() function might
be helpful in safely accessing that task's state remotely in
the common case where the task is blocked or preempted.
o Given a task that runs for a very long time on a CPU that has
nothing else to do (so that cond_resched() does nothing and
there are no preemptions), and does so in a code path that never
invokes cond_resched_tasks_rcu_qs(), it might be necessary to IPI
to CPU that this task is running on. Or to invoke resched_cpu()
in order to force a context switch on that CPU, whether it needs
one or not. (Which makes the scheduler do the IPI for us.)
o PREEMPT_RT kernels might want memory ordering on the nesting count
increments and decrements, along with READ_ONCE() and WRITE_ONCE()
or similar, in order to avoid the aforementioned IPIs. But this
increases overhead, so !PREEMPT_RT kernels would *not* want this.
And probably other things that I am not yet seeing. ;-)
quoted
- return_to_handler and the rethook/kretprobe trampolines are not
instrumented. Their C callees take the ftrace recursion lock before
touching any ops and the trampolines themselves are static text, so I
believe they do not need it, but I would like Steven and Masami to
confirm.
Note, there has been some work in the past (and may happen again in the
future) that will remove the preempt_disable() from the trace_recursion
locking. If that happens, then I believe the trace_recursion would need to
increment (and decrement) your counter. Probably need a comment there to
let whomever know about it if they decide to remove the preempt_disable().
quoted
- The register_ftrace_direct() contract change: out-of-line direct
trampolines now have to maintain the count themselves (the samples
are converted). I do not know of out-of-tree users beyond BPF, but
this is the one place an existing user could be silently weakened.
- Whether arm64 folks are comfortable with the ldr/add/str in
ftrace_caller and the BPF trampoline, and with treating all of
ftrace_caller as trampoline text for the IP check.
- If this holds up, cond_resched_tasks_rcu_qs() and
rcu_softirq_qs_periodic() become unnecessary on the opted-in
architectures; I have not touched them here.
I don't know. It may work, but I have a feeling there's a devil in the
details here that is waiting to bite us in the underside when we are not
(RCU) watching.