On Wed, 11 Mar 2026 12:54:26 +0100
Peter Zijlstra [off-list ref] wrote:
On Wed, Mar 11, 2026 at 07:52:47PM +0800, Yafang Shao wrote:
quoted
Recently, we resolved a latency spike issue caused by concurrently running
bpftrace processes. The root cause was high contention on the ftrace_lock
due to optimistic spinning. We can optimize this by disabling optimistic
spinning for ftrace_lock.
While semaphores may present similar challenges, I'm not currently aware of
specific instances that exhibit this exact issue. Should we encounter
problematic semaphores in production workloads, we can address them at that
time.
PATCH #1: introduce slow_mutex_[un]lock to disable optimistic spinning
PATCH #2: add variant for rtmutex
PATCH #3: disable optimistic spinning for ftrace_lock
So I really utterly hate this.
Yep...
Adding the extra parameter is likely to have a measurable impact
on everything else.
The problematic path is obvious: find_kallsyms_symbol+142
module_address_lookup+104
kallsyms_lookup_buildid+203
kallsyms_lookup+20
print_rec+64
t_show+67
seq_read_iter+709
seq_read+165
vfs_read+165
ksys_read+103
__x64_sys_read+25
do_syscall_64+56
entry_SYSCALL_64_after_hwframe+100
The code needs to drop the ftrace_lock across t_show.
Although there is a bigger issue of why on earth the code is reading the
list of filter functions at all - never mind all the time.
I'll do it by hand when debugging, but I'd have though anything using bpf
will know exactly where to add its hooks.
David