Re: [RFC PATCH v2 0/3] disable optimistic spinning for ftrace_lock

From
Yafang Shao <hidden>
Date
2026-03-11 13:41:10
Also in
lkml
Reply to
David Laight, 2026-03-11
On Wed, Mar 11, 2026 at 8:53 PM David Laight
[off-list ref] wrote:
On Wed, 11 Mar 2026 12:54:26 +0100
Peter Zijlstra [off-list ref] wrote:
quoted
On Wed, Mar 11, 2026 at 07:52:47PM +0800, Yafang Shao wrote:
quoted
Recently, we resolved a latency spike issue caused by concurrently running
bpftrace processes. The root cause was high contention on the ftrace_lock
due to optimistic spinning. We can optimize this by disabling optimistic
spinning for ftrace_lock.

While semaphores may present similar challenges, I'm not currently aware of
specific instances that exhibit this exact issue. Should we encounter
problematic semaphores in production workloads, we can address them at that
time.

PATCH #1: introduce slow_mutex_[un]lock to disable optimistic spinning
PATCH #2: add variant for rtmutex
PATCH #3: disable optimistic spinning for ftrace_lock
So I really utterly hate this.
Yep...
Adding the extra parameter is likely to have a measurable impact
on everything else.

The problematic path is obvious:        find_kallsyms_symbol+142
        module_address_lookup+104
        kallsyms_lookup_buildid+203
        kallsyms_lookup+20
        print_rec+64
        t_show+67
        seq_read_iter+709
        seq_read+165
        vfs_read+165
        ksys_read+103
        __x64_sys_read+25
        do_syscall_64+56
        entry_SYSCALL_64_after_hwframe+100

The code needs to drop the ftrace_lock across t_show.
It's unclear whether we can safely release ftrace_lock within t_show —
doing so would probably necessitate a major redesign of the current
implementation.
Although there is a bigger issue of why on earth the code is reading the
list of filter functions at all - never mind all the time.
bpftrace reads the complete list of available functions into
userspace, then performs matching against the target function to
determine if it is traceable.
I'll do it by hand when debugging, but I'd have though anything using bpf
will know exactly where to add its hooks.

-- 
Regards
Yafang
Thread (13 messages) 13 messages, 5 authors, last reply 2026-03-12
Keyboard shortcuts
hback out one level
jnext message in thread / next series-diff position
kprevious message in thread / previous series-diff position
ldrill in
Escclose help / fold thread tree
?toggle this help