Re: [Bug] soft lockup in syscall_exit_to_user_mode in Linux kernel v6.15-rc5

Hello Steven,

Thank you very much for your prompt reply and kind support. Based on
your analysis, I will disable KASAN and attempt to reproduce the crash
accordingly.

I will keep you updated with any progress. I truly appreciate your
time and assistance.

Best regards,
John

On Thu, May 22, 2025 at 1:30 AM Steven Rostedt [off-list ref] wrote:

On Thu, 22 May 2025 00:40:29 +0800
John [off-list ref] wrote:

quoted

Root Cause Analysis:
The root cause is unbounded recursion or excessive iteration in
lock_acquire() initiated via perf tracepoints that fire during slab
allocations and trace buffer updates. Specifically:
tracing_gen_ctx_irq_test() is invoked while tracing kernel contexts
(e.g., IRQ/softirq nesting).
This tracepoint triggers perf_trace_lock_acquire() and further invokes
lock_acquire() from lockdep.

tracing_gen_ctx_irq_test() is not a tracepoint. It's a simple routine to
find out how to fill the "common_flags" part of a trace event.

Here's the entire function:

unsigned int tracing_gen_ctx_irq_test(unsigned int irqs_status)
{
        unsigned int trace_flags = irqs_status;
        unsigned int pc;

        pc = preempt_count();

        if (pc & NMI_MASK)
                trace_flags |= TRACE_FLAG_NMI;
        if (pc & HARDIRQ_MASK)
                trace_flags |= TRACE_FLAG_HARDIRQ;
        if (in_serving_softirq())
                trace_flags |= TRACE_FLAG_SOFTIRQ;
        if (softirq_count() >> (SOFTIRQ_SHIFT + 1))
                trace_flags |= TRACE_FLAG_BH_OFF;

        if (tif_need_resched())
                trace_flags |= TRACE_FLAG_NEED_RESCHED;
        if (test_preempt_need_resched())
                trace_flags |= TRACE_FLAG_PREEMPT_RESCHED;
        if (IS_ENABLED(CONFIG_ARCH_HAS_PREEMPT_LAZY) && tif_test_bit(TIF_NEED_RESCHED_LAZY))
                trace_flags |= TRACE_FLAG_NEED_RESCHED_LAZY;
        return (trace_flags << 16) | (min_t(unsigned int, pc & 0xff, 0xf)) |
                (min_t(unsigned int, migration_disable_value(), 0xf)) << 4;
}

The functions it calls are:

static __always_inline int preempt_count(void)
{
        return raw_cpu_read_4(__preempt_count) & ~PREEMPT_NEED_RESCHED;
}

# define softirq_count()        (preempt_count() & SOFTIRQ_MASK)
#define in_serving_softirq()    (softirq_count() & SOFTIRQ_OFFSET)

static __always_inline bool tif_need_resched(void)
{
        return tif_test_bit(TIF_NEED_RESCHED);
}

static __always_inline bool test_preempt_need_resched(void)
{
        return !(raw_cpu_read_4(__preempt_count) & PREEMPT_NEED_RESCHED);
}

static unsigned short migration_disable_value(void)
{
#if defined(CONFIG_SMP)
        return current->migration_disabled;
#else
        return 0;
#endif
}

Nothing there should cause any recursion or issue. It's basically testing
various states and then returns a flags value.

It does not call lock_acquire().

quoted

Inside lock_acquire(), the kernel attempts to inspect instruction
addresses via __kernel_text_address(), which cascades into
unwind_get_return_address() and stack_trace_save().
However, these introspection functions are not expected to run in
real-time-sensitive softirq context and they do not contain preemption
or rescheduling points. With sufficient recursion or stress (e.g.,
slab allocation with tracepoints and lockdep active), CPU#0 gets
trapped and triggers the watchdog.

At present, I have not yet obtained a minimal reproducer for this
issue. However, I am actively working on reproducing it, and I will
promptly share any additional findings or a working reproducer as soon
as it becomes available.

Thank you very much for your time and attention to this matter. I
truly appreciate the efforts of the Linux kernel community.

Looking at the backtrace you have:

kernel_text_address+0x35/0xc0 kernel/extable.c:94
 __kernel_text_address+0xd/0x40 kernel/extable.c:79
 unwind_get_return_address arch/x86/kernel/unwind_orc.c:369 [inline]
 unwind_get_return_address+0x59/0xa0 arch/x86/kernel/unwind_orc.c:364
 arch_stack_walk+0x9c/0xf0 arch/x86/kernel/stacktrace.c:26
 stack_trace_save+0x8e/0xc0 kernel/stacktrace.c:122
 kasan_save_stack+0x24/0x50 mm/kasan/common.c:47
 kasan_save_track+0x14/0x30 mm/kasan/common.c:68
 unpoison_slab_object mm/kasan/common.c:319 [inline]
 __kasan_slab_alloc+0x59/0x70 mm/kasan/common.c:345
 kasan_slab_alloc include/linux/kasan.h:250 [inline]
 slab_post_alloc_hook mm/slub.c:4147 [inline]

KASAN is a very intrusive debugging utility that often causes soft lockups
and such when used with other debugging utilities.

If you can reproduce a softlockup without KASAN enabled, I'd then be more
worried about this. Usually when I trigger a softlockup and have KASAN
enabled, I just disable KASAN.

-- Steve