Re: [PATCH 1/4] tracing/osnoise: Per-cpu mutex and fd detachment

st 26. 8. 2026 v 23:21 odesílatel Crystal Wood [off-list ref] napsal:

On Wed, 2026-08-26 at 15:31 -0400, Steven Rostedt wrote:

quoted

On Mon, 24 Aug 2026 21:30:37 +0000
sashiko-bot@kernel.org wrote:

quoted

[Severity: High]
Can holding osn_var->lock while calling kthread_stop() cause a deadlock?

kthread_stop() blocks waiting for the target kthread to exit. However, if
the kthread is currently handling a migration request in
osnoise_migration_pending() (shown slightly earlier in the diff), it will
attempt to acquire this same osn_var->lock before it can clear its state and
exit:

This code has a lot of nastiness in the locking with regard to the thread.
I haven't looked deeply at your patch, but the fact that Sashiko pointed
out a possible deadlock I think you may be hitting the mess I was hitting.

Have you looked at this thread: https://lore.kernel.org/all/20240820130001.124768-1-tglozar@redhat.com/ (local)

That thread is about user fd sync, which is what this patch is trying to
address.

This particular deadlock is with kernel threads, in a corner case of
getting migrated even though we try to pin them to one cpu.  We can get
rid of this migration code and just have stop_kthread() take care of it,
adding a get/put_task_struct() so that it's OK for the thread to die
early.  This way we can also handle any other abnormal thread exits.

I'll respond to the rest of the Sashiko comments soon.

quoted

It has some tests I ran along with tracing that detected issues. Of course,
everything needs to be run with lockdep enabled too.

I did run with lockdep (and some custom hacky state tracking) but
forcing migration is a hole in my test coverage.

Is there a particular test you're suggesting?

-Crystal

The race window is practically really small. I had success of
reproducing the migration deadlock in a virtual machine with a bash
script [1] that deliberately slows down the kernel/osnoise tracer with
perf, ftrace, and bpftrace hooks:

[root@cs9 tglozar]#
/home/tglozar/dev/linux/tools/tracing/rtla/osnoise_migration_race.sh
13 10000
cpu=13 mask=0,1,2,3,4,5,6,7,8,9,10,11,12 iters=10000 stop=nop
iter 1 ok (kpid 265080)
iter 2 ok (kpid 265086)
...
iter 422 ok (kpid 267814)
*** hang iter 423  kthread=267821  affiner=267823  stopper=267827
    cat /proc/267821/stack /proc/267827/stack
    do not rmdir instances/ or write current_tracer/online; reboot
[root@cs9 tglozar]# cat /proc/267821/stack; echo -------; cat /proc/267827/stack
[<0>] timerlat_main+0x266/0x3f0
[<0>] kthread+0xe6/0x120
[<0>] ret_from_fork+0x1be/0x250
[<0>] ret_from_fork_asm+0x1a/0x30
-------
[<0>] kthread_stop+0x6b/0x180
[<0>] stop_kthread+0x86/0xc0
[<0>] stop_per_cpu_kthreads+0xf/0x40
[<0>] osnoise_workload_stop.part.0+0xc/0xa0
[<0>] timerlat_tracer_reset+0x9/0x40
[<0>] tracing_set_tracer+0x14d/0x300
[<0>] tracing_set_trace_write+0x7f/0xe0
[<0>] vfs_write+0xf8/0x410
[<0>] ksys_write+0x65/0xe0
[<0>] do_syscall_64+0xc1/0x460
[<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e

[1] https://gitlab.com/-/snippets/6045758

Note that any path that is calling kthread_stop() can trigger the
deadlock (together with the migration), that is, both making the CPU
offline and stopping the osnoise tracer. Sashiko flagged the former,
my reproducer used the latter (the script can do both).



Tomas