On 2025-02-04 03:51:48 [-0800], Paul E. McKenney wrote:

On Tue, Feb 04, 2025 at 11:26:11AM +0100, Sebastian Andrzej Siewior wrote:

quoted

On 2025-01-30 10:53:19 [-0800], Paul E. McKenney wrote:

quoted

The timer and hrtimer softirq processing has moved to dedicated threads
for kernels built with CONFIG_IRQ_FORCED_THREADING=y.  This results in
timers not expiring until later in early boot, which in turn causes the
RCU Tasks self-tests to hang in kernels built with CONFIG_PROVE_RCU=y,
which further causes the entire kernel to hang.  One fix would be to
make timers work during this time, but there are no known users of RCU
Tasks grace periods during that time, so no justification for the added
complexity.  Not yet, anyway.

This commit therefore moves the call to rcu_init_tasks_generic() from
kernel_init_freeable() to a core_initcall().  This works because the
timer and hrtimer kthreads are created at early_initcall() time.

Fixes: 49a17639508c3 ("softirq: Use a dedicated thread for timer wakeups on PREEMPT_RT.")
?

Quite possibly...  I freely confess that I was more focused on the fix
than on the bug's origin.  Would you be willing to try this commit and
its predecessor?

Yes. Just verified.
Tested-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Reviewed-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>

quoted

I played with it and I can reproduce the issue with !RT + threadirqs but
not with RT (which implies threadirqs).
Is there anything in RT that avoids the problem?

Not that I know of, but then again I did not try it.  To your point,

The change looks fine.

I do need to make a -rt rcutorture scenario.  TREE03 has been intended to
approximate this, and it uses the following Kconfig options:

------------------------------------------------------------------------

CONFIG_SMP=y
CONFIG_NR_CPUS=16
CONFIG_PREEMPT_NONE=n
CONFIG_PREEMPT_VOLUNTARY=n
CONFIG_PREEMPT=y
#CHECK#CONFIG_PREEMPT_RCU=y
CONFIG_HZ_PERIODIC=y
CONFIG_NO_HZ_IDLE=n
CONFIG_NO_HZ_FULL=n
CONFIG_RCU_TRACE=y
CONFIG_HOTPLUG_CPU=y
CONFIG_RCU_FANOUT=2
CONFIG_RCU_FANOUT_LEAF=2
CONFIG_RCU_NOCB_CPU=n
CONFIG_DEBUG_LOCK_ALLOC=n
CONFIG_RCU_BOOST=y
CONFIG_DEBUG_OBJECTS_RCU_HEAD=n
CONFIG_RCU_EXPERT=y

You could enable CONFIG_PREEMPT_RT ;)
CONFIG_PREEMPT_LAZY is probably also set a lot.

That should be it.

------------------------------------------------------------------------

And the following kernel-boot parameters:

------------------------------------------------------------------------

rcutorture.onoff_interval=200 rcutorture.onoff_holdoff=30
rcutree.gp_preinit_delay=12
rcutree.gp_init_delay=3
rcutree.gp_cleanup_delay=3
rcutree.kthread_prio=2
threadirqs
rcutree.use_softirq=0
rcutorture.preempt_duration=10

------------------------------------------------------------------------

Some of these are for RCU's benefit, but what should I change to more
closely approximate a typical real-time deployment?

See above.

quoted

Thank you for debugging and the patch. 

And to you for digging into it!

Thanks.

							Thanx, Paul

Sebastian