[RFC PATCH 09/12] arm64: Use the pre-counted hardirq lifecycle
flat view
COOLING7d
From: Karl Mehltretter <hidden>
Date: 2026-10-22 05:18:09
Also in:
linux-arm-kernel, linux-doc, linux-rt-devel, lkml, llvm
Subsystem:
arm64 port (aarch64 architecture), the rest · Maintainers:
Catalin Marinas, Will Deacon, Linus Torvalds
Ordinary EL0 and EL1 interrupt paths establish generic entry state before irq_enter_rcu() adds HARDIRQ_OFFSET and lockdep nesting. Callbacks in that early window are therefore not classified as hardirq work. Their exit paths also drop the classification before function-graph return callbacks and the kernel return tail have completed. After the EL1 pseudo-NMI decision, enter ordinary interrupts through irqentry_hardirq_enter(). Complete delayed timer and accounting work with irq_enter_rcu_preempt_counted(). Use the retained exit preparation followed immediately by raw count and lockdep teardown. For kernel returns, restore return-tail classification after masking DAIF and before MTE exit tracing. Keep the pseudo-NMI lifecycle unchanged. Force arm64 preempt_count() inline so noinstr callers cannot be outlined into instrumentable text. The noinstr path also depends on the raw preempt-count and __kern_my_cpu_offset() inlining changes carried as patches 1 and 2. Keep them separate because they are externally authored fixes. Link: https://lore.kernel.org/all/c3a90368dd6c096af9d918e036a27728696fb8a4.1786603168.git.hongyan.xia@transsion.com/ (local) Link: https://lore.kernel.org/r/20260918161407.2300-12-will@kernel.org (local) Assisted-by: LLM --- arch/arm64/include/asm/preempt.h | 2 +- arch/arm64/kernel/entry-common.c | 23 +++++++++++++++++------ 2 files changed, 18 insertions(+), 7 deletions(-)
diff --git a/arch/arm64/include/asm/preempt.h b/arch/arm64/include/asm/preempt.h
index 4f917596f387d..3d61be0f825fb 100644
--- a/arch/arm64/include/asm/preempt.h
+++ b/arch/arm64/include/asm/preempt.h@@ -7,7 +7,7 @@ #define PREEMPT_NEED_RESCHED BIT(32) #define PREEMPT_ENABLED (PREEMPT_NEED_RESCHED) -static inline int preempt_count(void) +static __always_inline int preempt_count(void) { return READ_ONCE(current_thread_info()->preempt.count); }
diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
index 93056a766fb20..fda807c61ff27 100644
--- a/arch/arm64/kernel/entry-common.c
+++ b/arch/arm64/kernel/entry-common.c@@ -56,6 +56,8 @@ static void noinstr __arm64_exit_to_kernel_mode(struct pt_regs *regs, irqentry_state_t state) { local_daif_mask(); + /* arm64 uses the split exit API, so enter the tail before MTE tracing. */ + irqentry_hardirq_tail_enter(state); mte_check_tfsr_exit(); irqentry_exit_to_kernel_mode_after_preempt(regs, state); }
@@ -523,12 +525,16 @@ static __always_inline void __el1_irq(struct pt_regs *regs, void (*handler)(struct pt_regs *)) { irqentry_state_t state; + bool do_sub; - state = arm64_enter_from_kernel_mode(regs); + state = irqentry_hardirq_enter(regs); + mte_check_tfsr_entry(); + mte_disable_tco_entry(current); - irq_enter_rcu(); + irq_enter_rcu_preempt_counted(); do_interrupt_handler(regs, handler); - irq_exit_rcu(); + do_sub = irq_exit_rcu_preempt_counted_prepare(); + irqentry_hardirq_exit_finish(do_sub); arm64_exit_to_kernel_mode_preempt(regs, state); }
@@ -839,16 +845,21 @@ asmlinkage void noinstr el0t_64_sync_handler(struct pt_regs *regs) static void noinstr el0_interrupt(struct pt_regs *regs, void (*handler)(struct pt_regs *)) { - arm64_enter_from_user_mode(regs); + bool do_sub; + + (void)irqentry_hardirq_enter(regs); + mte_disable_tco_entry(current); + sme_enter_from_user_mode(); write_sysreg(DAIF_PROCCTX_NOIRQ, daif); if (regs->pc & BIT(55)) arm64_apply_bp_hardening(); - irq_enter_rcu(); + irq_enter_rcu_preempt_counted(); do_interrupt_handler(regs, handler); - irq_exit_rcu(); + do_sub = irq_exit_rcu_preempt_counted_prepare(); + irqentry_hardirq_exit_finish(do_sub); arm64_exit_to_user_mode(regs); }
--
2.39.5 (Apple Git-154)