Re: [PATCH 3/4] arm64: errata: Work around early CME DVMSync acknowledgement
From: Catalin Marinas <catalin.marinas@arm.com>
Date: 2026-03-06 12:19:28
Also in:
kvmarm
Subsystem:
arm64 port (aarch64 architecture), the rest · Maintainers:
Catalin Marinas, Will Deacon, Linus Torvalds
On Fri, Mar 06, 2026 at 12:00:30PM +0000, Catalin Marinas wrote:
On Thu, Mar 05, 2026 at 02:32:11PM +0000, Will Deacon wrote:quoted
On Mon, Mar 02, 2026 at 04:57:56PM +0000, Catalin Marinas wrote:quoted
+void sme_do_dvmsync(void) +{ + /* + * This is called from the TLB maintenance functions after the DSB ISH + * to send hardware DVMSync message. If this CPU sees the mask as + * empty, the remote CPU executing sme_set_active() would have seen + * the DVMSync and no IPI required. + */ + if (cpumask_empty(sme_active_cpus)) + return; + + preempt_disable(); + smp_call_function_many(sme_active_cpus, sme_dvmsync_ipi, NULL, true); + preempt_enable(); +}Why do we care about all CPUs using SME, rather than limiting it to the set of CPUs using SME with the mm we've invalidated? This looks like it will result in unnecessary cross-calls when multiple tasks are using SME (especially as the mm flag is only cleared on fork).Yes, it's a possibility but I traded it for simplicity. We also have the TTU case where we don't have an mm and we don't want to broadcast to all CPUs either, hence an sme_active_cpus mask. As I just replied on patch 2, for the TLB batching we wouldn't be able to use a cpumask in the batching structure since, per the ordering above, we need the DVMSync before checking if/where to send the IPI to. For the typical TLBI (not TTU), we can track a per-mm mask passed down to this function (I have patches doing this but it didn't make a significant difference in benchmarks).
Reusing the current mm_cpumask(), something like below. We could also scrap the MMCF_SME_DVMSYNC flag, though we end up always call sme_do_dvmsync() and checking the mask, probably more expensive than a flag check.
diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h
index e3ea0246a4f4..2c77ca41cb14 100644
--- a/arch/arm64/include/asm/tlbflush.h
+++ b/arch/arm64/include/asm/tlbflush.h@@ -81,7 +81,7 @@ static inline unsigned long get_trans_granule(void) } #ifdef CONFIG_ARM64_ERRATUM_SME_DVMSYNC -void sme_do_dvmsync(void); +void sme_do_dvmsync(struct mm_struct *mm); static inline void sme_dvmsync(struct mm_struct *mm) {
@@ -90,7 +90,7 @@ static inline void sme_dvmsync(struct mm_struct *mm) if (mm && !test_bit(ilog2(MMCF_SME_DVMSYNC), &mm->context.flags)) return; - sme_do_dvmsync(); + sme_do_dvmsync(mm); } #else static inline void sme_dvmsync(struct mm_struct *mm) { }
diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
index 90015fc29722..37e215cd0f39 100644
--- a/arch/arm64/kernel/fpsimd.c
+++ b/arch/arm64/kernel/fpsimd.c@@ -1378,6 +1378,7 @@ void sme_set_active(unsigned int cpu) if (!test_bit(ilog2(MMCF_SME_DVMSYNC), ¤t->mm->context.flags)) set_bit(ilog2(MMCF_SME_DVMSYNC), ¤t->mm->context.flags); + cpumask_set_cpu(cpu, mm_cpumask(current->mm)); cpumask_set_cpu(cpu, sme_active_cpus); /*
@@ -1398,6 +1399,7 @@ void sme_clear_active(unsigned int cpu) * With SCTLR_EL1.IESB enabled, the SME memory transactions are * completed on entering EL1. */ + cpumask_clear_cpu(cpu, mm_cpumask(current->mm)); cpumask_clear_cpu(cpu, sme_active_cpus); }
@@ -1410,19 +1412,25 @@ static void sme_dvmsync_ipi(void *unused) */ } -void sme_do_dvmsync(void) +void sme_do_dvmsync(struct mm_struct *mm) { /* * This is called from the TLB maintenance functions after the DSB ISH * to send hardware DVMSync message. If this CPU sees the mask as * empty, the remote CPU executing sme_set_active() would have seen * the DVMSync and no IPI required. + * + * When an mm is provided, limit the IPI to CPUs that are actively + * running SME code for that mm (recorded in mm_cpumask()), otherwise + * fall back to the global sme_active_cpus mask. */ - if (cpumask_empty(sme_active_cpus)) + const struct cpumask *mask = mm ? mm_cpumask(mm) : sme_active_cpus; + + if (cpumask_empty(mask)) return; preempt_disable(); - smp_call_function_many(sme_active_cpus, sme_dvmsync_ipi, NULL, true); + smp_call_function_many(mask, sme_dvmsync_ipi, NULL, true); preempt_enable(); }