[PATCH v13 01/15] asm-generic: barrier: Add smp_cond_load_relaxed_timeout()
From: Ankur Arora <hidden>
Date: 2026-07-02 01:34:23
Also in:
bpf, linux-arch, linux-pm, lkml
Subsystem:
generic include/asm header files, the rest · Maintainers:
Arnd Bergmann, Linus Torvalds
Add smp_cond_load_relaxed_timeout(), which extends smp_cond_load_relaxed() to allow waiting for a duration. We loop around waiting for the condition variable to change while peridically doing a time-check. The loop uses cpu_poll_relax() to slow down the busy-wait, which, unless overridden by the architecture code, amounts to a cpu_relax(). Note that there are two ways for the time-check to fail: the timeout case or, @time_expr_ns returning an invalid value (negative or zero). The second failure mode allows for clocks attached to the clock-domain of @cond_expr -- which might cease to operate meaningfully once some state internal to @cond_expr has changed -- to fail. Evaluation of @time_expr_ns: in the fastpath we want to keep the performance close to smp_cond_load_relaxed(). So defer evaluation of the potentially costly @time_expr_ns to the slowpath. This also means that there will always be some hardware dependent duration that has passed in cpu_poll_relax() iterations at the time of first evaluation. Additionally cpu_poll_relax() is not guaranteed to return at timeout boundary. In sum, expect timeout overshoot when we exit due to expiration of the timeout. The number of spin iterations before time-check, SMP_TIMEOUT_POLL_COUNT is chosen to be 200 by default. With a cpu_poll_relax() iteration taking ~20-30 cycles (measured on a variety of x86 platforms), we expect a time-check every ~4000-6000 cycles. The outer limit of the overshoot is double that when working with the parameters above. This might be higher or lower depending on the implementation of cpu_poll_relax() across architectures. Lastly, config option ARCH_HAS_CPU_RELAX indicates availability of a cpu_poll_relax() that is cheaper than polling. This might be relevant for cases with a long timeout. Cc: Arnd Bergmann <arnd@arndb.de> Cc: Will Deacon <will@kernel.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: linux-arch@vger.kernel.org Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Ankur Arora <redacted> --- Notes: adds a justification for arguments taken in cpu_poll_relax() [1]. As mentioned in v12, I'm not addressing issues with potential timeout range issues (around S64_MAX, or during early boot) [1][2]. These are best handled in code review instead of overcomplicating the implementation. [1] https://lore.kernel.org/all/7d2189443a0181a123e53e5ea93f3a905b1cb4d71d5d6bdfa27a77ed4be65448@mail.kernel.org/ (local) [2] https://lore.kernel.org/all/20260608082501.BC5A91F00898@smtp.kernel.org/ (local) --- include/asm-generic/barrier.h | 83 +++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+)
diff --git a/include/asm-generic/barrier.h b/include/asm-generic/barrier.h
index b99cb57dfccc..d53e75a92598 100644
--- a/include/asm-generic/barrier.h
+++ b/include/asm-generic/barrier.h@@ -273,6 +273,85 @@ do { \ }) #endif +/* + * Number of times we iterate in the loop before doing the time check. + * Note that the iteration count assumes that the loop condition is + * relatively cheap. + */ +#ifndef SMP_TIMEOUT_POLL_COUNT +#define SMP_TIMEOUT_POLL_COUNT 200 +#endif + +/* + * cpu_poll_relax() stitches up two kinds of primitives: ones that provide + * a momentary blip in the pipeline (ex. cpu_relax() on x86). + * The second support waiting for @ptr value to change, coupled with a + * with a precise (or imprecise) timeout. + * + * cpu_poll_relax() keeps them together, because its utility is in minimizing + * expensive operations while polling on @ptr waiting for it to change. + * The arguments to cpu_poll_relax() are only needed for the waiting + * primitives. + * + * Also note that platforms with a suitable cpu_poll_relax() are expected + * to define ARCH_HAS_CPU_RELAX. + */ +#ifndef cpu_poll_relax +#define cpu_poll_relax(ptr, val, timeout_ns) cpu_relax() +#endif + +/** + * smp_cond_load_relaxed_timeout() - (Spin) wait for cond with no ordering + * guarantees until a timeout expires. + * @ptr: pointer to the variable to wait on. + * @cond_expr: boolean expression to wait for. + * @time_expr_ns: expression that evaluates to monotonic time (in ns) or, + * on failure, returns a negative value. + * @timeout_ns: timeout value in ns + * Both of the above are assumed to be compatible with s64; the signed + * value is used to handle the failure case in @time_expr_ns. + * + * Equivalent to using READ_ONCE() on the condition variable. + * + * Callers that expect to wait for prolonged durations might want + * to take into account the availability of ARCH_HAS_CPU_RELAX. + * + * Note that @ptr is expected to point to a memory address. Using this + * interface with MMIO will be slower (since SMP_TIMEOUT_POLL_COUNT is + * tuned for memory) and might also break in interesting architecture + * dependent ways. + */ +#ifndef smp_cond_load_relaxed_timeout +#define smp_cond_load_relaxed_timeout(ptr, cond_expr, \ + time_expr_ns, timeout_ns) \ +({ \ + typeof(ptr) __PTR = (ptr); \ + __unqual_scalar_typeof(*(ptr)) VAL; \ + u32 __count = 0, __spin = SMP_TIMEOUT_POLL_COUNT; \ + s64 __timeout = (s64)timeout_ns; \ + s64 __time_now, __time_end = 0; \ + \ + for (;;) { \ + VAL = READ_ONCE(*__PTR); \ + if (cond_expr) \ + break; \ + cpu_poll_relax(__PTR, VAL, (u64)__timeout); \ + if (++__count < __spin) \ + continue; \ + __time_now = (s64)(time_expr_ns); \ + if (unlikely(__time_end == 0)) \ + __time_end = __time_now + __timeout; \ + __timeout = __time_end - __time_now; \ + if (__time_now <= 0 || __timeout <= 0) { \ + VAL = READ_ONCE(*__PTR); \ + break; \ + } \ + __count = 0; \ + } \ + (typeof(*(ptr)))VAL; \ +}) +#endif + /* * pmem_wmb() ensures that all stores for which the modification * are written to persistent storage by preceding instructions have
--
2.43.7