Re: [PATCH RFC 4/4] arm64/io: Add {__raw_read|__raw_write}128 support
From: Mark Rutland <mark.rutland@arm.com>
Date: 2025-11-12 12:28:06
Also in:
linux-arch, linux-arm-kernel, linux-crypto, lkml
On Wed, Nov 12, 2025 at 09:58:46AM +0800, Chenghai Huang wrote:
From: Weili Qian <qianweili@huawei.com> Starting from ARMv8.4, stp and ldp instructions become atomic.
That's not true for accesses to Device memory types.
Per ARM DDI 0487, L.b, section B2.2.1.1 ("Changes to single-copy atomicity in
Armv8.4"):
If FEAT_LSE2 is implemented, LDP, LDNP, and STP instructions that load
or store two 64-bit registers are single-copy atomic when all of the
following conditions are true:
• The overall memory access is aligned to 16 bytes.
• Accesses are to Inner Write-Back, Outer Write-Back Normal cacheable memory.
IIUC when used for Device memory types, those can be split, and a part
of the access could be replayed multiple times (e.g. due to an
intetrupt).
I don't think we can add this generally. It is not atomic, and not
generally safe.
Mark.
quoted hunk ↗ jump to hunk
Currently, device drivers depend on 128-bit atomic memory IO access, but these are implemented within the drivers. Therefore, this introduces generic {__raw_read|__raw_write}128 function for 128-bit memory access. Signed-off-by: Weili Qian <qianweili@huawei.com> Signed-off-by: Chenghai Huang <redacted> --- arch/arm64/include/asm/io.h | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+)diff --git a/arch/arm64/include/asm/io.h b/arch/arm64/include/asm/io.h index 83e03abbb2ca..80430750a28c 100644 --- a/arch/arm64/include/asm/io.h +++ b/arch/arm64/include/asm/io.h@@ -50,6 +50,17 @@ static __always_inline void __raw_writeq(u64 val, volatile void __iomem *addr) asm volatile("str %x0, %1" : : "rZ" (val), "Qo" (*ptr)); } +#define __raw_write128 __raw_write128 +static __always_inline void __raw_write128(u128 val, volatile void __iomem *addr) +{ + u64 low, high; + + low = val; + high = (u64)(val >> 64); + + asm volatile ("stp %x0, %x1, [%2]\n" :: "rZ"(low), "rZ"(high), "r"(addr)); +} + #define __raw_readb __raw_readb static __always_inline u8 __raw_readb(const volatile void __iomem *addr) {@@ -95,6 +106,16 @@ static __always_inline u64 __raw_readq(const volatile void __iomem *addr) return val; } +#define __raw_read128 __raw_read128 +static __always_inline u128 __raw_read128(const volatile void __iomem *addr) +{ + u64 high, low; + + asm volatile("ldp %0, %1, [%2]" : "=r" (low), "=r" (high) : "r" (addr)); + + return (((u128)high << 64) | (u128)low); +} + /* IO barriers */ #define __io_ar(v) \ ({ \-- 2.33.0