RE: [RFC PATCH v2 2/2] dma: swiotlb: Initialize and size shared default pools for memory encryption

From: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org> Sent: Thursday, August 13, 2026 3:25 AM

Systems with memory encryption require shared or unencrypted buffers for
device DMA. Confidential guests may route all DMA through SWIOTLB, making
the default pool too small for I/O-intensive workloads. Host memory
encryption also requires a shared default pool when bouncing is needed,
but does not require the guest sizing policy.

Move the existing x86 sizing policy into the SWIOTLB core. Detect memory
encryption before allocating the default pool so that it is initialized
and marked shared even without DMA addressing limitations. Increase the
pool size to 6% of guest memory, clamped between the default size and
1 GiB, only for confidential guests.

The core can determine the confidential-computing requirement directly,
so remove SWIOTLB_INIT_CC_SHARED and its architecture uses. Keep the
pseries secure-guest setup before swiotlb_init() so that the pool is
allocated with SWIOTLB_ANY and is not released later.

A restricted DMA pool already supplies shared bounce buffers for its
devices. Record its presence during reserved-memory initialization and
avoid initializing or marking the default pool as shared solely because
memory encryption is enabled.

Signed-off-by: Aneesh Kumar K.V (Arm) <aneesh.kumar@kernel.org>
---
 arch/arm64/mm/init.c      |  6 +---
 arch/powerpc/mm/mem.c     |  2 +-
 arch/s390/mm/init.c       |  2 +-
 arch/x86/mm/mem_encrypt.c | 28 ---------------
 include/linux/swiotlb.h   |  2 --
 kernel/dma/swiotlb.c      | 72 +++++++++++++++++++++++++++++----------
 6 files changed, 57 insertions(+), 55 deletions(-)

[snip]

+static void __init swiotlb_adjust_cc_attributes(void)
+{
+	unsigned long size;
+	phys_addr_t total_mem = memblock_phys_mem_size();
+
+	/* Do not resize for host memory encryption. */
+	if (!cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT))
+		return;
+
+	/*
+	 * For SEV and TDX and CCA, all DMA has to occur via
+	 * shared/unencrypted pages. Kernel uses SWIOTLB to make this
+	 * happen without changing device drivers. However, depending on
+	 * the workload being run, the default 64MB of SWIOTLB may not be
+	 * enough and SWIOTLB may run out of buffers for DMA, resulting in
+	 * I/O errors and/or performance degradation especially with high
+	 * I/O workloads.
+	 *
+	 * Adjust the default size of SWIOTLB using a percentage of guest
+	 * memory for SWIOTLB buffers. Also, as the SWIOTLB bounce buffer
+	 * memory is allocated from low memory, ensure that the adjusted
+	 * size is within the limits of low available memory.
+	 *
+	 * The percentage of guest memory used here for SWIOTLB buffers is
+	 * more of an approximation of the static adjustment which 64MB for
+	 * <1G, and ~128M to 256M for 1G-to-4G, i.e., the 6%
+	 */
+	size = total_mem * 6 / 100;
+	size = clamp_val(size, IO_TLB_DEFAULT_SIZE, SZ_1G);
+	swiotlb_adjust_size(size);
+
+	if (!IS_ENABLED(CONFIG_SWIOTLB_DYNAMIC))
+		pr_info("Consider enabling CONFIG_SWIOTLB_DYNAMIC for memory-encrypted systems\n");

I'm not keen on this recommendation. The implementation of
SWIOTLB_DYNAMIC has two limitations that make it problematic for
large CoCo VMs. First, when swiotlb is unable to find sufficient space
in the default pool allocated at boot time, SWIOTLB_DYNAMC creates a
"transient" swiotlb pool that is allocated from the DMA atomic pool to
satisfy the immediate request. Growing the overall swiotlb is kicked off
asynchronously in the background. dma_unmap() frees the transient
pool and the memory is returned to the DMA atomic pool, but a spike
in swiotlb allocations in a big system still has substantial potential to
exhaust the DMA atomic pool. Then you get the same mapping failure
you'd get without SWIOTLB_DYNAMIC. The fundamental problem has
only been moved around.

Second, growing the swiotlb in the background adds a new pool, which
requires allocating contiguous physical memory. That's at most a
MAX_PAGE_ORDER allocation, so 4 MiB. The new pool is divided into
"areas" of minimum size 256 KiB so swiotlb allocations can proceed
in parallel under a per-area spin lock. 16 areas isn't too bad, but might
still have more contention than the original default pool, which, for
example, would have 64 areas on a 64 vCPU VM. But if memory is
fragmented, and you can only get 512 KiB, you'll have only 2 areas,
and substantially more contention. Contention for the swiotlb
spin lock at high vCPU counts has been shown to be a substantial
problem, which is why areas were introduced. But SWIOTLB_DYNAMIC
often can't preserve the parallelism in the dynamically added pools
because of the contiguous memory allocation limitations.

Net, in smaller CoCo VMs, there's some marginal value in
SWIOTLB_DYNAMIC, but that value diminishes as the VM gets
larger and more swiotlb parallelism is needed. And in either case,
there's still the potential of exhausting the DMA atomic pool due
to a spike in demand, in which case you haven't solved anything.

Michael