Re: [PATCH v6 net] octeontx2-af: switch qmem from coherent DMA alloc to streaming DMA mapping

On 2026-09-02 at 07:40:23, Ratheesh Kannoth (rkannoth@marvell.com) wrote:

qmem_alloc() uses dma_alloc_attrs() with DMA_ATTR_FORCE_CONTIGUOUS, which
allocates CPU-cache-coherent DMA memory and, with CMA enabled, draws from
the CMA pool. qmem backs NIX/NPA queue contexts, admin queues, and LMTST
regions (including CN10K LMTST areas that span page boundaries), so
consumption grows with enabled interfaces and is hard to provision in CMA.

Switch qmem to a streaming-DMA-style path: allocate physically contiguous
compound pages from the buddy allocator via __get_free_pages(), then map
them for device access with dma_map_page_attrs(). Add
otx2_dma_alloc_coherent() and otx2_dma_free_coherent() helpers that
enforce dev_is_dma_coherent(), retry with GFP_DMA32 when the physical
range is outside the device DMA mask, and wire qmem_alloc()/qmem_free()
through them instead of dma_alloc_attrs()/dma_free_attrs().

This works on Octeon because the octeontx2 driver is written for
DMA-coherent devices: Octeon platforms provide IO coherency (via SMMU), so
the driver already uses streaming DMA APIs for packet data while
deliberately skipping explicit CPU cache sync (DMA_ATTR_SKIP_CPU_SYNC).
The same IO coherency lets qmem use a streaming map of buddy-allocated
pages instead of a dedicated coherent allocator or CMA reservation. That
is valid because the platform is DMA-coherent, not because omitting
dma_sync_* magically makes memory coherent.

Allocations requiring more than MAX_PAGE_ORDER pages are still rejected,
since the buddy allocator cannot serve them without CMA.

My reply to sashiko comments

+----+------------+------------+-----+----------+----------------+---------------+-----------------+
| #  | Location   | Issue      | Sev | Preexist | When it bites  | Suggested fix | My comments     |
+----+------------+------------+-----+----------+----------------+---------------+-----------------+
| 1  | otx2_dma_  | alloc_gfp  | Med | No       | GFP_ATOMIC/    | Preserve      |  All existing callers of this static function
|    | alloc_     | ORs in     |     |          | NOWAIT may     | caller GFP;   |  use GFP_KERNEL. So in sleep in context.
|    | coherent() | __GFP_     |     |          | sleep in       | add RECLAIM   |                 |
|    | GFP        | RECLAIM;   |     |          | atomic ctx     | only when     |                 |
|    | handling   | violates   |     |          | (sched while   | allowed.      |                 |
|    |            | caller GFP |     |          | atomic).       |               |                 |
|    |            | semantics  |     |          | qmem uses      |               |                 |
|    |            |            |     |          | GFP_KERNEL.    |               |                 |
+----+------------+------------+-----+----------+----------------+---------------+-----------------+
| 2  | otx2_dma_  | Coherent   | Hi  | No       | SWIOTLB bounce | Keep coherent |                 |
|    | alloc_     | DMA ->     |     |          | (swiotlb=force | DMA or add    | Commit message clearly explains the reason
|    | coherent() | streaming  |     |          | ARM CCA): CPU  | dma_sync_*;   |                 |
|    | DMA map    | map; no    |     |          | and device use | detect bounce |                 |
|    |            | dma_sync_* |     |          | disjoint mem;  | fallback.     |                 |
|    |            | in queue   |     |          | queue broken.  |               |                 |
+----+------------+------------+-----+----------+----------------+---------------+-----------------+