Re: [PATCH net-next v4 1/3] net: devmem: allow rx-buf-size > PAGE_SIZE per dmabuf binding

From
Mina Almasry <hidden>
Date
2026-07-07 20:36:36
Also in
dri-devel, linux-kselftest, linux-media, lkml
Reply to
Bobby Eshleman, 2026-07-01
On Wed, Jul 1, 2026 at 12:22 PM Bobby Eshleman [off-list ref] wrote:
From: Bobby Eshleman <bobbyeshleman@meta.com>

Every devmem dmabuf binding today hands the page_pool PAGE_SIZE niovs.
This caps a single RX descriptor at PAGE_SIZE, burning CPU on buffer
churn for large flows.

Add a bind-time netlink attribute, NETDEV_A_DMABUF_RX_BUF_SIZE, that
lets userspace request a larger niov size. The value must be a power of
two >= PAGE_SIZE.

Measurements
------------
Setup: kperf in devmem RX/TX cuda mode, 4 flows, 64 MB messages, 60s,
dctcp, num-rx-queues=4, dmabuf-rx/tx-size-mb=2048, 10 runs per niov
size, mlx5.

CPU Util:

   niov        net sirq %        net idle %         app sys %        app idle %
  -----  ----------------  ----------------  ----------------  ----------------
     4K   62.38 +/-  8.27   33.40 +/-  7.51   54.15 +/- 10.23   43.67 +/- 10.53
    16K   58.91 +/-  5.35   35.23 +/-  5.88   41.05 +/-  8.87   56.42 +/-  9.24
    32K   64.12 +/-  0.68   31.09 +/-  1.48   44.54 +/-  3.51   52.63 +/-  3.65
    64K   54.69 +/-  5.54   39.67 +/-  5.81   35.47 +/-  3.11   61.97 +/-  3.27

RX app sys % drops ~19% from 4K to 64K.

Throughput:

   niov       RX dev Gbps   RX flow avg Gbps
  -----  ----------------  -----------------
     4K  300.63 +/- 53.21    75.16 +/- 13.30
    16K  321.35 +/- 28.20    80.34 +/-  7.05
    32K  347.63 +/-  2.20    86.91 +/-  0.55
    64K  332.11 +/- 14.26    83.03 +/-  3.56

Throughput seems to increase, but the stdev is pretty wide so could just
be noise.

kperf support (not yet merged):
https://github.com/facebookexperimental/kperf/commit/8837577f920876bce6986ec18869ac04439ebcd2

Signed-off-by: Bobby Eshleman <bobbyeshleman@meta.com>
Acked-by: Stanislav Fomichev <sdf@fomichev.me>
I'm pretty happy to see most of this patch being a spot-for-spot
replacement of PAGE_SIZE with a variable. FWIW:

Reviewed-by: Mina Almasry <redacted>
quoted hunk ↗ jump to hunk
---
 Documentation/netlink/specs/netdev.yaml |  8 +++++
 include/uapi/linux/netdev.h             |  1 +
 net/core/devmem.c                       | 55 +++++++++++++++++++--------------
 net/core/devmem.h                       | 13 +++++---
 net/core/netdev-genl-gen.c              |  5 +--
 net/core/netdev-genl.c                  | 19 ++++++++++--
 tools/include/uapi/linux/netdev.h       |  1 +
 7 files changed, 71 insertions(+), 31 deletions(-)
diff --git a/Documentation/netlink/specs/netdev.yaml b/Documentation/netlink/specs/netdev.yaml
index 5f143da7458c..70b902008bd3 100644
--- a/Documentation/netlink/specs/netdev.yaml
+++ b/Documentation/netlink/specs/netdev.yaml
@@ -598,6 +598,13 @@ attribute-sets:
         type: u32
         checks:
           min: 1
+      -
+        name: rx-buf-size
+        doc: |
+          Size in bytes of each RX buffer the NIC writes into from the bound
+          dmabuf. Must be a power of two and >= PAGE_SIZE; defaults to
+          PAGE_SIZE.
+        type: u32

 operations:
   list:
@@ -812,6 +819,7 @@ operations:
             - ifindex
             - fd
             - queues
+            - rx-buf-size
         reply:
           attributes:
             - id
diff --git a/include/uapi/linux/netdev.h b/include/uapi/linux/netdev.h
index 2f3ab75e8cc0..85e1d20c6268 100644
--- a/include/uapi/linux/netdev.h
+++ b/include/uapi/linux/netdev.h
@@ -219,6 +219,7 @@ enum {
        NETDEV_A_DMABUF_QUEUES,
        NETDEV_A_DMABUF_FD,
        NETDEV_A_DMABUF_ID,
+       NETDEV_A_DMABUF_RX_BUF_SIZE,

        __NETDEV_A_DMABUF_MAX,
        NETDEV_A_DMABUF_MAX = (__NETDEV_A_DMABUF_MAX - 1)
diff --git a/net/core/devmem.c b/net/core/devmem.c
index 957d6b96216b..3d6cf35e50f3 100644
--- a/net/core/devmem.c
+++ b/net/core/devmem.c
@@ -46,7 +46,7 @@ static dma_addr_t net_devmem_get_dma_addr(const struct net_iov *niov)

        owner = net_devmem_iov_to_chunk_owner(niov);
        return owner->base_dma_addr +
-              ((dma_addr_t)net_iov_idx(niov) << PAGE_SHIFT);
+              ((dma_addr_t)net_iov_idx(niov) << owner->binding->niov_shift);
 }

 static void net_devmem_dmabuf_binding_release(struct percpu_ref *ref)
@@ -90,16 +90,17 @@ net_devmem_alloc_dmabuf(struct net_devmem_dmabuf_binding *binding)
        struct dmabuf_genpool_chunk_owner *owner;
        unsigned long dma_addr;
        struct net_iov *niov;
-       ssize_t offset;
-       ssize_t index;
+       size_t offset;
+       size_t index;
nit: I would keep this signed. Some of the most frustrating issues I
ran into is some of the underflowing and then passing a > check or
something. Although if the LLM is not complaining about this
particular case, there is probably no issue with it. I also notice a
lot of existing code that deals with indexes and offsets goes for
signed.

-- 
Thanks,
Mina
Thread (9 messages) 9 messages, 3 authors, last reply 2026-07-08
Keyboard shortcuts
hback out one level
jnext message in thread / next series-diff position
kprevious message in thread / previous series-diff position
ldrill in
Escclose help / fold thread tree
?toggle this help