On Wed, Jul 1, 2026 at 12:22 PM Bobby Eshleman [off-list ref] wrote:
From: Bobby Eshleman <bobbyeshleman@meta.com>
Every devmem dmabuf binding today hands the page_pool PAGE_SIZE niovs.
This caps a single RX descriptor at PAGE_SIZE, burning CPU on buffer
churn for large flows.
Add a bind-time netlink attribute, NETDEV_A_DMABUF_RX_BUF_SIZE, that
lets userspace request a larger niov size. The value must be a power of
two >= PAGE_SIZE.
Measurements
------------
Setup: kperf in devmem RX/TX cuda mode, 4 flows, 64 MB messages, 60s,
dctcp, num-rx-queues=4, dmabuf-rx/tx-size-mb=2048, 10 runs per niov
size, mlx5.
CPU Util:
niov net sirq % net idle % app sys % app idle %
----- ---------------- ---------------- ---------------- ----------------
4K 62.38 +/- 8.27 33.40 +/- 7.51 54.15 +/- 10.23 43.67 +/- 10.53
16K 58.91 +/- 5.35 35.23 +/- 5.88 41.05 +/- 8.87 56.42 +/- 9.24
32K 64.12 +/- 0.68 31.09 +/- 1.48 44.54 +/- 3.51 52.63 +/- 3.65
64K 54.69 +/- 5.54 39.67 +/- 5.81 35.47 +/- 3.11 61.97 +/- 3.27
RX app sys % drops ~19% from 4K to 64K.
Throughput:
niov RX dev Gbps RX flow avg Gbps
----- ---------------- -----------------
4K 300.63 +/- 53.21 75.16 +/- 13.30
16K 321.35 +/- 28.20 80.34 +/- 7.05
32K 347.63 +/- 2.20 86.91 +/- 0.55
64K 332.11 +/- 14.26 83.03 +/- 3.56
Throughput seems to increase, but the stdev is pretty wide so could just
be noise.
kperf support (not yet merged):
https://github.com/facebookexperimental/kperf/commit/8837577f920876bce6986ec18869ac04439ebcd2
Signed-off-by: Bobby Eshleman <bobbyeshleman@meta.com>
Acked-by: Stanislav Fomichev <sdf@fomichev.me>
I'm pretty happy to see most of this patch being a spot-for-spot
replacement of PAGE_SIZE with a variable. FWIW:
Reviewed-by: Mina Almasry <redacted>
quoted hunk ↗ jump to hunk
---
Documentation/netlink/specs/netdev.yaml | 8 +++++
include/uapi/linux/netdev.h | 1 +
net/core/devmem.c | 55 +++++++++++++++++++--------------
net/core/devmem.h | 13 +++++---
net/core/netdev-genl-gen.c | 5 +--
net/core/netdev-genl.c | 19 ++++++++++--
tools/include/uapi/linux/netdev.h | 1 +
7 files changed, 71 insertions(+), 31 deletions(-)
diff --git a/Documentation/netlink/specs/netdev.yaml b/Documentation/netlink/specs/netdev.yaml
index 5f143da7458c..70b902008bd3 100644
--- a/Documentation/netlink/specs/netdev.yaml
+++ b/Documentation/netlink/specs/netdev.yaml
@@ -598,6 +598,13 @@ attribute-sets:
type: u32
checks:
min: 1
+ -
+ name: rx-buf-size
+ doc: |
+ Size in bytes of each RX buffer the NIC writes into from the bound
+ dmabuf. Must be a power of two and >= PAGE_SIZE; defaults to
+ PAGE_SIZE.
+ type: u32
operations:
list:@@ -812,6 +819,7 @@ operations:
- ifindex
- fd
- queues
+ - rx-buf-size
reply:
attributes:
- iddiff --git a/include/uapi/linux/netdev.h b/include/uapi/linux/netdev.h
index 2f3ab75e8cc0..85e1d20c6268 100644
--- a/include/uapi/linux/netdev.h
+++ b/include/uapi/linux/netdev.h
@@ -219,6 +219,7 @@ enum {
NETDEV_A_DMABUF_QUEUES,
NETDEV_A_DMABUF_FD,
NETDEV_A_DMABUF_ID,
+ NETDEV_A_DMABUF_RX_BUF_SIZE,
__NETDEV_A_DMABUF_MAX,
NETDEV_A_DMABUF_MAX = (__NETDEV_A_DMABUF_MAX - 1)diff --git a/net/core/devmem.c b/net/core/devmem.c
index 957d6b96216b..3d6cf35e50f3 100644
--- a/net/core/devmem.c
+++ b/net/core/devmem.c
@@ -46,7 +46,7 @@ static dma_addr_t net_devmem_get_dma_addr(const struct net_iov *niov)
owner = net_devmem_iov_to_chunk_owner(niov);
return owner->base_dma_addr +
- ((dma_addr_t)net_iov_idx(niov) << PAGE_SHIFT);
+ ((dma_addr_t)net_iov_idx(niov) << owner->binding->niov_shift);
}
static void net_devmem_dmabuf_binding_release(struct percpu_ref *ref)@@ -90,16 +90,17 @@ net_devmem_alloc_dmabuf(struct net_devmem_dmabuf_binding *binding)
struct dmabuf_genpool_chunk_owner *owner;
unsigned long dma_addr;
struct net_iov *niov;
- ssize_t offset;
- ssize_t index;
+ size_t offset;
+ size_t index;
nit: I would keep this signed. Some of the most frustrating issues I
ran into is some of the underflowing and then passing a > check or
something. Although if the LLM is not complaining about this
particular case, there is probably no issue with it. I also notice a
lot of existing code that deals with indexes and offsets goes for
signed.
--
Thanks,
Mina