[PATCH v2 net 2/5] xsk: drain continuation descs after overflow in xsk_build_skb()
From: Maciej Fijalkowski <maciej.fijalkowski@intel.com>
Date: 2026-07-10 19:45:00
Also in:
bpf
Subsystem:
networking [general], the rest, xdp sockets (af_xdp) · Maintainers:
"David S. Miller", Eric Dumazet, Jakub Kicinski, Paolo Abeni, Linus Torvalds, Magnus Karlsson, Maciej Fijalkowski
From: Jason Xing <kernelxing@tencent.com>
Fix generic xmit path multi-buffer logic when packets are either too big
(count of descriptors exceed MAX_SKB_FRAGS) or an invalid descriptor is
included in fragmented packet. Introduce xdp_sock::drain_cont and act
upon this flag - when it is set, keep on consuming descriptors from
AF_XDP Tx ring and put them directly onto Cq. Previously these
descriptors were silently lost and could never be reached again.
Fixes: cf24f5a5feea ("xsk: add support for AF_XDP multi-buffer on Tx path")
Closes: https://lore.kernel.org/all/20260425041726.85FB3C2BCB2@smtp.kernel.org/ (local)
Co-developed-by: Maciej Fijalkowski <maciej.fijalkowski@intel.com> # wrapped cq addr submission onto routine
Signed-off-by: Maciej Fijalkowski <maciej.fijalkowski@intel.com>
Signed-off-by: Jason Xing <kernelxing@tencent.com>
---
include/net/xdp_sock.h | 1 +
net/xdp/xsk.c | 39 +++++++++++++++++++++++++++++++++++++++
2 files changed, 40 insertions(+)
diff --git a/include/net/xdp_sock.h b/include/net/xdp_sock.h
index ebac60a3d8a1..8b51876efbed 100644
--- a/include/net/xdp_sock.h
+++ b/include/net/xdp_sock.h@@ -80,6 +80,7 @@ struct xdp_sock { * call of __xsk_generic_xmit(). */ struct sk_buff *skb; + bool drain_cont; struct list_head map_list; /* Protects map_list */
diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c
index a7a83dc4546a..43791647cf18 100644
--- a/net/xdp/xsk.c
+++ b/net/xdp/xsk.c@@ -737,6 +737,19 @@ static void xsk_cq_submit_addr_locked(struct xsk_buff_pool *pool, spin_unlock_irqrestore(&pool->cq_prod_lock, flags); } +static void xsk_cq_submit_addr_single_locked(struct xsk_buff_pool *pool, + struct xdp_desc *desc) +{ + unsigned long flags; + u32 idx; + + spin_lock_irqsave(&pool->cq_prod_lock, flags); + idx = xskq_get_prod(pool->cq); + xskq_prod_write_addr(pool->cq, idx, desc->addr); + xskq_prod_submit_n(pool->cq, 1); + spin_unlock_irqrestore(&pool->cq_prod_lock, flags); +} + static void xsk_cq_cancel_locked(struct xsk_buff_pool *pool, u32 n) { spin_lock(&pool->cq->cq_cached_prod_lock);
@@ -1063,11 +1076,22 @@ static int __xsk_generic_xmit(struct sock *sk) goto out; } + if (unlikely(xs->drain_cont)) { + xsk_cq_submit_addr_single_locked(xs->pool, &desc); + + xs->tx->invalid_descs++; + xskq_cons_release(xs->tx); + xs->drain_cont = xp_mb_desc(&desc); + continue; + } + skb = xsk_build_skb(xs, &desc); if (IS_ERR(skb)) { err = PTR_ERR(skb); if (err != -EOVERFLOW) goto out; + if (xp_mb_desc(&desc)) + xs->drain_cont = true; err = 0; continue; }
@@ -1101,9 +1125,24 @@ static int __xsk_generic_xmit(struct sock *sk) } if (xskq_has_descs(xs->tx)) { + bool reclaim_desc = xs->skb || xs->drain_cont; + + if (reclaim_desc) { + err = xsk_cq_reserve_locked(xs->pool); + if (err) { + err = -EAGAIN; + goto out; + } + } + if (xs->skb) xsk_drop_skb(xs->skb); + + if (reclaim_desc) + xsk_cq_submit_addr_single_locked(xs->pool, &desc); + xskq_cons_release(xs->tx); + xs->drain_cont = xp_mb_desc(&desc); } out:
--
2.43.0