Re: [REGRESSION][BISECTED] tun/tap & vhost-net: multi-threaded network performance
From: "Michael S. Tsirkin" <mst@redhat.com>
Date: 2026-07-02 22:44:36
Also in:
lkml, regressions
Subsystem:
networking drivers, the rest, tun/tap driver, virtio host (vhost) · Maintainers:
Andrew Lunn, "David S. Miller", Eric Dumazet, Jakub Kicinski, Paolo Abeni, Linus Torvalds, Willem de Bruijn, Jason Wang, "Michael S. Tsirkin"
On Thu, Jul 02, 2026 at 01:07:47PM +0200, Brett Sheffield wrote:
On 2026-07-02 09:24, Simon Schippers wrote:quoted
On 7/1/26 22:56, Michael S. Tsirkin wrote:quoted
On Wed, Jul 01, 2026 at 09:16:48PM +0200, Brett Sheffield wrote:quoted
TL;DR - Commit 1d6e569b7d0c0b2736636749e4be0a27f3cefcb3 causes significant performance regressions with TAP interfaces and multithreaded network code. Please revert. Librecast is an IPv6 multicast library. One of the tests (0055) fails under Linux 7.2-rc1. The test performs data synchronization over IPv6 multicast using a TAP interface. This test has run successfully on every stable, LTS and mainline RC released in the past year. Every kernel with my Tested-by has run this test. There have been a bunch of changes to MLDv2 so I started bisecting there, but the culprit is actually 1d6e569b7d0c0b2736636749e4be0a27f3cefcb3 "tun/tap & vhost-net: avoid ptr_ring tail-drop when a qdisc is present" Reverting this commit fixes the test. To eliminate my code and any multicast weirdness, I ran tests with iperf3 comparing the same host running 7.2-rc1 both with and without 1d6e569b7d0 reverted.Thank you very much for your bisect! As the author, I am sorry for that regression!No worries. That's why we test :-)quoted
quoted
- does it help to increase the tun queue size?I agree, this would be great to know. However, even then we must act. I am considering IFF_BACKPRESSURE as a feature flag, defaulting to off. It would just enable/disable the stopping logic in tun_net_xmit() and the waking logic in __tun_wake_queue(). If disabled, it would result in the same logic as before. I could provide such a patch as [net] material.I'm going to make myself a strong cup of tea and dig into it a bit more here and will let you know if I find anything worth reporting. If you need me to try re-testing with specific settings or test a patch I'm happy to do so. Cheers, Brett -- Brett Sheffield (he/him) Librecast - Decentralising the Internet with Multicast https://librecast.net/ https://blog.brettsheffield.com/
Well, the issue was with host to guest right? Then testing what does bql do might be interesting. Might help. Something like this? Lightly tested.
diff --git a/drivers/net/tun.c b/drivers/net/tun.c
index bfa49fa9e3a1..abc46354c107 100644
--- a/drivers/net/tun.c
+++ b/drivers/net/tun.c@@ -1076,6 +1076,7 @@ static netdev_tx_t tun_net_xmit(struct sk_buff *skb, struct net_device *dev) queue = netdev_get_tx_queue(dev, txq); spin_lock(&tfile->tx_ring.producer_lock); + netdev_tx_sent_queue(queue, len); ret = __ptr_ring_produce(&tfile->tx_ring, skb); if (!qdisc_txq_has_no_queue(queue) && __ptr_ring_check_produce(&tfile->tx_ring) == -ENOSPC) {
@@ -1088,6 +1089,7 @@ static netdev_tx_t tun_net_xmit(struct sk_buff *skb, struct net_device *dev) spin_unlock(&tfile->tx_ring.producer_lock); if (ret) { + netdev_tx_completed_queue(queue, 1, len); /* This should be a rare case if a qdisc is present, but * can happen due to lltx. * Since skb_tx_timestamp(), skb_orphan(),
@@ -2148,15 +2150,19 @@ static ssize_t tun_put_user(struct tun_struct *tun, /* Callers must hold ring.consumer_lock */ static void __tun_wake_queue(struct tun_struct *tun, - struct tun_file *tfile, int consumed) + struct tun_file *tfile, + unsigned int pkts, unsigned int bytes) { struct netdev_queue *txq = netdev_get_tx_queue(tun->dev, tfile->queue_index); + if (bytes) + netdev_tx_completed_queue(txq, pkts, bytes); + /* Paired with smp_mb__after_atomic() in tun_net_xmit() */ smp_mb(); if (netif_tx_queue_stopped(txq)) { - tfile->cons_cnt += consumed; + tfile->cons_cnt += pkts; if (tfile->cons_cnt >= tfile->tx_ring.size / 2 || __ptr_ring_empty(&tfile->tx_ring)) { netif_tx_wake_queue(txq);
@@ -2167,12 +2173,16 @@ static void __tun_wake_queue(struct tun_struct *tun, static void *tun_ring_consume(struct tun_struct *tun, struct tun_file *tfile) { + unsigned int bytes = 0; void *ptr; spin_lock(&tfile->tx_ring.consumer_lock); ptr = __ptr_ring_consume(&tfile->tx_ring); - if (ptr) - __tun_wake_queue(tun, tfile, 1); + if (ptr) { + if (!tun_is_xdp_frame(ptr)) + bytes = ((struct sk_buff *)ptr)->len; + __tun_wake_queue(tun, tfile, 1, bytes); + } spin_unlock(&tfile->tx_ring.consumer_lock); return ptr;
@@ -3805,7 +3815,7 @@ struct ptr_ring *tun_get_tx_ring(struct file *file) EXPORT_SYMBOL_GPL(tun_get_tx_ring); /* Callers must hold ring.consumer_lock */ -void tun_wake_queue(struct file *file, int consumed) +void tun_wake_queue(struct file *file, unsigned int pkts, unsigned int bytes) { struct tun_file *tfile; struct tun_struct *tun;
@@ -3821,7 +3831,7 @@ void tun_wake_queue(struct file *file, int consumed) tun = rcu_dereference(tfile->tun); if (tun) - __tun_wake_queue(tun, tfile, consumed); + __tun_wake_queue(tun, tfile, pkts, bytes); rcu_read_unlock(); }
diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
index db341c922673..5267b323bd59 100644
--- a/drivers/vhost/net.c
+++ b/drivers/vhost/net.c@@ -181,14 +181,23 @@ static int vhost_net_buf_produce(struct sock *sk, { struct file *file = sk->sk_socket->file; struct vhost_net_buf *rxq = &nvq->rxq; + unsigned int bytes = 0; + int i; rxq->head = 0; spin_lock(&nvq->rx_ring->consumer_lock); rxq->tail = __ptr_ring_consume_batched(nvq->rx_ring, rxq->queue, VHOST_NET_BATCH); - if (rxq->tail) - tun_wake_queue(file, rxq->tail); + if (rxq->tail) { + for (i = 0; i < rxq->tail; i++) { + void *ptr = rxq->queue[i]; + + if (!tun_is_xdp_frame(ptr)) + bytes += ((struct sk_buff *)ptr)->len; + } + tun_wake_queue(file, rxq->tail, bytes); + } spin_unlock(&nvq->rx_ring->consumer_lock); return rxq->tail;
diff --git a/include/linux/if_tun.h b/include/linux/if_tun.h
index 5f3e206c7a73..49b85bf4f828 100644
--- a/include/linux/if_tun.h
+++ b/include/linux/if_tun.h@@ -22,7 +22,7 @@ struct tun_msg_ctl { #if defined(CONFIG_TUN) || defined(CONFIG_TUN_MODULE) struct socket *tun_get_socket(struct file *); struct ptr_ring *tun_get_tx_ring(struct file *file); -void tun_wake_queue(struct file *file, int consumed); +void tun_wake_queue(struct file *file, unsigned int pkts, unsigned int bytes); static inline bool tun_is_xdp_frame(void *ptr) {
@@ -56,7 +56,8 @@ static inline struct ptr_ring *tun_get_tx_ring(struct file *f) return ERR_PTR(-EINVAL); } -static inline void tun_wake_queue(struct file *f, int consumed) {} +static inline void tun_wake_queue(struct file *f, + unsigned int pkts, unsigned int bytes) {} static inline bool tun_is_xdp_frame(void *ptr) {