Re: [PATCH net-next v3 0/3] net: hash uncached route lists by device
flat view
From: Kuniyuki Iwashima <kuniyu@google.com>
Date: 2026-09-18 04:21:11
Also in:
linux-kselftest, lkml
Subsystem:
networking [general], networking [ipv4/ipv6], the rest · Maintainers:
"David S. Miller", Eric Dumazet, Jakub Kicinski, Paolo Abeni, David Ahern, Ido Schimmel, Linus Torvalds
From: Chris Arges <redacted> Date: Thu, 17 Sep 2026 19:38:40 -0500
On 2026-09-17 22:10:22, Kuniyuki Iwashima wrote:quoted
From: Chris J Arges <redacted> Date: Thu, 17 Sep 2026 14:38:21 -0500quoted
We have observed hung tasks blocked on rtnl_mutex while network namespaces were being removed. The namespaces contained many network devices, and the host had accumulated a large population of entries on the global per-CPU uncached route lists. A perf profile collected during one incident attributed most of the cleanup worker's samples to rt_flush_dev():99.92% kworker/u384:3- worker_thread `-88.71% process_one_work `-81.02% cleanup_net `-81.00% unregister_netdevice_many_notify `-79.42% notifier_call_chain `-78.05% fib_netdev_event `-77.92% rt_flush_devFor each device, rt_flush_dev() visits every possible CPU and scans the global uncached route population while its caller holds rtnl_mutex. If N is the number of devices, C the number of possible CPUs, and R the number of uncached routes, the cost is O(N * (C + R)). During namespace cleanup, other processes that issue RTNETLINK operations requiring the RTNL lock can stall until cleanup releases the lock. A minimal reproducer is available here: https://github.com/arges/linux-reproducers/tree/main/rtnl-flush-storm This series replaces each per-CPU uncached route list with a hash table using the network device as its key. Each table uses 64 buckets.This sounds a bit overkill. Also, this series still leaves O(N * C) loops. Given unregistering a single device is less common than destroying netns, I think the right approach should be to make the route flush once in cleanup_net() + outside RTNL. Could you try this change ? (only compile-tested)Excellent, I'll test this and report back.
I found a pre-existing issue, which affects the previous diff, so on top of it, please apply this patch https://lore.kernel.org/netdev/20260918041439.2575935-1-kuniyu@google.com/T/#u (local) and this diff : ---8<---
diff --git a/net/ipv4/route.c b/net/ipv4/route.c
index d35b66b33bbc..c12e20e07749 100644
--- a/net/ipv4/route.c
+++ b/net/ipv4/route.c@@ -1567,12 +1567,13 @@ void rt_add_uncached_list(struct rtable *rt) { struct uncached_list *ul = raw_cpu_ptr(&rt_uncached_list); + rt->dst.rt_uncached_list = ul; + spin_lock_bh(&ul->lock); if (!check_net(dst_dev_net_rcu(&rt->dst))) { rt_replace_uncached_list(rt); } else { - rt->dst.rt_uncached_list = ul; list_add_tail(&rt->dst.rt_uncached, &ul->head); }
diff --git a/net/ipv6/route.c b/net/ipv6/route.c
index 6cffe8440b44..f22793abbbb8 100644
--- a/net/ipv6/route.c
+++ b/net/ipv6/route.c@@ -155,12 +155,13 @@ void rt6_uncached_list_add(struct rt6_info *rt) { struct uncached_list *ul = raw_cpu_ptr(&rt6_uncached_list); + rt->dst.rt_uncached_list = ul; + spin_lock_bh(&ul->lock); if (!check_net(dst_dev_net_rcu(&rt->dst))) { rt6_uncached_list_replace(rt); } else { - rt->dst.rt_uncached_list = ul; list_add_tail(&rt->dst.rt_uncached, &ul->head); } ---8<---
Thanks !