Questions about resilient next-hop groups (6.18.35)

Hello everyone,

I am running kernel 6.18.35 and I have the following multi path route setup:


10.181.0.0/16 nhid 400
      nexthop dev bond0-ip6tnl0 weight 1
      nexthop dev bond0-ip6tnl1 weight 1
100.80.0.0/12 nhid 400
      nexthop dev bond0-ip6tnl0 weight 1
      nexthop dev bond0-ip6tnl1 weight 1
100.112.0.0/13 nhid 400
      nexthop dev bond0-ip6tnl0 weight 1
      nexthop dev bond0-ip6tnl1 weight 1


The links are ip6tnl devices sending traffic over an IPv6 tunnel to the other end.

I am using L4 ECMP hash policy:

~ # sysctl net.ipv4.fib_multipath_hash_policy
net.ipv4.fib_multipath_hash_policy = 1
~ # sysctl net.ipv6.fib_multipath_hash_policy
net.ipv6.fib_multipath_hash_policy = 1

Here are the next hop definitions:

id 400 group 4000,128/4001,128 type resilient buckets 1024 idle_timer 60 unbalanced_timer 0 unbalanced_time 0
id 4000 dev bond0-ip6tnl0 scope host
id 4001 dev bond0-ip6tnl1 scope host


So I have two equal weighted next hops with 512 buckets each.

Now I am running a TCP connection in a shell on a host in the 10.181.0.0/16 network over one of the next hops that continuously produces traffic to keep its bucket busy:

while true; do 
  sleep 0.1
  date
done 


Thu Jul  9 13:27:03 UTC 2026
Thu Jul  9 13:27:03 UTC 2026
Thu Jul  9 13:27:03 UTC 2026
Thu Jul  9 13:27:04 UTC 2026
(…)

This roughly sends 10 pkts/s.
Now I want to drain next hop 4000 which is where my traffic is flowing:


ip nexthop replace id 400 group 4000,1/4001,255 type resilient buckets 1024 idle_timer 60 unbalanced_timer 0

Assigning it a weight of 1 should reduce the number of buckets assigned to next hop 4000 to 4 and move idle buckets to next hop 4001.

However, what I am seeing is that the connection is immediately terminated.


When I am changing the weights more slowly it kind of works:

~ # ip nexthop replace id 400 group 4000,128/4001,128 type resilient idle_timer 60 unbalanced_timer 0
~ # ip nexthop replace id 400 group 4000,64/4001,192 type resilient idle_timer 60 unbalanced_timer 0
~ # ip nexthop replace id 400 group 4000,5/4001,250 type resilient idle_timer 60 unbalanced_timer 0
~ # ip nexthop replace id 400 group 4000,1/4001,255 type resilient idle_timer 60 unbalanced_timer 0

I can then see the buckets reduced to just 4 over time as expected:

Every 1.0s: ip nexthop bucket list nhid 4000                                                                                                                               2026-07-09 13:44:22

id 400 index 447 idle_time 135.51 nhid 4000
id 400 index 631 idle_time 117.56 nhid 4000
id 400 index 720 idle_time 73.11 nhid 4000
id 400 index 742 idle_time 136.19 nhid 4000


My guess is that my connection is in bucket index 720 as it has the lowest idle_time reported. 

My questions:
- Why is the idle_time reported so high? There should be packets flowing through that bucket every 0.1 seconds, so I was expecting the idle_time to be near zero.
- Why does my connection get cut immediately when draining rapidly but kind of stays alive when changing weights more slowly? Shouldn't it be a busy (undrainable) bucket either way?
Bucket nr 720 already has idle_time higher than 60 so I think it would be migrated on any further weight changes - which would disrupt the existing connection again.
- When / how is the idle_time of a bucket updated? The docs mention that “Every time a packet is forwarded through a bucket, this timer is updated to current jiffies value.” I found that the bucket idle_time is refreshed more frequently when fever buckets are used. But why does the number of buckets even matter here? Fewer buckets mean more flows per bucket, I get that. But if there is at least one busy connection in a bucket, shouldn't that be enough to flag it busy?



Best regards,
Dom