Re: [PATCH net-next v9] net: reduce RFS/ARFS flow updates by checking LLC affinity

On Sun, Sep 13, 2026 at 12:14 PM Eric Dumazet [off-list ref] wrote:

On Sat, Sep 12, 2026 at 8:50 PM chuang [off-list ref] wrote:

quoted

Hi, let me restructure the issues.

quoted

1) Also not sure why you're trying to touch RFS, the change only helps

aRFS and you could avoid bulk of the refactoring issues.

The scenario is similar to the one described in "[RFC] problems with
RFS on bRPC applications"[1].

I attempted to enable ARFS on a Mellanox CX-6 NIC. While it performs
well for simple workloads, performance degrades significantly when
running a bRPC[2]  workload on a 2-node NUMA machine. After tracing, I
identified patterns that ARFS/RFS fails to handle efficiently:

- Multiple threads use epoll to read from the same socket, causing
frequent flow updates in sock_flow_table.
- Threads reading from the socket migrate frequently between CPUs.

I tested a PoC version using a bRPC service, utilizing funccount [3]
to monitor execution frequency and perf top to observe hotspots:

Before Patch

The mlx5e_rx_flow_steer frequency is over 380k/s, and queued_spin_lock
is a major hotspot (6.30% in perf top). The application also suffers
from a noticeable drop.

FUNC                                     COUNT
mlx5e_rx_flow_steer              387594

FUNC                                     COUNT
mlx5e_rx_flow_steer              390142

FUNC                                     COUNT
mlx5e_rx_flow_steer              386694

FUNC                                     COUNT
mlx5e_rx_flow_steer              389094

# perf top hotspot:
queued_spin_lock                         6.30%

After Patch

The ARFS update frequency is significantly reduced. queued_spin_lock
is no longer a hotspot in perf top, and the application's overall
performance has improved.

FUNC                                     COUNT
mlx5e_rx_flow_steer                      43

FUNC                                     COUNT
mlx5e_rx_flow_steer                        9

FUNC                                     COUNT
mlx5e_rx_flow_steer                    207

FUNC                                     COUNT
mlx5e_rx_flow_steer                      26

quoted

2) You put a very fast path function out-of-line, why ?

The reason is that tun uses sock_rps_record_flow_hash(). When I moved
all rps_record_sock_flow and rps_record_cond modifications into
include/net/rps.h, it triggered the following compilation errors due
to symbol visibility:

ERROR: modpost: "cpus_share_cache" [drivers/net/tun.ko] undefined!
ERROR: modpost: "cpus_share_cache" [net/sctp/sctp.ko] undefined!
make[2]: *** [scripts/Makefile.modpost:147: Module.symvers] Error 1

This arises because the patch uses cpus_share_cache() to limit the
RFS/ARFS update frequency at the LLC level. To keep this in the fast
path, I could move cpus_share_cache() to
include/linux/sched/topology.h.

Hmmm

If the LLC affinity check is performed in set_rps_cpu() inside
net/core/dev.c before triggering ndo_rx_flow_steer:

net/core/dev.c is built-in, so it can call cpus_share_cache() directly
without any module export issues.

sock_rps_record_flow() and sock_rps_record_flow_hash() in
include/net/rps.h remain 100% inline, retaining the zero-cost static
key NOP in the TCP/socket fast path.

No exports, no refactoring of tun.ko or sctp.ko, and no fast-path degradation.

Great idea, this indeed minimizes the changes.

Although there is a slight semantic difference compared to
net_hotdata.rps_sock_flow_table, meaning the CPU recorded in
`sock_flow_table[flow_id].ent` might not perfectly align with the one
used in `set_rps_cpu`. It is still the least intrusive approach.

How about the following simple change?

diff --git a/net/core/dev.c b/net/core/dev.c
index 3a0dd1f98084..7c513bf14221 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c

@@ -5236,7 +5236,7 @@ static int get_rps_cpu(struct net_device *dev,

struct sk_buff *skb,
                 *     have been dequeued, thus preserving in order delivery.
                 */
                if (unlikely(tcpu != next_cpu) &&
-                   (tcpu >= nr_cpu_ids || !cpu_online(tcpu) ||
+                   (tcpu >= nr_cpu_ids || !cpu_online(tcpu) ||
!rps_check_llc_affinity(tcpu, next_cpu) ||
                     ((int)(READ_ONCE(per_cpu(softnet_data,
tcpu).input_queue_head) -
                      rflow->last_qtail)) >= 0)) {
                        tcpu = next_cpu;