Re: [PATCH net-next v4 0/4] Add support to do threaded napi busy poll

On 2025-03-26 16:34, Samiullah Khawaja wrote:

On Tue, Mar 25, 2025 at 10:47 AM Martin Karsten [off-list ref] wrote:

quoted

On 2025-03-25 12:40, Samiullah Khawaja wrote:

quoted

On Sun, Mar 23, 2025 at 7:38 PM Martin Karsten [off-list ref] wrote:

quoted

On 2025-03-20 22:15, Samiullah Khawaja wrote:

quoted

Extend the already existing support of threaded napi poll to do continuous
busy polling.

This is used for doing continuous polling of napi to fetch descriptors
from backing RX/TX queues for low latency applications. Allow enabling
of threaded busypoll using netlink so this can be enabled on a set of
dedicated napis for low latency applications.

Once enabled user can fetch the PID of the kthread doing NAPI polling
and set affinity, priority and scheduler for it depending on the
low-latency requirements.

Currently threaded napi is only enabled at device level using sysfs. Add
support to enable/disable threaded mode for a napi individually. This
can be done using the netlink interface. Extend `napi-set` op in netlink
spec that allows setting the `threaded` attribute of a napi.

Extend the threaded attribute in napi struct to add an option to enable
continuous busy polling. Extend the netlink and sysfs interface to allow
enabling/disabling threaded busypolling at device or individual napi
level.

We use this for our AF_XDP based hard low-latency usecase with usecs
level latency requirement. For our usecase we want low jitter and stable
latency at P99.

Following is an analysis and comparison of available (and compatible)
busy poll interfaces for a low latency usecase with stable P99. Please
note that the throughput and cpu efficiency is a non-goal.

For analysis we use an AF_XDP based benchmarking tool `xdp_rr`. The
description of the tool and how it tries to simulate the real workload
is following,

- It sends UDP packets between 2 machines.
- The client machine sends packets at a fixed frequency. To maintain the
     frequency of the packet being sent, we use open-loop sampling. That is
     the packets are sent in a separate thread.
- The server replies to the packet inline by reading the pkt from the
     recv ring and replies using the tx ring.
- To simulate the application processing time, we use a configurable
     delay in usecs on the client side after a reply is received from the
     server.

The xdp_rr tool is posted separately as an RFC for tools/testing/selftest.

Thanks very much for sending the benchmark program and these specific
experiments. I am able to build the tool and run the experiments in
principle. While I don't have a complete picture yet, one observation
seems already clear, so I want to report back on it.

Thanks for reproducing this Martin. Really appreciate you reviewing
this and your interest in this.

quoted

quoted

We use this tool with following napi polling configurations,

- Interrupts only
- SO_BUSYPOLL (inline in the same thread where the client receives the
     packet).
- SO_BUSYPOLL (separate thread and separate core)
- Threaded NAPI busypoll

The configurations that you describe as SO_BUSYPOLL here are not using
the best busy-polling configuration. The best busy-polling strictly
alternates between application processing and network polling. No
asynchronous processing due to hardware irq delivery or softirq
processing should happen.

A high-level check is making sure that no softirq processing is reported
for the relevant cores (see, e.g., "%soft" in sar -P <cores> -u ALL 1).
In addition, interrupts can be counted in /proc/stat or /proc/interrupts.

Unfortunately it is not always straightforward to enter this pattern. In
this particular case, it seems that two pieces are missing:

1) Because the XPD socket is created with XDP_COPY, it is never marked
with its corresponding napi_id. Without the socket being marked with a
valid napi_id, sk_busy_loop (called from __xsk_recvmsg) never invokes
napi_busy_loop. Instead the gro_flush_timeout/napi_defer_hard_irqs
softirq loop controls packet delivery.

Nice catch. It seems a recent change broke the busy polling for AF_XDP
and there was a fix for the XDP_ZEROCOPY but the XDP_COPY remained
broken and seems in my experiments I didn't pick that up. During my
experimentation I confirmed that all experiment modes are invoking the
busypoll and not going through softirqs. I confirmed this through perf
traces. I sent out a fix for XDP_COPY busy polling here in the link
below. I will resent this for the net since the original commit has
already landed in 6.13.
https://lore.kernel.org/netdev/CAAywjhSEjaSgt7fCoiqJiMufGOi=oxa164_vTfk+3P43H60qwQ@mail.gmail.com/T/#t (local)

quoted

I found code at the end of xsk_bind in xsk.c that is conditional on xs->zc:

          if (xs->zc && qid < dev->real_num_rx_queues) {
                  struct netdev_rx_queue *rxq;

                  rxq = __netif_get_rx_queue(dev, qid);
                  if (rxq->napi)
                          __sk_mark_napi_id_once(sk, rxq->napi->napi_id);
          }

I am not an expert on XDP sockets, so I don't know why that is or what
would be an acceptable workaround/fix, but when I simply remove the
check for xs->zc, the socket is being marked and napi_busy_loop is being
called. But maybe there's a better way to accomplish this.

+1

quoted

2) SO_PREFER_BUSY_POLL needs to be set on the XDP socket to make sure
that busy polling stays in control after napi_busy_loop, regardless of
how many packets were found. Without this setting, the gro_flush_timeout
timer is not extended in busy_poll_stop.

With these two changes, both SO_BUSYPOLL alternatives perform noticeably
better in my experiments and come closer to Threaded NAPI busypoll, so I
was wondering if you could try that in your environment? While this
might not change the big picture, I think it's important to fully
understand and document the trade-offs.

I agree. In my experiments the SO_BUSYPOLL works properly, please see
the commit I mentioned above. But I will experiment with
SO_PREFER_BUSY_POLL to see whether it makes any significant change.

I'd like to clarify: Your original experiments cannot have used
busypoll, because it was broken for XDP_COPY. Did you rerun the

On my idpf test platform the AF_XDP support is broken with the latest
kernel, so I didn't have the original commit that broke AF_XDP
busypoll for zerocopy and copy mode. So in the experiments that I
shared XDP_COPY busy poll has been working. Please see the traces
below. Sorry for the confusion.

Ok, that explains it.

quoted

experiments with the XDP_COPY fix but without SO_PREFER_BUSY_POLL and

I tried with SO_PREFER_BUSY_POLL as you suggested, I see results
matching the previous observation:

12Kpkts/sec with 78usecs delay:

INLINE:
p5: 16700
p50: 17100
p95: 17200
p99: 17200

This comment applies to the experiments overall: I believe these 
carefully crafted period/delay configurations that just straddle the 
capacity limit do not show any additional benefits over and above what 
the basic experiments (without application delay) already show.

If you want to illustrate the fact that the slightly faster mechanism 
reaches capacity a little later, I would find experiments with a fixed 
period and varying the delay from 0 to overload more illustrative.

quoted

see the same latency numbers as before? Also, can you provide more
details about the perf tracing that you used to see that busypoll is
invoked, but softirq is not?

I used the following command to record the call graph and could see
the calls to napi_busy_loop going from xsk_rcvmsg. Confirmed with
SO_PREFER_BUSY_POLL also below,

perf record -o prefer.perf -a -e cycles -g sleep 10
perf report --stdio -i prefer.perf

  --1.35%--entry_SYSCALL_64
             |
              --1.31%--do_syscall_64
                        __x64_sys_recvfrom
                        __sys_recvfrom
                        sock_recvmsg
                        xsk_recvmsg
                        __xsk_recvmsg.constprop.0.isra.0
                        napi_busy_loop
                        __napi_busy_loop

I do see softirq getting triggered occasionally, when inline the
busy_poll thread is not able to pick up the packets. I used following
command to find number of samples for each in the trace,

perf report -g -n -i prefer.perf

Filtered the results to include only the interesting symbols

<
Children      Self       Samples  Command          Shared Object
           Symbol
+    1.48%     0.06%            46  xsk_rr           [idpf]
             [k] idpf_vport_splitq_napi_poll

+    1.28%     0.11%            86  xsk_rr           [kernel.kallsyms]
             [k] __napi_busy_loop

+    0.71%     0.02%            17  xsk_rr           [kernel.kallsyms]
             [k] net_rx_action

+    0.69%     0.01%             6  xsk_rr           [kernel.kallsyms]
             [k] __napi_poll

Thanks, this makes me realize that I forgot to mention something as well:

SO_PREFER_BUSY_POLL should eliminate the remaining softirq invocations, 
but only if gro_flush_timeout is big enough. In fact, in a full busypoll 
configuration, the value of gro_flush_timeout should not matter at all 
as long as its sufficiently higher than the application period. I have 
set it to 1000000 for these experiments as another litmus test that 
busypoll is actually working.

Last not least, I found that co-locating the single-threaded 
busy-polling application with the irq core improved the outcome. I.e., 
in your experiment setup you would taskset the application to Core 2. 
Not sure I have a rock-solid explanation, but it did make a difference.

Since net-next patch submission is closed, I thought I provide this 
feedback now, so you can decide whether to take it into account for the 
next go-around.

Best,
Martin