-----Original Message-----
From: Intel-wired-lan <redacted> On Behalf Of
kerneljasonxing@gmail.com
Sent: Wednesday, September 1, 2021 3:42 PM
To: Brandeburg, Jesse <redacted>; Nguyen, Anthony L
[off-list ref]; davem@davemloft.net; kuba@kernel.org;
ast@kernel.org; daniel@iogearbox.net; hawk@kernel.org;
john.fastabend@gmail.com; andrii@kernel.org; kafai@fb.com;
songliubraving@fb.com; yhs@fb.com; kpsingh@kernel.org
Cc: Shujin Li <redacted>; Jason Xing
[off-list ref]; kerneljasonxing@gmail.com;
netdev@vger.kernel.org; linux-kernel@vger.kernel.org; intel-wired-
lan@lists.osuosl.org; bpf@vger.kernel.org
Subject: [Intel-wired-lan] [PATCH v7] ixgbe: let the xdpdrv work with more
than 64 cpus

From: Jason Xing <redacted>

Originally, ixgbe driver doesn't allow the mounting of xdpdrv if the server is
equipped with more than 64 cpus online. So it turns out that the loading of
xdpdrv causes the "NOMEM" failure.

Actually, we can adjust the algorithm and then make it work through mapping
the current cpu to some xdp ring with the protect of @tx_lock.

Here're some numbers before/after applying this patch with xdp-example
loaded on the eth0X:

As client (tx path):
                    Before    After
TCP_STREAM send-64   734.14    714.20
TCP_STREAM send-128  1401.91   1395.05
TCP_STREAM send-512  5311.67   5292.84
TCP_STREAM send-1k   9277.40   9356.22 (not stable)
TCP_RR     send-1    22559.75  21844.22
TCP_RR     send-128  23169.54  22725.13
TCP_RR     send-512  21670.91  21412.56

As server (rx path):
                    Before    After
TCP_STREAM send-64   1416.49   1383.12
TCP_STREAM send-128  3141.49   3055.50
TCP_STREAM send-512  9488.73   9487.44
TCP_STREAM send-1k   9491.17   9356.22 (not stable)
TCP_RR     send-1    23617.74  23601.60
...

Notice: the TCP_RR mode is unstable as the official document explaines.

I tested many times with different parameters combined through netperf.
Though the result is not that accurate, I cannot see much influence on this
patch. The static key is places on the hot path, but it actually shouldn't cause a
huge regression theoretically.

Fixes: 33fdc82f08 ("ixgbe: add support for XDP_TX action")
Reported-by: kernel test robot <redacted>
Co-developed-by: Shujin Li <redacted>
Signed-off-by: Shujin Li <redacted>
Signed-off-by: Jason Xing <redacted>
---
v7:
- Factorized to a single spin_lock/unlock in ixgbe_xdp_xmit() (Eric)
- Handle other parts of lock/unlock in ixgbe_run_xdp()/_zc() (Jason)

v6:
- Move the declaration of static-key to the proper position (Test Robot)
- Add reported-by tag (Jason)
- Add more detailed performance test results (Jason)

v5:
- Change back to nr_cpu_ids (Eric)

v4:
- Update the wrong commit messages. (Jason)

v3:
- Change nr_cpu_ids to num_online_cpus() (Maciej)
- Rename MAX_XDP_QUEUES to IXGBE_MAX_XDP_QS (Maciej)
- Rename ixgbe_determine_xdp_cpu() to ixgbe_determine_xdp_q_idx()
(Maciej)
- Wrap ixgbe_xdp_ring_update_tail() with lock into one function (Maciej)

v2:
- Adjust cpu id in ixgbe_xdp_xmit(). (Jesper)
- Add a fallback path. (Maciej)
- Adjust other parts related to xdp ring.
---
drivers/net/ethernet/intel/ixgbe/ixgbe.h           | 23 +++++++++-
drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c       |  9 +++-
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c      | 50 ++++++++++++++++-----
-
.../net/ethernet/intel/ixgbe/ixgbe_txrx_common.h   |  3 +-
drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c       | 16 ++++---
5 files changed, 77 insertions(+), 24 deletions(-)

Tested-by: Sandeep Penigalapati <redacted>