From: Tony Nguyen <anthony.l.nguyen@intel.com> Date: 2021-12-28 17:59:04
Alexander Lobakin says:
napi_build_skb() I introduced earlier this year ([0]) aims
to decrease MM pressure and the overhead from in-place
kmem_cache_alloc() on each Rx entry processing by decaching
skbuff_heads from NAPI per-cpu cache filled prior to that by
napi_consume_skb() (so it is sort of a direct shortcut for
free -> mm -> alloc cycle).
Currently, no in-tree drivers use it. Switch all Intel Ethernet
drivers to it to get slight-to-medium perf boosts depending on
the frame size.
ice driver, 50 Gbps link, pktgen + XDP_PASS (local in) sample:
frame_size/nthreads 64/42 128/20 256/8 512/4 1024/2 1532/1
net-next (Kpps) 46062 34654 18248 9830 5343 2714
series 47438 34708 18330 9875 5435 2777
increase 2.9% 0.15% 0.45% 0.46% 1.72% 2.32%
Additionally, e1000's been switched to napi_consume_skb() as it's
safe and works fine there, and there's no point in napi_build_skb()
without paired NAPI cache feeding point.
[0] https://lore.kernel.org/all/20210213141021.87840-1-alobakin@pm.me
The following are changes since commit 0f1eae8e565e632f64670a5730894f22819fcaad:
net: caif: remove redundant assignment to variable expectlen
and are available in the git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/tnguy/next-queue 10GbE
Alexander Lobakin (9):
e1000: switch to napi_consume_skb()
e1000: switch to napi_build_skb()
i40e: switch to napi_build_skb()
iavf: switch to napi_build_skb()
ice: switch to napi_build_skb()
igb: switch to napi_build_skb()
igc: switch to napi_build_skb()
ixgbe: switch to napi_build_skb()
ixgbevf: switch to napi_build_skb()
drivers/net/ethernet/intel/e1000/e1000_main.c | 14 ++++++++------
drivers/net/ethernet/intel/i40e/i40e_txrx.c | 2 +-
drivers/net/ethernet/intel/iavf/iavf_txrx.c | 2 +-
drivers/net/ethernet/intel/ice/ice_txrx.c | 2 +-
drivers/net/ethernet/intel/igb/igb_main.c | 2 +-
drivers/net/ethernet/intel/igc/igc_main.c | 2 +-
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 2 +-
drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c | 2 +-
8 files changed, 15 insertions(+), 13 deletions(-)
--
2.31.1
From: Tony Nguyen <anthony.l.nguyen@intel.com> Date: 2021-12-28 17:59:05
From: Alexander Lobakin <redacted>
In order to take the best from per-cpu NAPI skbuff_head caches and
CPU cycles, let's switch from dev_kfree_skb_any(), which passes skb
back to the mm layer, to napi_consume_skb(), which feeds those
caches on non-zero budget instead (falls back to the former on 0).
Do the replacement in e1000_unmap_and_free_tx_resource(). There are
4 call sites of this function throughout the driver:
* e1000_clean_tx_ring(). Slowpath, process context, cleans the
whole Tx ring on ifdown. Use budget of 0 here;
* e1000_tx_map(). Hotpath, net Tx softirq, unmaps the buffers in
case of error. Use 0 as well;
* e1000_clean_tx_irq(). Hotpath, NAPI Tx completion polling cycle.
As the driver doesn't count completed Tx entries towards the NAPI
budget, just use the poll budget of 64 to utilize caches.
Apart from being a preparation for switching to napi_build_skb(),
this is useful on its own as well, as napi_consume_skb() flushes
skb caches by batches of 32 instead of one-at-a-time.
Signed-off-by: Alexander Lobakin <redacted>
Reviewed-by: Michal Swiatkowski <redacted>
Tested-by: Tony Brelinski <redacted>
Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
---
drivers/net/ethernet/intel/e1000/e1000_main.c | 12 +++++++-----
1 file changed, 7 insertions(+), 5 deletions(-)
From: Tony Nguyen <anthony.l.nguyen@intel.com> Date: 2021-12-28 17:59:06
From: Alexander Lobakin <redacted>
napi_build_skb() reuses per-cpu NAPI skbuff_head cache in order
to save some cycles on freeing/allocating skbuff_heads on every
new Rx or completed Tx element.
e1000 driver runs Tx completion polling cycle right before the Rx
one. Now that e1000 uses napi_consume_skb() to put skbuff_heads of
completed entries into the cache, it will never empty and always
warm at that moment. Switch to the napi_build_skb() to relax mm
pressure on heavy Rx and increase throughput.
Signed-off-by: Alexander Lobakin <redacted>
Reviewed-by: Michal Swiatkowski <redacted>
Tested-by: Tony Brelinski <redacted>
Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
---
drivers/net/ethernet/intel/e1000/e1000_main.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
From: Tony Nguyen <anthony.l.nguyen@intel.com> Date: 2021-12-28 17:59:08
From: Alexander Lobakin <redacted>
napi_build_skb() reuses per-cpu NAPI skbuff_head cache in order
to save some cycles on freeing/allocating skbuff_heads on every
new Rx or completed Tx.
i40e driver runs Tx completion polling cycle right before the Rx
one and uses napi_consume_skb() to feed the cache with skbuff_heads
of completed entries, so it's never empty and always warm at that
moment. Switch to the napi_build_skb() to relax mm pressure on
heavy Rx.
Signed-off-by: Alexander Lobakin <redacted>
Reviewed-by: Michal Swiatkowski <redacted>
Tested-by: Gurucharan G <redacted>
Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
---
drivers/net/ethernet/intel/i40e/i40e_txrx.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
From: Tony Nguyen <anthony.l.nguyen@intel.com> Date: 2021-12-28 17:59:08
From: Alexander Lobakin <redacted>
napi_build_skb() reuses per-cpu NAPI skbuff_head cache in order
to save some cycles on freeing/allocating skbuff_heads on every
new Rx or completed Tx.
iavf driver runs Tx completion polling cycle right before the Rx
one and uses napi_consume_skb() to feed the cache with skbuff_heads
of completed entries, so it's never empty and always warm at that
moment. Switch to the napi_build_skb() to relax mm pressure on
heavy Rx.
Signed-off-by: Alexander Lobakin <redacted>
Reviewed-by: Michal Swiatkowski <redacted>
Tested-by: Konrad Jankowski <redacted>
Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
---
drivers/net/ethernet/intel/iavf/iavf_txrx.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
From: Tony Nguyen <anthony.l.nguyen@intel.com> Date: 2021-12-28 17:59:08
From: Alexander Lobakin <redacted>
napi_build_skb() reuses per-cpu NAPI skbuff_head cache in order
to save some cycles on freeing/allocating skbuff_heads on every
new Rx or completed Tx.
ice driver runs Tx completion polling cycle right before the Rx
one and uses napi_consume_skb() to feed the cache with skbuff_heads
of completed entries, so it's never empty and always warm at that
moment. Switch to the napi_build_skb() to relax mm pressure on
heavy Rx.
Signed-off-by: Alexander Lobakin <redacted>
Reviewed-by: Michal Swiatkowski <redacted>
Tested-by: Gurucharan G <redacted>
Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
---
drivers/net/ethernet/intel/ice/ice_txrx.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
From: Tony Nguyen <anthony.l.nguyen@intel.com> Date: 2021-12-28 17:59:10
From: Alexander Lobakin <redacted>
napi_build_skb() reuses per-cpu NAPI skbuff_head cache in order
to save some cycles on freeing/allocating skbuff_heads on every
new Rx or completed Tx.
igb driver runs Tx completion polling cycle right before the Rx
one and uses napi_consume_skb() to feed the cache with skbuff_heads
of completed entries, so it's never empty and always warm at that
moment. Switch to the napi_build_skb() to relax mm pressure on
heavy Rx.
Signed-off-by: Alexander Lobakin <redacted>
Reviewed-by: Michal Swiatkowski <redacted>
Tested-by: Gurucharan G <redacted>
Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
---
drivers/net/ethernet/intel/igb/igb_main.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
From: Tony Nguyen <anthony.l.nguyen@intel.com> Date: 2021-12-28 17:59:10
From: Alexander Lobakin <redacted>
napi_build_skb() reuses per-cpu NAPI skbuff_head cache in order
to save some cycles on freeing/allocating skbuff_heads on every
new Rx or completed Tx.
igc driver runs Tx completion polling cycle right before the Rx
one and uses napi_consume_skb() to feed the cache with skbuff_heads
of completed entries, so it's never empty and always warm at that
moment. Switch to the napi_build_skb() to relax mm pressure on
heavy Rx.
Signed-off-by: Alexander Lobakin <redacted>
Reviewed-by: Michal Swiatkowski <redacted>
Tested-by: Nechama Kraus <redacted>
Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
---
drivers/net/ethernet/intel/igc/igc_main.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
From: Tony Nguyen <anthony.l.nguyen@intel.com> Date: 2021-12-28 17:59:11
From: Alexander Lobakin <redacted>
napi_build_skb() reuses per-cpu NAPI skbuff_head cache in order
to save some cycles on freeing/allocating skbuff_heads on every
new Rx or completed Tx.
ixgbe driver runs Tx completion polling cycle right before the Rx
one and uses napi_consume_skb() to feed the cache with skbuff_heads
of completed entries, so it's never empty and always warm at that
moment. Switch to the napi_build_skb() to relax mm pressure on
heavy Rx.
Signed-off-by: Alexander Lobakin <redacted>
Reviewed-by: Michal Swiatkowski <redacted>
Tested-by: Gurucharan G <redacted>
Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
---
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
@@ -2170,7 +2170,7 @@ static struct sk_buff *ixgbe_build_skb(struct ixgbe_ring *rx_ring,net_prefetch(xdp->data_meta);/* build an skb to around the page buffer */-skb=build_skb(xdp->data_hard_start,truesize);+skb=napi_build_skb(xdp->data_hard_start,truesize);if(unlikely(!skb))returnNULL;
From: Tony Nguyen <anthony.l.nguyen@intel.com> Date: 2021-12-28 17:59:11
From: Alexander Lobakin <redacted>
napi_build_skb() reuses per-cpu NAPI skbuff_head cache in order
to save some cycles on freeing/allocating skbuff_heads on every
new Rx or completed Tx.
ixgbevf driver runs Tx completion polling cycle right before the Rx
one and uses napi_consume_skb() to feed the cache with skbuff_heads
of completed entries, so it's never empty and always warm at that
moment. Switch to the napi_build_skb() to relax mm pressure on
heavy Rx.
Signed-off-by: Alexander Lobakin <redacted>
Reviewed-by: Michal Swiatkowski <redacted>
Tested-by: Konrad Jankowski <redacted>
Signed-off-by: Tony Nguyen <anthony.l.nguyen@intel.com>
---
drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
Hello:
This series was applied to netdev/net-next.git (master)
by Tony Nguyen [off-list ref]:
On Tue, 28 Dec 2021 09:58:06 -0800 you wrote:
Alexander Lobakin says:
napi_build_skb() I introduced earlier this year ([0]) aims
to decrease MM pressure and the overhead from in-place
kmem_cache_alloc() on each Rx entry processing by decaching
skbuff_heads from NAPI per-cpu cache filled prior to that by
napi_consume_skb() (so it is sort of a direct shortcut for
free -> mm -> alloc cycle).
Currently, no in-tree drivers use it. Switch all Intel Ethernet
drivers to it to get slight-to-medium perf boosts depending on
the frame size.
[...]