[PATCH net-next 0/3] i40e, ice, ixgbe: optimize for XDP_REDIRECT in xsk path

Optimize run_xdp_zc() for the XDP program verdict being XDP_REDIRECT
in the zsk zero-copy path. This path is only used when having AF_XDP
zero-copy on and in that case most packets will be directed to user
space. This provides around 100k extra packets in throughput on my
server when running l2fwd in xdpsock.

Thanks: Magnus

Magnus Karlsson (3):
  i40: optimize for XDP_REDIRECT in xsk path
  ixgbe: optimize for XDP_REDIRECT in xsk path
  ice: optimize for XDP_REDIRECT in xsk path

 drivers/net/ethernet/intel/i40e/i40e_xsk.c   | 11 +++++++----
 drivers/net/ethernet/intel/ice/ice_xsk.c     | 12 ++++++++----
 drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 11 +++++++----
 3 files changed, 22 insertions(+), 12 deletions(-)


base-commit: 6b4f503186b73e3da24c6716c8c7ea903e6b74d4
--
2.29.0

From: Magnus Karlsson <magnus.karlsson@intel.com>

Optimize i40e_run_xdp_zc() for the XDP program verdict being
XDP_REDIRECT in the zsk zero-copy path. This path is only used when
having AF_XDP zero-copy on and in that case most packets will be
directed to user space. This provides a little over 100k extra packets
in throughput on my server when running l2fwd in xdpsock.

Signed-off-by: Magnus Karlsson <magnus.karlsson@intel.com>
---
 drivers/net/ethernet/intel/i40e/i40e_xsk.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ethernet/intel/i40e/i40e_xsk.c b/drivers/net/ethernet/intel/i40e/i40e_xsk.c
index 67febc7b6798..fbf76c67d77d 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_xsk.c
+++ b/drivers/net/ethernet/intel/i40e/i40e_xsk.c

@@ -159,6 +159,13 @@ static int i40e_run_xdp_zc(struct i40e_ring *rx_ring, struct xdp_buff *xdp)
 	xdp_prog = READ_ONCE(rx_ring->xdp_prog);
 	act = bpf_prog_run_xdp(xdp_prog, xdp);
 
+	if (likely(act == XDP_REDIRECT)) {
+		err = xdp_do_redirect(rx_ring->netdev, xdp, xdp_prog);
+		result = !err ? I40E_XDP_REDIR : I40E_XDP_CONSUMED;
+		rcu_read_unlock();
+		return result;
+	}
+
 	switch (act) {
 	case XDP_PASS:
 		break;

@@ -166,10 +173,6 @@ static int i40e_run_xdp_zc(struct i40e_ring *rx_ring, struct xdp_buff *xdp)
 		xdp_ring = rx_ring->vsi->xdp_rings[rx_ring->queue_index];
 		result = i40e_xmit_xdp_tx_ring(xdp, xdp_ring);
 		break;
-	case XDP_REDIRECT:
-		err = xdp_do_redirect(rx_ring->netdev, xdp, xdp_prog);
-		result = !err ? I40E_XDP_REDIR : I40E_XDP_CONSUMED;
-		break;
 	default:
 		bpf_warn_invalid_xdp_action(act);
 		fallthrough;

-- 
2.29.0

From: Magnus Karlsson <magnus.karlsson@intel.com>

Optimize ixgbe_run_xdp_zc() for the XDP program verdict being
XDP_REDIRECT in the zsk zero-copy path. This path is only used when
having AF_XDP zero-copy on and in that case most packets will be
directed to user space. This provides a little under 100k extra
packets in throughput on my server when running l2fwd in xdpsock.

Signed-off-by: Magnus Karlsson <magnus.karlsson@intel.com>
---
 drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c
index 3771857cf887..91ad5b902673 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c

@@ -104,6 +104,13 @@ static int ixgbe_run_xdp_zc(struct ixgbe_adapter *adapter,
 	xdp_prog = READ_ONCE(rx_ring->xdp_prog);
 	act = bpf_prog_run_xdp(xdp_prog, xdp);
 
+	if (likely(act == XDP_REDIRECT)) {
+		err = xdp_do_redirect(rx_ring->netdev, xdp, xdp_prog);
+		result = !err ? IXGBE_XDP_REDIR : IXGBE_XDP_CONSUMED;
+		rcu_read_unlock();
+		return result;
+	}
+
 	switch (act) {
 	case XDP_PASS:
 		break;

@@ -115,10 +122,6 @@ static int ixgbe_run_xdp_zc(struct ixgbe_adapter *adapter,
 		}
 		result = ixgbe_xmit_xdp_ring(adapter, xdpf);
 		break;
-	case XDP_REDIRECT:
-		err = xdp_do_redirect(rx_ring->netdev, xdp, xdp_prog);
-		result = !err ? IXGBE_XDP_REDIR : IXGBE_XDP_CONSUMED;
-		break;
 	default:
 		bpf_warn_invalid_xdp_action(act);
 		fallthrough;

-- 
2.29.0

From: Magnus Karlsson <magnus.karlsson@intel.com>

Optimize ice_run_xdp_zc() for the XDP program verdict being
XDP_REDIRECT in the zsk zero-copy path. This path is only used when
having AF_XDP zero-copy on and in that case most packets will be
directed to user space. This provides a little over 100k extra packets
in throughput on my server when running l2fwd in xdpsock.

Signed-off-by: Magnus Karlsson <magnus.karlsson@intel.com>
---
 drivers/net/ethernet/intel/ice/ice_xsk.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ethernet/intel/ice/ice_xsk.c b/drivers/net/ethernet/intel/ice/ice_xsk.c
index 797886524054..3b180e52112f 100644
--- a/drivers/net/ethernet/intel/ice/ice_xsk.c
+++ b/drivers/net/ethernet/intel/ice/ice_xsk.c

@@ -521,6 +521,14 @@ ice_run_xdp_zc(struct ice_ring *rx_ring, struct xdp_buff *xdp)
 	}
 
 	act = bpf_prog_run_xdp(xdp_prog, xdp);
+
+	if (likely(act == XDP_REDIRECT)) {
+		err = xdp_do_redirect(rx_ring->netdev, xdp, xdp_prog);
+		result = !err ? ICE_XDP_REDIR : ICE_XDP_CONSUMED;
+		rcu_read_unlock();
+		return result;
+	}
+
 	switch (act) {
 	case XDP_PASS:
 		break;

@@ -528,10 +536,6 @@ ice_run_xdp_zc(struct ice_ring *rx_ring, struct xdp_buff *xdp)
 		xdp_ring = rx_ring->vsi->xdp_rings[rx_ring->q_index];
 		result = ice_xmit_xdp_buff(xdp, xdp_ring);
 		break;
-	case XDP_REDIRECT:
-		err = xdp_do_redirect(rx_ring->netdev, xdp, xdp_prog);
-		result = !err ? ICE_XDP_REDIR : ICE_XDP_CONSUMED;
-		break;
 	default:
 		bpf_warn_invalid_xdp_action(act);
 		fallthrough;

-- 
2.29.0

On Wed, Dec 02, 2020 at 04:07:21PM +0100, Magnus Karlsson wrote:

Optimize run_xdp_zc() for the XDP program verdict being XDP_REDIRECT
in the zsk zero-copy path. This path is only used when having AF_XDP
zero-copy on and in that case most packets will be directed to user
space. This provides around 100k extra packets in throughput on my
server when running l2fwd in xdpsock.

Thanks: Magnus

For series:
Acked-by: Maciej Fijalkowski <maciej.fijalkowski@intel.com>

You only ate 'e' from i40e subject line.

Magnus Karlsson (3):
  i40: optimize for XDP_REDIRECT in xsk path
  ixgbe: optimize for XDP_REDIRECT in xsk path
  ice: optimize for XDP_REDIRECT in xsk path

 drivers/net/ethernet/intel/i40e/i40e_xsk.c   | 11 +++++++----
 drivers/net/ethernet/intel/ice/ice_xsk.c     | 12 ++++++++----
 drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 11 +++++++----
 3 files changed, 22 insertions(+), 12 deletions(-)


base-commit: 6b4f503186b73e3da24c6716c8c7ea903e6b74d4
--
2.29.0

On Wed, Dec 2, 2020 at 9:49 PM Maciej Fijalkowski
[off-list ref] wrote:

On Wed, Dec 02, 2020 at 04:07:21PM +0100, Magnus Karlsson wrote:

quoted

Optimize run_xdp_zc() for the XDP program verdict being XDP_REDIRECT
in the zsk zero-copy path. This path is only used when having AF_XDP
zero-copy on and in that case most packets will be directed to user
space. This provides around 100k extra packets in throughput on my
server when running l2fwd in xdpsock.

Thanks: Magnus

For series:
Acked-by: Maciej Fijalkowski <maciej.fijalkowski@intel.com>

You only ate 'e' from i40e subject line.

Sorry, you are right. Tony, would you please be so kind to add this
missing "e" in the commit message before you send the pull request?

Thanks: Magnus

quoted

Magnus Karlsson (3):
  i40: optimize for XDP_REDIRECT in xsk path
  ixgbe: optimize for XDP_REDIRECT in xsk path
  ice: optimize for XDP_REDIRECT in xsk path

 drivers/net/ethernet/intel/i40e/i40e_xsk.c   | 11 +++++++----
 drivers/net/ethernet/intel/ice/ice_xsk.c     | 12 ++++++++----
 drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 11 +++++++----
 3 files changed, 22 insertions(+), 12 deletions(-)


base-commit: 6b4f503186b73e3da24c6716c8c7ea903e6b74d4
--
2.29.0

On Thu, 2020-12-03 at 07:58 +0100, Magnus Karlsson wrote:

On Wed, Dec 2, 2020 at 9:49 PM Maciej Fijalkowski
[off-list ref] wrote:

quoted

On Wed, Dec 02, 2020 at 04:07:21PM +0100, Magnus Karlsson wrote:

quoted

Optimize run_xdp_zc() for the XDP program verdict being
XDP_REDIRECT
in the zsk zero-copy path. This path is only used when having
AF_XDP
zero-copy on and in that case most packets will be directed to
user
space. This provides around 100k extra packets in throughput on
my
server when running l2fwd in xdpsock.

Thanks: Magnus

For series:
Acked-by: Maciej Fijalkowski <maciej.fijalkowski@intel.com>

You only ate 'e' from i40e subject line.

Sorry, you are right. Tony, would you please be so kind to add this
missing "e" in the commit message before you send the pull request?

Thanks: Magnus

I can do that.

Thanks,
Tony

quoted

quoted

Magnus Karlsson (3):
  i40: optimize for XDP_REDIRECT in xsk path
  ixgbe: optimize for XDP_REDIRECT in xsk path
  ice: optimize for XDP_REDIRECT in xsk path

 drivers/net/ethernet/intel/i40e/i40e_xsk.c   | 11 +++++++----
 drivers/net/ethernet/intel/ice/ice_xsk.c     | 12 ++++++++----
 drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 11 +++++++----
 3 files changed, 22 insertions(+), 12 deletions(-)


base-commit: 6b4f503186b73e3da24c6716c8c7ea903e6b74d4
--
2.29.0

From: Intel-wired-lan <redacted> On Behalf Of Magnus
Karlsson
Sent: Wednesday, December 2, 2020 8:37 PM
To: Karlsson, Magnus <magnus.karlsson@intel.com>; Topel, Bjorn
[off-list ref]; intel-wired-lan@lists.osuosl.org; Nguyen, Anthony L
[off-list ref]
Cc: netdev@vger.kernel.org; maciejromanfijalkowski@gmail.com
Subject: [Intel-wired-lan] [PATCH net-next 3/3] ice: optimize for XDP_REDIRECT in
xsk path

From: Magnus Karlsson <magnus.karlsson@intel.com>

Optimize ice_run_xdp_zc() for the XDP program verdict being XDP_REDIRECT in
the zsk zero-copy path. This path is only used when having AF_XDP zero-copy on
and in that case most packets will be directed to user space. This provides a little
over 100k extra packets in throughput on my server when running l2fwd in
xdpsock.

Signed-off-by: Magnus Karlsson <magnus.karlsson@intel.com>
---
 drivers/net/ethernet/intel/ice/ice_xsk.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

Tested-by: George Kuruvinakunnel <redacted>

From: Intel-wired-lan <redacted> On Behalf Of Magnus
Karlsson
Sent: Wednesday, December 2, 2020 8:37 PM
To: Karlsson, Magnus <magnus.karlsson@intel.com>; Topel, Bjorn
[off-list ref]; intel-wired-lan@lists.osuosl.org; Nguyen, Anthony L
[off-list ref]
Cc: netdev@vger.kernel.org; maciejromanfijalkowski@gmail.com
Subject: [Intel-wired-lan] [PATCH net-next 1/3] i40: optimize for XDP_REDIRECT
in xsk path

From: Magnus Karlsson <magnus.karlsson@intel.com>

Optimize i40e_run_xdp_zc() for the XDP program verdict being XDP_REDIRECT
in the zsk zero-copy path. This path is only used when having AF_XDP zero-copy
on and in that case most packets will be directed to user space. This provides a
little over 100k extra packets in throughput on my server when running l2fwd in
xdpsock.

Signed-off-by: Magnus Karlsson <magnus.karlsson@intel.com>
---
 drivers/net/ethernet/intel/i40e/i40e_xsk.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

Tested-by: George Kuruvinakunnel <redacted>

From: Intel-wired-lan <redacted> On Behalf Of
Magnus Karlsson
Sent: Wednesday, December 2, 2020 8:37 PM
To: Karlsson, Magnus <magnus.karlsson@intel.com>; Topel, Bjorn
[off-list ref]; intel-wired-lan@lists.osuosl.org; Nguyen, Anthony L
[off-list ref]
Cc: netdev@vger.kernel.org; maciejromanfijalkowski@gmail.com
Subject: [Intel-wired-lan] [PATCH net-next 2/3] ixgbe: optimize for
XDP_REDIRECT in xsk path

From: Magnus Karlsson <magnus.karlsson@intel.com>

Optimize ixgbe_run_xdp_zc() for the XDP program verdict being XDP_REDIRECT
in the zsk zero-copy path. This path is only used when having AF_XDP zero-copy
on and in that case most packets will be directed to user space. This provides a
little under 100k extra packets in throughput on my server when running l2fwd in
xdpsock.

Signed-off-by: Magnus Karlsson <magnus.karlsson@intel.com>
---
 drivers/net/ethernet/intel/ixgbe/ixgbe_xsk.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

Tested-by: Vishakha Jambekar <redacted>