[RFT net-next] net: ti: add pp skb recycling support

Subsystems: networking drivers, the rest, ti ethernet switch driver (cpsw)

8 messages, 6 authors, 2021-06-09 · open the first message on its own page

[RFT net-next] net: ti: add pp skb recycling support

From: Lorenzo Bianconi <lorenzo@kernel.org>
Date: 2021-06-09 12:01:41

As already done for mvneta and mvpp2, enable skb recycling for ti
ethernet drivers

Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
---
This patch has just compile-tested
---
 drivers/net/ethernet/ti/cpsw.c     | 4 ++--
 drivers/net/ethernet/ti/cpsw_new.c | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/drivers/net/ethernet/ti/cpsw.c b/drivers/net/ethernet/ti/cpsw.c
index c0cd7de88316..049508667a6d 100644
--- a/drivers/net/ethernet/ti/cpsw.c
+++ b/drivers/net/ethernet/ti/cpsw.c
@@ -430,8 +430,8 @@ static void cpsw_rx_handler(void *token, int len, int status)
 		cpts_rx_timestamp(cpsw->cpts, skb);
 	skb->protocol = eth_type_trans(skb, ndev);
 
-	/* unmap page as no netstack skb page recycling */
-	page_pool_release_page(pool, page);
+	/* mark skb for recycling */
+	skb_mark_for_recycle(skb, page, pool);
 	netif_receive_skb(skb);
 
 	ndev->stats.rx_bytes += len;
diff --git a/drivers/net/ethernet/ti/cpsw_new.c b/drivers/net/ethernet/ti/cpsw_new.c
index 69b7a4e0220a..11f536138495 100644
--- a/drivers/net/ethernet/ti/cpsw_new.c
+++ b/drivers/net/ethernet/ti/cpsw_new.c
@@ -373,8 +373,8 @@ static void cpsw_rx_handler(void *token, int len, int status)
 		cpts_rx_timestamp(cpsw->cpts, skb);
 	skb->protocol = eth_type_trans(skb, ndev);
 
-	/* unmap page as no netstack skb page recycling */
-	page_pool_release_page(pool, page);
+	/* mark skb for recycling */
+	skb_mark_for_recycle(skb, page, pool);
 	netif_receive_skb(skb);
 
 	ndev->stats.rx_bytes += len;
-- 
2.31.1

Re: [RFT net-next] net: ti: add pp skb recycling support

From: Matteo Croce <hidden>
Date: 2021-06-09 12:21:38

On Wed, Jun 9, 2021 at 2:01 PM Lorenzo Bianconi [off-list ref] wrote:
As already done for mvneta and mvpp2, enable skb recycling for ti
ethernet drivers

Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
Looks good! If someone with the HW could provide a with and without
the patch, that would be nice!

-- 
per aspera ad upstream

Re: [RFT net-next] net: ti: add pp skb recycling support

From: Grygorii Strashko <grygorii.strashko@ti.com>
Date: 2021-06-09 15:03:17

hi

On 09/06/2021 15:20, Matteo Croce wrote:
On Wed, Jun 9, 2021 at 2:01 PM Lorenzo Bianconi [off-list ref] wrote:
quoted
As already done for mvneta and mvpp2, enable skb recycling for ti
ethernet drivers

Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
Looks good! If someone with the HW could provide a with and without
the patch, that would be nice!
What test would you recommend to run?

-- 
Best regards,
grygorii

Re: [RFT net-next] net: ti: add pp skb recycling support

From: Lorenzo Bianconi <hidden>
Date: 2021-06-09 15:12:34

hi

On 09/06/2021 15:20, Matteo Croce wrote:
quoted
On Wed, Jun 9, 2021 at 2:01 PM Lorenzo Bianconi [off-list ref] wrote:
quoted
As already done for mvneta and mvpp2, enable skb recycling for ti
ethernet drivers

Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
Looks good! If someone with the HW could provide a with and without
the patch, that would be nice!
What test would you recommend to run?
Hi Grygorii,

Just pass the skb to the stack, so I think regular XDP_PASS use case is enough.

Regards,
Lorenzo
-- 
Best regards,
grygorii

Re: [RFT net-next] net: ti: add pp skb recycling support

From: Matteo Croce <hidden>
Date: 2021-06-09 15:44:40

On Wed, Jun 9, 2021 at 5:03 PM Grygorii Strashko
[off-list ref] wrote:
hi

On 09/06/2021 15:20, Matteo Croce wrote:
quoted
On Wed, Jun 9, 2021 at 2:01 PM Lorenzo Bianconi [off-list ref] wrote:
quoted
As already done for mvneta and mvpp2, enable skb recycling for ti
ethernet drivers

Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
Looks good! If someone with the HW could provide a with and without
the patch, that would be nice!
What test would you recommend to run?

--
Best regards,
grygorii
Hi Grygorii,

A test which benefits most from this kind of change is one in which
the frames are freed early.
One option would be to use mausezahn which by default sends frames
with an invalid ethertype, that are dropped very early from the stack
(I think in __netif_receive_skb_core() or near there).
Then, on the device I just watch the device statistics to count
packets per second:

mausezahn eth0 -c 0 -b $board_mac_address

This test should be precise enough for a gigabit link, on faster links
I usually use a DPDK or AF_XDP based one.

Regards,
-- 
per aspera ad upstream

Re: [RFT net-next] net: ti: add pp skb recycling support

From: Jesper Dangaard Brouer <hidden>
Date: 2021-06-09 15:55:47

On Wed, 9 Jun 2021 17:43:57 +0200
Matteo Croce [off-list ref] wrote:
On Wed, Jun 9, 2021 at 5:03 PM Grygorii Strashko
[off-list ref] wrote:
quoted
On 09/06/2021 15:20, Matteo Croce wrote:  
quoted
On Wed, Jun 9, 2021 at 2:01 PM Lorenzo Bianconi [off-list ref] wrote:  
quoted
As already done for mvneta and mvpp2, enable skb recycling for ti
ethernet drivers

Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>  
Looks good! If someone with the HW could provide a with and without
the patch, that would be nice!
 
What test would you recommend to run?
[...]
A test which benefits most from this kind of change is one in which
the frames are freed early.
I would also recommend running an XDP_PASS program, and then running
something that let the packets travel as deep as possible into netstack.
Not to test performance, but to make sure we didn't break something!

I've hacked up bnxt driver (it's not as straight forward as this driver
to convert) and is running some TCP tests.  Not problems so-far :-0

I wanted to ask if someone knows howto setup the zero-copy TCP stuff
that google did? (but is that TX only?)

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

Re: [RFT net-next] net: ti: add pp skb recycling support

From: Eric Dumazet <hidden>
Date: 2021-06-09 16:58:54


On 6/9/21 5:55 PM, Jesper Dangaard Brouer wrote:
On Wed, 9 Jun 2021 17:43:57 +0200
Matteo Croce [off-list ref] wrote:
quoted
On Wed, Jun 9, 2021 at 5:03 PM Grygorii Strashko
[off-list ref] wrote:
quoted
On 09/06/2021 15:20, Matteo Croce wrote:  
quoted
On Wed, Jun 9, 2021 at 2:01 PM Lorenzo Bianconi [off-list ref] wrote:  
quoted
As already done for mvneta and mvpp2, enable skb recycling for ti
ethernet drivers

Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>  
Looks good! If someone with the HW could provide a with and without
the patch, that would be nice!
 
What test would you recommend to run?
[...]
quoted
A test which benefits most from this kind of change is one in which
the frames are freed early.
I would also recommend running an XDP_PASS program, and then running
something that let the packets travel as deep as possible into netstack.
Not to test performance, but to make sure we didn't break something!

I've hacked up bnxt driver (it's not as straight forward as this driver
to convert) and is running some TCP tests.  Not problems so-far :-0

I wanted to ask if someone knows howto setup the zero-copy TCP stuff
that google did? (but is that TX only?)
TX does not need anything fancy really.

For RX part, you can look at https://netdevconf.info/0x14/pub/slides/62/Implementing%20TCP%20RX%20zero%20copy.pdf

Re: [RFT net-next] net: ti: add pp skb recycling support

From: Grygorii Strashko <grygorii.strashko@ti.com>
Date: 2021-06-09 17:03:52


On 09/06/2021 15:20, Matteo Croce wrote:
On Wed, Jun 9, 2021 at 2:01 PM Lorenzo Bianconi [off-list ref] wrote:
quoted
As already done for mvneta and mvpp2, enable skb recycling for ti
ethernet drivers

Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
Looks good! If someone with the HW could provide a with and without
the patch, that would be nice!
Not sure to which mail to answer, so answering here - thanks all

1) I've simulated packet drop using iperf
Host:
- arp -s <some IP> <unknown MAC>
- iperf -c <some IP> -u -l60 -b700M -t60 -i1

DUP:
- place interface in promisc mode
- check rx_packets stats

I see big improvement ~47Kpps vs ~64Kpps

2) I've run iperf3 tests - see no regressions, but also not too much improvements

3) I've applied 2 patches
- this one
- and [1]

Results are below, Thank you

Tested-by: Grygorii Strashko <grygorii.strashko@ti.com>
Reviewed-by: Grygorii Strashko <grygorii.strashko@ti.com>

[1] https://patchwork.kernel.org/project/netdevbpf/patch/20210609103326.278782-18-toke@redhat.com/

=========== Before:
[perf top]
  47.15%  [kernel]                   [k] _raw_spin_unlock_irqrestore
   11.77%  [kernel]                   [k] __cpdma_chan_free
    3.16%  [kernel]                   [k] ___bpf_prog_run
    2.52%  [kernel]                   [k] cpsw_rx_vlan_encap
    2.34%  [kernel]                   [k] __netif_receive_skb_core
    2.27%  [kernel]                   [k] free_unref_page
    2.26%  [kernel]                   [k] kmem_cache_free
    2.24%  [kernel]                   [k] kmem_cache_alloc
    1.69%  [kernel]                   [k] __softirqentry_text_start
    1.61%  [kernel]                   [k] cpsw_rx_handler
    1.19%  [kernel]                   [k] page_pool_release_page
    1.19%  [kernel]                   [k] clear_bits_ll
    1.15%  [kernel]                   [k] page_frag_free
    1.06%  [kernel]                   [k] __dma_page_dev_to_cpu
    0.99%  [kernel]                   [k] memset
    0.94%  [kernel]                   [k] __alloc_pages_bulk
    0.92%  [kernel]                   [k] kfree_skb
    0.85%  [kernel]                   [k] packet_rcv
    0.78%  [kernel]                   [k] page_address
    0.75%  [kernel]                   [k] v7_dma_inv_range
    0.71%  [kernel]                   [k] __lock_text_start

[rx packets - with packet drop]
rxdiff 48004
rxdiff 47630
rxdiff 47538Š


[iperf3 TCP]
iperf3 -c 192.168.1.1 -i1
[  5]   0.00-10.00  sec   873 MBytes   732 Mbits/sec    0             sender
[  5]   0.00-10.01  sec   866 MBytes   726 Mbits/sec                  receiver

=========== After:
[perf top - with packet drop]
  40.58%  [kernel]                   [k] _raw_spin_unlock_irqrestore
   16.18%  [kernel]                   [k] __softirqentry_text_start
   10.33%  [kernel]                   [k] __cpdma_chan_free
    2.62%  [kernel]                   [k] ___bpf_prog_run
    2.05%  [kernel]                   [k] cpsw_rx_vlan_encap
    2.00%  [kernel]                   [k] kmem_cache_alloc
    1.86%  [kernel]                   [k] __netif_receive_skb_core
    1.80%  [kernel]                   [k] kmem_cache_free
    1.63%  [kernel]                   [k] cpsw_rx_handler
    1.12%  [kernel]                   [k] cpsw_rx_mq_poll
    1.11%  [kernel]                   [k] page_pool_put_page
    1.04%  [kernel]                   [k] _raw_spin_unlock
    0.97%  [kernel]                   [k] clear_bits_ll
    0.90%  [kernel]                   [k] packet_rcv
    0.88%  [kernel]                   [k] __dma_page_dev_to_cpu
    0.85%  [kernel]                   [k] kfree_skb
    0.80%  [kernel]                   [k] memset
    0.71%  [kernel]                   [k] __lock_text_start
    0.66%  [kernel]                   [k] v7_dma_inv_range
    0.64%  [kernel]                   [k] gen_pool_free_owner
    0.58%  [kernel]                   [k] __rcu_read_unlock

[rx packets - with packet drop]
rxdiff 65843
rxdiff 66722
rxdiff 65264

[iperf3 TCP]
iperf3 -c 192.168.1.1 -i1
[  5]   0.00-10.00  sec   884 MBytes   742 Mbits/sec    0             sender
[  5]   0.00-10.01  sec   878 MBytes   735 Mbits/sec                  receiver


-- 
Best regards,
grygorii
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help