[PATCH net-next 2/3] mlx4_en: moderate frequency of TX completions

No need to ask for completion for every packet being sent.
So the method is to ask for a completion every 16 packets,
or when the queue is about to be full.

Signed-off-by: Yevgeny Petrilin <redacted>
---
 drivers/net/ethernet/mellanox/mlx4/en_tx.c |    7 +++++--
 1 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx4/en_tx.c b/drivers/net/ethernet/mellanox/mlx4/en_tx.c
index ff32505..e452bba 100644
--- a/drivers/net/ethernet/mellanox/mlx4/en_tx.c
+++ b/drivers/net/ethernet/mellanox/mlx4/en_tx.c

@@ -615,6 +615,7 @@ netdev_tx_t mlx4_en_xmit(struct sk_buff *skb, struct net_device *dev)
 	int lso_header_size;
 	void *fragptr;
 	bool bounce = false;
+	int ring_busy;
 
 	if (!priv->port_up)
 		goto tx_drop;

@@ -638,8 +639,8 @@ netdev_tx_t mlx4_en_xmit(struct sk_buff *skb, struct net_device *dev)
 		vlan_tag = vlan_tx_tag_get(skb);
 
 	/* Check available TXBBs And 2K spare for prefetch */
-	if (unlikely(((int)(ring->prod - ring->cons)) >
-		     ring->size - HEADROOM - MAX_DESC_TXBBS)) {
+	ring_busy = (int)(ring->prod - ring->cons) - (ring->size - HEADROOM - MAX_DESC_TXBBS);
+	if (unlikely(ring_busy > 0)) {
 		/* every full Tx ring stops queue */
 		netif_tx_stop_queue(netdev_get_tx_queue(dev, tx_ind));
 		ring->blocked = 1;

@@ -680,6 +681,8 @@ netdev_tx_t mlx4_en_xmit(struct sk_buff *skb, struct net_device *dev)
 		!!vlan_tx_tag_present(skb);
 	tx_desc->ctrl.fence_size = (real_size / 16) & 0x3f;
 	tx_desc->ctrl.srcrb_flags = priv->ctrl_flags;
+	if (!(!(index & 0xf) || vlan_tag || (ring_busy > -MAX_DESC_TXBBS)))
+		tx_desc->ctrl.srcrb_flags &= cpu_to_be32(~MLX4_WQE_CTRL_CQ_UPDATE);
 	if (likely(skb->ip_summed == CHECKSUM_PARTIAL)) {
 		tx_desc->ctrl.srcrb_flags |= cpu_to_be32(MLX4_WQE_CTRL_IP_CSUM |
 							 MLX4_WQE_CTRL_TCP_UDP_CSUM);

-- 
1.5.4.3

From: Yevgeny Petrilin <redacted>
Date: Thu, 23 Feb 2012 15:34:27 +0200

No need to ask for completion for every packet being sent.
So the method is to ask for a completion every 16 packets,
or when the queue is about to be full.

Signed-off-by: Yevgeny Petrilin <redacted>

You absolutely cannot do this, you must signal completion and free up
TX queue packets in a finite amount of time.

This means that if you suddenly stop getting new packets to send
you must still free up all the pending TX SKBs even if no more
packets are given to the driver.

Does your hardware unconditionally give a TX completion interrupt when
the TX queue empties completely?  If not, then you cannot make the
change contained in this patch.

On Thu, 2012-02-23 at 14:44 -0500, David Miller wrote:

From: Yevgeny Petrilin <redacted>
Date: Thu, 23 Feb 2012 15:34:27 +0200

quoted

No need to ask for completion for every packet being sent.
So the method is to ask for a completion every 16 packets,
or when the queue is about to be full.

Signed-off-by: Yevgeny Petrilin <redacted>

You absolutely cannot do this, you must signal completion and free up
TX queue packets in a finite amount of time.

Really, sfc has been doing this forever.

Ben.

This means that if you suddenly stop getting new packets to send
you must still free up all the pending TX SKBs even if no more
packets are given to the driver.

Does your hardware unconditionally give a TX completion interrupt when
the TX queue empties completely?  If not, then you cannot make the
change contained in this patch.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

On Thu, 2012-02-23 at 21:13 +0000, Ben Hutchings wrote:

On Thu, 2012-02-23 at 14:44 -0500, David Miller wrote:

quoted

From: Yevgeny Petrilin <redacted>
Date: Thu, 23 Feb 2012 15:34:27 +0200

quoted

No need to ask for completion for every packet being sent.
So the method is to ask for a completion every 16 packets,
or when the queue is about to be full.

Signed-off-by: Yevgeny Petrilin <redacted>

You absolutely cannot do this, you must signal completion and free up
TX queue packets in a finite amount of time.

Really, sfc has been doing this forever.

Ben.

quoted

This means that if you suddenly stop getting new packets to send
you must still free up all the pending TX SKBs even if no more
packets are given to the driver.

Does your hardware unconditionally give a TX completion interrupt when
the TX queue empties completely?  If not, then you cannot make the
change contained in this patch.

And it's pretty obvious the hardware has to also send a completion when
the queue is empty.  I hope.

Ben.

-- 
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

From: Ben Hutchings <redacted>
Date: Thu, 23 Feb 2012 21:13:56 +0000

On Thu, 2012-02-23 at 14:44 -0500, David Miller wrote:

quoted

From: Yevgeny Petrilin <redacted>
Date: Thu, 23 Feb 2012 15:34:27 +0200

quoted

No need to ask for completion for every packet being sent.
So the method is to ask for a completion every 16 packets,
or when the queue is about to be full.

Signed-off-by: Yevgeny Petrilin <redacted>

You absolutely cannot do this, you must signal completion and free up
TX queue packets in a finite amount of time.

Really, sfc has been doing this forever.

You have to stop, because otherwise sockets can hang since those
packets can hold a reference to the socket so they must be
released in a finite amount of time regardless of network
traffic.

From: Ben Hutchings <redacted>
Date: Thu, 23 Feb 2012 21:26:20 +0000

And it's pretty obvious the hardware has to also send a completion when
the queue is empty.  I hope.

Right and we know of several chips that don't.