[PATCH net-next] net/mlx4_en: avoid one cache line miss to ring doorbell

From: Eric Dumazet <edumazet@google.com>

This patch caches doorbell address directly in struct mlx4_en_tx_ring.

This removes the need to bring in cpu caches whole struct mlx4_uar
in fast path.

Note that mlx4_uar is not guaranteed to be on a local node,
because mlx4_bf_alloc() uses a single free list (priv->bf_list)
regardless of its node parameter.

This kind of change does matter in presence of light/moderate traffic.
In high stress, this read-only line would be kept hot in caches.

Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Tariq Toukan <tariqt@nvidia.com>
---
 drivers/net/ethernet/mellanox/mlx4/en_tx.c   | 4 ++--
 drivers/net/ethernet/mellanox/mlx4/mlx4_en.h | 1 +
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx4/en_tx.c b/drivers/net/ethernet/mellanox/mlx4/en_tx.c
index c56b9dba4c71898b61e87fd32e5fa523c313e445..817f4154b86d599cd593876ec83529051d95fe2f 100644
--- a/drivers/net/ethernet/mellanox/mlx4/en_tx.c
+++ b/drivers/net/ethernet/mellanox/mlx4/en_tx.c

@@ -130,6 +130,7 @@ int mlx4_en_create_tx_ring(struct mlx4_en_priv *priv,
 		ring->bf_enabled = !!(priv->pflags &
 				      MLX4_EN_PRIV_FLAGS_BLUEFLAME);
 	}
+	ring->doorbell_address = ring->bf.uar->map + MLX4_SEND_DOORBELL;
 
 	ring->hwtstamp_tx_type = priv->hwtstamp_config.tx_type;
 	ring->queue_index = queue_index;

@@ -753,8 +754,7 @@ void mlx4_en_xmit_doorbell(struct mlx4_en_tx_ring *ring)
 #else
 	iowrite32be(
 #endif
-		  (__force u32)ring->doorbell_qpn,
-		  ring->bf.uar->map + MLX4_SEND_DOORBELL);
+		  (__force u32)ring->doorbell_qpn, ring->doorbell_address);
 }
 
 static void mlx4_en_tx_write_desc(struct mlx4_en_tx_ring *ring,

diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h
index ad0a8b488832c8cdca2790e47fc778fe15686f7f..e1ad64cc0c8784352d18ae89df6d05bd9709e8a0 100644
--- a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h
+++ b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h

@@ -283,6 +283,7 @@ struct mlx4_en_tx_ring {
 	struct mlx4_bf		bf;
 
 	/* Following part should be mostly read */
+	void			*doorbell_address;
 	__be32			doorbell_qpn;
 	__be32			mr_key;
 	u32			size; /* number of TXBBs */

-- 
2.33.0.800.g4c38ced690-goog

On Thu, 30 Sep 2021 12:40:31 -0700 Eric Dumazet wrote:

From: Eric Dumazet <edumazet@google.com>

This patch caches doorbell address directly in struct mlx4_en_tx_ring.

This removes the need to bring in cpu caches whole struct mlx4_uar
in fast path.

Note that mlx4_uar is not guaranteed to be on a local node,
because mlx4_bf_alloc() uses a single free list (priv->bf_list)
regardless of its node parameter.

This kind of change does matter in presence of light/moderate traffic.
In high stress, this read-only line would be kept hot in caches.

Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Tariq Toukan <tariqt@nvidia.com>

 	/* Following part should be mostly read */
+	void			*doorbell_address;

We'll need to make sparse happy before applying:

drivers/net/ethernet/mellanox/mlx4/en_tx.c:133:32: warning: incorrect type in assignment (different address spaces)
drivers/net/ethernet/mellanox/mlx4/en_tx.c:133:32:    expected void *doorbell_address
drivers/net/ethernet/mellanox/mlx4/en_tx.c:133:32:    got void [noderef] __iomem *
drivers/net/ethernet/mellanox/mlx4/en_tx.c:757:56: warning: incorrect type in argument 2 (different address spaces)
drivers/net/ethernet/mellanox/mlx4/en_tx.c:757:56:    expected void [noderef] __iomem *
drivers/net/ethernet/mellanox/mlx4/en_tx.c:757:56:    got void *doorbell_address

On Thu, Sep 30, 2021 at 5:10 PM Jakub Kicinski [off-list ref] wrote:

On Thu, 30 Sep 2021 12:40:31 -0700 Eric Dumazet wrote:

quoted

From: Eric Dumazet <edumazet@google.com>

This patch caches doorbell address directly in struct mlx4_en_tx_ring.

This removes the need to bring in cpu caches whole struct mlx4_uar
in fast path.

Note that mlx4_uar is not guaranteed to be on a local node,
because mlx4_bf_alloc() uses a single free list (priv->bf_list)
regardless of its node parameter.

This kind of change does matter in presence of light/moderate traffic.
In high stress, this read-only line would be kept hot in caches.

Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Tariq Toukan <tariqt@nvidia.com>

quoted

      /* Following part should be mostly read */
+     void                    *doorbell_address;

We'll need to make sparse happy before applying:

drivers/net/ethernet/mellanox/mlx4/en_tx.c:133:32: warning: incorrect type in assignment (different address spaces)
drivers/net/ethernet/mellanox/mlx4/en_tx.c:133:32:    expected void *doorbell_address
drivers/net/ethernet/mellanox/mlx4/en_tx.c:133:32:    got void [noderef] __iomem *
drivers/net/ethernet/mellanox/mlx4/en_tx.c:757:56: warning: incorrect type in argument 2 (different address spaces)
drivers/net/ethernet/mellanox/mlx4/en_tx.c:757:56:    expected void [noderef] __iomem *
drivers/net/ethernet/mellanox/mlx4/en_tx.c:757:56:    got void *doorbell_address

Yes indeed, I'll send a V2 right away, thanks !