Re: [xdp-hints] Re: [PATCH bpf-next v7 15/17] net/mlx5e: Introduce wrapper for xdp_buff

On 13/01/2023 23:31, Toke Høiland-Jørgensen wrote:

Tariq Toukan [off-list ref] writes:

quoted

On 12/01/2023 23:55, Toke Høiland-Jørgensen wrote:

quoted

Toke Høiland-Jørgensen [off-list ref] writes:

quoted

Stanislav Fomichev [off-list ref] writes:

quoted

On Thu, Jan 12, 2023 at 12:07 AM Tariq Toukan [off-list ref] wrote:

quoted

On 12/01/2023 2:32, Stanislav Fomichev wrote:

quoted

From: Toke Høiland-Jørgensen <redacted>

Preparation for implementing HW metadata kfuncs. No functional change.

Cc: Tariq Toukan <tariqt@nvidia.com>
Cc: Saeed Mahameed <saeedm@nvidia.com>
Cc: John Fastabend <john.fastabend@gmail.com>
Cc: David Ahern <redacted>
Cc: Martin KaFai Lau <martin.lau@linux.dev>
Cc: Jakub Kicinski <kuba@kernel.org>
Cc: Willem de Bruijn <willemb@google.com>
Cc: Jesper Dangaard Brouer <redacted>
Cc: Anatoly Burakov <redacted>
Cc: Alexander Lobakin <redacted>
Cc: Magnus Karlsson <redacted>
Cc: Maryam Tahhan <redacted>
Cc: xdp-hints@xdp-project.net
Cc: netdev@vger.kernel.org
Signed-off-by: Toke Høiland-Jørgensen <redacted>
Signed-off-by: Stanislav Fomichev <redacted>
---
    drivers/net/ethernet/mellanox/mlx5/core/en.h  |  1 +
    .../net/ethernet/mellanox/mlx5/core/en/xdp.c  |  3 +-
    .../net/ethernet/mellanox/mlx5/core/en/xdp.h  |  6 +-
    .../ethernet/mellanox/mlx5/core/en/xsk/rx.c   | 25 ++++----
    .../net/ethernet/mellanox/mlx5/core/en_rx.c   | 58 +++++++++----------
    5 files changed, 50 insertions(+), 43 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index 2d77fb8a8a01..af663978d1b4 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h

@@ -469,6 +469,7 @@ struct mlx5e_txqsq {
    union mlx5e_alloc_unit {
        struct page *page;
        struct xdp_buff *xsk;
+     struct mlx5e_xdp_buff *mxbuf;

In XSK files below you mix usage of both alloc_units[page_idx].mxbuf and
alloc_units[page_idx].xsk, while both fields share the memory of a union.

As struct mlx5e_xdp_buff wraps struct xdp_buff, I think that you just
need to change the existing xsk field type from struct xdp_buff *xsk
into struct mlx5e_xdp_buff *xsk and align the usage.

Hmmm, good point. I'm actually not sure how it works currently.
mlx5e_alloc_unit.mxbuf doesn't seem to be initialized anywhere? Toke,
am I missing something?

It's initialised piecemeal in different places; but yeah, we're mixing
things a bit...

quoted

I'm thinking about something like this:

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h

b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index af663978d1b4..2d77fb8a8a01 100644

--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h

@@ -469,7 +469,6 @@ struct mlx5e_txqsq {
   union mlx5e_alloc_unit {
          struct page *page;
          struct xdp_buff *xsk;
-       struct mlx5e_xdp_buff *mxbuf;
   };

Hmm, for consistency with the non-XSK path we should rather go the other
direction and lose the xsk member, moving everything to mxbuf? Let me
give that a shot...

Something like the below?

-Toke

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index 6de02d8aeab8..cb9cdb6421c5 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h

@@ -468,7 +468,6 @@ struct mlx5e_txqsq {
   
   union mlx5e_alloc_unit {
   	struct page *page;
-	struct xdp_buff *xsk;
   	struct mlx5e_xdp_buff *mxbuf;
   };
   

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h
index cb568c62aba0..95694a25ec31 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.h

@@ -33,6 +33,7 @@
   #define __MLX5_EN_XDP_H__
   
   #include <linux/indirect_call_wrapper.h>
+#include <net/xdp_sock_drv.h>
   
   #include "en.h"
   #include "en/txrx.h"

@@ -112,6 +113,21 @@ static inline void mlx5e_xmit_xdp_doorbell(struct mlx5e_xdpsq *sq)
   	}
   }
   
+static inline struct mlx5e_xdp_buff *mlx5e_xsk_buff_alloc(struct xsk_buff_pool *pool)
+{
+	return (struct mlx5e_xdp_buff *)xsk_buff_alloc(pool);

What about the space needed for the rq / cqe fields? xsk_buff_alloc
won't allocate it.

It will! See patch 14 in the series that adds a 'cb' field to
xdp_buff_xsk, meaning there's actually space after the xdp_buff struct
being allocated by the xsk_buff_alloc API. The XSK_CHECK_PRIV_TYPE macro
call is there to ensure the cb field is big enough for the struct we're
casting to in the driver.

Oh okay, got it.

quoted

quoted

+}
+
+static inline void mlx5e_xsk_buff_free(struct mlx5e_xdp_buff *mxbuf)
+{
+	xsk_buff_free(&mxbuf->xdp);
+}
+
+static inline dma_addr_t mlx5e_xsk_buff_xdp_get_frame_dma(struct mlx5e_xdp_buff *mxbuf)
+{
+	return xsk_buff_xdp_get_frame_dma(&mxbuf->xdp);
+}
+
   /* Enable inline WQEs to shift some load from a congested HCA (HW) to
    * a less congested cpu (SW).
    */

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c
index 8bf3029abd3c..1f166dbb7f22 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c

@@ -3,7 +3,6 @@
   
   #include "rx.h"
   #include "en/xdp.h"
-#include <net/xdp_sock_drv.h>
   #include <linux/filter.h>
   
   /* RX data path */

@@ -21,7 +20,7 @@ int mlx5e_xsk_alloc_rx_mpwqe(struct mlx5e_rq *rq, u16 ix)
   	if (unlikely(!xsk_buff_can_alloc(rq->xsk_pool, rq->mpwqe.pages_per_wqe)))
   		goto err;
   
-	BUILD_BUG_ON(sizeof(wi->alloc_units[0]) != sizeof(wi->alloc_units[0].xsk));
+	BUILD_BUG_ON(sizeof(wi->alloc_units[0]) != sizeof(wi->alloc_units[0].mxbuf));
   	XSK_CHECK_PRIV_TYPE(struct mlx5e_xdp_buff);
   	batch = xsk_buff_alloc_batch(rq->xsk_pool, (struct xdp_buff **)wi->alloc_units,
   				     rq->mpwqe.pages_per_wqe);

This batching API gets broken as well...
xsk_buff_alloc_batch fills an array of struct xdp_buff pointers, it
cannot correctly act on the array of struct mlx5e_xdp_buff, as it
contains additional fields.

See above for why this does, in fact, work. I agree it's not totally
obvious, and in any case there's going to be a point where the cast
happens where type safety will break, which is what I was alluding to in
my reply to Stanislav.

I guess we could try to rework the API in xdp_sock_drv.h to make this
more obvious instead of using the casting driver-specific wrappers I
suggested here. Or we could go with Stanislav's suggestion of keeping
allocation etc using xdp_buff and only casting to mlx5e_xdp_buff in the
function where it's used; then we can keep the casting localised to that
function, and put a comment there explaining why it works?

Stanislav's proposal LGTM.
Let's keep the casting localised, and make sure there's a comment there.

quoted

Maybe letting mlx5e_xdp_buff point to its struct xdp_buff (instead of
wrapping it) will solve the problems here, then we'll loop over the
xdp_buff * array and copy the pointers into the struct mlx5e_xdp_buff *
array.
Need to give it deeper thoughts...

struct mlx5e_xdp_buff {
	struct xdp_buff *xdp;
	struct mlx5_cqe64 *cqe;
	struct mlx5e_rq *rq;
};

This was actually my original proposal; we discussed this back on v2 of
this patch series. People generally felt that the 'cb' field approach
(originally suggested by Jakub) was better.

I agree.

See the discussion starting
from here:

https://lore.kernel.org/r/20221123111431.7b54668e@kernel.org (local)

-Toke