On Thu, 30 Apr 2020 20:07:43 +0300
Tariq Toukan [off-list ref] wrote:

On 4/30/2020 2:22 PM, Jesper Dangaard Brouer wrote:

quoted

The mlx5 driver have multiple memory models, which are also changed
according to whether a XDP bpf_prog is attached.

The 'rx_striding_rq' setting is adjusted via ethtool priv-flags e.g.:
  # ethtool --set-priv-flags mlx5p2 rx_striding_rq off

On the general case with 4K page_size and regular MTU packet, then
the frame_sz is 2048 and 4096 when XDP is enabled, in both modes.

The info on the given frame size is stored differently depending on the
RQ-mode and encoded in a union in struct mlx5e_rq union wqe/mpwqe.
In rx striding mode rq->mpwqe.log_stride_sz is either 11 or 12, which
corresponds to 2048 or 4096 (MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ).
In non-striding mode (MLX5_WQ_TYPE_CYCLIC) the frag_stride is stored
in rq->wqe.info.arr[0].frag_stride, for the first fragment, which is
what the XDP case cares about.

To reduce effect on fast-path, this patch determine the frame_sz at
setup time, to avoid determining the memory model runtime. Variable
is named first_frame_sz to make it clear that this is only the frame
size of the first fragment.

This mlx5 driver does a DMA-sync on XDP_TX action, but grow is safe
as it have done a DMA-map on the entire PAGE_SIZE. The driver also
already does a XDP length check against sq->hw_mtu on the possible
XDP xmit paths mlx5e_xmit_xdp_frame() + mlx5e_xmit_xdp_frame_mpwqe().

V2: Fix that frag_size need to be recalc before creating SKB.

Cc: Tariq Toukan <redacted>
Cc: Saeed Mahameed <redacted>
Signed-off-by: Jesper Dangaard Brouer <redacted>
---
  drivers/net/ethernet/mellanox/mlx5/core/en.h      |    1 +
  drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c  |    1 +
  drivers/net/ethernet/mellanox/mlx5/core/en_main.c |    6 ++++++
  drivers/net/ethernet/mellanox/mlx5/core/en_rx.c   |    2 ++
  4 files changed, 10 insertions(+)

[... cut ...]

quoted

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
index e2beb89c1832..04671ed977a5 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c

@@ -1084,6 +1084,7 @@ mlx5e_skb_from_cqe_linear(struct mlx5e_rq *rq, struct mlx5_cqe64 *cqe,
  	if (consumed)
  		return NULL; /* page/packet was consumed by XDP */
  
+	frag_size = MLX5_SKB_FRAG_SZ(rx_headroom + cqe_bcnt);  

This is a re-calculation of frag_size, using the exact same command used 
earlier in this function, but with a newer value of rx_headroom.
This wasn't part of the previous patchset. I understand the need.

Yes, kernel will crash without this change.

However, this code repetition looks weird and non-optimal to me. I think 
we can come up with something better.

quoted

  	skb = mlx5e_build_linear_skb(rq, va, frag_size, rx_headroom, cqe_bcnt);
  	if (unlikely(!skb))
  		return NULL;

@@ -1385,6 +1386,7 @@ mlx5e_skb_from_cqe_mpwrq_linear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *wi,
  		return NULL; /* page/packet was consumed by XDP */
  	}
  
+	frag_size = MLX5_SKB_FRAG_SZ(rx_headroom + cqe_bcnt32);  

Same here.

quoted

  	skb = mlx5e_build_linear_skb(rq, va, frag_size, rx_headroom, cqe_bcnt32);
  	if (unlikely(!skb))
  		return NULL;

  

My suggetion is:
Pass &frag_size to mlx5e_xdp_handle(), and update it within it, just 
next to the update of rx_headroom.
All the needed information is there: the new rx_headroom, and cqe_bcnt.

First of all, passing yet-another argument to mlx5e_xdp_handle(), also
looks weird, and is on the brink of becoming a performance issue, as on
x86_64 you can pass max 6 arguments in registers before they get pushed
on the stack. Adding this would be the 7th argument.

Second the MLX5_SKB_FRAG_SZ() calculation is also weird, because it
does not provide any tailroom in the packet, I guess it is for
supporting another memory mode, as in case XDP is activated there are
plenty of tailroom.
  I though about increasing the frag_size, but I choose not to, because
then this patch would change the driver behavior beyond adding frame_sz
for XDP.

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer



bool mlx5e_xdp_handle(struct mlx5e_rq *rq, struct mlx5e_dma_info *di,
		      void *va, u16 *rx_headroom, u32 *len, bool xsk)
{
	[...]
}