On Sat, 2023-09-30 at 10:36 +0300, Leon Romanovsky wrote:

On Fri, Sep 29, 2023 at 02:15:49PM +0200, Niklas Schnelle wrote:

quoted

Since commit 06cd555f73ca ("net/mlx5: split mlx5_cmd_init() to probe and
reload routines") mlx5_cmd_init() is called in mlx5_mdev_init() which is
called in probe_one() before mlx5_pci_init(). This is a problem because
mlx5_pci_init() is where the DMA and coherent mask is set but
mlx5_cmd_init() already does a dma_alloc_coherent(). Thus a DMA
allocation is done during probe before the correct mask is set. This
causes probe to fail initialization of the cmdif SW structs on s390x
after that is converted to the common dma-iommu code. This is because on
s390x DMA addresses below 4 GiB are reserved on current machines and
unlike the old s390x specific DMA API implementation common code
enforces DMA masks.

Fix this by moving set_dma_caps() out of mlx5_pci_init() and into
probe_one() before mlx5_mdev_init(). To match the overall naming scheme
rename it to mlx5_dma_init().

Link: https://lore.kernel.org/linux-iommu/cfc9e9128ed5571d2e36421e347301057662a09e.camel@linux.ibm.com/ (local)
Fixes: 06cd555f73ca ("net/mlx5: split mlx5_cmd_init() to probe and reload routines")
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
---
Note: I ran into this while testing the linked series for converting
s390x to use dma-iommu. The existing s390x specific DMA API
implementation doesn't respect DMA masks and is thus not affected
despite of course also only supporting DMA addresses above 4 GiB.
---
Changes in v2:
- Instead of moving the whole mlx5_pci_init() only move the
  set_dma_caps() call so as to keep pci_enable_device() after the FW
  command interface initialization (Leon)
- Link to v1: https://lore.kernel.org/r/20230928-mlx5_init_fix-v1-1-79749d45ce60@linux.ibm.com (local)
---
 drivers/net/ethernet/mellanox/mlx5/core/main.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

Thanks,
Reviewed-by: Leon Romanovsky <leonro@nvidia.com>

Thank you for the review. Assuming the mlx5 tree is included in linux-
next I think it would be easiest if this goes via that tree thereby
unbreaking linux-next for s390. Or do you prefer Joerg to take this via
the IOMMU tree or even some other tree?

Thanks,
Niklas