Re: [PATCH v4 1/4] PCI/MSI: Conservatively generalize no_64bit_msi into msi_addr_mask
From: Mark Bloch <mbloch@nvidia.com>
Date: 2026-02-26 18:25:46
Also in:
amd-gfx, dri-devel, linux-pci, linux-riscv, linux-sound, linuxppc-dev, lkml
On 29/01/2026 3:56, Vivian Wang wrote:
Some PCI devices have PCI_MSI_FLAGS_64BIT in the MSI capability, but implement less than 64 address bits. This breaks on platforms where such a device is assigned an MSI address higher than what's reachable. Currently, the no_64bit_msi bit is set for these devices, meaning that only 32-bit MSI addresses are allowed for them. However, on some platforms the MSI doorbell address is above the 32-bit limit but within the addressable range of the device. As a first step to enabling MSI on those combinations of devices and platforms, conservatively generalize the single-bit flag no_64bit_msi into msi_addr_mask. (The name msi_addr_mask is chosen to avoid confusion with msi_mask.) The translation is essentially: - no_64bit_msi = 1 -> msi_addr_mask = DMA_BIT_MASK(32) - no_64bit_msi = 0 -> msi_addr_mask = DMA_BIT_MASK(64) - if (no_64bit_msi) -> if (msi_addr_mask < DMA_BIT_MASK(64))
Hey Vivian, We are seeing issues while reloading mlx5 on a PPC64 platform. We see the following messages in dmesg: mlx5_core 0000:00:08.0: mlx5_load:1266:(pid 1283): Failed to alloc IRQs mlx5_core 0000:00:08.0: E-Switch: cleanup mlx5_core 0000:00:08.0: probe_one:1959:(pid 1283): mlx5_init_one failed with error code -19 mlx5_core 0000:00:08.1: lsa_required: 0, lsa_enabled: 0, direct mapping: 0 mlx5_core 0000:00:08.1: lsa_required: 0, lsa_enabled: 0, direct mapping: 0 mlx5_core 0000:00:08.1: firmware version: 16.35.4506 mlx5_core 0000:00:08.1: 0.000 Gb/s available PCIe bandwidth (Unknown x16 link) mlx5_core 0000:00:08.1: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps mlx5_core 0000:00:08.1: E-Switch: Total vports 2, per vport: max uc(128) max mc(2048) mlx5_core 0000:00:08.1: Flow counters bulk query buffer size increased, bulk_query_len(8) mlx5_core 0000:00:08.1: mlx5_load:1266:(pid 1283): Failed to alloc IRQs mlx5_core 0000:00:08.1: E-Switch: cleanup mlx5_core 0000:00:08.1: probe_one:1959:(pid 1283): mlx5_init_one failed with error code -19 We've bisectedthe issue to this patch. We've pointed some AI tool to look at the reproducation script and this patch and it came up with the following idea: " The bug is a missing initialization of msi_addr_mask on PPC64LE, caused by a PPC-specific PCI device creation path. On most architectures, PCI devices are enumerated via the standard path:
pci_scan_device() -> pci_alloc_dev() (kzalloc) -> pci_setup_device() -> pci_device_add()
But PPC has its own device-tree-based PCI enumeration in arch/powerpc/kernel/pci_of_scan.c.
The function of_create_pci_dev() does this:
pci_of_scan.cLines 215-235
dev = pci_alloc_dev(bus); // kzalloc -> all fields zeroed, msi_addr_mask = 0
// ... manually sets vendor, device, class, dma_mask, etc.
// ... does NOT call pci_setup_device() !!!
pci_device_add(dev, bus); // calls pci_init_capabilities -> pci_msi_init
Inside pci_msi_init(), the code only sets msi_addr_mask = DMA_BIT_MASK(32) when 64-bit MSI is not supported.
If the device does support 64-bit MSI (like mlx5), it leaves msi_addr_mask untouched -- meaning it stays at 0 from the kzalloc.
"
We tried a patch that moved dev->msi_addr_mask = DMA_BIT_MASK(64) from pci_setup_device()
to pci_alloc_dev() and it solved the issue we were seeing (don't know if this is the proper fix).
Can you please have a look?
Mark
quoted hunk ↗ jump to hunk
Since no values other than DMA_BIT_MASK(32) and DMA_BIT_MASK(64) are used, no functional change is intended. Future patches that make use of intermediate values of msi_addr_mask will follow, allowing devices that cannot use full 64-bit addresses for MSI to work on platforms with MSI doorbell above the 32-bit limit. Acked-by: Takashi Iwai <redacted> # sound Reviewed-by: Brett Creeley <brett.creeley@amd.com> # ionic Reviewed-by: Thomas Gleixner <tglx@kernel.org> Signed-off-by: Vivian Wang <redacted> --- v4: Patch message rewording (Thomas) checkpatch complains about the comment include/linux/pci.h, which I have formatted similarly with other comments in the vicinity. --- arch/powerpc/platforms/powernv/pci-ioda.c | 2 +- arch/powerpc/platforms/pseries/msi.c | 4 ++-- drivers/gpu/drm/radeon/radeon_irq_kms.c | 2 +- drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c | 2 +- drivers/pci/msi/msi.c | 2 +- drivers/pci/msi/pcidev_msi.c | 2 +- drivers/pci/probe.c | 7 +++++++ include/linux/pci.h | 8 +++++++- sound/hda/controllers/intel.c | 2 +- 9 files changed, 22 insertions(+), 9 deletions(-)diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c index b0c1d9d16fb5..1c78fdfb7b03 100644 --- a/arch/powerpc/platforms/powernv/pci-ioda.c +++ b/arch/powerpc/platforms/powernv/pci-ioda.c@@ -1666,7 +1666,7 @@ static int __pnv_pci_ioda_msi_setup(struct pnv_phb *phb, struct pci_dev *dev, return -ENXIO; /* Force 32-bit MSI on some broken devices */ - if (dev->no_64bit_msi) + if (dev->msi_addr_mask < DMA_BIT_MASK(64)) is_64 = 0; /* Assign XIVE to PE */diff --git a/arch/powerpc/platforms/pseries/msi.c b/arch/powerpc/platforms/pseries/msi.c index a82aaa786e9e..7473c7ca1db0 100644 --- a/arch/powerpc/platforms/pseries/msi.c +++ b/arch/powerpc/platforms/pseries/msi.c@@ -383,7 +383,7 @@ static int rtas_prepare_msi_irqs(struct pci_dev *pdev, int nvec_in, int type, */ again: if (type == PCI_CAP_ID_MSI) { - if (pdev->no_64bit_msi) { + if (pdev->msi_addr_mask < DMA_BIT_MASK(64)) { rc = rtas_change_msi(pdn, RTAS_CHANGE_32MSI_FN, nvec); if (rc < 0) { /*@@ -409,7 +409,7 @@ static int rtas_prepare_msi_irqs(struct pci_dev *pdev, int nvec_in, int type, if (use_32bit_msi_hack && rc > 0) rtas_hack_32bit_msi_gen2(pdev); } else { - if (pdev->no_64bit_msi) + if (pdev->msi_addr_mask < DMA_BIT_MASK(64)) rc = rtas_change_msi(pdn, RTAS_CHANGE_32MSIX_FN, nvec); else rc = rtas_change_msi(pdn, RTAS_CHANGE_MSIX_FN, nvec);diff --git a/drivers/gpu/drm/radeon/radeon_irq_kms.c b/drivers/gpu/drm/radeon/radeon_irq_kms.c index 9961251b44ba..d550554a6f3f 100644 --- a/drivers/gpu/drm/radeon/radeon_irq_kms.c +++ b/drivers/gpu/drm/radeon/radeon_irq_kms.c@@ -252,7 +252,7 @@ static bool radeon_msi_ok(struct radeon_device *rdev) */ if (rdev->family < CHIP_BONAIRE) { dev_info(rdev->dev, "radeon: MSI limited to 32-bit\n"); - rdev->pdev->no_64bit_msi = 1; + rdev->pdev->msi_addr_mask = DMA_BIT_MASK(32); } /* force MSI on */diff --git a/drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c b/drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c index 70d86c5f52fb..0671deae9a28 100644 --- a/drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c +++ b/drivers/net/ethernet/pensando/ionic/ionic_bus_pci.c@@ -331,7 +331,7 @@ static int ionic_probe(struct pci_dev *pdev, const struct pci_device_id *ent) #ifdef CONFIG_PPC64 /* Ensure MSI/MSI-X interrupts lie within addressable physical memory */ - pdev->no_64bit_msi = 1; + pdev->msi_addr_mask = DMA_BIT_MASK(32); #endif err = ionic_setup_one(ionic);diff --git a/drivers/pci/msi/msi.c b/drivers/pci/msi/msi.c index 34d664139f48..48f5f03d1479 100644 --- a/drivers/pci/msi/msi.c +++ b/drivers/pci/msi/msi.c@@ -322,7 +322,7 @@ static int msi_verify_entries(struct pci_dev *dev) { struct msi_desc *entry; - if (!dev->no_64bit_msi) + if (dev->msi_addr_mask == DMA_BIT_MASK(64)) return 0; msi_for_each_desc(entry, &dev->dev, MSI_DESC_ALL) {diff --git a/drivers/pci/msi/pcidev_msi.c b/drivers/pci/msi/pcidev_msi.c index 5520aff53b56..0b0346813092 100644 --- a/drivers/pci/msi/pcidev_msi.c +++ b/drivers/pci/msi/pcidev_msi.c@@ -24,7 +24,7 @@ void pci_msi_init(struct pci_dev *dev) } if (!(ctrl & PCI_MSI_FLAGS_64BIT)) - dev->no_64bit_msi = 1; + dev->msi_addr_mask = DMA_BIT_MASK(32); } void pci_msix_init(struct pci_dev *dev)diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index 41183aed8f5d..a2bff57176a3 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c@@ -2047,6 +2047,13 @@ int pci_setup_device(struct pci_dev *dev) */ dev->dma_mask = 0xffffffff; + /* + * Assume 64-bit addresses for MSI initially. Will be changed to 32-bit + * if MSI (rather than MSI-X) capability does not have + * PCI_MSI_FLAGS_64BIT. Can also be overridden by driver. + */ + dev->msi_addr_mask = DMA_BIT_MASK(64); + dev_set_name(&dev->dev, "%04x:%02x:%02x.%d", pci_domain_nr(dev->bus), dev->bus->number, PCI_SLOT(dev->devfn), PCI_FUNC(dev->devfn));diff --git a/include/linux/pci.h b/include/linux/pci.h index 864775651c6f..0fe32fef0331 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h@@ -377,6 +377,13 @@ struct pci_dev { 0xffffffff. You only need to change this if your device has broken DMA or supports 64-bit transfers. */ + u64 msi_addr_mask; /* Mask of the bits of bus address for + MSI that this device implements. + Normally set based on device + capabilities. You only need to + change this if your device claims + to support 64-bit MSI but implements + fewer than 64 address bits. */ struct device_dma_parameters dma_parms;@@ -441,7 +448,6 @@ struct pci_dev { unsigned int is_busmaster:1; /* Is busmaster */ unsigned int no_msi:1; /* May not use MSI */ - unsigned int no_64bit_msi:1; /* May only use 32-bit MSIs */ unsigned int block_cfg_access:1; /* Config space access blocked */ unsigned int broken_parity_status:1; /* Generates false positive parity */ unsigned int irq_reroute_variant:2; /* Needs IRQ rerouting variant */diff --git a/sound/hda/controllers/intel.c b/sound/hda/controllers/intel.c index 1e8e3d61291a..c9542ebdf7e2 100644 --- a/sound/hda/controllers/intel.c +++ b/sound/hda/controllers/intel.c@@ -1905,7 +1905,7 @@ static int azx_first_init(struct azx *chip) if (chip->msi && chip->driver_caps & AZX_DCAPS_NO_MSI64) { dev_dbg(card->dev, "Disabling 64bit MSI\n"); - pci->no_64bit_msi = true; + pci->msi_addr_mask = DMA_BIT_MASK(32); } pci_set_master(pci);