Hi Frank,

Am Mittwoch, 26. Februar 2025, 17:31:26 CET schrieb Frank Li:

On Wed, Feb 26, 2025 at 01:11:37PM +0100, Alexander Stein wrote:

quoted

Hi Frank,

Am Dienstag, 28. Januar 2025, 22:15:58 CET schrieb Frank Li:

quoted

Add PCIe's msi-map and iommu-map property because i.MX95 support smmu and
its.

Signed-off-by: Frank Li <Frank.Li@nxp.com>
---
 arch/arm64/boot/dts/freescale/imx95.dtsi | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/arch/arm64/boot/dts/freescale/imx95.dtsi b/arch/arm64/boot/dts/freescale/imx95.dtsi
index 6b8470cb3461a..2cebeda43a52d 100644
--- a/arch/arm64/boot/dts/freescale/imx95.dtsi
+++ b/arch/arm64/boot/dts/freescale/imx95.dtsi

@@ -1573,6 +1573,12 @@ pcie0: pcie@4c300000 {
 			assigned-clock-parents = <0>, <0>,
 						 <&scmi_clk IMX95_CLK_SYSPLL1_PFD1_DIV2>;
 			power-domains = <&scmi_devpd IMX95_PD_HSIO_TOP>;
+			/* pcie0's Devid(BIT[7:6]) is 0x00, stream id(BIT[5:0]) is 0x10~0x17 */
+			msi-map = <0x0 &its 0x10 0x1>,
+				  <0x100 &its 0x11 0x7>;

Aren't you missing msi-map-mask = <0x1ff>; here? Similar to pcie1.
Either way, with this change PCIe on pcie0 is not working anymore,
regardless of msi-map-mask.

Yes, it should have msi-map-mask. During my test, I have not enable enetc
so I have not found this problem.

Just to be clear: This is not about enetc. This works fine here.

quoted

Without msi-map-mask:

quoted

OF: /soc/pcie@4c300000: iommu-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011
OF: /soc/pcie@4c300000: no msi-map translation for id 0x300 on (null)
r8169 0000:03:00.0: error -EINVAL: enable failure
r8169 0000:03:00.0: probe with driver r8169 failed with error -22

With msi-map-mask:

quoted

OF: /soc/pcie@4c300000: iommu-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011
OF: /soc/pcie@4c300000: msi-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011
r8169 0000:03:00.0: enabling device (0000 -> 0003)
r8169 0000:03:00.0: enabling Mem-Wr-Inval
r8169 0000:03:00.0: error -EIO: PCI read failed
r8169 0000:03:00.0: probe with driver r8169 failed with error -5

Can you try remove iommu-map and keep msi-map? then remove msi-map and
keep iommu-map to check which one cause this problem.

With only msi-map removed, but smmu enabled:

arm-smmu-v3 490d0000.iommu: event 0x10 received:
arm-smmu-v3 490d0000.iommu:      0x0000001100000010
arm-smmu-v3 490d0000.iommu:      0x0000020a00000000
arm-smmu-v3 490d0000.iommu:      0x000000009b0cc000
arm-smmu-v3 490d0000.iommu:      0x0000000000000000
arm-smmu-v3 490d0000.iommu: event: F_TRANSLATION client: 0000:01:00.0 sid: 0x11 ssid: 0x0 iova: 0x9b0cc000 ipa: 0x0
arm-smmu-v3 490d0000.iommu: priv data read s1 "Input address caused fault" stag: 0x0 r8169 0000:03:00.0
enp3s0: Link is Down

With only iommu-map removed, both smmu enabled or disabled:

OF: /soc/pcie@4c300000: msi-map, using mask 000001ff, id-base: 00000100, out-base: 00000011, length: 00000007, id: 00000300 -> 00000011
r8169 0000:03:00.0: enabling device (0000 -> 0003)
r8169 0000:03:00.0: enabling Mem-Wr-Inval
r8169 0000:03:00.0: error -EIO: PCI read failed
r8169 0000:03:00.0: probe with driver r8169 failed with error -5

Only if smmu is disabled and msi-map is removed the driver probes
successfully:

r8169 0000:03:00.0: enabling device (0000 -> 0003)
r8169 0000:03:00.0: enabling Mem-Wr-Inval
r8169 0000:03:00.0 eth0: RTL8168g/8111g, d8:9d:b9:00:16:10, XID 4c0, IRQ 160
r8169 0000:03:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko]
r8169 0000:03:00.0 enp3s0: renamed from eth0
r8169 0000:03:00.0: enabling bus mastering
r8169 0000:03:00.0 enp3s0: Link is Down

quoted

Without msi-map/iommu-map:

quoted

r8169 0000:03:00.0: enabling device (0000 -> 0003)
r8169 0000:03:00.0: enabling Mem-Wr-Inval
r8169 0000:03:00.0 eth0: RTL8168g/8111g, d8:9d:b9:00:16:10, XID 4c0, IRQ 166
r8169 0000:03:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko]
r8169 0000:03:00.0 enp3s0: renamed from eth0
r8169 0000:03:00.0: enabling bus mastering
r8169 0000:03:00.0 enp3s0: Link is Down

pcie1 works as expected. But this is only a single PCIe device, rather than
having a PCIe bridge.
Any idea what's wrong here?

Can you help dump more information at for PCIe bridge case:

imx_pcie_add_lut(), need rid and sid information.
drivers/pci/controller/dwc/pci-imx6.c

Just to be clear, without msi-map and iommu-map I get:

imx6q-pcie 4c380000.pcie: rid: 0x0, sid: 0x18
imx6q-pcie 4c380000.pcie: rid: 0x100, sid: 0x19

This function get called once for each device.
Maybe the whole PCIe bus might help here, so I've put lspci output here as well.

$ lspci
0000:00:00.0 PCI bridge: Philips Semiconductors Device 0000
0000:01:00.0 PCI bridge: Pericom Semiconductor Device a303 (rev 03)
0000:02:01.0 PCI bridge: Pericom Semiconductor Device a303 (rev 03)
0000:02:02.0 PCI bridge: Pericom Semiconductor Device a303 (rev 03)
0000:03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
0000:04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8211/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
0001:00:00.0 PCI bridge: Philips Semiconductors Device 0000
0001:01:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
0002:00:00.0 Ethernet controller: Philips Semiconductors Device e101 (rev 04)
0002:00:01.0 Generic system peripheral [0807]: Philips Semiconductors Device e001 (rev 03)
0002:00:08.0 Ethernet controller: Philips Semiconductors Device e101 (rev 04)
0002:00:10.0 Ethernet controller: Philips Semiconductors Device e101 (rev 04)
0002:00:18.0 System peripheral: Philips Semiconductors Device ee02 (rev 04)
0003:01:00.0 System peripheral: Philips Semiconductors Device ee00 (rev 04)
0003:01:01.0 Generic system peripheral [0807]: Philips Semiconductors Device e001 (rev 03)

$ lspci -t
-[0000:00]---00.0-[01-ff]----00.0-[02-04]--+-01.0-[03]----00.0
                                           \-02.0-[04]----00.0
-[0001:00]---00.0-[01-ff]----00.0
-[0002:00]-+-00.0
           +-01.0
           +-08.0
           +-10.0
           \-18.0
-[0003:01]-+-00.0
           \-01.0

Thanks and best regards,
Alexander
-- 
TQ-Systems GmbH | Mühlstraße 2, Gut Delling | 82229 Seefeld, Germany
Amtsgericht München, HRB 105018
Geschäftsführer: Detlef Schneider, Rüdiger Stahl, Stefan Schneider
http://www.tq-group.com/