This patch series contains various fixes for pci-mvebu.c driver. Only
bugfixes, no new features.
For pci-mvebu.c I have prepared another 30+ patches with cleanups and
new features, they are currently available in my git branch:
https://git.kernel.org/pub/scm/linux/kernel/git/pali/linux.git/log/?h=pci-mvebu
Pali Rohár (15):
PCI: mvebu: Check for valid ports
PCI: mvebu: Check for errors from pci_bridge_emul_init() call
PCI: mvebu: Check that PCI bridge specified in DT has function number
zero
PCI: mvebu: Handle invalid size of read config request
PCI: mvebu: Disallow mapping interrupts on emulated bridges
PCI: mvebu: Fix support for bus mastering and PCI_COMMAND on emulated
bridge
PCI: mvebu: Do not modify PCI IO type bits in conf_write
PCI: mvebu: Propagate errors when updating PCI_IO_BASE and
PCI_MEM_BASE registers
PCI: mvebu: Setup PCIe controller to Root Complex mode
PCI: mvebu: Set PCI Bridge Class Code to PCI Bridge
PCI: mvebu: Fix configuring secondary bus of PCIe Root Port via
emulated bridge
PCI: mvebu: Fix support for PCI_BRIDGE_CTL_BUS_RESET on emulated
bridge
PCI: mvebu: Fix support for PCI_EXP_DEVCTL on emulated bridge
PCI: mvebu: Fix support for PCI_EXP_RTSTA on emulated bridge
PCI: mvebu: Fix support for DEVCAP2, DEVCTL2 and LNKCTL2 registers on
emulated bridge
drivers/pci/controller/pci-mvebu.c | 380 ++++++++++++++++++++++++-----
1 file changed, 313 insertions(+), 67 deletions(-)
--
2.20.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
@@ -864,6 +864,11 @@ static int mvebu_pcie_parse_port(struct mvebu_pcie *pcie,port->devfn=of_pci_get_devfn(child);if(port->devfn<0)gotoskip;+if(PCI_FUNC(port->devfn)!=0){+dev_err(dev,"%s: invalid function number, must be zero\n",+port->name);+gotoskip;+}ret=mvebu_get_tgt_attr(dev->of_node,port->devfn,IORESOURCE_MEM,&port->mem_target,&port->mem_attr);
--
2.20.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Some mvebu ports do not have to be initialized. So skip these uninitialized
mvebu ports in every port iteration function to prevent access to unmapped
memory or dereferencing NULL pointers. Uninitialized mvebu port has base
address set to NULL.
Signed-off-by: Pali Rohár <pali@kernel.org>
Cc: stable@vger.kernel.org
---
drivers/pci/controller/pci-mvebu.c | 7 +++++++
1 file changed, 7 insertions(+)
Function mvebu_pcie_hw_rd_conf() does not handle invalid size. So correctly
set read value to all-ones and return appropriate error return value
PCIBIOS_BAD_REGISTER_NUMBER like in mvebu_pcie_hw_wr_conf() function.
Signed-off-by: Pali Rohár <pali@kernel.org>
Cc: stable@vger.kernel.org
---
drivers/pci/controller/pci-mvebu.c | 3 +++
1 file changed, 3 insertions(+)
Interrupt support on mvebu emulated bridges is not implemented yet.
So properly indicate return value to callers that they cannot request
interrupts from emulated bridge.
Signed-off-by: Pali Rohár <pali@kernel.org>
Cc: stable@vger.kernel.org
---
drivers/pci/controller/pci-mvebu.c | 10 ++++++++++
1 file changed, 10 insertions(+)
Armada XP and new hardware supports access to DEVCAP2, DEVCTL2 and LNKCTL2
configuration registers of PCIe core via PCIE_CAP_PCIEXP. So export them
via emulated software root bridge.
Pre-XP hardware does not support these registers and returns zeros.
Signed-off-by: Pali Rohár <pali@kernel.org>
Fixes: 1f08673eef12 ("PCI: mvebu: Convert to PCI emulated bridge config space")
Cc: stable@vger.kernel.org
---
drivers/pci/controller/pci-mvebu.c | 23 +++++++++++++++++++++++
1 file changed, 23 insertions(+)
It looks like that mvebu PCIe controller has for each PCIe link fully
independent PCIe host bridge and so every PCIe Root Port is isolated not
only on its own bus but also isolated from each others. But in past device
tree structure was defined to put all PCIe Root Ports (as PCI Bridge
devices) into one root bus 0 and this bus is emulated by pci-mvebu.c
driver.
Probably reason for this decision was incorrect understanding of PCIe
topology of these Armada SoCs and also reason of misunderstanding how is
PCIe controller generating Type 0 and Type 1 config requests (it is fully
different compared to other drivers). Probably incorrect setup leaded to
very surprised things like having PCIe Root Port (PCI Bridge device, with
even incorrect Device Class set to Memory Controller) and the PCIe device
behind the Root Port on the same PCI bus, which obviously was needed to
somehow hack (as these two devices cannot be in reality on the same bus).
Properly set mvebu local bus number and mvebu local device number based on
PCI Bridge secondary bus number configuration. Also correctly report
configured secondary bus number in config space. And explain in driver
comment why this setup is correct.
Signed-off-by: Pali Rohár <pali@kernel.org>
Fixes: 1f08673eef12 ("PCI: mvebu: Convert to PCI emulated bridge config space")
Cc: stable@vger.kernel.org
---
drivers/pci/controller/pci-mvebu.c | 98 +++++++++++++++++++++++++++++-
1 file changed, 97 insertions(+), 1 deletion(-)
Hardware supports PCIe Hot Reset via PCIE_CTRL_OFF register. Use it for
implementing PCI_BRIDGE_CTL_BUS_RESET bit of PCI_BRIDGE_CONTROL register on
emulated bridge.
With this change the function pci_reset_secondary_bus() starts working and
can reset connected PCIe card.
Signed-off-by: Pali Rohár <pali@kernel.org>
Fixes: 1f08673eef12 ("PCI: mvebu: Convert to PCI emulated bridge config space")
Cc: stable@vger.kernel.org
---
drivers/pci/controller/pci-mvebu.c | 28 ++++++++++++++++++++++++++++
1 file changed, 28 insertions(+)
The default value of Class Code of this bridge corresponds to a Memory
controller, though. This is probably relict from the past when old
Marvell/Galileo PCI-based controllers were used as standalone PCI device
for connecting SDRAM or workaround for PCs with broken BIOS. Details are
in commit 36de23a4c5f0 ("MIPS: Cobalt: Explain GT64111 early PCI fixup").
Change the Class Code to correspond to a PCI Bridge.
Add comment explaining this change.
Signed-off-by: Pali Rohár <pali@kernel.org>
Cc: stable@vger.kernel.org
---
drivers/pci/controller/pci-mvebu.c | 28 +++++++++++++++++++++++++++-
1 file changed, 27 insertions(+), 1 deletion(-)
@@ -237,6 +237,32 @@ static void mvebu_pcie_setup_hw(struct mvebu_pcie_port *port)cmd&=~(PCI_COMMAND_IO|PCI_COMMAND_MEMORY|PCI_COMMAND_MASTER);mvebu_writel(port,cmd,PCIE_CMD_OFF);+/*+*ChangeClassCodeofPCIBridgedevicetoPCIBridge(0x6004)+*becausedefaultvalueisMemorycontroller(0x5080).+*+*NotethatthismvebuPCIBridgedoesnothavecompliantType1+*ConfigurationSpace.HeaderTypeisreportedasType0andit+*hasformatofType0configspace.+*+*MoreoverType0BARregisters(ranges0x10-0x28and0x30-0x34)+*havethesameformatinMarvell'sspecificationasinPCIe+*specification,buttheirmeaningistotallydifferentandtheydo+*differentthings:theyarealiasedintointernalmveburegisters+*(e.g.PCIE_BAR_LO_OFF)andtheseshouldnotbechangedor+*reconfiguredbypcidevicedrivers.+*+*ThereforedriverusesemulationofPCIBridgewhichemulates+*accesstoconfigurationspaceviainternalmveburegistersor+*emulatedconfigurationbuffer.DriveraccessthesePCIBridge+*directlyforsimplification,buttheseregisterscanbeaccessed+*alsoviastandardmvebuwayforaccessingPCIconfigspace.+*/+dev_rev=mvebu_readl(port,PCIE_DEV_REV_OFF);+dev_rev&=~0xffffff00;+dev_rev|=(PCI_CLASS_BRIDGE_PCI<<8)<<8;+mvebu_writel(port,dev_rev,PCIE_DEV_REV_OFF);+/* Point PCIe unit MBUS decode windows to DRAM space. */mvebu_pcie_setup_wins(port);
--
2.20.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Properly propagate failure from mvebu_pcie_add_windows() function back to
the caller mvebu_pci_bridge_emul_base_conf_write() and correctly updates
PCI_IO_BASE, PCI_MEM_BASE and PCI_IO_BASE_UPPER16 registers on error.
On error set base value higher than limit value which indicates that
address range is disabled. When IO is unsupported then let IO registers
zeroed as required by PCIe base specification.
Signed-off-by: Pali Rohár <pali@kernel.org>
Cc: stable@vger.kernel.org
---
drivers/pci/controller/pci-mvebu.c | 82 ++++++++++++++++++++----------
1 file changed, 55 insertions(+), 27 deletions(-)
@@ -369,30 +373,35 @@ static void mvebu_pcie_set_window(struct mvebu_pcie_port *port,}if(desired->size==0)-return;+return0;++ret=mvebu_pcie_add_windows(port,target,attribute,desired->base,+desired->size,desired->remap);+if(ret){+cur->size=0;+cur->base=0;+returnret;+}-mvebu_pcie_add_windows(port,target,attribute,desired->base,-desired->size,desired->remap);*cur=*desired;+return0;}-staticvoidmvebu_pcie_handle_iobase_change(structmvebu_pcie_port*port)+staticintmvebu_pcie_handle_iobase_change(structmvebu_pcie_port*port){structmvebu_pcie_windowdesired={};structpci_bridge_emul_conf*conf=&port->bridge.conf;/* Are the new iobase/iolimit values invalid? */if(conf->iolimit<conf->iobase||-conf->iolimitupper<conf->iobaseupper){-mvebu_pcie_set_window(port,port->io_target,port->io_attr,-&desired,&port->iowin);-return;-}+conf->iolimitupper<conf->iobaseupper)+returnmvebu_pcie_set_window(port,port->io_target,port->io_attr,+&desired,&port->iowin);if(!mvebu_has_ioport(port)){dev_WARN(&port->pcie->pdev->dev,"Attempt to set IO when IO is disabled\n");-return;+return-EOPNOTSUPP;}/*
@@ -410,21 +419,19 @@ static void mvebu_pcie_handle_iobase_change(struct mvebu_pcie_port *port)desired.remap)+1;-mvebu_pcie_set_window(port,port->io_target,port->io_attr,&desired,-&port->iowin);+returnmvebu_pcie_set_window(port,port->io_target,port->io_attr,&desired,+&port->iowin);}-staticvoidmvebu_pcie_handle_membase_change(structmvebu_pcie_port*port)+staticintmvebu_pcie_handle_membase_change(structmvebu_pcie_port*port){structmvebu_pcie_windowdesired={.remap=MVEBU_MBUS_NO_REMAP};structpci_bridge_emul_conf*conf=&port->bridge.conf;/* Are the new membase/memlimit values invalid? */-if(conf->memlimit<conf->membase){-mvebu_pcie_set_window(port,port->mem_target,port->mem_attr,-&desired,&port->memwin);-return;-}+if(conf->memlimit<conf->membase)+returnmvebu_pcie_set_window(port,port->mem_target,port->mem_attr,+&desired,&port->memwin);/**WereadthePCI-to-PCIbridgeemulatedregisters,and
@@ -522,15 +529,36 @@ mvebu_pci_bridge_emul_base_conf_write(struct pci_bridge_emul *bridge,break;casePCI_IO_BASE:-mvebu_pcie_handle_iobase_change(port);+if((mask&0xffff)&&mvebu_pcie_handle_iobase_change(port)){+/* On error disable IO range */+conf->iobase&=~0xf0;+conf->iolimit&=~0xf0;+conf->iobaseupper=cpu_to_le16(0x0000);+conf->iolimitupper=cpu_to_le16(0x0000);+if(mvebu_has_ioport(port))+conf->iobase|=0xf0;+}break;casePCI_MEMORY_BASE:-mvebu_pcie_handle_membase_change(port);+if(mvebu_pcie_handle_membase_change(port)){+/* On error disable mem range */+conf->membase=cpu_to_le16(le16_to_cpu(conf->membase)&~0xfff0);+conf->memlimit=cpu_to_le16(le16_to_cpu(conf->memlimit)&~0xfff0);+conf->membase=cpu_to_le16(le16_to_cpu(conf->membase)|0xfff0);+}break;casePCI_IO_BASE_UPPER16:-mvebu_pcie_handle_iobase_change(port);+if(mvebu_pcie_handle_iobase_change(port)){+/* On error disable IO range */+conf->iobase&=~0xf0;+conf->iolimit&=~0xf0;+conf->iobaseupper=cpu_to_le16(0x0000);+conf->iolimitupper=cpu_to_le16(0x0000);+if(mvebu_has_ioport(port))+conf->iobase|=0xf0;+}break;casePCI_PRIMARY_BUS:
--
2.20.1
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
According to PCI specifications bits [0:2] of Command Register, this should
be by default disabled on reset. So explicitly disable these bits at early
beginning of driver initialization.
Also remove code which unconditionally enables all 3 bits and let kernel
code (via pci_set_master() function) to handle bus mastering of PCI Bridge
via emulated PCI_COMMAND on emulated bridge.
Adjust existing functions mvebu_pcie_handle_iobase_change() and
mvebu_pcie_handle_membase_change() to handle PCI_IO_BASE and PCI_MEM_BASE
registers correctly even when bus mastering on emulated bridge is disabled.
Signed-off-by: Pali Rohár <pali@kernel.org>
Cc: stable@vger.kernel.org
---
drivers/pci/controller/pci-mvebu.c | 52 ++++++++++++++++++------------
1 file changed, 32 insertions(+), 20 deletions(-)
@@ -226,16 +226,14 @@ static void mvebu_pcie_setup_hw(struct mvebu_pcie_port *port){u32cmd,mask;-/* Point PCIe unit MBUS decode windows to DRAM space. */-mvebu_pcie_setup_wins(port);--/* Master + slave enable. */+/* Disable Root Bridge I/O space, memory space and bus mastering. */cmd=mvebu_readl(port,PCIE_CMD_OFF);-cmd|=PCI_COMMAND_IO;-cmd|=PCI_COMMAND_MEMORY;-cmd|=PCI_COMMAND_MASTER;+cmd&=~(PCI_COMMAND_IO|PCI_COMMAND_MEMORY|PCI_COMMAND_MASTER);mvebu_writel(port,cmd,PCIE_CMD_OFF);+/* Point PCIe unit MBUS decode windows to DRAM space. */+mvebu_pcie_setup_wins(port);+/* Enable interrupt lines A-D. */mask=mvebu_readl(port,PCIE_MASK_OFF);mask|=PCIE_MASK_ENABLE_INTS;
@@ -385,8 +383,7 @@ static void mvebu_pcie_handle_iobase_change(struct mvebu_pcie_port *port)/* Are the new iobase/iolimit values invalid? */if(conf->iolimit<conf->iobase||-conf->iolimitupper<conf->iobaseupper||-!(conf->command&PCI_COMMAND_IO)){+conf->iolimitupper<conf->iobaseupper){mvebu_pcie_set_window(port,port->io_target,port->io_attr,&desired,&port->iowin);return;
@@ -423,8 +420,7 @@ static void mvebu_pcie_handle_membase_change(struct mvebu_pcie_port *port)structpci_bridge_emul_conf*conf=&port->bridge.conf;/* Are the new membase/memlimit values invalid? */-if(conf->memlimit<conf->membase||-!(conf->command&PCI_COMMAND_MEMORY)){+if(conf->memlimit<conf->membase){mvebu_pcie_set_window(port,port->mem_target,port->mem_attr,&desired,&port->memwin);return;
PCI IO type bits are already initialized in mvebu_pci_bridge_emul_init()
function and only when IO support is enabled. These type bits are read-only
and pci-bridge-emul.c code already does not allow to modify them from upper
layers.
When IO support is disabled then all IO registers should be read-only and
return zeros. Therefore do not modify PCI IO type bits in
mvebu_pci_bridge_emul_base_conf_write() callback.
Signed-off-by: Pali Rohár <pali@kernel.org>
Fixes: 1f08673eef12 ("PCI: mvebu: Convert to PCI emulated bridge config space")
Cc: stable@vger.kernel.org
---
drivers/pci/controller/pci-mvebu.c | 7 -------
1 file changed, 7 deletions(-)
PME Status bit in Root Status Register (PCIE_RC_RTSTA_OFF) is read-only and
can be cleared only by writing 0b to the Interrupt Cause RW0C register
(PCIE_INT_CAUSE_OFF).
Signed-off-by: Pali Rohár <pali@kernel.org>
Fixes: 1f08673eef12 ("PCI: mvebu: Convert to PCI emulated bridge config space")
Cc: stable@vger.kernel.org
---
drivers/pci/controller/pci-mvebu.c | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
On Thu, Nov 25, 2021 at 01:45:53PM +0100, Pali Rohár wrote:
Driver cannot handle PCI bridges at non-zero function address. So add
appropriate check. Currently all in-tree kernel DTS files set PCI bridge
function to zero.
Why can the driver not handle bridges at non-zero function addresses?
The PCI spec allows that, doesn't it? Is this a hardware limitation?
@@ -864,6 +864,11 @@ static int mvebu_pcie_parse_port(struct mvebu_pcie *pcie,port->devfn=of_pci_get_devfn(child);if(port->devfn<0)gotoskip;+if(PCI_FUNC(port->devfn)!=0){+dev_err(dev,"%s: invalid function number, must be zero\n",+port->name);+gotoskip;+}ret=mvebu_get_tgt_attr(dev->of_node,port->devfn,IORESOURCE_MEM,&port->mem_target,&port->mem_attr);
On Friday 07 January 2022 12:15:12 Bjorn Helgaas wrote:
On Thu, Nov 25, 2021 at 01:45:53PM +0100, Pali Rohár wrote:
quoted
Driver cannot handle PCI bridges at non-zero function address. So add
appropriate check. Currently all in-tree kernel DTS files set PCI bridge
function to zero.
Why can the driver not handle bridges at non-zero function addresses?
The PCI spec allows that, doesn't it? Is this a hardware limitation?
It is software / kernel limitation.
Because this bridge is virtual, emulated by pci-bridge-emul.c driver and
this driver can emulate only single function PCI-to-PCI bridge device.
@@ -864,6 +864,11 @@ static int mvebu_pcie_parse_port(struct mvebu_pcie *pcie,port->devfn=of_pci_get_devfn(child);if(port->devfn<0)gotoskip;+if(PCI_FUNC(port->devfn)!=0){+dev_err(dev,"%s: invalid function number, must be zero\n",+port->name);+gotoskip;+}ret=mvebu_get_tgt_attr(dev->of_node,port->devfn,IORESOURCE_MEM,&port->mem_target,&port->mem_attr);
On Thu, Nov 25, 2021 at 01:45:54PM +0100, Pali Rohár wrote:
Function mvebu_pcie_hw_rd_conf() does not handle invalid size. So correctly
set read value to all-ones and return appropriate error return value
PCIBIOS_BAD_REGISTER_NUMBER like in mvebu_pcie_hw_wr_conf() function.
Signed-off-by: Pali Rohár <pali@kernel.org>
Cc: stable@vger.kernel.org
Is there a bug that this fixes? If not, I would drop the stable tag
(as I see Lorenzo already did, thanks!).
@@ -261,6 +261,9 @@ static int mvebu_pcie_hw_rd_conf(struct mvebu_pcie_port *port,case4:*val=readl_relaxed(conf_data);break;+default:+*val=0xffffffff;+returnPCIBIOS_BAD_REGISTER_NUMBER;
Might be the right thing to do, but there are many config accessors
that do not set *val to ~0 before returning
PCIBIOS_BAD_REGISTER_NUMBER:
pci_bus_read_config_byte (and word, dword) # PCI_OP_READ(), *val unchanged
pci_generic_config_read # *val = 32-bit value
pci_user_read_config_byte (...) # PCI_USER_READ_CONFIG(), *val unchanged
sh7786_pcie_read # *val unchanged
dw_pcie_read # *val = 0
mobiveil_pcie_read # *val = 0
faraday_raw_pci_read_config # *val = 32-bit value
ixp4xx_pci_read_config # *val unchanged
orion5x_pci_hw_rd_conf # *val = 32-bit value
orion_pcie_rd_conf # *val = 32-bit value
bonito64_pcibios_read # *val = 32-bit value
loongson_pcibios_read # *val = 32-bit value
msc_pcibios_read # *val = 32-bit value
ar724x_pci_read # *val unchanged
bcm1480_pcibios_read # *val = 32-bit value
_altera_pcie_cfg_read # *val = 32-bit value
rockchip_pcie_rd_own_conf # *val = 0
rockchip_pcie_rd_other_conf # *val = 0
pci_bridge_emul_conf_read # may depend on op?
There are more, but I got tired of looking. I actually didn't see any
that set *val to ~0.
I think the check in PCI_OP_READ() means that most accessors will
never see an invalid "size".
From: "Russell King (Oracle)" <linux@armlinux.org.uk> Date: 2022-01-07 19:15:39
On Fri, Jan 07, 2022 at 12:45:48PM -0600, Bjorn Helgaas wrote:
On Thu, Nov 25, 2021 at 01:45:54PM +0100, Pali Rohár wrote:
quoted
Function mvebu_pcie_hw_rd_conf() does not handle invalid size. So correctly
set read value to all-ones and return appropriate error return value
PCIBIOS_BAD_REGISTER_NUMBER like in mvebu_pcie_hw_wr_conf() function.
Signed-off-by: Pali Rohár <pali@kernel.org>
Cc: stable@vger.kernel.org
Is there a bug that this fixes? If not, I would drop the stable tag
(as I see Lorenzo already did, thanks!).
@@ -261,6 +261,9 @@ static int mvebu_pcie_hw_rd_conf(struct mvebu_pcie_port *port,case4:*val=readl_relaxed(conf_data);break;+default:+*val=0xffffffff;+returnPCIBIOS_BAD_REGISTER_NUMBER;
Might be the right thing to do, but there are many config accessors
that do not set *val to ~0 before returning
PCIBIOS_BAD_REGISTER_NUMBER:
I think a better question would be - how can this function be called
with a size that isn't 1, 2 or 4? I suppose if someone were to add
another PCI_OP_READ/PCI_OP_WRITE. However... they really need to audit
every implementation if they do that.
The generic implementation does this:
if (size == 1)
*val = readb(addr);
else if (size == 2)
*val = readw(addr);
else
*val = readl(addr);
and therefore completely ignores the size if it isn't 1 or 2. So I
don't think this is something that needs fixing.
If we're going to fix this in drivers, shouldn't we fix the generic
implementation too?
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
On Fri, Jan 07, 2022 at 07:18:19PM +0100, Pali Rohár wrote:
On Friday 07 January 2022 12:15:12 Bjorn Helgaas wrote:
quoted
On Thu, Nov 25, 2021 at 01:45:53PM +0100, Pali Rohár wrote:
quoted
Driver cannot handle PCI bridges at non-zero function address. So add
appropriate check. Currently all in-tree kernel DTS files set PCI bridge
function to zero.
Why can the driver not handle bridges at non-zero function addresses?
The PCI spec allows that, doesn't it? Is this a hardware limitation?
It is software / kernel limitation.
Because this bridge is virtual, emulated by pci-bridge-emul.c driver and
this driver can emulate only single function PCI-to-PCI bridge device.
That's weird. Why does pci-bridge-emul.c care about the function
number? Or maybe you're saying that pci-mvebu.c isn't smart enough to
build an emulated bridge at a non-zero function? Or, since this is
emulated, maybe there's just no *reason* to ever use a non-zero
function?
It would really be nice to have the commit log and maybe even a
comment allude to what's going on here . Otherwise this check sort of
dangles without having an obvious reason for existence.
Does this issue also affect pci-aardvark.c (which looks like the only
other user of pci_bridge_emul_init())?
@@ -864,6 +864,11 @@ static int mvebu_pcie_parse_port(struct mvebu_pcie *pcie,port->devfn=of_pci_get_devfn(child);if(port->devfn<0)gotoskip;+if(PCI_FUNC(port->devfn)!=0){+dev_err(dev,"%s: invalid function number, must be zero\n",+port->name);+gotoskip;+}ret=mvebu_get_tgt_attr(dev->of_node,port->devfn,IORESOURCE_MEM,&port->mem_target,&port->mem_attr);
On Thu, Nov 25, 2021 at 01:45:55PM +0100, Pali Rohár wrote:
Interrupt support on mvebu emulated bridges is not implemented yet.
Is this mvebu-specific, or is aardvar also affected?
So properly indicate return value to callers that they cannot request
interrupts from emulated bridge.
Pet peeve: descriptions that say "do this *properly*". As though the
previous authors were just ignorant or intentionally did something
*improperly* :)
I assume this means INTx doesn't work for some devices? Which ones?
I guess anything on the root bus? But INTx for devices *below* these
emulated Root Ports *does* work?
On Thu, Nov 25, 2021 at 01:45:58PM +0100, Pali Rohár wrote:
Properly propagate failure from mvebu_pcie_add_windows() function back to
the caller mvebu_pci_bridge_emul_base_conf_write() and correctly updates
PCI_IO_BASE, PCI_MEM_BASE and PCI_IO_BASE_UPPER16 registers on error.
On error set base value higher than limit value which indicates that
address range is disabled.
Does the spec say that if software programs something invalid,
hardware should proactively set the base and limit registers to
disable the window?
I'm not sure I've seen hardware that does this, and it seems ... maybe
a little aggressive.
What happens if software writes the base and limit in the wrong order,
so the window is invalid after the first write but valid after the
second? That actually sounds like it could be a sensible strategy to
prevent a partially-configured window from being active.
Bjorn
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
On Friday 07 January 2022 15:09:02 Bjorn Helgaas wrote:
On Fri, Jan 07, 2022 at 07:18:19PM +0100, Pali Rohár wrote:
quoted
On Friday 07 January 2022 12:15:12 Bjorn Helgaas wrote:
quoted
On Thu, Nov 25, 2021 at 01:45:53PM +0100, Pali Rohár wrote:
quoted
Driver cannot handle PCI bridges at non-zero function address. So add
appropriate check. Currently all in-tree kernel DTS files set PCI bridge
function to zero.
Why can the driver not handle bridges at non-zero function addresses?
The PCI spec allows that, doesn't it? Is this a hardware limitation?
It is software / kernel limitation.
Because this bridge is virtual, emulated by pci-bridge-emul.c driver and
this driver can emulate only single function PCI-to-PCI bridge device.
That's weird. Why does pci-bridge-emul.c care about the function
number?
pci-bridge-emul.c emulates whole PCI config space and multifunction PCI
device needs to have Multi-Function Device bit set. Which
pci-bridge-emul.c does not do as it "emulates" only singe-function
device. Also some extended PCIe registers needs to be aligned for
multifunction device. And for simplification nothing from this is
implemented in that pci-bridge-emul.c driver. Basically single function
device is easily to emulate than multi function device. And for
simplicity of driver it is just better to do not implement more stuff
if it is not required.
Or maybe you're saying that pci-mvebu.c isn't smart enough to
build an emulated bridge at a non-zero function? Or, since this is
emulated, maybe there's just no *reason* to ever use a non-zero
function?
These PCIe root ports are basically on different PCI domains, every root
port with its subtree has its own configuration, including own access
to config space of subdevices. And I do not think that there is a reason
to try putting root port (as emulated pci-to-pci bridge) on non-zero
function and putting separate root ports into "one" multifunction
device.
Technically it could be possible to implement it and also properly, as
it is just emulation of PCI device. But why? Just big complication
without any benefit. At least I do not see benefit.
It would really be nice to have the commit log and maybe even a
comment allude to what's going on here . Otherwise this check sort of
dangles without having an obvious reason for existence.
Does this issue also affect pci-aardvark.c (which looks like the only
other user of pci_bridge_emul_init())?
Theoretically yes. But pci-aardvark is single-root-port hardware and
therefore it is single-function device. And emulated device is
registered by pci-aardvark driver, not by DT. Because it is
single-root-port HW there is no DT node for root port like for any other
single-root-port PCIe controllers. So practically no.
@@ -864,6 +864,11 @@ static int mvebu_pcie_parse_port(struct mvebu_pcie *pcie,port->devfn=of_pci_get_devfn(child);if(port->devfn<0)gotoskip;+if(PCI_FUNC(port->devfn)!=0){+dev_err(dev,"%s: invalid function number, must be zero\n",+port->name);+gotoskip;+}ret=mvebu_get_tgt_attr(dev->of_node,port->devfn,IORESOURCE_MEM,&port->mem_target,&port->mem_attr);
On Friday 07 January 2022 15:32:16 Bjorn Helgaas wrote:
On Thu, Nov 25, 2021 at 01:45:55PM +0100, Pali Rohár wrote:
quoted
Interrupt support on mvebu emulated bridges is not implemented yet.
Is this mvebu-specific, or is aardvar also affected?
This is pci-mvebu.c driver specific, it does not implement emulation of
neither INTx, nor MSI interrupts for emulated pci bridge (root port). As
we know this HW does not have compliant pci root port, it needs to be
emulated in driver, and emulation for interrupts is missing. (it means
that also AER interrupt is missing).
And pci-aardvark.c driver has same issue and similar patch is required
for pci-aardvark.c too. Marek should take care of it. But for
pci-aardvark we already have implementation which emulates INTx
interrupts and it is waiting for review on the list:
https://lore.kernel.org/linux-pci/20211208061851.31867-1-kabel@kernel.org/
quoted
So properly indicate return value to callers that they cannot request
interrupts from emulated bridge.
Pet peeve: descriptions that say "do this *properly*". As though the
previous authors were just ignorant or intentionally did something
*improperly* :)
@@ -705,6 +705,15 @@ static struct pci_ops mvebu_pcie_ops = {.write=mvebu_pcie_wr_conf,};+staticintmvebu_pcie_map_irq(conststructpci_dev*dev,u8slot,u8pin)+{+/* Interrupt support on mvebu emulated bridges is not implemented yet */+if(dev->bus->number==0)+return0;/* Proper return code 0 == NO_IRQ */++returnof_irq_parse_and_map_pci(dev,slot,pin);
Is this something that could be done with a .read_base() op, e.g.,
make PCI_INTERRUPT_PIN contain zero (PCI_INTERRUPT_UNKNOWN)?
I'm not sure... maybe. I choose this style as after I implement
emulation of INTx interrupts it allows me just to replace "return 0;" by
"return my_mapping_function_for_root_port(...);". Similarly like it is
in pending pci-aardvark.c patch:
https://lore.kernel.org/linux-pci/20211208061851.31867-15-kabel@kernel.org/
I assume this means INTx doesn't work for some devices? Which ones?
I guess anything on the root bus? But INTx for devices *below* these
emulated Root Ports *does* work?
Exactly. All devices except emulated root ports (which are on bus 0)
have working MSI, MSI-X and INTx interrupts.
On Friday 07 January 2022 15:55:04 Bjorn Helgaas wrote:
On Thu, Nov 25, 2021 at 01:45:58PM +0100, Pali Rohár wrote:
quoted
Properly propagate failure from mvebu_pcie_add_windows() function back to
the caller mvebu_pci_bridge_emul_base_conf_write() and correctly updates
PCI_IO_BASE, PCI_MEM_BASE and PCI_IO_BASE_UPPER16 registers on error.
On error set base value higher than limit value which indicates that
address range is disabled.
Does the spec say that if software programs something invalid,
hardware should proactively set the base and limit registers to
disable the window?
No. But this patch address something totally different. Software can do
fully valid operation, e.g. try to set forwarding memory window as large
as possible. But because this driver "emulates" pci bridge by calling
software/kernel function (mvebu_pcie_add_windows), some operations which
in real HW cannot happen, are possible in software.
For example there are limitations in sizes of forwarding memory windows,
because it is done by mvebu-mbus driver, which is responsible for
configuring mapping and forwarding of PCIe I/O and MEM windows. And due
to Marvell HW, there are restrictions which are not in PCIe HW.
Currently if such error happens, obviously kernel is not able to set
PCIe windows and it just print warnings to dmesg. Trying to access these
windows would result in the worst case in crashes.
With this change when mvebu_pcie_add_windows() function fails then into
emulated config space is put information that particular forwarding
window is disabled. I think that it is better to indicate it in config
space what is the current "reality" of hardware configuration. If window
is disabled in real-HW (meaning in mvebu-mbus driver) then show it also
in emulated config space of pci bridge.
Do you have better idea what should emulated pci bridge do, if software
try to set fully valid configuration of forwarding window, but it is not
possible to achieve it (even compliant PCI bridge must be able to do
it)?
I'm not sure I've seen hardware that does this, and it seems ... maybe
a little aggressive.
What happens if software writes the base and limit in the wrong order,
so the window is invalid after the first write but valid after the
second? That actually sounds like it could be a sensible strategy to
prevent a partially-configured window from being active.
Bjorn
Invalid window (limit < base) means that window is disabled. And
pci-mvebu.c in its callbacks from pci-bridge-emul.c should correctly
handle it and propagates information about disablement to mvebu-mbus
driver.
After window is valid again (limit > base) then pci-mvebu.c call
mvebu-mbus to setup new mapping.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
On Fri, Jan 07, 2022 at 11:13:48PM +0100, Pali Rohár wrote:
On Friday 07 January 2022 15:32:16 Bjorn Helgaas wrote:
quoted
On Thu, Nov 25, 2021 at 01:45:55PM +0100, Pali Rohár wrote:
quoted
Interrupt support on mvebu emulated bridges is not implemented yet.
Is this mvebu-specific, or is aardvar also affected?
This is pci-mvebu.c driver specific, it does not implement emulation of
neither INTx, nor MSI interrupts for emulated pci bridge (root port). As
we know this HW does not have compliant pci root port, it needs to be
emulated in driver, and emulation for interrupts is missing. (it means
that also AER interrupt is missing).
And pci-aardvark.c driver has same issue and similar patch is required
for pci-aardvark.c too. Marek should take care of it. But for
pci-aardvark we already have implementation which emulates INTx
interrupts and it is waiting for review on the list:
https://lore.kernel.org/linux-pci/20211208061851.31867-1-kabel@kernel.org/
quoted
quoted
So properly indicate return value to callers that they cannot request
interrupts from emulated bridge.
Pet peeve: descriptions that say "do this *properly*". As though the
previous authors were just ignorant or intentionally did something
*improperly* :)
@@ -705,6 +705,15 @@ static struct pci_ops mvebu_pcie_ops = {.write=mvebu_pcie_wr_conf,};+staticintmvebu_pcie_map_irq(conststructpci_dev*dev,u8slot,u8pin)+{+/* Interrupt support on mvebu emulated bridges is not implemented yet */+if(dev->bus->number==0)+return0;/* Proper return code 0 == NO_IRQ */++returnof_irq_parse_and_map_pci(dev,slot,pin);
Is this something that could be done with a .read_base() op, e.g.,
make PCI_INTERRUPT_PIN contain zero (PCI_INTERRUPT_UNKNOWN)?
I'm not sure... maybe. I choose this style as after I implement
emulation of INTx interrupts it allows me just to replace "return 0;" by
"return my_mapping_function_for_root_port(...);".
OK, so even after you implement INTx for the emulated Root Ports, the
default of_irq_parse_and_map_pci() is insufficient, and you will
require an mvebu .map_irq() function. That's reasonable.
"PCI_INTERRUPT_PIN == 0" is the way software learns that a device
doesn't use INTx, of course, and I suppose PCI_INTERRUPT_PIN already
reads as zero, since mvebu_pci_bridge_emul_init() doesn't set
bridge->conf.intpin, and I assume the default value would be zero?
Bjorn
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
On Friday 07 January 2022 17:01:55 Bjorn Helgaas wrote:
On Fri, Jan 07, 2022 at 11:13:48PM +0100, Pali Rohár wrote:
quoted
On Friday 07 January 2022 15:32:16 Bjorn Helgaas wrote:
quoted
On Thu, Nov 25, 2021 at 01:45:55PM +0100, Pali Rohár wrote:
quoted
Interrupt support on mvebu emulated bridges is not implemented yet.
Is this mvebu-specific, or is aardvar also affected?
This is pci-mvebu.c driver specific, it does not implement emulation of
neither INTx, nor MSI interrupts for emulated pci bridge (root port). As
we know this HW does not have compliant pci root port, it needs to be
emulated in driver, and emulation for interrupts is missing. (it means
that also AER interrupt is missing).
And pci-aardvark.c driver has same issue and similar patch is required
for pci-aardvark.c too. Marek should take care of it. But for
pci-aardvark we already have implementation which emulates INTx
interrupts and it is waiting for review on the list:
https://lore.kernel.org/linux-pci/20211208061851.31867-1-kabel@kernel.org/
quoted
quoted
So properly indicate return value to callers that they cannot request
interrupts from emulated bridge.
Pet peeve: descriptions that say "do this *properly*". As though the
previous authors were just ignorant or intentionally did something
*improperly* :)
@@ -705,6 +705,15 @@ static struct pci_ops mvebu_pcie_ops = {.write=mvebu_pcie_wr_conf,};+staticintmvebu_pcie_map_irq(conststructpci_dev*dev,u8slot,u8pin)+{+/* Interrupt support on mvebu emulated bridges is not implemented yet */+if(dev->bus->number==0)+return0;/* Proper return code 0 == NO_IRQ */++returnof_irq_parse_and_map_pci(dev,slot,pin);
Is this something that could be done with a .read_base() op, e.g.,
make PCI_INTERRUPT_PIN contain zero (PCI_INTERRUPT_UNKNOWN)?
I'm not sure... maybe. I choose this style as after I implement
emulation of INTx interrupts it allows me just to replace "return 0;" by
"return my_mapping_function_for_root_port(...);".
OK, so even after you implement INTx for the emulated Root Ports, the
default of_irq_parse_and_map_pci() is insufficient, and you will
require an mvebu .map_irq() function. That's reasonable.
"PCI_INTERRUPT_PIN == 0" is the way software learns that a device
doesn't use INTx, of course, and I suppose PCI_INTERRUPT_PIN already
reads as zero, since mvebu_pci_bridge_emul_init() doesn't set
bridge->conf.intpin, and I assume the default value would be zero?
Bjorn
Yes, looks like that zeros are in emulated config space for fields not
explicitly initialized. Which is the pci-mvebu.c case.
But now I'm looking at pci-aardvark.c driver and it sets
PCI_INTERRUPT_PIN register to A:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/pci-aardvark.c?h=v5.16-rc8#n953
And that comment "/* Support interrupt A for MSI feature */" must be
total nonsense as INTA for sure is not required for MSI... Plus we know
that pci-aardvark.c driver does not implement for pci bridge neither
INTx nor MSI... Ach... seems that this code is here since beginning and
needs to be fixed...
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
On Fri, Jan 07, 2022 at 11:28:26PM +0100, Pali Rohár wrote:
On Friday 07 January 2022 15:55:04 Bjorn Helgaas wrote:
quoted
On Thu, Nov 25, 2021 at 01:45:58PM +0100, Pali Rohár wrote:
quoted
Properly propagate failure from mvebu_pcie_add_windows() function back to
the caller mvebu_pci_bridge_emul_base_conf_write() and correctly updates
PCI_IO_BASE, PCI_MEM_BASE and PCI_IO_BASE_UPPER16 registers on error.
On error set base value higher than limit value which indicates that
address range is disabled.
Does the spec say that if software programs something invalid,
hardware should proactively set the base and limit registers to
disable the window?
No. But this patch address something totally different. Software can do
fully valid operation, e.g. try to set forwarding memory window as large
as possible. But because this driver "emulates" pci bridge by calling
software/kernel function (mvebu_pcie_add_windows), some operations which
in real HW cannot happen, are possible in software.
For example there are limitations in sizes of forwarding memory windows,
because it is done by mvebu-mbus driver, which is responsible for
configuring mapping and forwarding of PCIe I/O and MEM windows. And due
to Marvell HW, there are restrictions which are not in PCIe HW.
Currently if such error happens, obviously kernel is not able to set
PCIe windows and it just print warnings to dmesg. Trying to access these
windows would result in the worst case in crashes.
With this change when mvebu_pcie_add_windows() function fails then into
emulated config space is put information that particular forwarding
window is disabled. I think that it is better to indicate it in config
space what is the current "reality" of hardware configuration. If window
is disabled in real-HW (meaning in mvebu-mbus driver) then show it also
in emulated config space of pci bridge.
Do you have better idea what should emulated pci bridge do, if software
try to set fully valid configuration of forwarding window, but it is not
possible to achieve it (even compliant PCI bridge must be able to do
it)?
On an ACPI system, the host bridge window sizes are constrained by the
host bridge _CRS method. I assume there's a similar constraint in DT.
Is the fact that mvebu_pcie_add_windows() can fail a symptom of a DT
that describes more available space than mvebu-bus can map?
quoted
I'm not sure I've seen hardware that does this, and it seems ... maybe
a little aggressive.
What happens if software writes the base and limit in the wrong order,
so the window is invalid after the first write but valid after the
second? That actually sounds like it could be a sensible strategy to
prevent a partially-configured window from being active.
Invalid window (limit < base) means that window is disabled. And
pci-mvebu.c in its callbacks from pci-bridge-emul.c should correctly
handle it and propagates information about disablement to mvebu-mbus
driver.
After window is valid again (limit > base) then pci-mvebu.c call
mvebu-mbus to setup new mapping.
Not sure I'm understanding the code correctly. Here's the sort of
thing I'm worried about, but maybe this is actually impossible:
Let's say software writes (0x00, 0xff) to the I/O (base, limit), which
describes the [io 0x0000-0xffff] window. If mvebu-mbus can't handle
that, it looks like you set the (base, limit) to (0xf0, 0x0f), which
would describe [io 0xf000-0x0fff], which is invalid.
The software writes 0x40 to the limit, so now we have (0xf0, 0x40), or
[io 0xf000-0x40ff]. That's still invalid, but software thinks the
0x00 it wrote to the base is still there.
Bjorn
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
On Friday 07 January 2022 17:16:17 Bjorn Helgaas wrote:
On Fri, Jan 07, 2022 at 11:28:26PM +0100, Pali Rohár wrote:
quoted
On Friday 07 January 2022 15:55:04 Bjorn Helgaas wrote:
quoted
On Thu, Nov 25, 2021 at 01:45:58PM +0100, Pali Rohár wrote:
quoted
Properly propagate failure from mvebu_pcie_add_windows() function back to
the caller mvebu_pci_bridge_emul_base_conf_write() and correctly updates
PCI_IO_BASE, PCI_MEM_BASE and PCI_IO_BASE_UPPER16 registers on error.
On error set base value higher than limit value which indicates that
address range is disabled.
Does the spec say that if software programs something invalid,
hardware should proactively set the base and limit registers to
disable the window?
No. But this patch address something totally different. Software can do
fully valid operation, e.g. try to set forwarding memory window as large
as possible. But because this driver "emulates" pci bridge by calling
software/kernel function (mvebu_pcie_add_windows), some operations which
in real HW cannot happen, are possible in software.
For example there are limitations in sizes of forwarding memory windows,
because it is done by mvebu-mbus driver, which is responsible for
configuring mapping and forwarding of PCIe I/O and MEM windows. And due
to Marvell HW, there are restrictions which are not in PCIe HW.
Currently if such error happens, obviously kernel is not able to set
PCIe windows and it just print warnings to dmesg. Trying to access these
windows would result in the worst case in crashes.
With this change when mvebu_pcie_add_windows() function fails then into
emulated config space is put information that particular forwarding
window is disabled. I think that it is better to indicate it in config
space what is the current "reality" of hardware configuration. If window
is disabled in real-HW (meaning in mvebu-mbus driver) then show it also
in emulated config space of pci bridge.
Do you have better idea what should emulated pci bridge do, if software
try to set fully valid configuration of forwarding window, but it is not
possible to achieve it (even compliant PCI bridge must be able to do
it)?
On an ACPI system, the host bridge window sizes are constrained by the
host bridge _CRS method. I assume there's a similar constraint in DT.
Is the fact that mvebu_pcie_add_windows() can fail a symptom of a DT
that describes more available space than mvebu-bus can map?
Memory maps for mvebu are more complicated. There is no explicit size in
DT ranges property as it is dynamically allocated by mvebu-mbus:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm/boot/dts/armada-385.dtsi?h=v5.15#n47
On some Armada platform (I think it is AXP) there is lot of SerDeses
with PCIe functionality (I think six or seven?) but "shared memory pool"
which mvebu-mbus allocates to consumers is not big enough to allow e.g.
256 MB + 64 kB for every PCIe port.
There is upper limit of mvebu memory slots in HW and each slot has size
restrictions. So mvebu-mbus has to deal with splitting requested
PCIe window to more mvebu memory slots... So even if there is available
memory for assigning then there does not have to be a free slot.
Due to nature of plugable PCIe expansion cards, you are basically free
to put any PCIe card which you like into any PCIe slot. So it would be
up to the user to choose combination of such cards which pass all those
mvebu windows and slots restrictions... Otherwise kernel just say that
cannot satisfy card's BAR assignment because it is not possible to set
forwarding windows correctly.
Moreover it is possible to bind / unbind pci mvebu device dynamically at
runtime (also by rmmod), so whole resource allocation in mvebu-bus is
dynamic even during system runtime. So theoretically user can unbind one
driver to free some memory and then can bind another (which needs more
memory).
I think that this pci-mvebu driver and HW is very unusual in both
resource assignment and supported features and requirements from SW.
quoted
quoted
I'm not sure I've seen hardware that does this, and it seems ... maybe
a little aggressive.
What happens if software writes the base and limit in the wrong order,
so the window is invalid after the first write but valid after the
second? That actually sounds like it could be a sensible strategy to
prevent a partially-configured window from being active.
Invalid window (limit < base) means that window is disabled. And
pci-mvebu.c in its callbacks from pci-bridge-emul.c should correctly
handle it and propagates information about disablement to mvebu-mbus
driver.
After window is valid again (limit > base) then pci-mvebu.c call
mvebu-mbus to setup new mapping.
Not sure I'm understanding the code correctly. Here's the sort of
thing I'm worried about, but maybe this is actually impossible:
Let's say software writes (0x00, 0xff) to the I/O (base, limit), which
describes the [io 0x0000-0xffff] window. If mvebu-mbus can't handle
that, it looks like you set the (base, limit) to (0xf0, 0x0f), which
would describe [io 0xf000-0x0fff], which is invalid.
The software writes 0x40 to the limit, so now we have (0xf0, 0x40), or
[io 0xf000-0x40ff]. That's still invalid, but software thinks the
0x00 it wrote to the base is still there.
Bjorn
I see. In this situation it does not work correctly.
But is not kernel itself "privileged" to setup forwarding windows?
Because currently kernel does not do it and therefore do we need to care
for it?
Or do you have idea how to handle this kind of situation? Or how to
handle these kinds of errors?
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
On Sat, Jan 08, 2022 at 12:46:58AM +0100, Pali Rohár wrote:
On Friday 07 January 2022 17:16:17 Bjorn Helgaas wrote:
quoted
On Fri, Jan 07, 2022 at 11:28:26PM +0100, Pali Rohár wrote:
quoted
On Friday 07 January 2022 15:55:04 Bjorn Helgaas wrote:
quoted
On Thu, Nov 25, 2021 at 01:45:58PM +0100, Pali Rohár wrote:
quoted
Properly propagate failure from mvebu_pcie_add_windows() function back to
the caller mvebu_pci_bridge_emul_base_conf_write() and correctly updates
PCI_IO_BASE, PCI_MEM_BASE and PCI_IO_BASE_UPPER16 registers on error.
On error set base value higher than limit value which indicates that
address range is disabled.
Does the spec say that if software programs something invalid,
hardware should proactively set the base and limit registers to
disable the window?
No. But this patch address something totally different. Software can do
fully valid operation, e.g. try to set forwarding memory window as large
as possible. But because this driver "emulates" pci bridge by calling
software/kernel function (mvebu_pcie_add_windows), some operations which
in real HW cannot happen, are possible in software.
For example there are limitations in sizes of forwarding memory windows,
because it is done by mvebu-mbus driver, which is responsible for
configuring mapping and forwarding of PCIe I/O and MEM windows. And due
to Marvell HW, there are restrictions which are not in PCIe HW.
Currently if such error happens, obviously kernel is not able to set
PCIe windows and it just print warnings to dmesg. Trying to access these
windows would result in the worst case in crashes.
With this change when mvebu_pcie_add_windows() function fails then into
emulated config space is put information that particular forwarding
window is disabled. I think that it is better to indicate it in config
space what is the current "reality" of hardware configuration. If window
is disabled in real-HW (meaning in mvebu-mbus driver) then show it also
in emulated config space of pci bridge.
Do you have better idea what should emulated pci bridge do, if software
try to set fully valid configuration of forwarding window, but it is not
possible to achieve it (even compliant PCI bridge must be able to do
it)?
On an ACPI system, the host bridge window sizes are constrained by the
host bridge _CRS method. I assume there's a similar constraint in DT.
Is the fact that mvebu_pcie_add_windows() can fail a symptom of a DT
that describes more available space than mvebu-bus can map?
I wish I knew how to really interpret those "ranges" properties. (Is
there a good description in Documentation/ somewhere? All I've found
so far is https://elinux.org/Device_Tree_Usage, which is good, but
doesn't match this example completely.)
I see:
pciec: pcie {
ranges = <...>;
pcie1: pcie@1,0 {
ranges = <0x82000000 0 0 0x82000000 0x1 0 1 0
0x81000000 0 0 0x81000000 0x1 0 1 0>;
};
pcie2: pcie@2,0 {
ranges = <0x82000000 0 0 0x82000000 0x2 0 1 0
0x81000000 0 0 0x81000000 0x2 0 1 0>;
};
pcie3: pcie@3,0 {
ranges = <0x82000000 0 0 0x82000000 0x3 0 1 0
0x81000000 0 0 0x81000000 0x3 0 1 0>;
};
pcie4: pcie@4,0 {
ranges = <0x82000000 0 0 0x82000000 0x4 0 1 0
0x81000000 0 0 0x81000000 0x4 0 1 0>;
};
};
What does this look like in dmesg, i.e., what CPU address ranges are
mapped to what PCI bus addresses?
Are pcie1, pcie2, etc Root Ports? Or are they each separate host
bridges (they each have "bus-range = <0x00 0xff>")?
Is space from pciec dynamically assigned to pcie1, pcie2, etc? If so,
I assume there are more restrictions on the size and alignment than on
PCI bridge windows, which allow size/alignment down to 1MB?
I'm trying to see how this could be described in ACPI because that's a
fairly general model that accommodates most machines. Possibly
describing mvebu in ACPI would involve losing some flexibility.
Bjorn
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
On Wednesday 12 January 2022 18:19:21 Bjorn Helgaas wrote:
On Sat, Jan 08, 2022 at 12:46:58AM +0100, Pali Rohár wrote:
quoted
On Friday 07 January 2022 17:16:17 Bjorn Helgaas wrote:
quoted
On Fri, Jan 07, 2022 at 11:28:26PM +0100, Pali Rohár wrote:
quoted
On Friday 07 January 2022 15:55:04 Bjorn Helgaas wrote:
quoted
On Thu, Nov 25, 2021 at 01:45:58PM +0100, Pali Rohár wrote:
quoted
Properly propagate failure from mvebu_pcie_add_windows() function back to
the caller mvebu_pci_bridge_emul_base_conf_write() and correctly updates
PCI_IO_BASE, PCI_MEM_BASE and PCI_IO_BASE_UPPER16 registers on error.
On error set base value higher than limit value which indicates that
address range is disabled.
Does the spec say that if software programs something invalid,
hardware should proactively set the base and limit registers to
disable the window?
No. But this patch address something totally different. Software can do
fully valid operation, e.g. try to set forwarding memory window as large
as possible. But because this driver "emulates" pci bridge by calling
software/kernel function (mvebu_pcie_add_windows), some operations which
in real HW cannot happen, are possible in software.
For example there are limitations in sizes of forwarding memory windows,
because it is done by mvebu-mbus driver, which is responsible for
configuring mapping and forwarding of PCIe I/O and MEM windows. And due
to Marvell HW, there are restrictions which are not in PCIe HW.
Currently if such error happens, obviously kernel is not able to set
PCIe windows and it just print warnings to dmesg. Trying to access these
windows would result in the worst case in crashes.
With this change when mvebu_pcie_add_windows() function fails then into
emulated config space is put information that particular forwarding
window is disabled. I think that it is better to indicate it in config
space what is the current "reality" of hardware configuration. If window
is disabled in real-HW (meaning in mvebu-mbus driver) then show it also
in emulated config space of pci bridge.
Do you have better idea what should emulated pci bridge do, if software
try to set fully valid configuration of forwarding window, but it is not
possible to achieve it (even compliant PCI bridge must be able to do
it)?
On an ACPI system, the host bridge window sizes are constrained by the
host bridge _CRS method. I assume there's a similar constraint in DT.
Is the fact that mvebu_pcie_add_windows() can fail a symptom of a DT
that describes more available space than mvebu-bus can map?
I wish I knew how to really interpret those "ranges" properties. (Is
there a good description in Documentation/ somewhere? All I've found
so far is https://elinux.org/Device_Tree_Usage, which is good, but
doesn't match this example completely.)
I see:
pciec: pcie {
ranges = <...>;
pcie1: pcie@1,0 {
ranges = <0x82000000 0 0 0x82000000 0x1 0 1 0
0x81000000 0 0 0x81000000 0x1 0 1 0>;
};
pcie2: pcie@2,0 {
ranges = <0x82000000 0 0 0x82000000 0x2 0 1 0
0x81000000 0 0 0x81000000 0x2 0 1 0>;
};
pcie3: pcie@3,0 {
ranges = <0x82000000 0 0 0x82000000 0x3 0 1 0
0x81000000 0 0 0x81000000 0x3 0 1 0>;
};
pcie4: pcie@4,0 {
ranges = <0x82000000 0 0 0x82000000 0x4 0 1 0
0x81000000 0 0 0x81000000 0x4 0 1 0>;
};
};
What does this look like in dmesg, i.e., what CPU address ranges are
mapped to what PCI bus addresses?
These explicit ranges in DT are probably ignored as they are invalid.
You can see them (0xffffffffffffffff) in dmesg. MEM and I/O resources
are parsed in pci-mvebu.c driver in mvebu_pcie_parse_request_resources()
function.
Here is relevant dmesg output:
[ 0.671607] mvebu-pcie soc:pcie: host bridge /soc/pcie ranges:
[ 0.677474] mvebu-pcie soc:pcie: MEM 0x00f1080000..0x00f1081fff -> 0x0000080000
[ 0.685245] mvebu-pcie soc:pcie: MEM 0x00f1040000..0x00f1041fff -> 0x0000040000
[ 0.693017] mvebu-pcie soc:pcie: MEM 0x00f1044000..0x00f1045fff -> 0x0000044000
[ 0.700788] mvebu-pcie soc:pcie: MEM 0x00f1048000..0x00f1049fff -> 0x0000048000
[ 0.708562] mvebu-pcie soc:pcie: MEM 0xffffffffffffffff..0x00fffffffe -> 0x0100000000
[ 0.716856] mvebu-pcie soc:pcie: IO 0xffffffffffffffff..0x00fffffffe -> 0x0100000000
[ 0.725146] mvebu-pcie soc:pcie: MEM 0xffffffffffffffff..0x00fffffffe -> 0x0200000000
[ 0.733438] mvebu-pcie soc:pcie: IO 0xffffffffffffffff..0x00fffffffe -> 0x0200000000
[ 0.741730] mvebu-pcie soc:pcie: MEM 0xffffffffffffffff..0x00fffffffe -> 0x0300000000
[ 0.750022] mvebu-pcie soc:pcie: IO 0xffffffffffffffff..0x00fffffffe -> 0x0300000000
[ 0.758314] mvebu-pcie soc:pcie: MEM 0xffffffffffffffff..0x00fffffffe -> 0x0400000000
[ 0.766606] mvebu-pcie soc:pcie: IO 0xffffffffffffffff..0x00fffffffe -> 0x0400000000
[ 0.907763] mvebu-pcie soc:pcie: PCI host bridge to bus 0000:00
[ 0.913698] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 0.919203] pci_bus 0000:00: root bus resource [mem 0xf1080000-0xf1081fff] (bus address [0x00080000-0x00081fff])
[ 0.929406] pci_bus 0000:00: root bus resource [mem 0xf1040000-0xf1041fff] (bus address [0x00040000-0x00041fff])
[ 0.939608] pci_bus 0000:00: root bus resource [mem 0xf1044000-0xf1045fff] (bus address [0x00044000-0x00045fff])
[ 0.949812] pci_bus 0000:00: root bus resource [mem 0xf1048000-0xf1049fff] (bus address [0x00048000-0x00049fff])
[ 0.960014] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xe7ffffff]
[ 0.966908] pci_bus 0000:00: root bus resource [io 0x1000-0xeffff]
[ 0.973261] pci 0000:00:01.0: [11ab:6820] type 01 class 0x060400
[ 0.979458] pci 0000:00:02.0: [11ab:6820] type 01 class 0x060400
[ 0.985643] pci 0000:00:03.0: [11ab:6820] type 01 class 0x060400
[ 0.992482] PCI: bus0: Fast back to back transfers disabled
[ 0.998075] pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 1.006108] pci 0000:00:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 1.014134] pci 0000:00:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[ 1.022242] pci 0000:01:00.0: [1e0f:0001] type 00 class 0x010802
[ 1.028289] pci 0000:01:00.0: reg 0x10: [mem 0xc0000000-0xc0003fff 64bit]
[ 1.035233] pci 0000:01:00.0: PME# supported from D0 D3hot
[ 1.040787] pci 0000:01:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x1 link at 0000:00:01.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link)
[ 1.056630] PCI: bus1: Fast back to back transfers disabled
[ 1.062217] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[ 1.068935] pci 0000:02:00.0: [168c:003c] type 00 class 0x028000
[ 1.074975] pci 0000:02:00.0: reg 0x10: [mem 0xc8000000-0xc81fffff 64bit]
[ 1.081817] pci 0000:02:00.0: reg 0x30: [mem 0xc8200000-0xc820ffff pref]
[ 1.088611] pci 0000:02:00.0: supports D1 D2
[ 1.093765] PCI: bus2: Fast back to back transfers disabled
[ 1.099356] pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 02
[ 1.106070] pci 0000:03:00.0: [168c:002e] type 00 class 0x028000
[ 1.112110] pci 0000:03:00.0: reg 0x10: [mem 0xd0000000-0xd000ffff 64bit]
[ 1.119031] pci 0000:03:00.0: supports D1
[ 1.123049] pci 0000:03:00.0: PME# supported from D0 D1 D3hot
[ 1.129688] PCI: bus3: Fast back to back transfers disabled
[ 1.135274] pci_bus 0000:03: busn_res: [bus 03-ff] end is updated to 03
[ 1.141926] pci 0000:00:01.0: BAR 8: assigned [mem 0xe0000000-0xe00fffff]
[ 1.148738] pci 0000:00:02.0: BAR 8: assigned [mem 0xe0200000-0xe04fffff]
[ 1.155544] pci 0000:00:03.0: BAR 8: assigned [mem 0xe0100000-0xe01fffff]
[ 1.162357] pci 0000:01:00.0: BAR 0: assigned [mem 0xe0000000-0xe0003fff 64bit]
[ 1.169696] pci 0000:00:01.0: PCI bridge to [bus 01]
[ 1.174674] pci 0000:00:01.0: bridge window [mem 0xe0000000-0xe00fffff]
[ 1.181490] pci 0000:02:00.0: BAR 0: assigned [mem 0xe0200000-0xe03fffff 64bit]
[ 1.188830] pci 0000:02:00.0: BAR 6: assigned [mem 0xe0400000-0xe040ffff pref]
[ 1.196074] pci 0000:00:02.0: PCI bridge to [bus 02]
[ 1.201050] pci 0000:00:02.0: bridge window [mem 0xe0200000-0xe04fffff]
[ 1.207867] pci 0000:03:00.0: BAR 0: assigned [mem 0xe0100000-0xe010ffff 64bit]
[ 1.215200] pci 0000:00:03.0: PCI bridge to [bus 03]
[ 1.220179] pci 0000:00:03.0: bridge window [mem 0xe0100000-0xe01fffff]
[ 1.227028] pcieport 0000:00:01.0: enabling device (0140 -> 0142)
[ 1.233186] pcieport 0000:00:02.0: enabling device (0140 -> 0142)
[ 1.239339] pcieport 0000:00:03.0: enabling device (0140 -> 0142)
And some more information from /proc:
$ cat /proc/ioports
00001000-000effff : PCI I/O
$ cat /proc/iomem
...
e0000000-e7ffffff : PCI MEM
e0000000-e00fffff : PCI Bus 0000:01
e0000000-e0003fff : 0000:01:00.0
e0000000-e0003fff : nvme
e0100000-e01fffff : PCI Bus 0000:03
e0100000-e010ffff : 0000:03:00.0
e0100000-e010ffff : ath9k
e0200000-e04fffff : PCI Bus 0000:02
e0200000-e03fffff : 0000:02:00.0
e0200000-e03fffff : ath
e0400000-e040ffff : 0000:02:00.0
...
Are pcie1, pcie2, etc Root Ports? Or are they each separate host
bridges (they each have "bus-range = <0x00 0xff>")?
From kernel point of view they are root ports. But in reality every of
these root port is on separate bus segment, but kernel pci-mvebu.c
driver merges all these segments/domains into one host bridge and put
all root ports into bus 0.
Here is lspci -tvnn output with topology:
$ lspci -tvnn
-[0000:00]-+-01.0-[01]----00.0 Device [1e0f:0001]
+-02.0-[02]----00.0 Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter [168c:003c]
\-03.0-[03]----00.0 Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) [168c:002e]
And without topology:
$ lspci -nn
00:01.0 PCI bridge [0604]: Marvell Technology Group Ltd. Device [11ab:6820] (rev 04)
00:02.0 PCI bridge [0604]: Marvell Technology Group Ltd. Device [11ab:6820] (rev 04)
00:03.0 PCI bridge [0604]: Marvell Technology Group Ltd. Device [11ab:6820] (rev 04)
01:00.0 Non-Volatile memory controller [0108]: Device [1e0f:0001]
02:00.0 Network controller [0280]: Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter [168c:003c]
03:00.0 Network controller [0280]: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) [168c:002e] (rev 01)
Buses 1, 2 and 3 represents mPCIe cards, all of them are in reality in
separate bus segments and on different HW host bridges. So they do *not*
share access to config space, do *not* share INTx interrupts, etc...
Is space from pciec dynamically assigned to pcie1, pcie2, etc? If so,
I assume there are more restrictions on the size and alignment than on
PCI bridge windows, which allow size/alignment down to 1MB?
Yes, exactly. I do not know now all restrictions. At least there are
fixed number of memory slots and each has to be of size 2^N. They are
dynamically assigned by kernel mbus driver at time when somebody updates
BASE/LIMIT registers. And that kernel mbus driver takes care to split
non-aligned window size to more slots of size 2^N. And resources are
shared from pool with other HW parts (e.g. DMA), so other drivers loaded
in kernel can "eat" available slots before pci-mvebu and then there does
not have to be nothing to allocate for PCI.
But most Armada boards do not have exported all peripherals from SoC,
unconnected are disabled in DT and therefore exhaustion should not
happen.
I'm trying to see how this could be described in ACPI because that's a
fairly general model that accommodates most machines. Possibly
describing mvebu in ACPI would involve losing some flexibility.
Bjorn
I do not understand APCI model very well and I'm in impression that it
is impossible to represent mvebu in ACPI.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
On Thu, Jan 13, 2022 at 11:35:23AM +0100, Pali Rohár wrote:
On Wednesday 12 January 2022 18:19:21 Bjorn Helgaas wrote:
quoted
On Sat, Jan 08, 2022 at 12:46:58AM +0100, Pali Rohár wrote:
quoted
On Friday 07 January 2022 17:16:17 Bjorn Helgaas wrote:
quoted
On Fri, Jan 07, 2022 at 11:28:26PM +0100, Pali Rohár wrote:
quoted
On Friday 07 January 2022 15:55:04 Bjorn Helgaas wrote:
quoted
On Thu, Nov 25, 2021 at 01:45:58PM +0100, Pali Rohár wrote:
quoted
Properly propagate failure from mvebu_pcie_add_windows()
function back to the caller
mvebu_pci_bridge_emul_base_conf_write() and correctly
updates PCI_IO_BASE, PCI_MEM_BASE and
PCI_IO_BASE_UPPER16 registers on error. On error set
base value higher than limit value which indicates that
address range is disabled.
Does the spec say that if software programs something
invalid, hardware should proactively set the base and
limit registers to disable the window?
No. But this patch address something totally different.
Software can do fully valid operation, e.g. try to set
forwarding memory window as large as possible. But because
this driver "emulates" pci bridge by calling software/kernel
function (mvebu_pcie_add_windows), some operations which in
real HW cannot happen, are possible in software.
For example there are limitations in sizes of forwarding
memory windows, because it is done by mvebu-mbus driver,
which is responsible for configuring mapping and forwarding
of PCIe I/O and MEM windows. And due to Marvell HW, there
are restrictions which are not in PCIe HW.
Currently if such error happens, obviously kernel is not
able to set PCIe windows and it just print warnings to
dmesg. Trying to access these windows would result in the
worst case in crashes.
With this change when mvebu_pcie_add_windows() function
fails then into emulated config space is put information
that particular forwarding window is disabled. I think that
it is better to indicate it in config space what is the
current "reality" of hardware configuration. If window is
disabled in real-HW (meaning in mvebu-mbus driver) then show
it also in emulated config space of pci bridge.
Do you have better idea what should emulated pci bridge do,
if software try to set fully valid configuration of
forwarding window, but it is not possible to achieve it
(even compliant PCI bridge must be able to do it)?
On an ACPI system, the host bridge window sizes are
constrained by the host bridge _CRS method. I assume there's
a similar constraint in DT.
Is the fact that mvebu_pcie_add_windows() can fail a symptom
of a DT that describes more available space than mvebu-bus can
map?
I wish I knew how to really interpret those "ranges" properties.
(Is there a good description in Documentation/ somewhere? All
I've found so far is https://elinux.org/Device_Tree_Usage, which
is good, but doesn't match this example completely.)
I see:
pciec: pcie {
ranges = <...>;
pcie1: pcie@1,0 {
ranges = <0x82000000 0 0 0x82000000 0x1 0 1 0
0x81000000 0 0 0x81000000 0x1 0 1 0>;
};
pcie2: pcie@2,0 {
ranges = <0x82000000 0 0 0x82000000 0x2 0 1 0
0x81000000 0 0 0x81000000 0x2 0 1 0>;
};
pcie3: pcie@3,0 {
ranges = <0x82000000 0 0 0x82000000 0x3 0 1 0
0x81000000 0 0 0x81000000 0x3 0 1 0>;
};
pcie4: pcie@4,0 {
ranges = <0x82000000 0 0 0x82000000 0x4 0 1 0
0x81000000 0 0 0x81000000 0x4 0 1 0>;
};
};
What does this look like in dmesg, i.e., what CPU address ranges are
mapped to what PCI bus addresses?
These explicit ranges in DT are probably ignored as they are invalid.
You can see them (0xffffffffffffffff) in dmesg.
Are you saying that this DT ranges and the dmesg line are connected?
ranges = <0x82000000 0 0 0x82000000 0x1 0 1 0
0x81000000 0 0 0x81000000 0x1 0 1 0>;
mvebu-pcie soc:pcie: MEM 0xffffffffffffffff..0x00fffffffe -> 0x0100000000
1) It would be nice if there were a hint somewhere in Documentation/
that would allow mere mortals to see the connection there.
2) Why do we have these DT entries if they are invalid and useless?
MEM and I/O resources are parsed in pci-mvebu.c driver in
mvebu_pcie_parse_request_resources() function.
So mvebu-mbus.c fills in the static mbus_state from the DT
"pcie-mem-aperture", which seems unconnected to the DT descriptions of
the PCI controllers:
static struct mvebu_mbus_state mbus_state;
mvebu_mbus_dt_init
mvebu_mbus_get_pcie_resources(&mbus_state.pcie_mem_aperture)
of_property_read_u32_array("pcie-mem-aperture")
mvebu_pcie_probe
mvebu_pcie_parse_request_resources
mvebu_mbus_get_pcie_mem_aperture(&pcie->mem)
*res = mbus_state.pcie_mem_aperture
pci_add_resource(&bridge->windows, &pcie->mem)
Here is relevant dmesg output:
mvebu-pcie soc:pcie: host bridge /soc/pcie ranges:
mvebu-pcie soc:pcie: MEM 0x00f1080000..0x00f1081fff -> 0x0000080000
mvebu-pcie soc:pcie: MEM 0x00f1040000..0x00f1041fff -> 0x0000040000
mvebu-pcie soc:pcie: MEM 0x00f1044000..0x00f1045fff -> 0x0000044000
mvebu-pcie soc:pcie: MEM 0x00f1048000..0x00f1049fff -> 0x0000048000
mvebu-pcie soc:pcie: MEM 0xffffffffffffffff..0x00fffffffe -> 0x0100000000
mvebu-pcie soc:pcie: IO 0xffffffffffffffff..0x00fffffffe -> 0x0100000000
mvebu-pcie soc:pcie: MEM 0xffffffffffffffff..0x00fffffffe -> 0x0200000000
mvebu-pcie soc:pcie: IO 0xffffffffffffffff..0x00fffffffe -> 0x0200000000
mvebu-pcie soc:pcie: MEM 0xffffffffffffffff..0x00fffffffe -> 0x0300000000
mvebu-pcie soc:pcie: IO 0xffffffffffffffff..0x00fffffffe -> 0x0300000000
mvebu-pcie soc:pcie: MEM 0xffffffffffffffff..0x00fffffffe -> 0x0400000000
mvebu-pcie soc:pcie: IO 0xffffffffffffffff..0x00fffffffe -> 0x0400000000
mvebu-pcie soc:pcie: PCI host bridge to bus 0000:00
pci_bus 0000:00: root bus resource [bus 00-ff]
pci_bus 0000:00: root bus resource [mem 0xf1080000-0xf1081fff] (bus address [0x00080000-0x00081fff])
pci_bus 0000:00: root bus resource [mem 0xf1040000-0xf1041fff] (bus address [0x00040000-0x00041fff])
pci_bus 0000:00: root bus resource [mem 0xf1044000-0xf1045fff] (bus address [0x00044000-0x00045fff])
pci_bus 0000:00: root bus resource [mem 0xf1048000-0xf1049fff] (bus address [0x00048000-0x00049fff])
pci_bus 0000:00: root bus resource [mem 0xe0000000-0xe7ffffff]
I see 0xf1080000-0xf1081fff, 0xf1040000-0xf1041fff, etc mentioned in
the DT info above, but I don't see where [mem 0xe0000000-0xe7ffffff]
came from.
Regardless, this means PCI thinks [mem 0xe0000000-0xe7ffffff] is
available on bus 00 and can be assigned to devices on bus 00 according
to the normal PCI rules (BARs aligned on size, PCI bridge windows
aligned on 1MB and multiple of 1MB in size). IIUC, mvebu imposes
additional alignment constraints on the bridge windows.
These are the bridge window assignments from your dmesg:
pci 0000:00:01.0: BAR 8: assigned [mem 0xe0000000-0xe00fffff]
pci 0000:00:02.0: BAR 8: assigned [mem 0xe0200000-0xe04fffff]
pci 0000:00:03.0: BAR 8: assigned [mem 0xe0100000-0xe01fffff]
The PCI core knows nothing about the mvebu constraints. Are we just
lucky here that when PCI assigned these bridge windows, they happen to
be supported on mvebu? What happens if PCI decides it needs 29MB on
bus 01?
quoted
Are pcie1, pcie2, etc Root Ports? Or are they each separate host
bridges (they each have "bus-range = <0x00 0xff>")?
From kernel point of view they are root ports. But in reality every of
these root port is on separate bus segment, but kernel pci-mvebu.c
driver merges all these segments/domains into one host bridge and put
all root ports into bus 0.
Here is lspci -tvnn output with topology:
$ lspci -tvnn
-[0000:00]-+-01.0-[01]----00.0 Device [1e0f:0001]
+-02.0-[02]----00.0 Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter [168c:003c]
\-03.0-[03]----00.0 Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) [168c:002e]
Buses 1, 2 and 3 represents mPCIe cards, all of them are in reality
in separate bus segments and on different HW host bridges. So they
do *not* share access to config space, do *not* share INTx
interrupts, etc...
quoted
Is space from pciec dynamically assigned to pcie1, pcie2, etc? If
so, I assume there are more restrictions on the size and alignment
than on PCI bridge windows, which allow size/alignment down to
1MB?
Yes, exactly. I do not know now all restrictions. At least there are
fixed number of memory slots and each has to be of size 2^N. They
are dynamically assigned by kernel mbus driver at time when somebody
updates BASE/LIMIT registers. And that kernel mbus driver takes care
to split non-aligned window size to more slots of size 2^N. And
resources are shared from pool with other HW parts (e.g. DMA), so
other drivers loaded in kernel can "eat" available slots before
pci-mvebu and then there does not have to be nothing to allocate for
PCI.
So IIUC,
pcie1 == 00:01.0 Root Port
pcie2 == 00:02.0 Root Port
pcie3 == 00:03.0 Root Port
From a software point of view, they're all under a single host bridge,
and Linux assumes everything under a host bridge plays by the PCI
rules.
In this case, the root ports *don't* play by the rules since they have
additional alignment restrictions, so I think these really should be
described as separate host bridges in DT with the address space
carved up statically among them.
It's common on x86 to have multiple host bridges that all appear to
software to be in domain 0000. The bus number ranges under each are
static, e.g., one bridge has [bus 00-7f] and another has [bus 80-ff].
But most Armada boards do not have exported all peripherals from SoC,
unconnected are disabled in DT and therefore exhaustion should not
happen.
quoted
I'm trying to see how this could be described in ACPI because that's a
fairly general model that accommodates most machines. Possibly
describing mvebu in ACPI would involve losing some flexibility.
I do not understand APCI model very well and I'm in impression that it
is impossible to represent mvebu in ACPI.
It could be described as a separate host bridge for every root port.
ACPI uses _CRS (current resource settings) to describe the apertures
to PCI and any address translation. Currently the _CRS description is
static, but ACPI does allow those resource assignments to be modified
via _PRS (possible resource settings) and _SRS (set resource
settings).
Bjorn
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
On Thursday 20 January 2022 11:50:47 Bjorn Helgaas wrote:
On Thu, Jan 13, 2022 at 11:35:23AM +0100, Pali Rohár wrote:
quoted
On Wednesday 12 January 2022 18:19:21 Bjorn Helgaas wrote:
quoted
On Sat, Jan 08, 2022 at 12:46:58AM +0100, Pali Rohár wrote:
quoted
On Friday 07 January 2022 17:16:17 Bjorn Helgaas wrote:
quoted
On Fri, Jan 07, 2022 at 11:28:26PM +0100, Pali Rohár wrote:
quoted
On Friday 07 January 2022 15:55:04 Bjorn Helgaas wrote:
quoted
On Thu, Nov 25, 2021 at 01:45:58PM +0100, Pali Rohár wrote:
quoted
Properly propagate failure from mvebu_pcie_add_windows()
function back to the caller
mvebu_pci_bridge_emul_base_conf_write() and correctly
updates PCI_IO_BASE, PCI_MEM_BASE and
PCI_IO_BASE_UPPER16 registers on error. On error set
base value higher than limit value which indicates that
address range is disabled.
Does the spec say that if software programs something
invalid, hardware should proactively set the base and
limit registers to disable the window?
No. But this patch address something totally different.
Software can do fully valid operation, e.g. try to set
forwarding memory window as large as possible. But because
this driver "emulates" pci bridge by calling software/kernel
function (mvebu_pcie_add_windows), some operations which in
real HW cannot happen, are possible in software.
For example there are limitations in sizes of forwarding
memory windows, because it is done by mvebu-mbus driver,
which is responsible for configuring mapping and forwarding
of PCIe I/O and MEM windows. And due to Marvell HW, there
are restrictions which are not in PCIe HW.
Currently if such error happens, obviously kernel is not
able to set PCIe windows and it just print warnings to
dmesg. Trying to access these windows would result in the
worst case in crashes.
With this change when mvebu_pcie_add_windows() function
fails then into emulated config space is put information
that particular forwarding window is disabled. I think that
it is better to indicate it in config space what is the
current "reality" of hardware configuration. If window is
disabled in real-HW (meaning in mvebu-mbus driver) then show
it also in emulated config space of pci bridge.
Do you have better idea what should emulated pci bridge do,
if software try to set fully valid configuration of
forwarding window, but it is not possible to achieve it
(even compliant PCI bridge must be able to do it)?
On an ACPI system, the host bridge window sizes are
constrained by the host bridge _CRS method. I assume there's
a similar constraint in DT.
Is the fact that mvebu_pcie_add_windows() can fail a symptom
of a DT that describes more available space than mvebu-bus can
map?
I wish I knew how to really interpret those "ranges" properties.
(Is there a good description in Documentation/ somewhere? All
I've found so far is https://elinux.org/Device_Tree_Usage, which
is good, but doesn't match this example completely.)
I see:
pciec: pcie {
ranges = <...>;
pcie1: pcie@1,0 {
ranges = <0x82000000 0 0 0x82000000 0x1 0 1 0
0x81000000 0 0 0x81000000 0x1 0 1 0>;
};
pcie2: pcie@2,0 {
ranges = <0x82000000 0 0 0x82000000 0x2 0 1 0
0x81000000 0 0 0x81000000 0x2 0 1 0>;
};
pcie3: pcie@3,0 {
ranges = <0x82000000 0 0 0x82000000 0x3 0 1 0
0x81000000 0 0 0x81000000 0x3 0 1 0>;
};
pcie4: pcie@4,0 {
ranges = <0x82000000 0 0 0x82000000 0x4 0 1 0
0x81000000 0 0 0x81000000 0x4 0 1 0>;
};
};
What does this look like in dmesg, i.e., what CPU address ranges are
mapped to what PCI bus addresses?
These explicit ranges in DT are probably ignored as they are invalid.
You can see them (0xffffffffffffffff) in dmesg.
Are you saying that this DT ranges and the dmesg line are connected?
ranges = <0x82000000 0 0 0x82000000 0x1 0 1 0
0x81000000 0 0 0x81000000 0x1 0 1 0>;
mvebu-pcie soc:pcie: MEM 0xffffffffffffffff..0x00fffffffe -> 0x0100000000
1) It would be nice if there were a hint somewhere in Documentation/
that would allow mere mortals to see the connection there.
I agree, that there is missing lot of documentation related to Marvell
PCIe controllers, both pci-mvebu.c and pci-aardvark.c. The main problem
now is that it is hard to find information which explain everything...
like mysterious Memory Controller (which is now explained):
https://lore.kernel.org/linux-pci/20211003120944.3lmwxylnhlp2kfj7@pali/
2) Why do we have these DT entries if they are invalid and useless?
Sorry, I do not know. I was not involved during introducing of DT files
for this platform. I'm just observing how it works...
quoted
MEM and I/O resources are parsed in pci-mvebu.c driver in
mvebu_pcie_parse_request_resources() function.
So mvebu-mbus.c fills in the static mbus_state from the DT
"pcie-mem-aperture", which seems unconnected to the DT descriptions of
the PCI controllers:
static struct mvebu_mbus_state mbus_state;
mvebu_mbus_dt_init
mvebu_mbus_get_pcie_resources(&mbus_state.pcie_mem_aperture)
of_property_read_u32_array("pcie-mem-aperture")
mvebu_pcie_probe
mvebu_pcie_parse_request_resources
mvebu_mbus_get_pcie_mem_aperture(&pcie->mem)
*res = mbus_state.pcie_mem_aperture
pci_add_resource(&bridge->windows, &pcie->mem)
quoted
Here is relevant dmesg output:
mvebu-pcie soc:pcie: host bridge /soc/pcie ranges:
mvebu-pcie soc:pcie: MEM 0x00f1080000..0x00f1081fff -> 0x0000080000
mvebu-pcie soc:pcie: MEM 0x00f1040000..0x00f1041fff -> 0x0000040000
mvebu-pcie soc:pcie: MEM 0x00f1044000..0x00f1045fff -> 0x0000044000
mvebu-pcie soc:pcie: MEM 0x00f1048000..0x00f1049fff -> 0x0000048000
mvebu-pcie soc:pcie: MEM 0xffffffffffffffff..0x00fffffffe -> 0x0100000000
mvebu-pcie soc:pcie: IO 0xffffffffffffffff..0x00fffffffe -> 0x0100000000
mvebu-pcie soc:pcie: MEM 0xffffffffffffffff..0x00fffffffe -> 0x0200000000
mvebu-pcie soc:pcie: IO 0xffffffffffffffff..0x00fffffffe -> 0x0200000000
mvebu-pcie soc:pcie: MEM 0xffffffffffffffff..0x00fffffffe -> 0x0300000000
mvebu-pcie soc:pcie: IO 0xffffffffffffffff..0x00fffffffe -> 0x0300000000
mvebu-pcie soc:pcie: MEM 0xffffffffffffffff..0x00fffffffe -> 0x0400000000
mvebu-pcie soc:pcie: IO 0xffffffffffffffff..0x00fffffffe -> 0x0400000000
mvebu-pcie soc:pcie: PCI host bridge to bus 0000:00
pci_bus 0000:00: root bus resource [bus 00-ff]
pci_bus 0000:00: root bus resource [mem 0xf1080000-0xf1081fff] (bus address [0x00080000-0x00081fff])
pci_bus 0000:00: root bus resource [mem 0xf1040000-0xf1041fff] (bus address [0x00040000-0x00041fff])
pci_bus 0000:00: root bus resource [mem 0xf1044000-0xf1045fff] (bus address [0x00044000-0x00045fff])
pci_bus 0000:00: root bus resource [mem 0xf1048000-0xf1049fff] (bus address [0x00048000-0x00049fff])
pci_bus 0000:00: root bus resource [mem 0xe0000000-0xe7ffffff]
I see 0xf1080000-0xf1081fff, 0xf1040000-0xf1041fff, etc mentioned in
the DT info above, but I don't see where [mem 0xe0000000-0xe7ffffff]
came from.
Regardless, this means PCI thinks [mem 0xe0000000-0xe7ffffff] is
available on bus 00 and can be assigned to devices on bus 00 according
to the normal PCI rules (BARs aligned on size, PCI bridge windows
aligned on 1MB and multiple of 1MB in size). IIUC, mvebu imposes
additional alignment constraints on the bridge windows.
These are the bridge window assignments from your dmesg:
quoted
pci 0000:00:01.0: BAR 8: assigned [mem 0xe0000000-0xe00fffff]
pci 0000:00:02.0: BAR 8: assigned [mem 0xe0200000-0xe04fffff]
pci 0000:00:03.0: BAR 8: assigned [mem 0xe0100000-0xe01fffff]
The PCI core knows nothing about the mvebu constraints. Are we just
lucky here that when PCI assigned these bridge windows, they happen to
be supported on mvebu? What happens if PCI decides it needs 29MB on
bus 01?
In this case pci-mvebu.c split 29MB window into continuous ranges of
power of two (16MB + 8MB + 4MB + 1MB) and then register each range to
mbus slot. Code is in function mvebu_pcie_add_windows():
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/pci-mvebu.c?h=v5.15#n300
So at the end there is continuous space of 29MB PCIe window, just it
"eats" 4 mbus slots.
This function may fail (if there is not enough free mbus slots) and this
patch is propagating that failure back to the caller.
quoted
quoted
Are pcie1, pcie2, etc Root Ports? Or are they each separate host
bridges (they each have "bus-range = <0x00 0xff>")?
From kernel point of view they are root ports. But in reality every of
these root port is on separate bus segment, but kernel pci-mvebu.c
driver merges all these segments/domains into one host bridge and put
all root ports into bus 0.
Here is lspci -tvnn output with topology:
$ lspci -tvnn
-[0000:00]-+-01.0-[01]----00.0 Device [1e0f:0001]
+-02.0-[02]----00.0 Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter [168c:003c]
\-03.0-[03]----00.0 Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) [168c:002e]
quoted
Buses 1, 2 and 3 represents mPCIe cards, all of them are in reality
in separate bus segments and on different HW host bridges. So they
do *not* share access to config space, do *not* share INTx
interrupts, etc...
quoted
Is space from pciec dynamically assigned to pcie1, pcie2, etc? If
so, I assume there are more restrictions on the size and alignment
than on PCI bridge windows, which allow size/alignment down to
1MB?
Yes, exactly. I do not know now all restrictions. At least there are
fixed number of memory slots and each has to be of size 2^N. They
are dynamically assigned by kernel mbus driver at time when somebody
updates BASE/LIMIT registers. And that kernel mbus driver takes care
to split non-aligned window size to more slots of size 2^N. And
resources are shared from pool with other HW parts (e.g. DMA), so
other drivers loaded in kernel can "eat" available slots before
pci-mvebu and then there does not have to be nothing to allocate for
PCI.
So IIUC,
pcie1 == 00:01.0 Root Port
pcie2 == 00:02.0 Root Port
pcie3 == 00:03.0 Root Port
From a software point of view, they're all under a single host bridge,
and Linux assumes everything under a host bridge plays by the PCI
rules.
Yes.
In this case, the root ports *don't* play by the rules since they have
additional alignment restrictions, so I think these really should be
described as separate host bridges in DT with the address space
carved up statically among them.
I fully agree with you.
But pci-mvebu.c driver and also its DT bindings are written differently.
Changing it probably would not be simple due to backward compatibility
and will take development resources...
It's common on x86 to have multiple host bridges that all appear to
software to be in domain 0000. The bus number ranges under each are
static, e.g., one bridge has [bus 00-7f] and another has [bus 80-ff].
For mvebu they are dynamic and kernel assigns them at boot. As my above
printed lspci topology is simple, first bridge has assigned [bus 01-01],
second bridge [bus 02-02] and third bridge [bus 03-03].
quoted
But most Armada boards do not have exported all peripherals from SoC,
unconnected are disabled in DT and therefore exhaustion should not
happen.
quoted
I'm trying to see how this could be described in ACPI because that's a
fairly general model that accommodates most machines. Possibly
describing mvebu in ACPI would involve losing some flexibility.
I do not understand APCI model very well and I'm in impression that it
is impossible to represent mvebu in ACPI.
It could be described as a separate host bridge for every root port.
ACPI uses _CRS (current resource settings) to describe the apertures
to PCI and any address translation. Currently the _CRS description is
static, but ACPI does allow those resource assignments to be modified
via _PRS (possible resource settings) and _SRS (set resource
settings).
Bjorn
On Thu, Jan 20, 2022 at 08:08:26PM +0100, Pali Rohár wrote:
On Thursday 20 January 2022 11:50:47 Bjorn Helgaas wrote:
quoted
On Thu, Jan 13, 2022 at 11:35:23AM +0100, Pali Rohár wrote:
quoted
On Wednesday 12 January 2022 18:19:21 Bjorn Helgaas wrote:
quoted
On Sat, Jan 08, 2022 at 12:46:58AM +0100, Pali Rohár wrote:
quoted
On Friday 07 January 2022 17:16:17 Bjorn Helgaas wrote:
quoted
On Fri, Jan 07, 2022 at 11:28:26PM +0100, Pali Rohár wrote:
quoted
On Friday 07 January 2022 15:55:04 Bjorn Helgaas wrote:
quoted
On Thu, Nov 25, 2021 at 01:45:58PM +0100, Pali Rohár wrote:
quoted
Properly propagate failure from
mvebu_pcie_add_windows() function back to the caller
mvebu_pci_bridge_emul_base_conf_write() and
correctly updates PCI_IO_BASE, PCI_MEM_BASE and
PCI_IO_BASE_UPPER16 registers on error. On error
set base value higher than limit value which
indicates that address range is disabled.
quoted
Regardless, this means PCI thinks [mem 0xe0000000-0xe7ffffff] is
available on bus 00 and can be assigned to devices on bus 00
according to the normal PCI rules (BARs aligned on size, PCI
bridge windows aligned on 1MB and multiple of 1MB in size). IIUC,
mvebu imposes additional alignment constraints on the bridge
windows.
These are the bridge window assignments from your dmesg:
quoted
pci 0000:00:01.0: BAR 8: assigned [mem 0xe0000000-0xe00fffff]
pci 0000:00:02.0: BAR 8: assigned [mem 0xe0200000-0xe04fffff]
pci 0000:00:03.0: BAR 8: assigned [mem 0xe0100000-0xe01fffff]
The PCI core knows nothing about the mvebu constraints. Are we
just lucky here that when PCI assigned these bridge windows, they
happen to be supported on mvebu? What happens if PCI decides it
needs 29MB on bus 01?
In this case pci-mvebu.c split 29MB window into continuous ranges of
power of two (16MB + 8MB + 4MB + 1MB) and then register each range
to mbus slot. Code is in function mvebu_pcie_add_windows():
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/pci-mvebu.c?h=v5.15#n300
So at the end there is continuous space of 29MB PCIe window, just it
"eats" 4 mbus slots.
This function may fail (if there is not enough free mbus slots) and
this patch is propagating that failure back to the caller.
This failure cannot occur in conforming PCI hardware. I guess if you
want to propagate the error from mvebu_pcie_add_windows() back to
mvebu_pci_bridge_emul_base_conf_write() and do something there, I'm OK
with that.
But change the commit log so it doesn't say "... and correctly update
PCI_IO_BASE, PCI_MEM_BASE and PCI_IO_BASE_UPPER16" because this is
completely device-specific behavior and is not "correct" per any PCI
spec.
Instead, say something about how mvebu doesn't support arbitrary
windows and we're disabling the window completely if we can't provide
what's requested.
Maybe this error warrants a clue in dmesg? How would a user figure
out what's going on in this situation? From the patch, it looks like
we would assign resources to a device, but the device just would not
work because the root port window was silently disabled?
Bjorn
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel