[PATCH v8 00/45] powerpc/powernv: PCI hotplug support

STALE3804d

Revision v8 of 4 in this series.

103 messages, 7 authors, 2016-05-02 · page 2 of 2 · open the first message on its own page

Re: [PATCH v8 30/45] powerpc/pci: Delay populating pdn

From: Alexey Kardashevskiy <hidden>
Date: 2016-04-19 08:19:20

On 02/17/2016 02:44 PM, Gavin Shan wrote:
The pdn (struct pci_dn) instances are allocated from memblock or
bootmem when creating PCI controller (hoses) in setup_arch(). PCI
hotplug, which will be supported by proceeding patches, releases
PCI device nodes and their corresponding pdn on unplugging event.
The memory chunks for pdn instances allocated from memblock or
bootmem are hard to reused after being released.

This delays creating pdn by pci_devs_phb_init() from setup_arch()
to core_initcall() so that they are allocated from slab. The memory
consumed by pdn can be released to system without problem during
PCI unplugging time. It indicates that pci_dn is unavailable in
setup_arch() and the the fixup on pdn (like AGP's) can't be carried
out that time. We have to do that in ppc_md.pcibios_root_bridge_prepare()
on maple/pasemi/powermac platforms where/when the pdn is available.

At the mean while, the EEH device is created when pdn is populated,
meaning pdn and EEH device have same life cycle. In turn, we needn't
call eeh_dev_init() to create EEH device explicitly.

Signed-off-by: Gavin Shan <redacted>

Uff. It would not hurt to mention that  pcibios_root_bridge_prepare is 
called from subsys_initcall() which is executed after core_initcall() so 
the code flow does not change.

Have you checked if there is anything in between 
core_initcall(pci_devs_phb_init) and subsys_initcall(pcibios_init) which 
might need device tree nodes? For example, subsys_initcall(pcibios_init) 
calls (eventually) pnv_pci_ioda_fixup(), if we are unlucky and 
pcibios_init() (and therefore pnv_pci_ioda_fixup() or what pseries/others 
do) is called before pcibios_init() - won't we crash or something?




-- 
Alexey

Re: [PATCH v8 31/45] powerpc/pci: Don't scan empty slot

From: Alexey Kardashevskiy <hidden>
Date: 2016-04-19 08:19:58

On 02/17/2016 02:44 PM, Gavin Shan wrote:
In hotplug case, function pci_add_pci_devices() is called to rescan
the specified PCI bus, which might not have any child devices. Access
to the PCI bus's child device node will cause kernel crash without
exception.

This adds one more check to skip scanning PCI bus that doesn't have
any subordinate devices from device-tree, in order to avoid kernel
crash.

Signed-off-by: Gavin Shan <redacted>

Reviewed-by: Alexey Kardashevskiy <redacted>

quoted hunk
---
  arch/powerpc/kernel/pci-hotplug.c | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/arch/powerpc/kernel/pci-hotplug.c b/arch/powerpc/kernel/pci-hotplug.c
index 7929a1c..3628c38 100644
--- a/arch/powerpc/kernel/pci-hotplug.c
+++ b/arch/powerpc/kernel/pci-hotplug.c
@@ -120,7 +120,8 @@ void pci_add_pci_devices(struct pci_bus *bus)
  	if (mode == PCI_PROBE_DEVTREE) {
  		/* use ofdt-based probe */
  		of_rescan_bus(dn, bus);
-	} else if (mode == PCI_PROBE_NORMAL) {
+	} else if (mode == PCI_PROBE_NORMAL &&
+		   dn->child && PCI_DN(dn->child)) {
  		/*
  		 * Use legacy probe. In the partial hotplug case, we
  		 * probably have grandchildren devices unplugged. So

-- 
Alexey

Re: [PATCH v8 32/45] powerpc/pci: Update bridge windows on PCI plug

From: Alexey Kardashevskiy <hidden>
Date: 2016-04-19 08:47:21

On 02/17/2016 02:44 PM, Gavin Shan wrote:
On the PCI plugging event, PCI slot's subordinate devices are
scanned and their (IO and MMIO) resources are assigned. Platform
dependent resources (PE#, IO/MMIO/DMA windows) are allocated or
created on updating windows of the slot's upstream bridge.

This updates the windows of the hot plugged slot's upstream bridge
in pcibios_finish_adding_to_bus() so that the platform resources
(PE#, IO/MMIO/DMA segments) are allocated or created accordingly.

Signed-off-by: Gavin Shan <redacted>

To my very limited knowledge of the common PCI code, looks good.


Reviewed-by: Alexey Kardashevskiy <redacted>



quoted hunk
---
  arch/powerpc/kernel/pci-common.c | 8 ++++++--
  1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/arch/powerpc/kernel/pci-common.c b/arch/powerpc/kernel/pci-common.c
index 40df3a5..be9e515 100644
--- a/arch/powerpc/kernel/pci-common.c
+++ b/arch/powerpc/kernel/pci-common.c
@@ -1444,8 +1444,12 @@ void pcibios_finish_adding_to_bus(struct pci_bus *bus)
  	/* Allocate bus and devices resources */
  	pcibios_allocate_bus_resources(bus);
  	pcibios_claim_one_bus(bus);
-	if (!pci_has_flag(PCI_PROBE_ONLY))
-		pci_assign_unassigned_bus_resources(bus);
+	if (!pci_has_flag(PCI_PROBE_ONLY)) {
+		if (bus->self)
+			pci_assign_unassigned_bridge_resources(bus->self);
+		else
+			pci_assign_unassigned_bus_resources(bus);
+	}

  	/* Fixup EEH */
  	eeh_add_device_tree_late(bus);

-- 
Alexey

Re: [PATCH v8 33/45] powerpc/powernv: Simplify pnv_eeh_reset()

From: Alexey Kardashevskiy <hidden>
Date: 2016-04-19 08:49:26

On 02/17/2016 02:44 PM, Gavin Shan wrote:
This drops unnecessary nested if statements in pnv_eeh_reset() to
improve the code readability. After the changes, the unused local
variable "ret" is dropped as well. No logical changes introduced.

Signed-off-by: Gavin Shan <redacted>


Reviewed-by: Alexey Kardashevskiy <redacted>


quoted hunk
---
  arch/powerpc/platforms/powernv/eeh-powernv.c | 67 +++++++++++++---------------
  1 file changed, 31 insertions(+), 36 deletions(-)
diff --git a/arch/powerpc/platforms/powernv/eeh-powernv.c b/arch/powerpc/platforms/powernv/eeh-powernv.c
index 69e41ce..9226df1 100644
--- a/arch/powerpc/platforms/powernv/eeh-powernv.c
+++ b/arch/powerpc/platforms/powernv/eeh-powernv.c
@@ -1009,8 +1009,9 @@ static int pnv_eeh_reset_vf_pe(struct eeh_pe *pe, int option)
  static int pnv_eeh_reset(struct eeh_pe *pe, int option)
  {
  	struct pci_controller *hose = pe->phb;
+	struct pnv_phb *phb;
  	struct pci_bus *bus;
-	int ret;
+	int64_t rc;

  	/*
  	 * For PHB reset, we always have complete reset. For those PEs whose
@@ -1026,45 +1027,39 @@ static int pnv_eeh_reset(struct eeh_pe *pe, int option)
  	 * reset. The side effect is that EEH core has to clear the frozen
  	 * state explicitly after BAR restore.
  	 */
-	if (pe->type & EEH_PE_PHB) {
-		ret = pnv_eeh_phb_reset(hose, option);
-	} else {
-		struct pnv_phb *phb;
-		s64 rc;
+	if (pe->type & EEH_PE_PHB)
+		return pnv_eeh_phb_reset(hose, option);

-		/*
-		 * The frozen PE might be caused by PAPR error injection
-		 * registers, which are expected to be cleared after hitting
-		 * frozen PE as stated in the hardware spec. Unfortunately,
-		 * that's not true on P7IOC. So we have to clear it manually
-		 * to avoid recursive EEH errors during recovery.
-		 */
-		phb = hose->private_data;
-		if (phb->model == PNV_PHB_MODEL_P7IOC &&
-		    (option == EEH_RESET_HOT ||
-		    option == EEH_RESET_FUNDAMENTAL)) {
-			rc = opal_pci_reset(phb->opal_id,
-					    OPAL_RESET_PHB_ERROR,
-					    OPAL_ASSERT_RESET);
-			if (rc != OPAL_SUCCESS) {
-				pr_warn("%s: Failure %lld clearing "
-					"error injection registers\n",
-					__func__, rc);
-				return -EIO;
-			}
+	/*
+	 * The frozen PE might be caused by PAPR error injection
+	 * registers, which are expected to be cleared after hitting
+	 * frozen PE as stated in the hardware spec. Unfortunately,
+	 * that's not true on P7IOC. So we have to clear it manually
+	 * to avoid recursive EEH errors during recovery.
+	 */
+	phb = hose->private_data;
+	if (phb->model == PNV_PHB_MODEL_P7IOC &&
+	    (option == EEH_RESET_HOT ||
+	     option == EEH_RESET_FUNDAMENTAL)) {
+		rc = opal_pci_reset(phb->opal_id,
+				    OPAL_RESET_PHB_ERROR,
+				    OPAL_ASSERT_RESET);
+		if (rc != OPAL_SUCCESS) {
+			pr_warn("%s: Failure %lld clearing error injection registers\n",
+				__func__, rc);
+			return -EIO;
  		}
-
-		bus = eeh_pe_bus_get(pe);
-		if (pe->type & EEH_PE_VF)
-			ret = pnv_eeh_reset_vf_pe(pe, option);
-		else if (pci_is_root_bus(bus) ||
-			pci_is_root_bus(bus->parent))
-			ret = pnv_eeh_root_reset(hose, option);
-		else
-			ret = pnv_eeh_bridge_reset(bus->self, option);
  	}

-	return ret;
+	bus = eeh_pe_bus_get(pe);
+	if (pe->type & EEH_PE_VF)
+		return pnv_eeh_reset_vf_pe(pe, option);
+
+	if (pci_is_root_bus(bus) ||
+	    pci_is_root_bus(bus->parent))
+		return pnv_eeh_root_reset(hose, option);
+
+	return pnv_eeh_bridge_reset(bus->self, option);
  }

  /**

-- 
Alexey

Re: [PATCH v8 34/45] powerpc/powernv: Exclude root bus in pnv_pci_reset_secondary_bus()

From: Alexey Kardashevskiy <hidden>
Date: 2016-04-19 08:57:37

On 02/17/2016 02:44 PM, Gavin Shan wrote:
The function pnv_pci_reset_secondary_bus() is called like below.
It's impossible for call the function on root bus. So it's safe
to remove the root bus case in the function. No functional changes
introduced.

    pci_parent_bus_reset() / pci_bus_reset() / pci_try_reset_bus()
    pci_reset_bridge_secondary_bus()
    pcibios_reset_secondary_bus()
    pnv_pci_reset_secondary_bus()

Signed-off-by: Gavin Shan <redacted>
Reviewed-by: Daniel Axtens <redacted>


Reviewed-by: Alexey Kardashevskiy <redacted>

quoted hunk
---
  arch/powerpc/platforms/powernv/eeh-powernv.c | 12 ++----------
  1 file changed, 2 insertions(+), 10 deletions(-)
diff --git a/arch/powerpc/platforms/powernv/eeh-powernv.c b/arch/powerpc/platforms/powernv/eeh-powernv.c
index 9226df1..593b8dc 100644
--- a/arch/powerpc/platforms/powernv/eeh-powernv.c
+++ b/arch/powerpc/platforms/powernv/eeh-powernv.c
@@ -868,16 +868,8 @@ static int pnv_eeh_bridge_reset(struct pci_dev *dev, int option)

  void pnv_pci_reset_secondary_bus(struct pci_dev *dev)
  {
-	struct pci_controller *hose;
-
-	if (pci_is_root_bus(dev->bus)) {
-		hose = pci_bus_to_host(dev->bus);
-		pnv_eeh_root_reset(hose, EEH_RESET_HOT);
-		pnv_eeh_root_reset(hose, EEH_RESET_DEACTIVATE);
-	} else {
-		pnv_eeh_bridge_reset(dev, EEH_RESET_HOT);
-		pnv_eeh_bridge_reset(dev, EEH_RESET_DEACTIVATE);
-	}
+	pnv_eeh_bridge_reset(dev, EEH_RESET_HOT);
+	pnv_eeh_bridge_reset(dev, EEH_RESET_DEACTIVATE);
  }

  static void pnv_eeh_wait_for_pending(struct pci_dn *pdn, const char *type,

-- 
Alexey

Re: [PATCH v8 37/45] powerpc/powernv: Use firmware PCI slot reset infrastructure

From: Alexey Kardashevskiy <hidden>
Date: 2016-04-19 09:34:55

On 02/17/2016 02:44 PM, Gavin Shan wrote:
quoted hunk
The skiboot firmware might provide the PCI slot reset capability
which is identified by property "ibm,reset-by-firmware" on the
PCI slot associated device node.

This checks the property. If it exists, the reset request is routed
to firmware. Otherwise, the reset is done by kernel as before.

Signed-off-by: Gavin Shan <redacted>
---
  arch/powerpc/platforms/powernv/eeh-powernv.c | 41 +++++++++++++++++++++++++++-
  1 file changed, 40 insertions(+), 1 deletion(-)
diff --git a/arch/powerpc/platforms/powernv/eeh-powernv.c b/arch/powerpc/platforms/powernv/eeh-powernv.c
index e23b063..c8a5217 100644
--- a/arch/powerpc/platforms/powernv/eeh-powernv.c
+++ b/arch/powerpc/platforms/powernv/eeh-powernv.c
@@ -789,7 +789,7 @@ static int pnv_eeh_root_reset(struct pci_controller *hose, int option)
  	return ret;
  }

-static int pnv_eeh_bridge_reset(struct pci_dev *dev, int option)
+static int __pnv_eeh_bridge_reset(struct pci_dev *dev, int option)
  {
  	struct pci_dn *pdn = pci_get_pdn_by_devfn(dev->bus, dev->devfn);
  	struct eeh_dev *edev = pdn_to_eeh_dev(pdn);
@@ -840,6 +840,45 @@ static int pnv_eeh_bridge_reset(struct pci_dev *dev, int option)
  	return 0;
  }

+static int pnv_eeh_bridge_reset(struct pci_dev *pdev, int option)
+{
+	struct pci_controller *hose;
+	struct pnv_phb *phb;
+	struct device_node *dn = pdev ? pci_device_to_OF_node(pdev) : NULL;
+	uint64_t id = (0x1ul << 60);

What is this 1<<60 for?

+	uint8_t scope;
+	int64_t rc;
+
+	/*
+	 * If the firmware can't handle it, we will issue hot reset
+	 * on the secondary bus despite the requested reset type.
+	 */
+	if (!dn || !of_get_property(dn, "ibm,reset-by-firmware", NULL))
+		return __pnv_eeh_bridge_reset(pdev, option);
+
+	/* The firmware can handle the request */
+	switch (option) {
+	case EEH_RESET_HOT:
+		scope = OPAL_RESET_PCI_HOT;
+		break;
+	case EEH_RESET_FUNDAMENTAL:
+		scope = OPAL_RESET_PCI_FUNDAMENTAL;
+		break;
+	case EEH_RESET_DEACTIVATE:
+		return 0;
+	default:
+		dev_warn(&pdev->dev, "%s: Unsupported reset %d\n",
+			 __func__, option);

Can the userspace trigger this case (via VFIO-EEH) and flood dmesg?


+		return -EINVAL;
+	}
+
+	hose = pci_bus_to_host(pdev->bus);
+	phb = hose->private_data;
+	id |= (pdev->bus->number << 24) | (pdev->devfn << 16) | phb->opal_id;
+	rc = opal_pci_reset(id, scope, OPAL_ASSERT_RESET);
+	return pnv_pci_poll(id, rc, NULL);
+}
+
  static int pnv_pci_dev_reset_type(struct pci_dev *pdev, void *data)
  {
  	int *freset = data;

-- 
Alexey

Re: [PATCH v8 39/45] powerpc/powernv: Select OF_DYNAMIC

From: Alexey Kardashevskiy <hidden>
Date: 2016-04-19 09:42:01

On 02/17/2016 02:44 PM, Gavin Shan wrote:
quoted hunk
The device tree will change dynamically in PowerNV PCI hotplug
driver. This enables CONFIG_OF_DYNAMIC to support that.

Signed-off-by: Gavin Shan <redacted>
---
  arch/powerpc/platforms/powernv/Kconfig | 1 +
  1 file changed, 1 insertion(+)
diff --git a/arch/powerpc/platforms/powernv/Kconfig b/arch/powerpc/platforms/powernv/Kconfig
index 604190c..e7b1ad7 100644
--- a/arch/powerpc/platforms/powernv/Kconfig
+++ b/arch/powerpc/platforms/powernv/Kconfig
@@ -18,6 +18,7 @@ config PPC_POWERNV
  	select CPU_FREQ_GOV_ONDEMAND
  	select CPU_FREQ_GOV_CONSERVATIVE
  	select PPC_DOORBELL
+	select OF_DYNAMIC

Why not to enable it in 45/45 under config HOTPLUG_PCI_POWERNV? Is there 
any benefit of having it always on if HOTPLUG_PCI_POWERNV is not enabled?

  	default y

  config OPAL_PRD

-- 
Alexey

Re: [PATCH v8 09/45] powerpc/powernv: Simplify pnv_ioda_setup_pe_seg()

From: Gavin Shan <hidden>
Date: 2016-04-20 00:04:32

On Wed, Apr 13, 2016 at 04:45:39PM +1000, Alexey Kardashevskiy wrote:
On 02/17/2016 02:43 PM, Gavin Shan wrote:
quoted
The original implementation of pnv_ioda_setup_pe_seg() configures
IO and M32 segments by separate logics, which can be merged by
by caching @segmap, @seg_size, @win in advance. This shouldn't
cause any behavioural changes.

Signed-off-by: Gavin Shan <redacted>
---
 arch/powerpc/platforms/powernv/pci-ioda.c | 62 ++++++++++++++-----------------
 1 file changed, 28 insertions(+), 34 deletions(-)
diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index 44cc5f3..fd7d382 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -2940,8 +2940,10 @@ static void pnv_ioda_setup_pe_seg(struct pci_controller *hose,
 	struct pnv_phb *phb = hose->private_data;
 	struct pci_bus_region region;
 	struct resource *res;
-	int i, index;
-	int rc;
+	unsigned int segsize;
+	int *segmap, index, i;
+	uint16_t win;
+	int64_t rc;

 	/*
 	 * NOTE: We only care PCI bus based PE for now. For PCI
@@ -2958,23 +2960,9 @@ static void pnv_ioda_setup_pe_seg(struct pci_controller *hose,
 		if (res->flags & IORESOURCE_IO) {
 			region.start = res->start - phb->ioda.io_pci_base;
 			region.end   = res->end - phb->ioda.io_pci_base;
-			index = region.start / phb->ioda.io_segsize;
-
-			while (index < phb->ioda.total_pe_num &&
-			       region.start <= region.end) {
-				phb->ioda.io_segmap[index] = pe->pe_number;
-				rc = opal_pci_map_pe_mmio_window(phb->opal_id,
-					pe->pe_number, OPAL_IO_WINDOW_TYPE, 0, index);
-				if (rc != OPAL_SUCCESS) {
-					pr_err("%s: OPAL error %d when mapping IO "
-					       "segment #%d to PE#%d\n",
-					       __func__, rc, index, pe->pe_number);
-					break;
-				}
-
-				region.start += phb->ioda.io_segsize;
-				index++;
-			}
+			segsize      = phb->ioda.io_segsize;
+			segmap       = phb->ioda.io_segmap;
+			win          = OPAL_IO_WINDOW_TYPE;
 		} else if ((res->flags & IORESOURCE_MEM) &&
 			   !pnv_pci_is_mem_pref_64(res->flags)) {
 			region.start = res->start -
@@ -2983,23 +2971,29 @@ static void pnv_ioda_setup_pe_seg(struct pci_controller *hose,
 			region.end   = res->end -
 				       hose->mem_offset[0] -
 				       phb->ioda.m32_pci_base;
-			index = region.start / phb->ioda.m32_segsize;
-
-			while (index < phb->ioda.total_pe_num &&
-			       region.start <= region.end) {
-				phb->ioda.m32_segmap[index] = pe->pe_number;
-				rc = opal_pci_map_pe_mmio_window(phb->opal_id,
-					pe->pe_number, OPAL_M32_WINDOW_TYPE, 0, index);
-				if (rc != OPAL_SUCCESS) {
-					pr_err("%s: OPAL error %d when mapping M32 "
-					       "segment#%d to PE#%d",
-					       __func__, rc, index, pe->pe_number);
-					break;
-				}
+			segsize      = phb->ioda.m32_segsize;
+			segmap       = phb->ioda.m32_segmap;
+			win          = OPAL_M32_WINDOW_TYPE;
+		} else {
+			continue;
+		}

-				region.start += phb->ioda.m32_segsize;
-				index++;
+		index = region.start / segsize;
+		while (index < phb->ioda.total_pe_num &&
+		       region.start <= region.end) {
+			segmap[index] = pe->pe_number;
+			rc = opal_pci_map_pe_mmio_window(phb->opal_id,
+					pe->pe_number, win, 0, index);
+			if (rc != OPAL_SUCCESS) {
+				pr_warn("%s: Error %lld mapping (%d) seg#%d to PHB#%d-PE#%d\n",
+					__func__, rc, win, index,
+					pe->phb->hose->global_number,
+					pe->pe_number);
+				break;
Please move this loop to a helper and stop caching segsize/segmap/win; this
will make the code easier to read and the next patch will look much cleaner
as it will not have to move this exact loop.
Thanks. It's good idea and I'll change the code accordingly in next revision.
quoted
 			}
+
+			region.start += segsize;
+			index++;
 		}
 	}
 }

-- 
Alexey

Re: [PATCH v8 11/45] powerpc/powernv: Track M64 segment consumption

From: Gavin Shan <hidden>
Date: 2016-04-20 00:05:45

On Wed, Apr 13, 2016 at 05:09:45PM +1000, Alexey Kardashevskiy wrote:
On 02/17/2016 02:43 PM, Gavin Shan wrote:
quoted
When unplugging PCI devices, their parent PEs might be offline.
The consumed M64 resource by the PEs should be released at that
time. As we track M32 segment consumption, this introduces an
array to the PHB to track the mapping between M64 segment and
PE number.

Signed-off-by: Gavin Shan <redacted>

Reviewed-by: Alexey Kardashevskiy <redacted>

but it would not hurt to mention in the commit log why M64 segment is not
tracked/setup by the existing (at this point, at least)
pnv_ioda_setup_one_res().
Right, I'll add something for it to the commit log in next revision, thanks!
quoted
---
 arch/powerpc/platforms/powernv/pci-ioda.c | 10 ++++++++--
 arch/powerpc/platforms/powernv/pci.h      |  1 +
 2 files changed, 9 insertions(+), 2 deletions(-)
diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index 7330a73..fc0374a 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -305,6 +305,7 @@ static int pnv_ioda2_pick_m64_pe(struct pci_bus *bus, bool all)
 		phb->ioda.total_pe_num) {
 		pe = &phb->ioda.pe_array[i];

+		phb->ioda.m64_segmap[pe->pe_number] = pe->pe_number;
 		if (!master_pe) {
 			pe->flags |= PNV_IODA_PE_MASTER;
 			INIT_LIST_HEAD(&pe->slaves);
@@ -3245,7 +3246,7 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
 {
 	struct pci_controller *hose;
 	struct pnv_phb *phb;
-	unsigned long size, m32map_off, pemap_off, iomap_off = 0;
+	unsigned long size, m64map_off, m32map_off, pemap_off, iomap_off = 0;
 	const __be64 *prop64;
 	const __be32 *prop32;
 	int i, len;
@@ -3332,6 +3333,8 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,

 	/* Allocate aux data & arrays. We don't have IO ports on PHB3 */
 	size = _ALIGN_UP(phb->ioda.total_pe_num / 8, sizeof(unsigned long));
+	m64map_off = size;
+	size += phb->ioda.total_pe_num * sizeof(phb->ioda.m64_segmap[0]);
 	m32map_off = size;
 	size += phb->ioda.total_pe_num * sizeof(phb->ioda.m32_segmap[0]);
 	if (phb->type == PNV_PHB_IODA1) {
@@ -3342,9 +3345,12 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
 	size += phb->ioda.total_pe_num * sizeof(struct pnv_ioda_pe);
 	aux = memblock_virt_alloc(size, 0);
 	phb->ioda.pe_alloc = aux;
+	phb->ioda.m64_segmap = aux + m64map_off;
 	phb->ioda.m32_segmap = aux + m32map_off;
-	for (i = 0; i < phb->ioda.total_pe_num; i++)
+	for (i = 0; i < phb->ioda.total_pe_num; i++) {
+		phb->ioda.m64_segmap[i] = IODA_INVALID_PE;
 		phb->ioda.m32_segmap[i] = IODA_INVALID_PE;
+	}
 	if (phb->type == PNV_PHB_IODA1) {
 		phb->ioda.io_segmap = aux + iomap_off;
 		for (i = 0; i < phb->ioda.total_pe_num; i++)
diff --git a/arch/powerpc/platforms/powernv/pci.h b/arch/powerpc/platforms/powernv/pci.h
index 36c4965..866a5ea 100644
--- a/arch/powerpc/platforms/powernv/pci.h
+++ b/arch/powerpc/platforms/powernv/pci.h
@@ -146,6 +146,7 @@ struct pnv_phb {
 		struct pnv_ioda_pe	*pe_array;

 		/* M32 & IO segment maps */
+		int			*m64_segmap;
 		int			*m32_segmap;
 		int			*io_segmap;

-- 
Alexey

Re: [PATCH v8 16/45] powerpc/powernv: Remove DMA32 PE list

From: Gavin Shan <hidden>
Date: 2016-04-20 00:34:04

On Wed, Apr 13, 2016 at 06:59:40PM +1000, Alexey Kardashevskiy wrote:
On 02/17/2016 02:43 PM, Gavin Shan wrote:
quoted
PEs are put into PHB DMA32 list (phb->ioda.pe_dma_list) according
to their DMA32 weight. The PEs on the list are iterated to setup
their TCE32 tables at system booting time. The list is used for
once and there is for keep having it.
"there is no need to keep it" may be?
Sorry, I should have fixed it in early revision. Will fix it
up in next revision.
quoted
This moves the logic calculating DMA32 weight of PHB and PE to
pnv_ioda_setup_dma() to drop PHB's DMA32 list. Also, every PE
traces the consumed DMA32 segment by @tce32_seg and @tce32_segcount
are useless and they're removed.

Signed-off-by: Gavin Shan <redacted>

Reviewed-by: Alexey Kardashevskiy <redacted>

with few comments below...
quoted
---
 arch/powerpc/platforms/powernv/pci-ioda.c | 168 +++++++++++++-----------------
 arch/powerpc/platforms/powernv/pci.h      |  19 ----
 2 files changed, 75 insertions(+), 112 deletions(-)
diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index e60cff6..0fc2309 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -886,44 +886,6 @@ out:
 	return 0;
 }

-static void pnv_ioda_link_pe_by_weight(struct pnv_phb *phb,
-				       struct pnv_ioda_pe *pe)
-{
-	struct pnv_ioda_pe *lpe;
-
-	list_for_each_entry(lpe, &phb->ioda.pe_dma_list, dma_link) {
-		if (lpe->dma_weight < pe->dma_weight) {
-			list_add_tail(&pe->dma_link, &lpe->dma_link);
-			return;
-		}
-	}
-	list_add_tail(&pe->dma_link, &phb->ioda.pe_dma_list);
-}
-
-static unsigned int pnv_ioda_dma_weight(struct pci_dev *dev)
-{
-	/* This is quite simplistic. The "base" weight of a device
-	 * is 10. 0 means no DMA is to be accounted for it.
-	 */
-
-	/* If it's a bridge, no DMA */
-	if (dev->hdr_type != PCI_HEADER_TYPE_NORMAL)
-		return 0;
-
-	/* Reduce the weight of slow USB controllers */
-	if (dev->class == PCI_CLASS_SERIAL_USB_UHCI ||
-	    dev->class == PCI_CLASS_SERIAL_USB_OHCI ||
-	    dev->class == PCI_CLASS_SERIAL_USB_EHCI)
-		return 3;
-
-	/* Increase the weight of RAID (includes Obsidian) */
-	if ((dev->class >> 8) == PCI_CLASS_STORAGE_RAID)
-		return 15;
-
-	/* Default */
-	return 10;
-}
-
 #ifdef CONFIG_PCI_IOV
 static int pnv_pci_vf_resource_shift(struct pci_dev *dev, int offset)
 {
@@ -1028,7 +990,6 @@ static struct pnv_ioda_pe *pnv_ioda_setup_dev_PE(struct pci_dev *dev)
 	pe->flags = PNV_IODA_PE_DEV;
 	pe->pdev = dev;
 	pe->pbus = NULL;
-	pe->tce32_seg = -1;
 	pe->mve_number = -1;
 	pe->rid = dev->bus->number << 8 | pdn->devfn;
@@ -1044,16 +1005,6 @@ static struct pnv_ioda_pe *pnv_ioda_setup_dev_PE(struct pci_dev *dev)
 		return NULL;
 	}

-	/* Assign a DMA weight to the device */
-	pe->dma_weight = pnv_ioda_dma_weight(dev);
-	if (pe->dma_weight != 0) {
-		phb->ioda.dma_weight += pe->dma_weight;
-		phb->ioda.dma_pe_count++;
-	}
-
-	/* Link the PE */
-	pnv_ioda_link_pe_by_weight(phb, pe);
-
 	return pe;
 }
@@ -1071,7 +1022,6 @@ static void pnv_ioda_setup_same_PE(struct pci_bus *bus, struct pnv_ioda_pe *pe)
 		}
 		pdn->pcidev = dev;
 		pdn->pe_number = pe->pe_number;
-		pe->dma_weight += pnv_ioda_dma_weight(dev);
 		if ((pe->flags & PNV_IODA_PE_BUS_ALL) && dev->subordinate)
 			pnv_ioda_setup_same_PE(dev->subordinate, pe);
 	}
@@ -1108,10 +1058,8 @@ static void pnv_ioda_setup_bus_PE(struct pci_bus *bus, bool all)
 	pe->flags |= (all ? PNV_IODA_PE_BUS_ALL : PNV_IODA_PE_BUS);
 	pe->pbus = bus;
 	pe->pdev = NULL;
-	pe->tce32_seg = -1;
 	pe->mve_number = -1;
 	pe->rid = bus->busn_res.start << 8;
-	pe->dma_weight = 0;

 	if (all)
 		pe_info(pe, "Secondary bus %d..%d associated with PE#%d\n",
@@ -1133,17 +1081,6 @@ static void pnv_ioda_setup_bus_PE(struct pci_bus *bus, bool all)

 	/* Put PE to the list */
 	list_add_tail(&pe->list, &phb->ioda.pe_list);
-
-	/* Account for one DMA PE if at least one DMA capable device exist
-	 * below the bridge
-	 */
-	if (pe->dma_weight != 0) {
-		phb->ioda.dma_weight += pe->dma_weight;
-		phb->ioda.dma_pe_count++;
-	}
-
-	/* Link the PE */
-	pnv_ioda_link_pe_by_weight(phb, pe);
 }

 static struct pnv_ioda_pe *pnv_ioda_setup_npu_PE(struct pci_dev *npu_pdev)
@@ -1184,7 +1121,6 @@ static struct pnv_ioda_pe *pnv_ioda_setup_npu_PE(struct pci_dev *npu_pdev)
 			rid = npu_pdev->bus->number << 8 | npu_pdn->devfn;
 			npu_pdn->pcidev = npu_pdev;
 			npu_pdn->pe_number = pe_num;
-			pe->dma_weight += pnv_ioda_dma_weight(npu_pdev);
 			phb->ioda.pe_rmap[rid] = pe->pe_number;

 			/* Map the PE to this link */
@@ -1532,7 +1468,6 @@ static void pnv_ioda_setup_vf_PE(struct pci_dev *pdev, u16 num_vfs)
 		pe->flags = PNV_IODA_PE_VF;
 		pe->pbus = NULL;
 		pe->parent_dev = pdev;
-		pe->tce32_seg = -1;
 		pe->mve_number = -1;
 		pe->rid = (pci_iov_virtfn_bus(pdev, vf_index) << 8) |
 			   pci_iov_virtfn_devfn(pdev, vf_index);
@@ -2023,6 +1958,54 @@ static struct iommu_table_ops pnv_ioda2_iommu_ops = {
 	.free = pnv_ioda2_table_free,
 };

+static int pnv_pci_ioda_dev_dma_weight(struct pci_dev *dev, void *data)
+{
+	unsigned int *weight = (unsigned int *)data;
+
+	/* This is quite simplistic. The "base" weight of a device
+	 * is 10. 0 means no DMA is to be accounted for it.
+	 */
+	if (dev->hdr_type != PCI_HEADER_TYPE_NORMAL)
+		return 0;
+
+	if (dev->class == PCI_CLASS_SERIAL_USB_UHCI ||
+	    dev->class == PCI_CLASS_SERIAL_USB_OHCI ||
+	    dev->class == PCI_CLASS_SERIAL_USB_EHCI)
+		*weight += 3;
+	else if ((dev->class >> 8) == PCI_CLASS_STORAGE_RAID)
+		*weight += 15;
+	else
+		*weight += 10;
+
+	return 0;
+}
+
+static unsigned int pnv_pci_ioda_pe_dma_weight(struct pnv_ioda_pe *pe)
+{
+	unsigned int weight = 0;
+
+	if ((pe->flags & PNV_IODA_PE_DEV) && pe->pdev) {
+		pnv_pci_ioda_dev_dma_weight(pe->pdev, &weight);
+	} else if ((pe->flags & PNV_IODA_PE_BUS) && pe->pbus) {
+		struct pci_dev *pdev;
+
+		list_for_each_entry(pdev, &pe->pbus->devices, bus_list)
+			pnv_pci_ioda_dev_dma_weight(pdev, &weight);
+	} else if ((pe->flags & PNV_IODA_PE_BUS_ALL) && pe->pbus) {
+		pci_walk_bus(pe->pbus, pnv_pci_ioda_dev_dma_weight, &weight);
+	}
+
+	return weight;
+}
+
+static unsigned int pnv_pci_ioda_total_dma_weight(struct pnv_phb *phb)

s/pnv_pci_ioda_total_dma_weight/pnv_pci_ioda1_phb_dma_weight/ ? "total" does
not say much. Or just merge it into pnv_pci_ioda1_setup_dma_pe() as it is
useless for anything but IODA1.
Nice suggestion. I will merge it to pnv_pci_ioda1_setup_dma_pe().
quoted
+{
+	unsigned int weight = 0;
+
+	pci_walk_bus(phb->hose->bus, pnv_pci_ioda_dev_dma_weight, &weight);
+	return weight;
+}
+
 static void pnv_pci_ioda1_setup_dma_pe(struct pnv_phb *phb,
 				       struct pnv_ioda_pe *pe,
 				       unsigned int base,
@@ -2039,17 +2022,12 @@ static void pnv_pci_ioda1_setup_dma_pe(struct pnv_phb *phb,
 	/* XXX FIXME: Provide 64-bit DMA facilities & non-4K TCE tables etc.. */
 	/* XXX FIXME: Allocate multi-level tables on PHB3 */

-	/* We shouldn't already have a 32-bit DMA associated */
-	if (WARN_ON(pe->tce32_seg >= 0))
-		return;
-
 	tbl = pnv_pci_table_alloc(phb->hose->node);
 	iommu_register_group(&pe->table_group, phb->hose->global_number,
 			pe->pe_number);
 	pnv_pci_link_table_and_group(phb->hose->node, 0, tbl, &pe->table_group);

 	/* Grab a 32-bit TCE table */
-	pe->tce32_seg = base;
 	pe_info(pe, " Setting up 32-bit TCE table at %08x..%08x\n",
 		base * PNV_IODA1_DMA32_SEGSIZE,
 		(base + segs) * PNV_IODA1_DMA32_SEGSIZE - 1);
@@ -2116,8 +2094,6 @@ static void pnv_pci_ioda1_setup_dma_pe(struct pnv_phb *phb,
 	return;
  fail:
 	/* XXX Failure: Try to fallback to 64-bit only ? */
-	if (pe->tce32_seg >= 0)
-		pe->tce32_seg = -1;
 	if (tce_mem)
 		__free_pages(tce_mem, get_order(tce32_segsz * segs));
 	if (tbl) {
@@ -2528,10 +2504,6 @@ static void pnv_pci_ioda2_setup_dma_pe(struct pnv_phb *phb,
 {
 	int64_t rc;

-	/* We shouldn't already have a 32-bit DMA associated */
-	if (WARN_ON(pe->tce32_seg >= 0))
-		return;
-
 	/* TVE #1 is selected by PCI address bit 59 */
 	pe->tce_bypass_base = 1ull << 59;
@@ -2539,7 +2511,6 @@ static void pnv_pci_ioda2_setup_dma_pe(struct pnv_phb *phb,
 			pe->pe_number);

 	/* The PE will reserve all possible 32-bits space */
-	pe->tce32_seg = 0;
 	pe_info(pe, "Setting up 32-bit TCE table at 0..%08x\n",
 		phb->ioda.m32_pci_base);
@@ -2555,11 +2526,8 @@ static void pnv_pci_ioda2_setup_dma_pe(struct pnv_phb *phb,
 #endif

 	rc = pnv_pci_ioda2_setup_default_config(pe);
-	if (rc) {
-		if (pe->tce32_seg >= 0)
-			pe->tce32_seg = -1;
+	if (rc)
 		return;
-	}

 	if (pe->flags & PNV_IODA_PE_DEV)
 		iommu_add_device(&pe->pdev->dev);
@@ -2570,24 +2538,32 @@ static void pnv_pci_ioda2_setup_dma_pe(struct pnv_phb *phb,
 static void pnv_ioda_setup_dma(struct pnv_phb *phb)
 {
 	struct pci_controller *hose = phb->hose;
-	unsigned int residual, remaining, segs, tw, base;
+	unsigned int weight, total_weight, dma_pe_count;
+	unsigned int residual, remaining, segs, base;
 	struct pnv_ioda_pe *pe;

+	total_weight = pnv_pci_ioda_total_dma_weight(phb);
+	dma_pe_count = 0;
+	list_for_each_entry(pe, &phb->ioda.pe_list, list) {
+		weight = pnv_pci_ioda_pe_dma_weight(pe);
+		if (weight > 0)
+			dma_pe_count++;
+	}
+
 	/* If we have more PE# than segments available, hand out one
 	 * per PE until we run out and let the rest fail. If not,
 	 * then we assign at least one segment per PE, plus more based
 	 * on the amount of devices under that PE
 	 */
-	if (phb->ioda.dma_pe_count > phb->ioda.tce32_count)
+	if (dma_pe_count > phb->ioda.tce32_count)
 		residual = 0;
 	else
-		residual = phb->ioda.tce32_count -
-			phb->ioda.dma_pe_count;
+		residual = phb->ioda.tce32_count - dma_pe_count;

 	pr_info("PCI: Domain %04x has %ld available 32-bit DMA segments\n",
 		hose->global_number, phb->ioda.tce32_count);
 	pr_info("PCI: %d PE# for a total weight of %d\n",
-		phb->ioda.dma_pe_count, phb->ioda.dma_weight);
+		dma_pe_count, total_weight);

 	pnv_pci_ioda_setup_opal_tce_kill(phb);
@@ -2596,18 +2572,20 @@ static void pnv_ioda_setup_dma(struct pnv_phb *phb)
 	 * weight
 	 */
 	remaining = phb->ioda.tce32_count;
-	tw = phb->ioda.dma_weight;
 	base = 0;
-	list_for_each_entry(pe, &phb->ioda.pe_dma_list, dma_link) {
-		if (!pe->dma_weight)
+	list_for_each_entry(pe, &phb->ioda.pe_list, list) {
+		weight = pnv_pci_ioda_pe_dma_weight(pe);
+		if (!weight)
 			continue;
+
 		if (!remaining) {
 			pe_warn(pe, "No DMA32 resources available\n");
 			continue;
 		}
 		segs = 1;
 		if (residual) {
-			segs += ((pe->dma_weight * residual)  + (tw / 2)) / tw;
+			segs += ((weight * residual) + (total_weight / 2)) /
+				total_weight;
 			if (segs > remaining)
 				segs = remaining;
 		}
@@ -2619,7 +2597,7 @@ static void pnv_ioda_setup_dma(struct pnv_phb *phb)
 		 */
 		if (phb->type == PNV_PHB_IODA1) {
 			pe_info(pe, "DMA weight %d, assigned %d DMA32 segments\n",
-				pe->dma_weight, segs);
+				weight, segs);
 			pnv_pci_ioda1_setup_dma_pe(phb, pe, base, segs);
 		} else if (phb->type == PNV_PHB_IODA2) {
 			pe_info(pe, "Assign DMA32 space\n");
@@ -3156,13 +3134,18 @@ static void pnv_npu_ioda_fixup(void)
 	struct pci_controller *hose, *tmp;
 	struct pnv_phb *phb;
 	struct pnv_ioda_pe *pe;
+	unsigned int weight;

 	list_for_each_entry_safe(hose, tmp, &hose_list, list_node) {
 		phb = hose->private_data;
 		if (phb->type != PNV_PHB_NPU)
 			continue;

-		list_for_each_entry(pe, &phb->ioda.pe_dma_list, dma_link) {
+		list_for_each_entry(pe, &phb->ioda.pe_list, list) {
+			weight = pnv_pci_ioda_pe_dma_weight(pe);
+			if (!weight)
+				continue;
Is this even possible for NPU PE to get weight==0? WARN_ON()? BUG_ON()?
It's impossible and worthy to have a WARN_ON() here. Will address it
in next revision.
quoted
+
 			enable_bypass = dma_get_mask(&pe->pdev->dev) ==
 				DMA_BIT_MASK(64);
 			pnv_npu_init_dma_pe(pe);
@@ -3443,7 +3426,6 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
 	phb->ioda.pe_array = aux + pemap_off;
 	set_bit(phb->ioda.reserved_pe_idx, phb->ioda.pe_alloc);

-	INIT_LIST_HEAD(&phb->ioda.pe_dma_list);
 	INIT_LIST_HEAD(&phb->ioda.pe_list);
 	mutex_init(&phb->ioda.pe_list_mutex);
diff --git a/arch/powerpc/platforms/powernv/pci.h b/arch/powerpc/platforms/powernv/pci.h
index 1d8e775..e90bcbe 100644
--- a/arch/powerpc/platforms/powernv/pci.h
+++ b/arch/powerpc/platforms/powernv/pci.h
@@ -53,14 +53,7 @@ struct pnv_ioda_pe {
 	/* PE number */
 	unsigned int		pe_number;

-	/* "Weight" assigned to the PE for the sake of DMA resource
-	 * allocations
-	 */
-	unsigned int		dma_weight;
-
 	/* "Base" iommu table, ie, 4K TCEs, 32-bit DMA */
-	int			tce32_seg;
-	int			tce32_segcount;
 	struct iommu_table_group table_group;

 	/* 64-bit TCE bypass region */
@@ -78,7 +71,6 @@ struct pnv_ioda_pe {
 	struct list_head	slaves;

 	/* Link in list of PE#s */
-	struct list_head	dma_link;
 	struct list_head	list;
 };
@@ -173,17 +165,6 @@ struct pnv_phb {
 		/* 32-bit TCE tables allocation */
 		unsigned long		tce32_count;

-		/* Total "weight" for the sake of DMA resources
-		 * allocation
-		 */
-		unsigned int		dma_weight;
-		unsigned int		dma_pe_count;
-
-		/* Sorted list of used PE's, sorted at
-		 * boot for resource allocation purposes
-		 */
-		struct list_head	pe_dma_list;
-
 		/* TCE cache invalidate registers (physical and
 		 * remapped)
 		 */

-- 
Alexey

Re: [PATCH v8 18/45] powerpc/powernv: Increase PE# capacity

From: Gavin Shan <hidden>
Date: 2016-04-20 00:52:46

On Tue, Apr 19, 2016 at 12:02:23PM +1000, Alexey Kardashevskiy wrote:
On 02/17/2016 02:44 PM, Gavin Shan wrote:
quoted
Each PHB maintains an array helping to translate 2-bytes Request
ID (RID) to PE# with the assumption that PE# takes one byte, meaning
that we can't have more than 256 PEs. However, pci_dn->pe_number
already had 4-bytes for the PE#.

This extends the PE# capacity for every PHB. After that, the PE number
is represented by 4-bytes value. Then we can reuse IODA_INVALID_PE to
check the PE# in phb->pe_rmap[] is valid or not.

This should be merged into "[PATCH v8 21/45] powerpc/powernv: Create PEs at
PCI hot plugging time" as it does not make sense alone (this patch does the
initialization but only 3 patches apart this default value is analyzed ->
hard to review).
Indeed, will move accordingly in next revision.
quoted
Signed-off-by: Gavin Shan <redacted>
Reviewed-by: Daniel Axtens <redacted>
---
 arch/powerpc/platforms/powernv/pci-ioda.c | 6 +++++-
 arch/powerpc/platforms/powernv/pci.h      | 7 ++-----
 2 files changed, 7 insertions(+), 6 deletions(-)
diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index 59782fba..7800897 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -757,7 +757,7 @@ static int pnv_ioda_deconfigure_pe(struct pnv_phb *phb, struct pnv_ioda_pe *pe)

 	/* Clear the reverse map */
 	for (rid = pe->rid; rid < rid_end; rid++)
-		phb->ioda.pe_rmap[rid] = 0;
+		phb->ioda.pe_rmap[rid] = IODA_INVALID_PE;

 	/* Release from all parents PELT-V */
 	while (parent) {
@@ -3387,6 +3387,10 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
 	if (prop32)
 		phb->ioda.reserved_pe_idx = be32_to_cpup(prop32);

+	/* Invalidate RID to PE# mapping */
+	for (i = 0; i < ARRAY_SIZE(phb->ioda.pe_rmap); ++i)
+		phb->ioda.pe_rmap[i] = IODA_INVALID_PE;
+
 	/* Parse 64-bit MMIO range */
 	pnv_ioda_parse_m64_window(phb);
diff --git a/arch/powerpc/platforms/powernv/pci.h b/arch/powerpc/platforms/powernv/pci.h
index 350e630..928cf81 100644
--- a/arch/powerpc/platforms/powernv/pci.h
+++ b/arch/powerpc/platforms/powernv/pci.h
@@ -160,11 +160,8 @@ struct pnv_phb {
 		struct list_head	pe_list;
 		struct mutex            pe_list_mutex;

-		/* Reverse map of PEs, will have to extend if
-		 * we are to support more than 256 PEs, indexed
-		 * bus { bus, devfn }
-		 */
-		unsigned char		pe_rmap[0x10000];
+		/* Reverse map of PEs, indexed by {bus, devfn} */
+		int			pe_rmap[0x10000];

 		/* TCE cache invalidate registers (physical and
 		 * remapped)

-- 
Alexey

Re: [PATCH v8 20/45] powerpc/powernv: Allocate PE# in reverse order

From: Gavin Shan <hidden>
Date: 2016-04-20 01:04:35

On Tue, Apr 19, 2016 at 01:07:59PM +1000, Alexey Kardashevskiy wrote:
On 02/17/2016 02:44 PM, Gavin Shan wrote:
quoted
PE number for one particular PE can be allocated dynamically or
reserved according to the consumed M64 (64-bits prefetchable)
segments of the PE. The M64 resources, and hence their segments
and PE number are assigned/reserved in ascending order. The PE
numbers are allocated dynamically in ascending order as well.
It's not a problem as the PE numbers are reserved and then
allocated all at once in fine order. However, it will introduce
conflicts when PCI hotplug is supported: the PE number to be
reserved for newly added PE might have been assigned.

To resolve above conflicts, this forces the PE number to be
allocated dynamically in reverse order. With this patch applied,
the PE numbers are reserved in ascending order, but allocated
dynamically in reverse order.

The patch is probably is ok, the commit log is not - I do not follow it. Some
PEs are reserved (for what? why does the absolute PE number matter? put it in
the commit log), that means that the corresponding bits in pe_alloc[] should
be set so when you will be allocating PEs for a just plugged device, you
won't pick them and you will pick free ones, and the order should not matter.
I would think that "reservation" happens once at the boot time so you set
"used" bits for the reserved PEs then and after that the dynamic allocator
will skip them.
I will enhance the commit log in next revision, perhaps just pick part of
below words: On PHB3, there are 16 M64 BARs in hardware. The last one is
split ovenly into 256 segments. Each segment can be associated/assigned
to fixed PE# (segment#x <-> PE#x) which is how the hardware was designed.
If one plugged PE has M64 (64-bits prefetchable memory) resources, its
PE# is equal to the segment#. Otherwise, the PE# is allocated dynamically
if the PE doesn't contain M64 resource.

The M64 resources are assigned from low to high end, meaning the reserved
PE# (according to the M64 segments) are grown from low to high end. It's
most likely to get a dynamically allocated PE# which should be reserved
because of M64 segment. It's the conflicts the patch tries to resolve.

The PE# reservation doesn't happen once at boot time because it's
unknow how many PEs and how much M64 resources will be hot added.
quoted
Signed-off-by: Gavin Shan <redacted>
---
 arch/powerpc/platforms/powernv/pci-ioda.c | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)
diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index f182ca7..565725b 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -144,16 +144,14 @@ static void pnv_ioda_reserve_pe(struct pnv_phb *phb, int pe_no)

 static struct pnv_ioda_pe *pnv_ioda_alloc_pe(struct pnv_phb *phb)
 {
-	unsigned long pe;
+	unsigned long pe = phb->ioda.total_pe_num - 1;

-	do {
-		pe = find_next_zero_bit(phb->ioda.pe_alloc,
-					phb->ioda.total_pe_num, 0);
-		if (pe >= phb->ioda.total_pe_num)
-			return NULL;
-	} while(test_and_set_bit(pe, phb->ioda.pe_alloc));
+	for (pe = phb->ioda.total_pe_num - 1; pe >= 0; pe--) {
+		if (!test_and_set_bit(pe, phb->ioda.pe_alloc))
+			return pnv_ioda_init_pe(phb, pe);
+	}

-	return pnv_ioda_init_pe(phb, pe);
+	return NULL;
 }

 static void pnv_ioda_free_pe(struct pnv_ioda_pe *pe)

-- 
Alexey

Re: [PATCH v8 21/45] powerpc/powernv: Create PEs at PCI hot plugging time

From: Gavin Shan <hidden>
Date: 2016-04-20 01:12:05

On Tue, Apr 19, 2016 at 02:16:42PM +1000, Alexey Kardashevskiy wrote:
On 02/17/2016 02:44 PM, Gavin Shan wrote:
quoted
Currently, the PEs and their associated resources are assigned
in ppc_md.pcibios_fixup() except those used by SRIOV VFs.
But this new code does not affect IOV and VF's PEs will still be created
somewhere else rather than pnv_pci_setup_bridge()?
Correct. VF PEs cannot be created in pnv_pci_setup_bridge() as the PF's
IOV capability isn't enabled at that point.
quoted
The
function is called for once after PCI probing and resources
assignment is completed. So it isn't hotplug friendly.

This creates PEs dynamically by ppc_md.pcibios_setup_bridge(), which
is called on the event during system bootup and PCI hotplug: updating
PCI bridge's windows after resource assignment/reassignment are done.
For partial hotplug case, where not all PCI devices belonging to the
PE are unplugged and plugged again, we just need unbinding/binding
the affected PCI devices with the corresponding PE without creating
new one.

As there is no upstream bridge for root bus that needs to be covered
by PE, we have to create PE for root bus in ppc_md.pcibios_setup_bridge()
before any other PEs can be created, as PE for root bus is the ancestor
to anyone else.
We did not need a root bus PE before? What is the other PE reserved for?
Comments only say "reserved"...
No, A PE for root bus is needed before. other PEs can be for the PCI bus
originated from root port and the subordinate domains.
 
quoted
Also, the windows of root port or the upstream port of PCIe switch behind
root port are extended to be PHB's apertures to accommodate the additional
resources needed by newly plugged devices based on the fact: hotpluggable
slot is behind root port or downstream port of the PCIe switch behind
root port. The extension for those PCI brdiges' windows is done in
ppc_md.pcibios_setup_bridge() as well.

This patch seems to be doing way too many things, hard to follow.

Could you please split the patch into smaller chunks? For example (you can do
it totally different):
- move pnv_pci_ioda_setup_opal_tce_kill()
- move PE creation from pnv_pci_ioda_fixup() to pnv_pci_setup_bridge();
- add pnv_pci_fixup_bridge_resources()
- add an extra reserved PE for the root bus (and all this magic with
root_pe_idx/root_pe_populated)
- ...
I'll evaluate it later. It's always nice to have small patches. Thanks
for the comments.


-- 
Alexey

Re: [PATCH v8 22/45] powerpc/powernv/ioda1: Support releasing IODA1 TCE table

From: Gavin Shan <hidden>
Date: 2016-04-20 01:15:17

On Tue, Apr 19, 2016 at 02:28:51PM +1000, Alexey Kardashevskiy wrote:
On 02/17/2016 02:44 PM, Gavin Shan wrote:
quoted
pnv_pci_ioda_table_free_pages() can be reused to release the IODA1
TCE table when releasing IODA1 PE in subsequent patches.

This renames the following functions to support releasing IODA1 TCE
table: pnv_pci_ioda2_table_free_pages() to pnv_pci_ioda_table_free_pages(),
pnv_pci_ioda2_table_do_free_pages() to pnv_pci_ioda_table_do_free_pages().
No logical changes introduced.
I can only see renaming here but it seems (from
IODA_architecture_04-14-2008.pdf) that IODA1 does not support multi-level TCE
tables in the way IODA2 does.
Note that the change was proposed by you in last round. Yes, TVE on P7IOC
doesn't support multiple levels of TCE tables. In this case, we will always
have "tbl->it_indirect_levels" to 1, right?
quoted
Signed-off-by: Gavin Shan <redacted>
---
 arch/powerpc/platforms/powernv/pci-ioda.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)
diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index d360607..077f9db 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -51,7 +51,7 @@
 #define POWERNV_IOMMU_DEFAULT_LEVELS	1
 #define POWERNV_IOMMU_MAX_LEVELS	5

-static void pnv_pci_ioda2_table_free_pages(struct iommu_table *tbl);
+static void pnv_pci_ioda_table_free_pages(struct iommu_table *tbl);

 static void pe_level_printk(const struct pnv_ioda_pe *pe, const char *level,
 			    const char *fmt, ...)
@@ -1352,7 +1352,7 @@ static void pnv_pci_ioda2_release_dma_pe(struct pci_dev *dev, struct pnv_ioda_pe
 		iommu_group_put(pe->table_group.group);
 		BUG_ON(pe->table_group.group);
 	}
-	pnv_pci_ioda2_table_free_pages(tbl);
+	pnv_pci_ioda_table_free_pages(tbl);
 	iommu_free_table(tbl, of_node_full_name(dev->dev.of_node));
 }
@@ -1946,7 +1946,7 @@ static void pnv_ioda2_tce_free(struct iommu_table *tbl, long index,

 static void pnv_ioda2_table_free(struct iommu_table *tbl)
 {
-	pnv_pci_ioda2_table_free_pages(tbl);
+	pnv_pci_ioda_table_free_pages(tbl);
 	iommu_free_table(tbl, "pnv");
 }
@@ -2448,7 +2448,7 @@ static __be64 *pnv_pci_ioda2_table_do_alloc_pages(int nid, unsigned shift,
 	return addr;
 }

-static void pnv_pci_ioda2_table_do_free_pages(__be64 *addr,
+static void pnv_pci_ioda_table_do_free_pages(__be64 *addr,
 		unsigned long size, unsigned level);

 static long pnv_pci_ioda2_table_alloc_pages(int nid, __u64 bus_offset,
@@ -2487,7 +2487,7 @@ static long pnv_pci_ioda2_table_alloc_pages(int nid, __u64 bus_offset,
 	 * release partially allocated table.
 	 */
 	if (offset < tce_table_size) {
-		pnv_pci_ioda2_table_do_free_pages(addr,
+		pnv_pci_ioda_table_do_free_pages(addr,
 				1ULL << (level_shift - 3), levels - 1);
 		return -ENOMEM;
 	}
@@ -2505,7 +2505,7 @@ static long pnv_pci_ioda2_table_alloc_pages(int nid, __u64 bus_offset,
 	return 0;
 }

-static void pnv_pci_ioda2_table_do_free_pages(__be64 *addr,
+static void pnv_pci_ioda_table_do_free_pages(__be64 *addr,
 		unsigned long size, unsigned level)
 {
 	const unsigned long addr_ul = (unsigned long) addr &
@@ -2521,7 +2521,7 @@ static void pnv_pci_ioda2_table_do_free_pages(__be64 *addr,
 			if (!(hpa & (TCE_PCI_READ | TCE_PCI_WRITE)))
 				continue;

-			pnv_pci_ioda2_table_do_free_pages(__va(hpa), size,
+			pnv_pci_ioda_table_do_free_pages(__va(hpa), size,
 					level - 1);
 		}
 	}
@@ -2529,7 +2529,7 @@ static void pnv_pci_ioda2_table_do_free_pages(__be64 *addr,
 	free_pages(addr_ul, get_order(size << 3));
 }

-static void pnv_pci_ioda2_table_free_pages(struct iommu_table *tbl)
+static void pnv_pci_ioda_table_free_pages(struct iommu_table *tbl)
 {
 	const unsigned long size = tbl->it_indirect_levels ?
 			tbl->it_level_size : tbl->it_size;
@@ -2537,7 +2537,7 @@ static void pnv_pci_ioda2_table_free_pages(struct iommu_table *tbl)
 	if (!tbl->it_size)
 		return;

-	pnv_pci_ioda2_table_do_free_pages((__be64 *)tbl->it_base, size,
+	pnv_pci_ioda_table_do_free_pages((__be64 *)tbl->it_base, size,
 			tbl->it_indirect_levels);
 }

-- 
Alexey

Re: [PATCH v8 24/45] powerpc/pci: Rename pcibios_{add,remove}_pci_devices()

From: Gavin Shan <hidden>
Date: 2016-04-20 01:23:04

On Tue, Apr 19, 2016 at 03:28:36PM +1000, Alexey Kardashevskiy wrote:
On 02/17/2016 02:44 PM, Gavin Shan wrote:
quoted
This renames pcibios_{add,remove}_pci_devices() to avoid conflicts
with names of the weak functions in PCI subsystem, which have the
prefix "pcibios". No logical changes introduced.

Signed-off-by: Gavin Shan <redacted>
---
 arch/powerpc/include/asm/pci-bridge.h |  4 ++--
 arch/powerpc/kernel/eeh_driver.c      | 12 ++++++------
 arch/powerpc/kernel/pci-hotplug.c     | 15 +++++++--------
 drivers/pci/hotplug/rpadlpar_core.c   |  2 +-
 drivers/pci/hotplug/rpaphp_core.c     |  4 ++--
 drivers/pci/hotplug/rpaphp_pci.c      |  2 +-
 6 files changed, 19 insertions(+), 20 deletions(-)
diff --git a/arch/powerpc/include/asm/pci-bridge.h b/arch/powerpc/include/asm/pci-bridge.h
index 4dd6ef4..c817f38 100644
--- a/arch/powerpc/include/asm/pci-bridge.h
+++ b/arch/powerpc/include/asm/pci-bridge.h
@@ -263,10 +263,10 @@ static inline struct eeh_dev *pdn_to_eeh_dev(struct pci_dn *pdn)
 extern struct pci_bus *pcibios_find_pci_bus(struct device_node *dn);

 /** Remove all of the PCI devices under this bus */
-extern void pcibios_remove_pci_devices(struct pci_bus *bus);
+extern void pci_remove_pci_devices(struct pci_bus *bus);

pci_lala_pci_lala() ("pci" is used twice) looks weird, if the prefix is
"pci", what other device types can they handle?...

May be pcihp_add_devices(), pcihp_remove_devices() as these as defined in
pci-hotplug.c?
I assume you're talking about drivers/pci/hotplug/pci_hotplug_core.c.
pci_hotplug_core.c uses pci_hp_ prefix rather than pcihp_. I will
rename them to pci_hp_*() in next revision.

gwshan@gwshan:~/sandbox/linux$ find . -name pci-hotplug.c
./arch/powerpc/kernel/pci-hotplug.c
gwshan@gwshan:~/sandbox/linux$ grep pci*hp arch/powerpc/kernel/pci-hotplug.c 
quoted
 /** Discover new pci devices under this bus, and add them */
-extern void pcibios_add_pci_devices(struct pci_bus *bus);
+extern void pci_add_pci_devices(struct pci_bus *bus);


 extern void isa_bridge_find_early(struct pci_controller *hose);
diff --git a/arch/powerpc/kernel/eeh_driver.c b/arch/powerpc/kernel/eeh_driver.c
index fb6207d..59e53fe 100644
--- a/arch/powerpc/kernel/eeh_driver.c
+++ b/arch/powerpc/kernel/eeh_driver.c
@@ -621,7 +621,7 @@ static int eeh_reset_device(struct eeh_pe *pe, struct pci_bus *bus,
 	 * We don't remove the corresponding PE instances because
 	 * we need the information afterwords. The attached EEH
 	 * devices are expected to be attached soon when calling
-	 * into pcibios_add_pci_devices().
+	 * into pci_add_pci_devices().
 	 */
 	eeh_pe_state_mark(pe, EEH_PE_KEEP);
 	if (bus) {
@@ -630,7 +630,7 @@ static int eeh_reset_device(struct eeh_pe *pe, struct pci_bus *bus,
 		} else {
 			eeh_pe_state_clear(pe, EEH_PE_PRI_BUS);
 			pci_lock_rescan_remove();
-			pcibios_remove_pci_devices(bus);
+			pci_remove_pci_devices(bus);
 			pci_unlock_rescan_remove();
 		}
 	} else if (frozen_bus) {
@@ -681,7 +681,7 @@ static int eeh_reset_device(struct eeh_pe *pe, struct pci_bus *bus,
 		if (pe->type & EEH_PE_VF)
 			eeh_add_virt_device(edev, NULL);
 		else
-			pcibios_add_pci_devices(bus);
+			pci_add_pci_devices(bus);
 	} else if (frozen_bus && rmv_data->removed) {
 		pr_info("EEH: Sleep 5s ahead of partial hotplug\n");
 		ssleep(5);
@@ -691,7 +691,7 @@ static int eeh_reset_device(struct eeh_pe *pe, struct pci_bus *bus,
 		if (pe->type & EEH_PE_VF)
 			eeh_add_virt_device(edev, NULL);
 		else
-			pcibios_add_pci_devices(frozen_bus);
+			pci_add_pci_devices(frozen_bus);
 	}
 	eeh_pe_state_clear(pe, EEH_PE_KEEP);
@@ -896,7 +896,7 @@ perm_error:
 			eeh_pe_dev_mode_mark(pe, EEH_DEV_REMOVED);

 			pci_lock_rescan_remove();
-			pcibios_remove_pci_devices(frozen_bus);
+			pci_remove_pci_devices(frozen_bus);
 			pci_unlock_rescan_remove();
 		}
 	}
@@ -981,7 +981,7 @@ static void eeh_handle_special_event(void)
 				bus = eeh_pe_bus_get(phb_pe);
 				eeh_pe_dev_traverse(pe,
 					eeh_report_failure, NULL);
-				pcibios_remove_pci_devices(bus);
+				pci_remove_pci_devices(bus);
 			}
 			pci_unlock_rescan_remove();
 		}
diff --git a/arch/powerpc/kernel/pci-hotplug.c b/arch/powerpc/kernel/pci-hotplug.c
index 59c4361..78bf2a1 100644
--- a/arch/powerpc/kernel/pci-hotplug.c
+++ b/arch/powerpc/kernel/pci-hotplug.c
@@ -38,20 +38,20 @@ void pcibios_release_device(struct pci_dev *dev)
 }

 /**
- * pcibios_remove_pci_devices - remove all devices under this bus
+ * pci_remove_pci_devices - remove all devices under this bus
  * @bus: the indicated PCI bus
  *
  * Remove all of the PCI devices under this bus both from the
  * linux pci device tree, and from the powerpc EEH address cache.
  */
-void pcibios_remove_pci_devices(struct pci_bus *bus)
+void pci_remove_pci_devices(struct pci_bus *bus)
 {
 	struct pci_dev *dev, *tmp;
 	struct pci_bus *child_bus;

 	/* First go down child busses */
 	list_for_each_entry(child_bus, &bus->children, node)
-		pcibios_remove_pci_devices(child_bus);
+		pci_remove_pci_devices(child_bus);

 	pr_debug("PCI: Removing devices on bus %04x:%02x\n",
 		 pci_domain_nr(bus),  bus->number);
@@ -60,11 +60,10 @@ void pcibios_remove_pci_devices(struct pci_bus *bus)
 		pci_stop_and_remove_bus_device(dev);
 	}
 }
-
-EXPORT_SYMBOL_GPL(pcibios_remove_pci_devices);
+EXPORT_SYMBOL_GPL(pci_remove_pci_devices);

 /**
- * pcibios_add_pci_devices - adds new pci devices to bus
+ * pci_add_pci_devices - adds new pci devices to bus
  * @bus: the indicated PCI bus
  *
  * This routine will find and fixup new pci devices under
@@ -74,7 +73,7 @@ EXPORT_SYMBOL_GPL(pcibios_remove_pci_devices);
  * is how this routine differs from other, similar pcibios
  * routines.)
  */
-void pcibios_add_pci_devices(struct pci_bus * bus)
+void pci_add_pci_devices(struct pci_bus *bus)
 {
 	int slotno, mode, pass, max;
 	struct pci_dev *dev;
@@ -114,4 +113,4 @@ void pcibios_add_pci_devices(struct pci_bus * bus)
 	}
 	pcibios_finish_adding_to_bus(bus);
 }
-EXPORT_SYMBOL_GPL(pcibios_add_pci_devices);
+EXPORT_SYMBOL_GPL(pci_add_pci_devices);
diff --git a/drivers/pci/hotplug/rpadlpar_core.c b/drivers/pci/hotplug/rpadlpar_core.c
index b46b57d..730982b 100644
--- a/drivers/pci/hotplug/rpadlpar_core.c
+++ b/drivers/pci/hotplug/rpadlpar_core.c
@@ -380,7 +380,7 @@ int dlpar_remove_pci_slot(char *drc_name, struct device_node *dn)
 	}

 	/* Remove all devices below slot */
-	pcibios_remove_pci_devices(bus);
+	pci_remove_pci_devices(bus);

 	/* Unmap PCI IO space */
 	if (pcibios_unmap_io_space(bus)) {
diff --git a/drivers/pci/hotplug/rpaphp_core.c b/drivers/pci/hotplug/rpaphp_core.c
index 611f605..bba07b3 100644
--- a/drivers/pci/hotplug/rpaphp_core.c
+++ b/drivers/pci/hotplug/rpaphp_core.c
@@ -404,7 +404,7 @@ static int enable_slot(struct hotplug_slot *hotplug_slot)

 	if (state == PRESENT) {
 		pci_lock_rescan_remove();
-		pcibios_add_pci_devices(slot->bus);
+		pci_add_pci_devices(slot->bus);
 		pci_unlock_rescan_remove();
 		slot->state = CONFIGURED;
 	} else if (state == EMPTY) {
@@ -426,7 +426,7 @@ static int disable_slot(struct hotplug_slot *hotplug_slot)
 		return -EINVAL;

 	pci_lock_rescan_remove();
-	pcibios_remove_pci_devices(slot->bus);
+	pci_remove_pci_devices(slot->bus);
 	pci_unlock_rescan_remove();
 	vm_unmap_aliases();
diff --git a/drivers/pci/hotplug/rpaphp_pci.c b/drivers/pci/hotplug/rpaphp_pci.c
index 7836d69..1099b38 100644
--- a/drivers/pci/hotplug/rpaphp_pci.c
+++ b/drivers/pci/hotplug/rpaphp_pci.c
@@ -116,7 +116,7 @@ int rpaphp_enable_slot(struct slot *slot)
 		}

 		if (list_empty(&bus->devices))
-			pcibios_add_pci_devices(bus);
+			pci_add_pci_devices(bus);

 		if (!list_empty(&bus->devices)) {
 			info->adapter_status = CONFIGURED;

-- 
Alexey

Re: [PATCH v8 28/45] powerpc/pci: Introduce pci_remove_device_node_info()

From: Gavin Shan <hidden>
Date: 2016-04-20 01:25:23

On Tue, Apr 19, 2016 at 03:48:26PM +1000, Alexey Kardashevskiy wrote:
On 02/17/2016 02:44 PM, Gavin Shan wrote:
quoted
This implements and exports pci_remove_device_node_info(). It's
used to remove the pdn (struct pci_dn) for the indicated device
node. The function is going to be used by PowerNV PCI hotplug
driver.

Signed-off-by: Gavin Shan <redacted>
Kind of strange that there is no such helper for pseries, is there?
I don't find one actually. If you find one, pls let me know, thanks!
Reviewed-by: Alexey Kardashevskiy <redacted>

quoted
---
 arch/powerpc/include/asm/pci-bridge.h |  1 +
 arch/powerpc/kernel/pci_dn.c          | 23 +++++++++++++++++++++++
 2 files changed, 24 insertions(+)
diff --git a/arch/powerpc/include/asm/pci-bridge.h b/arch/powerpc/include/asm/pci-bridge.h
index 72a9d4e..c6310e2 100644
--- a/arch/powerpc/include/asm/pci-bridge.h
+++ b/arch/powerpc/include/asm/pci-bridge.h
@@ -240,6 +240,7 @@ extern struct pci_dn *add_dev_pci_data(struct pci_dev *pdev);
 extern void remove_dev_pci_data(struct pci_dev *pdev);
 extern struct pci_dn *pci_add_device_node_info(struct pci_controller *hose,
 					       struct device_node *dn);
+extern void pci_remove_device_node_info(struct device_node *dn);

 static inline int pci_device_from_OF_node(struct device_node *np,
 					  u8 *bus, u8 *devfn)
diff --git a/arch/powerpc/kernel/pci_dn.c b/arch/powerpc/kernel/pci_dn.c
index 0a249ff..ce10281 100644
--- a/arch/powerpc/kernel/pci_dn.c
+++ b/arch/powerpc/kernel/pci_dn.c
@@ -331,6 +331,29 @@ struct pci_dn *pci_add_device_node_info(struct pci_controller *hose,
 }
 EXPORT_SYMBOL_GPL(pci_add_device_node_info);

+void pci_remove_device_node_info(struct device_node *dn)
+{
+	struct pci_dn *pdn = dn ? PCI_DN(dn) : NULL;
+#ifdef CONFIG_EEH
+	struct eeh_dev *edev = pdn_to_eeh_dev(pdn);
+
+	if (edev)
+		edev->pdn = NULL;
+#endif
+
+	if (!pdn)
+		return;
+
+	WARN_ON(!list_empty(&pdn->child_list));
+	list_del(&pdn->list);
+	if (pdn->parent)
+		of_node_put(pdn->parent->node);
+
+	dn->data = NULL;
+	kfree(pdn);
+}
+EXPORT_SYMBOL_GPL(pci_remove_device_node_info);
+
 /*
  * Traverse a device tree stopping each PCI device in the tree.
  * This is done depth first.  As each node is processed, a "pre"

-- 
Alexey

Re: [PATCH v8 30/45] powerpc/pci: Delay populating pdn

From: Gavin Shan <hidden>
Date: 2016-04-20 02:13:18

On Tue, Apr 19, 2016 at 06:19:20PM +1000, Alexey Kardashevskiy wrote:
On 02/17/2016 02:44 PM, Gavin Shan wrote:
quoted
The pdn (struct pci_dn) instances are allocated from memblock or
bootmem when creating PCI controller (hoses) in setup_arch(). PCI
hotplug, which will be supported by proceeding patches, releases
PCI device nodes and their corresponding pdn on unplugging event.
The memory chunks for pdn instances allocated from memblock or
bootmem are hard to reused after being released.

This delays creating pdn by pci_devs_phb_init() from setup_arch()
to core_initcall() so that they are allocated from slab. The memory
consumed by pdn can be released to system without problem during
PCI unplugging time. It indicates that pci_dn is unavailable in
setup_arch() and the the fixup on pdn (like AGP's) can't be carried
out that time. We have to do that in ppc_md.pcibios_root_bridge_prepare()
on maple/pasemi/powermac platforms where/when the pdn is available.

At the mean while, the EEH device is created when pdn is populated,
meaning pdn and EEH device have same life cycle. In turn, we needn't
call eeh_dev_init() to create EEH device explicitly.

Signed-off-by: Gavin Shan <redacted>

Uff. It would not hurt to mention that  pcibios_root_bridge_prepare is called
from subsys_initcall() which is executed after core_initcall() so the code
flow does not change.
Yes, will do in next revision.
Have you checked if there is anything in between
core_initcall(pci_devs_phb_init) and subsys_initcall(pcibios_init) which
might need device tree nodes? For example, subsys_initcall(pcibios_init)
calls (eventually) pnv_pci_ioda_fixup(), if we are unlucky and pcibios_init()
(and therefore pnv_pci_ioda_fixup() or what pseries/others do) is called
before pcibios_init() - won't we crash or something?
I don't catch what you were asking. device-tree nodes (struct device_node)
are always there. This patch doesn't affect them. Perhaps you were talking
about pdn (PCI_DN). If it's the case, this patch delays creating pdn from
setup_arch() to core_initcall(pci_devs_phb_init). I don't see anything need
pdn between setup_arch() and core_initcall().

The changes introduced to powermac/pasemi platforms are: move fixing the child
pdns of the specifiec PHB's pdn from setup_arch() to subsys_initcall(pcibios_init).
I don't see anything between them needs the fixed pdns.

I don't understand how pcibios_init() is called before pcibios_init() in your
context. Sorry for my bad English. Perhaps you're asking the the called sequence
on core_initcall() and subsys_init()? If so, they're defined like below:

#define core_initcall(fn)		__define_initcall(fn, 1)
#define subsys_initcall(fn)		__define_initcall(fn, 4)
-- 
Alexey

Re: [PATCH v8 37/45] powerpc/powernv: Use firmware PCI slot reset infrastructure

From: Gavin Shan <hidden>
Date: 2016-04-20 02:33:06

On Tue, Apr 19, 2016 at 07:34:55PM +1000, Alexey Kardashevskiy wrote:
On 02/17/2016 02:44 PM, Gavin Shan wrote:
quoted
The skiboot firmware might provide the PCI slot reset capability
which is identified by property "ibm,reset-by-firmware" on the
PCI slot associated device node.

This checks the property. If it exists, the reset request is routed
to firmware. Otherwise, the reset is done by kernel as before.

Signed-off-by: Gavin Shan <redacted>
---
 arch/powerpc/platforms/powernv/eeh-powernv.c | 41 +++++++++++++++++++++++++++-
 1 file changed, 40 insertions(+), 1 deletion(-)
diff --git a/arch/powerpc/platforms/powernv/eeh-powernv.c b/arch/powerpc/platforms/powernv/eeh-powernv.c
index e23b063..c8a5217 100644
--- a/arch/powerpc/platforms/powernv/eeh-powernv.c
+++ b/arch/powerpc/platforms/powernv/eeh-powernv.c
@@ -789,7 +789,7 @@ static int pnv_eeh_root_reset(struct pci_controller *hose, int option)
 	return ret;
 }

-static int pnv_eeh_bridge_reset(struct pci_dev *dev, int option)
+static int __pnv_eeh_bridge_reset(struct pci_dev *dev, int option)
 {
 	struct pci_dn *pdn = pci_get_pdn_by_devfn(dev->bus, dev->devfn);
 	struct eeh_dev *edev = pdn_to_eeh_dev(pdn);
@@ -840,6 +840,45 @@ static int pnv_eeh_bridge_reset(struct pci_dev *dev, int option)
 	return 0;
 }

+static int pnv_eeh_bridge_reset(struct pci_dev *pdev, int option)
+{
+	struct pci_controller *hose;
+	struct pnv_phb *phb;
+	struct device_node *dn = pdev ? pci_device_to_OF_node(pdev) : NULL;
+	uint64_t id = (0x1ul << 60);

What is this 1<<60 for?
As you replied in other threads, it's worthy to have some macros for this
piece of business. This bit indicates the ID of the slot behind a switch
port. If this bit is cleared, the ID represents a PHB slot.
quoted
+	uint8_t scope;
+	int64_t rc;
+
+	/*
+	 * If the firmware can't handle it, we will issue hot reset
+	 * on the secondary bus despite the requested reset type.
+	 */
+	if (!dn || !of_get_property(dn, "ibm,reset-by-firmware", NULL))
+		return __pnv_eeh_bridge_reset(pdev, option);
+
+	/* The firmware can handle the request */
+	switch (option) {
+	case EEH_RESET_HOT:
+		scope = OPAL_RESET_PCI_HOT;
+		break;
+	case EEH_RESET_FUNDAMENTAL:
+		scope = OPAL_RESET_PCI_FUNDAMENTAL;
+		break;
+	case EEH_RESET_DEACTIVATE:
+		return 0;
+	default:
+		dev_warn(&pdev->dev, "%s: Unsupported reset %d\n",
+			 __func__, option);

Can the userspace trigger this case (via VFIO-EEH) and flood dmesg?
It depends on how you defined message flooding actually. It's abnormal
path caused by program internal error, not external users.
quoted
+		return -EINVAL;
+	}
+
+	hose = pci_bus_to_host(pdev->bus);
+	phb = hose->private_data;
+	id |= (pdev->bus->number << 24) | (pdev->devfn << 16) | phb->opal_id;
+	rc = opal_pci_reset(id, scope, OPAL_ASSERT_RESET);
+	return pnv_pci_poll(id, rc, NULL);
+}
+
 static int pnv_pci_dev_reset_type(struct pci_dev *pdev, void *data)
 {
 	int *freset = data;

-- 
Alexey

Re: [PATCH v8 39/45] powerpc/powernv: Select OF_DYNAMIC

From: Gavin Shan <hidden>
Date: 2016-04-20 02:38:04

On Tue, Apr 19, 2016 at 07:42:01PM +1000, Alexey Kardashevskiy wrote:
On 02/17/2016 02:44 PM, Gavin Shan wrote:
quoted
The device tree will change dynamically in PowerNV PCI hotplug
driver. This enables CONFIG_OF_DYNAMIC to support that.

Signed-off-by: Gavin Shan <redacted>
---
 arch/powerpc/platforms/powernv/Kconfig | 1 +
 1 file changed, 1 insertion(+)
diff --git a/arch/powerpc/platforms/powernv/Kconfig b/arch/powerpc/platforms/powernv/Kconfig
index 604190c..e7b1ad7 100644
--- a/arch/powerpc/platforms/powernv/Kconfig
+++ b/arch/powerpc/platforms/powernv/Kconfig
@@ -18,6 +18,7 @@ config PPC_POWERNV
 	select CPU_FREQ_GOV_ONDEMAND
 	select CPU_FREQ_GOV_CONSERVATIVE
 	select PPC_DOORBELL
+	select OF_DYNAMIC

Why not to enable it in 45/45 under config HOTPLUG_PCI_POWERNV? Is there any
benefit of having it always on if HOTPLUG_PCI_POWERNV is not enabled?
Agree, I will move accordingly in next revision. Note that we have to move
it back here once something else depends on OF_DYNAMIC in future.
quoted
 	default y

 config OPAL_PRD

-- 
Alexey

Re: [PATCH v8 40/45] drivers/of: Split unflatten_dt_node()

From: Gavin Shan <hidden>
Date: 2016-04-20 02:38:49

On Wed, Feb 17, 2016 at 08:30:42AM -0600, Rob Herring wrote:
On Tue, Feb 16, 2016 at 9:44 PM, Gavin Shan [off-list ref] wrote:
quoted
The function unflatten_dt_node() is called recursively to unflatten
device nodes and properties in the FDT blob. It looks complicated
and hard to be understood.

This splits the function into 3 functions: populate_properties(),
populate_node() and unflatten_dt_node(). populate_properties(),
which is called by populate_node(), creates properties for the
indicated device node. The later one creates the device nodes
from FDT blob. populate_node() gets the offset in FDT blob for
next device nodes and then calls populate_node(). No logical
changes introduced.

Signed-off-by: Gavin Shan <redacted>
---
 drivers/of/fdt.c | 249 ++++++++++++++++++++++++++++++++-----------------------
 1 file changed, 147 insertions(+), 102 deletions(-)
One nit, otherwise:

Acked-by: Rob Herring <robh@kernel.org>

[...]
quoted
+               /* And we process the "ibm,phandle" property
+                * used in pSeries dynamic device tree
+                * stuff
+                */
+               if (!strcmp(pname, "ibm,phandle"))
+                       np->phandle = be32_to_cpup(val);
+
+               pp->name   = (char *)pname;
+               pp->length = sz;
+               pp->value  = (__be32 *)val;
This cast should not be needed.
Rob, very sorry to response so lately. I will fix it up in next revision.
quoted
+               *pprev     = pp;
+               pprev      = &pp->next;
+       }

Re: [PATCH v8 00/45] powerpc/powernv: PCI hotplug support

From: Gavin Shan <hidden>
Date: 2016-04-20 02:40:58

On Fri, Apr 15, 2016 at 11:10:21AM -0500, Rob Herring wrote:
On Wed, Apr 13, 2016 at 8:30 PM, Gavin Shan [off-list ref] wrote:
quoted
On Thu, Apr 14, 2016 at 09:57:32AM +1000, Alistair Popple wrote:
quoted
Hi Gavin,

<snip>
quoted
quoted
Why exactly cannot EEH reset changes go to a smaller separate patchset
(before hotplug)?
As I explained before, the patchset's order is: PCI generic part,
PowerNV PCI related, EEH related, device-tree part and hotplug driver.

The EEH reset change is included in PATCH[37/45]. There is no point
to reorder the patches.
I don't understand all of the dependencies but if possible splitting the
series up into a set of smaller self-contained patch series makes things
easier to review and may make it easier for you to get this functionality
reviewed and accepted into upstream.
Thanks, Alistair. I will move those cleanup/refactor related patches
to form a separate series which is expected to be merged first. That
will helps the reviewers to focus on the patches with complicated
changes as you suggested. Alexey, please let me know if that way is
you like to see or not.
As I said last cycle, I'll happily take the DT refactoring patches
separately, but you have to tell me if you want me to apply them and
it has to be well before the merge window.
Thanks, Rob. I hope to post next revision (v9) soon and the device-tree
related cleanup patches should be ready for next merge window in it.
Rob

Re: [PATCH v8 21/45] powerpc/powernv: Create PEs at PCI hot plugging time

From: Alexey Kardashevskiy <hidden>
Date: 2016-04-20 03:00:38

On 04/20/2016 11:12 AM, Gavin Shan wrote:
On Tue, Apr 19, 2016 at 02:16:42PM +1000, Alexey Kardashevskiy wrote:
quoted
On 02/17/2016 02:44 PM, Gavin Shan wrote:
quoted
Currently, the PEs and their associated resources are assigned
in ppc_md.pcibios_fixup() except those used by SRIOV VFs.
But this new code does not affect IOV and VF's PEs will still be created
somewhere else rather than pnv_pci_setup_bridge()?
Correct. VF PEs cannot be created in pnv_pci_setup_bridge() as the PF's
IOV capability isn't enabled at that point.
quoted
quoted
The
function is called for once after PCI probing and resources
assignment is completed. So it isn't hotplug friendly.

This creates PEs dynamically by ppc_md.pcibios_setup_bridge(), which
is called on the event during system bootup and PCI hotplug: updating
PCI bridge's windows after resource assignment/reassignment are done.
For partial hotplug case, where not all PCI devices belonging to the
PE are unplugged and plugged again, we just need unbinding/binding
the affected PCI devices with the corresponding PE without creating
new one.

As there is no upstream bridge for root bus that needs to be covered
by PE, we have to create PE for root bus in ppc_md.pcibios_setup_bridge()
before any other PEs can be created, as PE for root bus is the ancestor
to anyone else.
We did not need a root bus PE before? What is the other PE reserved for?
Comments only say "reserved"...
No, A PE for root bus is needed before.
Ok. We needed a PE for the root bus and we need it now. What changed? Why 
do you reserve another PE?

other PEs can be for the PCI bus
originated from root port and the subordinate domains.
quoted
quoted
Also, the windows of root port or the upstream port of PCIe switch behind
root port are extended to be PHB's apertures to accommodate the additional
resources needed by newly plugged devices based on the fact: hotpluggable
slot is behind root port or downstream port of the PCIe switch behind
root port. The extension for those PCI brdiges' windows is done in
ppc_md.pcibios_setup_bridge() as well.

This patch seems to be doing way too many things, hard to follow.

Could you please split the patch into smaller chunks? For example (you can do
it totally different):
- move pnv_pci_ioda_setup_opal_tce_kill()
- move PE creation from pnv_pci_ioda_fixup() to pnv_pci_setup_bridge();
- add pnv_pci_fixup_bridge_resources()
- add an extra reserved PE for the root bus (and all this magic with
root_pe_idx/root_pe_populated)
- ...
I'll evaluate it later. It's always nice to have small patches. Thanks
for the comments.
quoted


--
Alexey
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Alexey

Re: [PATCH v8 22/45] powerpc/powernv/ioda1: Support releasing IODA1 TCE table

From: Alexey Kardashevskiy <hidden>
Date: 2016-04-20 03:17:01

On 04/20/2016 11:15 AM, Gavin Shan wrote:
On Tue, Apr 19, 2016 at 02:28:51PM +1000, Alexey Kardashevskiy wrote:
quoted
On 02/17/2016 02:44 PM, Gavin Shan wrote:
quoted
pnv_pci_ioda_table_free_pages() can be reused to release the IODA1
TCE table when releasing IODA1 PE in subsequent patches.

This renames the following functions to support releasing IODA1 TCE
table: pnv_pci_ioda2_table_free_pages() to pnv_pci_ioda_table_free_pages(),
pnv_pci_ioda2_table_do_free_pages() to pnv_pci_ioda_table_do_free_pages().
No logical changes introduced.
I can only see renaming here but it seems (from
IODA_architecture_04-14-2008.pdf) that IODA1 does not support multi-level TCE
tables in the way IODA2 does.
Note that the change was proposed by you in last round.
Hm. I do not recall proposing exactly that :-/
Yes, TVE on P7IOC
doesn't support multiple levels of TCE tables.
I thought it supports 2 levels.
In this case, we will always
have "tbl->it_indirect_levels" to 1, right?
Nope, it will be 0. But it is still ugly to use release function but not to 
use its allocating counterpart which is pnv_pci_ioda2_table_alloc_pages().

I suggest having pnv_pci_ioda1_table_free_pages() which will be just a 
single free_pages() call. If you need some ioda*-common code to free a 
table, then define pnv_ioda1_iommu_ops::free().
quoted
quoted
Signed-off-by: Gavin Shan <redacted>
---
  arch/powerpc/platforms/powernv/pci-ioda.c | 18 +++++++++---------
  1 file changed, 9 insertions(+), 9 deletions(-)
diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index d360607..077f9db 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -51,7 +51,7 @@
  #define POWERNV_IOMMU_DEFAULT_LEVELS	1
  #define POWERNV_IOMMU_MAX_LEVELS	5

-static void pnv_pci_ioda2_table_free_pages(struct iommu_table *tbl);
+static void pnv_pci_ioda_table_free_pages(struct iommu_table *tbl);

  static void pe_level_printk(const struct pnv_ioda_pe *pe, const char *level,
  			    const char *fmt, ...)
@@ -1352,7 +1352,7 @@ static void pnv_pci_ioda2_release_dma_pe(struct pci_dev *dev, struct pnv_ioda_pe
  		iommu_group_put(pe->table_group.group);
  		BUG_ON(pe->table_group.group);
  	}
-	pnv_pci_ioda2_table_free_pages(tbl);
+	pnv_pci_ioda_table_free_pages(tbl);
  	iommu_free_table(tbl, of_node_full_name(dev->dev.of_node));
  }
@@ -1946,7 +1946,7 @@ static void pnv_ioda2_tce_free(struct iommu_table *tbl, long index,

  static void pnv_ioda2_table_free(struct iommu_table *tbl)
  {
-	pnv_pci_ioda2_table_free_pages(tbl);
+	pnv_pci_ioda_table_free_pages(tbl);
  	iommu_free_table(tbl, "pnv");
  }
@@ -2448,7 +2448,7 @@ static __be64 *pnv_pci_ioda2_table_do_alloc_pages(int nid, unsigned shift,
  	return addr;
  }

-static void pnv_pci_ioda2_table_do_free_pages(__be64 *addr,
+static void pnv_pci_ioda_table_do_free_pages(__be64 *addr,
  		unsigned long size, unsigned level);

  static long pnv_pci_ioda2_table_alloc_pages(int nid, __u64 bus_offset,
@@ -2487,7 +2487,7 @@ static long pnv_pci_ioda2_table_alloc_pages(int nid, __u64 bus_offset,
  	 * release partially allocated table.
  	 */
  	if (offset < tce_table_size) {
-		pnv_pci_ioda2_table_do_free_pages(addr,
+		pnv_pci_ioda_table_do_free_pages(addr,
  				1ULL << (level_shift - 3), levels - 1);
  		return -ENOMEM;
  	}
@@ -2505,7 +2505,7 @@ static long pnv_pci_ioda2_table_alloc_pages(int nid, __u64 bus_offset,
  	return 0;
  }

-static void pnv_pci_ioda2_table_do_free_pages(__be64 *addr,
+static void pnv_pci_ioda_table_do_free_pages(__be64 *addr,
  		unsigned long size, unsigned level)
  {
  	const unsigned long addr_ul = (unsigned long) addr &
@@ -2521,7 +2521,7 @@ static void pnv_pci_ioda2_table_do_free_pages(__be64 *addr,
  			if (!(hpa & (TCE_PCI_READ | TCE_PCI_WRITE)))
  				continue;

-			pnv_pci_ioda2_table_do_free_pages(__va(hpa), size,
+			pnv_pci_ioda_table_do_free_pages(__va(hpa), size,
  					level - 1);
  		}
  	}
@@ -2529,7 +2529,7 @@ static void pnv_pci_ioda2_table_do_free_pages(__be64 *addr,
  	free_pages(addr_ul, get_order(size << 3));
  }

-static void pnv_pci_ioda2_table_free_pages(struct iommu_table *tbl)
+static void pnv_pci_ioda_table_free_pages(struct iommu_table *tbl)
  {
  	const unsigned long size = tbl->it_indirect_levels ?
  			tbl->it_level_size : tbl->it_size;
@@ -2537,7 +2537,7 @@ static void pnv_pci_ioda2_table_free_pages(struct iommu_table *tbl)
  	if (!tbl->it_size)
  		return;

-	pnv_pci_ioda2_table_do_free_pages((__be64 *)tbl->it_base, size,
+	pnv_pci_ioda_table_do_free_pages((__be64 *)tbl->it_base, size,
  			tbl->it_indirect_levels);
  }

--
Alexey

-- 
Alexey

Re: [PATCH v8 24/45] powerpc/pci: Rename pcibios_{add,remove}_pci_devices()

From: Alexey Kardashevskiy <hidden>
Date: 2016-04-20 03:21:21

On 04/20/2016 11:23 AM, Gavin Shan wrote:
On Tue, Apr 19, 2016 at 03:28:36PM +1000, Alexey Kardashevskiy wrote:
quoted
On 02/17/2016 02:44 PM, Gavin Shan wrote:
quoted
This renames pcibios_{add,remove}_pci_devices() to avoid conflicts
with names of the weak functions in PCI subsystem, which have the
prefix "pcibios". No logical changes introduced.

Signed-off-by: Gavin Shan <redacted>
---
  arch/powerpc/include/asm/pci-bridge.h |  4 ++--
  arch/powerpc/kernel/eeh_driver.c      | 12 ++++++------
  arch/powerpc/kernel/pci-hotplug.c     | 15 +++++++--------
  drivers/pci/hotplug/rpadlpar_core.c   |  2 +-
  drivers/pci/hotplug/rpaphp_core.c     |  4 ++--
  drivers/pci/hotplug/rpaphp_pci.c      |  2 +-
  6 files changed, 19 insertions(+), 20 deletions(-)
diff --git a/arch/powerpc/include/asm/pci-bridge.h b/arch/powerpc/include/asm/pci-bridge.h
index 4dd6ef4..c817f38 100644
--- a/arch/powerpc/include/asm/pci-bridge.h
+++ b/arch/powerpc/include/asm/pci-bridge.h
@@ -263,10 +263,10 @@ static inline struct eeh_dev *pdn_to_eeh_dev(struct pci_dn *pdn)
  extern struct pci_bus *pcibios_find_pci_bus(struct device_node *dn);

  /** Remove all of the PCI devices under this bus */
-extern void pcibios_remove_pci_devices(struct pci_bus *bus);
+extern void pci_remove_pci_devices(struct pci_bus *bus);

pci_lala_pci_lala() ("pci" is used twice) looks weird, if the prefix is
"pci", what other device types can they handle?...

May be pcihp_add_devices(), pcihp_remove_devices() as these as defined in
pci-hotplug.c?
I assume you're talking about drivers/pci/hotplug/pci_hotplug_core.c.
No, the helpers you are renaming are in pci-hotplug.c which uses "pci_" as 
a prefix even though the file is supposed to be about hotplug.

pci_hotplug_core.c uses pci_hp_ prefix rather than pcihp_. I will
rename them to pci_hp_*() in next revision.
Anyway, this will work too.

gwshan@gwshan:~/sandbox/linux$ find . -name pci-hotplug.c
./arch/powerpc/kernel/pci-hotplug.c
gwshan@gwshan:~/sandbox/linux$ grep pci*hp arch/powerpc/kernel/pci-hotplug.c
quoted
quoted
  /** Discover new pci devices under this bus, and add them */
-extern void pcibios_add_pci_devices(struct pci_bus *bus);
+extern void pci_add_pci_devices(struct pci_bus *bus);


  extern void isa_bridge_find_early(struct pci_controller *hose);
diff --git a/arch/powerpc/kernel/eeh_driver.c b/arch/powerpc/kernel/eeh_driver.c
index fb6207d..59e53fe 100644
--- a/arch/powerpc/kernel/eeh_driver.c
+++ b/arch/powerpc/kernel/eeh_driver.c
@@ -621,7 +621,7 @@ static int eeh_reset_device(struct eeh_pe *pe, struct pci_bus *bus,
  	 * We don't remove the corresponding PE instances because
  	 * we need the information afterwords. The attached EEH
  	 * devices are expected to be attached soon when calling
-	 * into pcibios_add_pci_devices().
+	 * into pci_add_pci_devices().
  	 */
  	eeh_pe_state_mark(pe, EEH_PE_KEEP);
  	if (bus) {
@@ -630,7 +630,7 @@ static int eeh_reset_device(struct eeh_pe *pe, struct pci_bus *bus,
  		} else {
  			eeh_pe_state_clear(pe, EEH_PE_PRI_BUS);
  			pci_lock_rescan_remove();
-			pcibios_remove_pci_devices(bus);
+			pci_remove_pci_devices(bus);
  			pci_unlock_rescan_remove();
  		}
  	} else if (frozen_bus) {
@@ -681,7 +681,7 @@ static int eeh_reset_device(struct eeh_pe *pe, struct pci_bus *bus,
  		if (pe->type & EEH_PE_VF)
  			eeh_add_virt_device(edev, NULL);
  		else
-			pcibios_add_pci_devices(bus);
+			pci_add_pci_devices(bus);
  	} else if (frozen_bus && rmv_data->removed) {
  		pr_info("EEH: Sleep 5s ahead of partial hotplug\n");
  		ssleep(5);
@@ -691,7 +691,7 @@ static int eeh_reset_device(struct eeh_pe *pe, struct pci_bus *bus,
  		if (pe->type & EEH_PE_VF)
  			eeh_add_virt_device(edev, NULL);
  		else
-			pcibios_add_pci_devices(frozen_bus);
+			pci_add_pci_devices(frozen_bus);
  	}
  	eeh_pe_state_clear(pe, EEH_PE_KEEP);
@@ -896,7 +896,7 @@ perm_error:
  			eeh_pe_dev_mode_mark(pe, EEH_DEV_REMOVED);

  			pci_lock_rescan_remove();
-			pcibios_remove_pci_devices(frozen_bus);
+			pci_remove_pci_devices(frozen_bus);
  			pci_unlock_rescan_remove();
  		}
  	}
@@ -981,7 +981,7 @@ static void eeh_handle_special_event(void)
  				bus = eeh_pe_bus_get(phb_pe);
  				eeh_pe_dev_traverse(pe,
  					eeh_report_failure, NULL);
-				pcibios_remove_pci_devices(bus);
+				pci_remove_pci_devices(bus);
  			}
  			pci_unlock_rescan_remove();
  		}
diff --git a/arch/powerpc/kernel/pci-hotplug.c b/arch/powerpc/kernel/pci-hotplug.c
index 59c4361..78bf2a1 100644
--- a/arch/powerpc/kernel/pci-hotplug.c
+++ b/arch/powerpc/kernel/pci-hotplug.c
@@ -38,20 +38,20 @@ void pcibios_release_device(struct pci_dev *dev)
  }

  /**
- * pcibios_remove_pci_devices - remove all devices under this bus
+ * pci_remove_pci_devices - remove all devices under this bus
   * @bus: the indicated PCI bus
   *
   * Remove all of the PCI devices under this bus both from the
   * linux pci device tree, and from the powerpc EEH address cache.
   */
-void pcibios_remove_pci_devices(struct pci_bus *bus)
+void pci_remove_pci_devices(struct pci_bus *bus)
  {
  	struct pci_dev *dev, *tmp;
  	struct pci_bus *child_bus;

  	/* First go down child busses */
  	list_for_each_entry(child_bus, &bus->children, node)
-		pcibios_remove_pci_devices(child_bus);
+		pci_remove_pci_devices(child_bus);

  	pr_debug("PCI: Removing devices on bus %04x:%02x\n",
  		 pci_domain_nr(bus),  bus->number);
@@ -60,11 +60,10 @@ void pcibios_remove_pci_devices(struct pci_bus *bus)
  		pci_stop_and_remove_bus_device(dev);
  	}
  }
-
-EXPORT_SYMBOL_GPL(pcibios_remove_pci_devices);
+EXPORT_SYMBOL_GPL(pci_remove_pci_devices);

  /**
- * pcibios_add_pci_devices - adds new pci devices to bus
+ * pci_add_pci_devices - adds new pci devices to bus
   * @bus: the indicated PCI bus
   *
   * This routine will find and fixup new pci devices under
@@ -74,7 +73,7 @@ EXPORT_SYMBOL_GPL(pcibios_remove_pci_devices);
   * is how this routine differs from other, similar pcibios
   * routines.)
   */
-void pcibios_add_pci_devices(struct pci_bus * bus)
+void pci_add_pci_devices(struct pci_bus *bus)
  {
  	int slotno, mode, pass, max;
  	struct pci_dev *dev;
@@ -114,4 +113,4 @@ void pcibios_add_pci_devices(struct pci_bus * bus)
  	}
  	pcibios_finish_adding_to_bus(bus);
  }
-EXPORT_SYMBOL_GPL(pcibios_add_pci_devices);
+EXPORT_SYMBOL_GPL(pci_add_pci_devices);
diff --git a/drivers/pci/hotplug/rpadlpar_core.c b/drivers/pci/hotplug/rpadlpar_core.c
index b46b57d..730982b 100644
--- a/drivers/pci/hotplug/rpadlpar_core.c
+++ b/drivers/pci/hotplug/rpadlpar_core.c
@@ -380,7 +380,7 @@ int dlpar_remove_pci_slot(char *drc_name, struct device_node *dn)
  	}

  	/* Remove all devices below slot */
-	pcibios_remove_pci_devices(bus);
+	pci_remove_pci_devices(bus);

  	/* Unmap PCI IO space */
  	if (pcibios_unmap_io_space(bus)) {
diff --git a/drivers/pci/hotplug/rpaphp_core.c b/drivers/pci/hotplug/rpaphp_core.c
index 611f605..bba07b3 100644
--- a/drivers/pci/hotplug/rpaphp_core.c
+++ b/drivers/pci/hotplug/rpaphp_core.c
@@ -404,7 +404,7 @@ static int enable_slot(struct hotplug_slot *hotplug_slot)

  	if (state == PRESENT) {
  		pci_lock_rescan_remove();
-		pcibios_add_pci_devices(slot->bus);
+		pci_add_pci_devices(slot->bus);
  		pci_unlock_rescan_remove();
  		slot->state = CONFIGURED;
  	} else if (state == EMPTY) {
@@ -426,7 +426,7 @@ static int disable_slot(struct hotplug_slot *hotplug_slot)
  		return -EINVAL;

  	pci_lock_rescan_remove();
-	pcibios_remove_pci_devices(slot->bus);
+	pci_remove_pci_devices(slot->bus);
  	pci_unlock_rescan_remove();
  	vm_unmap_aliases();
diff --git a/drivers/pci/hotplug/rpaphp_pci.c b/drivers/pci/hotplug/rpaphp_pci.c
index 7836d69..1099b38 100644
--- a/drivers/pci/hotplug/rpaphp_pci.c
+++ b/drivers/pci/hotplug/rpaphp_pci.c
@@ -116,7 +116,7 @@ int rpaphp_enable_slot(struct slot *slot)
  		}

  		if (list_empty(&bus->devices))
-			pcibios_add_pci_devices(bus);
+			pci_add_pci_devices(bus);

  		if (!list_empty(&bus->devices)) {
  			info->adapter_status = CONFIGURED;

--
Alexey
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Alexey

Re: [PATCH v8 21/45] powerpc/powernv: Create PEs at PCI hot plugging time

From: Gavin Shan <hidden>
Date: 2016-04-20 03:35:11

On Wed, Apr 20, 2016 at 01:00:38PM +1000, Alexey Kardashevskiy wrote:
On 04/20/2016 11:12 AM, Gavin Shan wrote:
quoted
On Tue, Apr 19, 2016 at 02:16:42PM +1000, Alexey Kardashevskiy wrote:
quoted
On 02/17/2016 02:44 PM, Gavin Shan wrote:
quoted
Currently, the PEs and their associated resources are assigned
in ppc_md.pcibios_fixup() except those used by SRIOV VFs.
But this new code does not affect IOV and VF's PEs will still be created
somewhere else rather than pnv_pci_setup_bridge()?
Correct. VF PEs cannot be created in pnv_pci_setup_bridge() as the PF's
IOV capability isn't enabled at that point.
quoted
quoted
The
function is called for once after PCI probing and resources
assignment is completed. So it isn't hotplug friendly.

This creates PEs dynamically by ppc_md.pcibios_setup_bridge(), which
is called on the event during system bootup and PCI hotplug: updating
PCI bridge's windows after resource assignment/reassignment are done.
For partial hotplug case, where not all PCI devices belonging to the
PE are unplugged and plugged again, we just need unbinding/binding
the affected PCI devices with the corresponding PE without creating
new one.

As there is no upstream bridge for root bus that needs to be covered
by PE, we have to create PE for root bus in ppc_md.pcibios_setup_bridge()
before any other PEs can be created, as PE for root bus is the ancestor
to anyone else.
We did not need a root bus PE before? What is the other PE reserved for?
Comments only say "reserved"...
No, A PE for root bus is needed before.
Ok. We needed a PE for the root bus and we need it now. What changed? Why do
you reserve another PE?
Originally, all PEs (include the one for root bus) were created at PHB fixup time
in pnv_pci_ioda_fixup(). With this patch, all PEs are created in pnv_pci_setup_bridge().
pnv_pci_setup_bridge() is called for every PCI buses other than root bus. It means
pnv_pci_setup_bridge() isn't called for root bus. So we have to create PE for root
bus before the left PEs are created there. The PE# for root bus is reserved in advance
and used in pnv_pci_setup_bridge() at that point.
quoted
other PEs can be for the PCI bus
quoted
originated from root port and the subordinate domains.
quoted
quoted
Also, the windows of root port or the upstream port of PCIe switch behind
root port are extended to be PHB's apertures to accommodate the additional
resources needed by newly plugged devices based on the fact: hotpluggable
slot is behind root port or downstream port of the PCIe switch behind
root port. The extension for those PCI brdiges' windows is done in
ppc_md.pcibios_setup_bridge() as well.

This patch seems to be doing way too many things, hard to follow.

Could you please split the patch into smaller chunks? For example (you can do
it totally different):
- move pnv_pci_ioda_setup_opal_tce_kill()
- move PE creation from pnv_pci_ioda_fixup() to pnv_pci_setup_bridge();
- add pnv_pci_fixup_bridge_resources()
- add an extra reserved PE for the root bus (and all this magic with
root_pe_idx/root_pe_populated)
- ...
I'll evaluate it later. It's always nice to have small patches. Thanks
for the comments.
quoted


--
Alexey
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Alexey

Re: [PATCH v8 30/45] powerpc/pci: Delay populating pdn

From: Alexey Kardashevskiy <hidden>
Date: 2016-04-20 03:54:54

On 04/20/2016 12:13 PM, Gavin Shan wrote:
On Tue, Apr 19, 2016 at 06:19:20PM +1000, Alexey Kardashevskiy wrote:
quoted
On 02/17/2016 02:44 PM, Gavin Shan wrote:
quoted
The pdn (struct pci_dn) instances are allocated from memblock or
bootmem when creating PCI controller (hoses) in setup_arch(). PCI
hotplug, which will be supported by proceeding patches, releases
PCI device nodes and their corresponding pdn on unplugging event.
The memory chunks for pdn instances allocated from memblock or
bootmem are hard to reused after being released.

This delays creating pdn by pci_devs_phb_init() from setup_arch()
to core_initcall() so that they are allocated from slab. The memory
consumed by pdn can be released to system without problem during
PCI unplugging time. It indicates that pci_dn is unavailable in
setup_arch() and the the fixup on pdn (like AGP's) can't be carried
out that time. We have to do that in ppc_md.pcibios_root_bridge_prepare()
on maple/pasemi/powermac platforms where/when the pdn is available.

At the mean while, the EEH device is created when pdn is populated,
meaning pdn and EEH device have same life cycle. In turn, we needn't
call eeh_dev_init() to create EEH device explicitly.

Signed-off-by: Gavin Shan <redacted>

Uff. It would not hurt to mention that  pcibios_root_bridge_prepare is called
from subsys_initcall() which is executed after core_initcall() so the code
flow does not change.
Yes, will do in next revision.
quoted
Have you checked if there is anything in between
core_initcall(pci_devs_phb_init) and subsys_initcall(pcibios_init) which
might need device tree nodes? For example, subsys_initcall(pcibios_init)
calls (eventually) pnv_pci_ioda_fixup(), if we are unlucky and pcibios_init()
(and therefore pnv_pci_ioda_fixup() or what pseries/others do) is called
before pcibios_init() - won't we crash or something?
I don't catch what you were asking. device-tree nodes (struct device_node)
are always there. This patch doesn't affect them. Perhaps you were talking
about pdn (PCI_DN). If it's the case, this patch delays creating pdn from
setup_arch() to core_initcall(pci_devs_phb_init).

While thinking of explaining what I wanted to ask, I found my answer :)

pcibios_init() calls ppc_md.pcibios_root_bridge_prepare() first, then 
ppc_md.pcibios_fixup() so we are fine here with ordering.

I don't see anything need pdn between setup_arch() and core_initcall().
The changes introduced to powermac/pasemi platforms are: move fixing the child
pdns of the specifiec PHB's pdn from setup_arch() to subsys_initcall(pcibios_init).
I don't see anything between them needs the fixed pdns.

I don't understand how pcibios_init() is called before pcibios_init() in your
pcibios_init() is used twice in the sentence above :)

Anyway,


Reviewed-by: Alexey Kardashevskiy <redacted>



context. Sorry for my bad English. Perhaps you're asking the the called sequence
on core_initcall() and subsys_init()? If so, they're defined like below:

#define core_initcall(fn)		__define_initcall(fn, 1)
#define subsys_initcall(fn)		__define_initcall(fn, 4)
 >
quoted
--
Alexey
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Alexey

Re: [PATCH v8 37/45] powerpc/powernv: Use firmware PCI slot reset infrastructure

From: Alexey Kardashevskiy <hidden>
Date: 2016-04-20 04:17:07

On 04/20/2016 12:33 PM, Gavin Shan wrote:
On Tue, Apr 19, 2016 at 07:34:55PM +1000, Alexey Kardashevskiy wrote:
quoted
On 02/17/2016 02:44 PM, Gavin Shan wrote:
quoted
The skiboot firmware might provide the PCI slot reset capability
which is identified by property "ibm,reset-by-firmware" on the
PCI slot associated device node.

This checks the property. If it exists, the reset request is routed
to firmware. Otherwise, the reset is done by kernel as before.

Signed-off-by: Gavin Shan <redacted>
---
  arch/powerpc/platforms/powernv/eeh-powernv.c | 41 +++++++++++++++++++++++++++-
  1 file changed, 40 insertions(+), 1 deletion(-)
diff --git a/arch/powerpc/platforms/powernv/eeh-powernv.c b/arch/powerpc/platforms/powernv/eeh-powernv.c
index e23b063..c8a5217 100644
--- a/arch/powerpc/platforms/powernv/eeh-powernv.c
+++ b/arch/powerpc/platforms/powernv/eeh-powernv.c
@@ -789,7 +789,7 @@ static int pnv_eeh_root_reset(struct pci_controller *hose, int option)
  	return ret;
  }

-static int pnv_eeh_bridge_reset(struct pci_dev *dev, int option)
+static int __pnv_eeh_bridge_reset(struct pci_dev *dev, int option)
  {
  	struct pci_dn *pdn = pci_get_pdn_by_devfn(dev->bus, dev->devfn);
  	struct eeh_dev *edev = pdn_to_eeh_dev(pdn);
@@ -840,6 +840,45 @@ static int pnv_eeh_bridge_reset(struct pci_dev *dev, int option)
  	return 0;
  }

+static int pnv_eeh_bridge_reset(struct pci_dev *pdev, int option)
+{
+	struct pci_controller *hose;
+	struct pnv_phb *phb;
+	struct device_node *dn = pdev ? pci_device_to_OF_node(pdev) : NULL;
+	uint64_t id = (0x1ul << 60);

What is this 1<<60 for?
As you replied in other threads, it's worthy to have some macros for this
piece of business. This bit indicates the ID of the slot behind a switch
port. If this bit is cleared, the ID represents a PHB slot.
quoted
quoted
+	uint8_t scope;
+	int64_t rc;
+
+	/*
+	 * If the firmware can't handle it, we will issue hot reset
+	 * on the secondary bus despite the requested reset type.
+	 */
+	if (!dn || !of_get_property(dn, "ibm,reset-by-firmware", NULL))
+		return __pnv_eeh_bridge_reset(pdev, option);
+
+	/* The firmware can handle the request */
+	switch (option) {
+	case EEH_RESET_HOT:
+		scope = OPAL_RESET_PCI_HOT;
+		break;
+	case EEH_RESET_FUNDAMENTAL:
+		scope = OPAL_RESET_PCI_FUNDAMENTAL;
+		break;
+	case EEH_RESET_DEACTIVATE:
+		return 0;
+	default:
+		dev_warn(&pdev->dev, "%s: Unsupported reset %d\n",
+			 __func__, option);

Can the userspace trigger this case (via VFIO-EEH) and flood dmesg?
It depends on how you defined message flooding actually. It's abnormal
path caused by program internal error, not external users.

Can QEMU be changed to do something special (cause reset with a wrong 
option) via VFIO/EEH interface in a loop to make this message appear? Or 
the call with a wrong option will never reach this point?

quoted
quoted
+		return -EINVAL;
+	}
+
+	hose = pci_bus_to_host(pdev->bus);
+	phb = hose->private_data;
+	id |= (pdev->bus->number << 24) | (pdev->devfn << 16) | phb->opal_id;
+	rc = opal_pci_reset(id, scope, OPAL_ASSERT_RESET);
+	return pnv_pci_poll(id, rc, NULL);
+}
+
  static int pnv_pci_dev_reset_type(struct pci_dev *pdev, void *data)
  {
  	int *freset = data;

--
Alexey

-- 
Alexey

Re: [PATCH v8 40/45] drivers/of: Split unflatten_dt_node()

From: Gavin Shan <hidden>
Date: 2016-05-02 02:02:21

On Wed, Feb 17, 2016 at 08:30:42AM -0600, Rob Herring wrote:
On Tue, Feb 16, 2016 at 9:44 PM, Gavin Shan [off-list ref] wrote:
quoted
The function unflatten_dt_node() is called recursively to unflatten
device nodes and properties in the FDT blob. It looks complicated
and hard to be understood.

This splits the function into 3 functions: populate_properties(),
populate_node() and unflatten_dt_node(). populate_properties(),
which is called by populate_node(), creates properties for the
indicated device node. The later one creates the device nodes
from FDT blob. populate_node() gets the offset in FDT blob for
next device nodes and then calls populate_node(). No logical
changes introduced.

Signed-off-by: Gavin Shan <redacted>
---
 drivers/of/fdt.c | 249 ++++++++++++++++++++++++++++++++-----------------------
 1 file changed, 147 insertions(+), 102 deletions(-)
One nit, otherwise:

Acked-by: Rob Herring <robh@kernel.org>

[...]
quoted
+               /* And we process the "ibm,phandle" property
+                * used in pSeries dynamic device tree
+                * stuff
+                */
+               if (!strcmp(pname, "ibm,phandle"))
+                       np->phandle = be32_to_cpup(val);
+
+               pp->name   = (char *)pname;
+               pp->length = sz;
+               pp->value  = (__be32 *)val;
This cast should not be needed.
It's needed. Otherwise, we will have warning. So I will keep it. I just
went through this one for next revision and sorry for late response.

drivers/of/fdt.c:225:14: warning: assignment discards ‘const’ qualifier from pointer target type
   pp->value  = val;
              ^

Thanks,
Gavin
quoted
+               *pprev     = pp;
+               pprev      = &pp->next;
+       }

Previous page

Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help