[RFC 0/3] powerpc/powernv: support CAPI + kexec

Currently, if you attempt to kexec into a new kernel from a machine with a
CAPI card and the cxl driver loaded, you are going to have an exceedingly
bad time. It turns out that the hardware doesn't really cope very well with
going through the standard Linux PCI initialisation process while a PHB is
still in CAPI mode. Checkstops everywhere!

I've submitted RFC patches for skiboot[0][1] to disable CAPI mode when we
do a complete reset on a PHB. This series ensures that when we enter the
new kernel, we ask skiboot to do a complete reset on any PHBs that have
been left in CAPI mode before we initialise everything.

At this stage, I haven't thought too hard about getting to the point where
we can do this after Linux has booted for stuff like PCI hotplug...
triggering a creset after the Linux EEH handling is all set up can get
interesting.

This has only gotten the very lightest of testing - I've kexec-ed quite a
few times with no real problems, and I've run some basic CAPI tests that
don't seem to fail too much more than they normally fail. It does look like
we get spammed with a tonne of HMIs and frozen PE messages in the skiboot
log, not sure how bad this is.

Thanks to Vaibhav Jain (who made a previous attempt at this), Mikey
Neuling, Ben Herrenschmidt, Gavin Shan, Bill Daly and JT Kellington for
advice on various bits of this.

Andrew

[0] http://patchwork.ozlabs.org/patch/670781/
[1] http://patchwork.ozlabs.org/patch/670782/

Andrew Donnellan (3):
  powerpc/powernv: fix comment style and spelling
  powerpc/powernv: add opal_pci_get_phb_capi_mode() call
  powerpc/powernv: reset any PHBs in CAPI mode during initialisation

 arch/powerpc/include/asm/opal-api.h            |  3 ++-
 arch/powerpc/include/asm/opal.h                |  1 +
 arch/powerpc/platforms/powernv/opal-wrappers.S |  1 +
 arch/powerpc/platforms/powernv/pci-ioda.c      | 17 ++++++++++++++---
 4 files changed, 18 insertions(+), 4 deletions(-)

base-commit: 024c7e3756d8a42fc41fe8a9488488b9b09d1dcc
-- 
git-series 0.8.10

Signed-off-by: Andrew Donnellan <redacted>
---
 arch/powerpc/platforms/powernv/pci-ioda.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index c16d790..ca5e9b5 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c

@@ -3736,10 +3736,11 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
 	if (rc)
 		pr_warning("  OPAL Error %ld performing IODA table reset !\n", rc);
 
-	/* If we're running in kdump kerenl, the previous kerenl never
+	/*
+	 * If we're running in kdump kernel, the previous kernel never
 	 * shutdown PCI devices correctly. We already got IODA table
 	 * cleaned out. So we have to issue PHB reset to stop all PCI
-	 * transactions from previous kerenl.
+	 * transactions from previous kernel.
 	 */
 	if (is_kdump_kernel()) {
 		pr_info("  Issue PHB reset ...\n");

-- 
git-series 0.8.10

opal_pci_get_phb_capi_mode() returns OPAL_PHB_CAPI_MODE_CAPI if the PHB is
in CAPI mode, and OPAL_PHB_CAPI_MODE_PCIE if it isn't.

We're going to use this call to determine if a PHB requires a complete
reset during initialisation in order to disable CAPI mode (on sufficiently
new skiboots that support this).

Signed-off-by: Andrew Donnellan <redacted>

---

Corresponding skiboot RFC: http://patchwork.ozlabs.org/patch/670781/
---
 arch/powerpc/include/asm/opal-api.h            | 3 ++-
 arch/powerpc/include/asm/opal.h                | 1 +
 arch/powerpc/platforms/powernv/opal-wrappers.S | 1 +
 3 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/arch/powerpc/include/asm/opal-api.h b/arch/powerpc/include/asm/opal-api.h
index 0e2e57b..078ce77 100644
--- a/arch/powerpc/include/asm/opal-api.h
+++ b/arch/powerpc/include/asm/opal-api.h

@@ -167,7 +167,8 @@
 #define OPAL_INT_EOI				124
 #define OPAL_INT_SET_MFRR			125
 #define OPAL_PCI_TCE_KILL			126
-#define OPAL_LAST				126
+#define OPAL_PCI_GET_PHB_CAPI_MODE		128
+#define OPAL_LAST				128
 
 /* Device tree flags */
 

diff --git a/arch/powerpc/include/asm/opal.h b/arch/powerpc/include/asm/opal.h
index ee05bd2..501d32a 100644
--- a/arch/powerpc/include/asm/opal.h
+++ b/arch/powerpc/include/asm/opal.h

@@ -228,6 +228,7 @@ int64_t opal_pci_tce_kill(uint64_t phb_id, uint32_t kill_type,
 int64_t opal_rm_pci_tce_kill(uint64_t phb_id, uint32_t kill_type,
 			     uint32_t pe_num, uint32_t tce_size,
 			     uint64_t dma_addr, uint32_t npages);
+int64_t opal_pci_get_phb_capi_mode(uint64_t phb_id);
 
 /* Internal functions */
 extern int early_init_dt_scan_opal(unsigned long node, const char *uname,

diff --git a/arch/powerpc/platforms/powernv/opal-wrappers.S b/arch/powerpc/platforms/powernv/opal-wrappers.S
index 3d29d40..338d034 100644
--- a/arch/powerpc/platforms/powernv/opal-wrappers.S
+++ b/arch/powerpc/platforms/powernv/opal-wrappers.S

@@ -308,3 +308,4 @@ OPAL_CALL(opal_int_eoi,				OPAL_INT_EOI);
 OPAL_CALL(opal_int_set_mfrr,			OPAL_INT_SET_MFRR);
 OPAL_CALL(opal_pci_tce_kill,			OPAL_PCI_TCE_KILL);
 OPAL_CALL_REAL(opal_rm_pci_tce_kill,		OPAL_PCI_TCE_KILL);
+OPAL_CALL(opal_pci_get_phb_capi_mode,		OPAL_PCI_GET_PHB_CAPI_MODE);

-- 
git-series 0.8.10

If we kexec into a new kernel on a machine where a PHB has been switched
into CAPI mode, we need to disable CAPI mode in the new kernel before
traffic begins to flow on the PHB and causes a machine checkstop.

During PHB initialisation, ask OPAL whether each PHB is in CAPI mode, and
if so, do a complete reset in order to disable CAPI mode.

This requires a version of skiboot that implements the
OPAL_PCI_GET_PHB_CAPI_MODE call (introduced at the same time that the
capability to disable CAPI mode during complete resets was introduced).

Signed-off-by: Andrew Donnellan <redacted>

---

Corresponding skiboot code: http://patchwork.ozlabs.org/patch/670782/
---
 arch/powerpc/platforms/powernv/pci-ioda.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index ca5e9b5..bd76651 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c

@@ -3526,6 +3526,7 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
 	const __be32 *prop32;
 	int len;
 	unsigned int segno;
+	bool capi_mode = false;
 	u64 phb_id;
 	void *aux;
 	long rc;

@@ -3741,8 +3742,17 @@ static void __init pnv_pci_init_ioda_phb(struct device_node *np,
 	 * shutdown PCI devices correctly. We already got IODA table
 	 * cleaned out. So we have to issue PHB reset to stop all PCI
 	 * transactions from previous kernel.
+	 *
+	 * Additionally, if the PHB is in CAPI mode, we also need to
+	 * reset it to get it out of CAPI mode.
 	 */
-	if (is_kdump_kernel()) {
+	if (opal_check_token(OPAL_PCI_GET_PHB_CAPI_MODE) &&
+	    opal_pci_get_phb_capi_mode(phb_id) == OPAL_PHB_CAPI_MODE_CAPI) {
+		pr_info("  PHB in CAPI mode, reset required\n");
+		capi_mode = true;
+	}
+
+	if (is_kdump_kernel() || capi_mode) {
 		pr_info("  Issue PHB reset ...\n");
 		pnv_eeh_phb_reset(hose, EEH_RESET_FUNDAMENTAL);
 		pnv_eeh_phb_reset(hose, EEH_RESET_DEACTIVATE);

-- 
git-series 0.8.10

On Fri, 2016-16-09 at 10:39:44 UTC, Andrew Donnellan wrote:

Signed-off-by: Andrew Donnellan <redacted>

Applied to powerpc next, thanks.

https://git.kernel.org/powerpc/c/6060e9ea8d2b69fff8fc70ed6a

cheers