[PATCH v4 1/2] cxl: Set the PBCQ Tunnel BAR register when enabling capi mode

Skiboot used to set the default Tunnel BAR register value when capi mode
was enabled. This approach was ok for the cxl driver, but prevented other
drivers from choosing different values.

Skiboot versions > 5.11 will not set the default value any longer. This
patch modifies the cxl driver to set/reset the Tunnel BAR register when
entering/exiting the cxl mode, with pnv_pci_set_tunnel_bar().

That should work with old skiboot (since we are re-writing the value
already set) and new skiboot.

Signed-off-by: Philippe Bergheaud <redacted>
Reviewed-by: Christophe Lombard <redacted>
Acked-by: Frederic Barrat <redacted>
---
v2: Restrict tunnel bar setting to power9.
    Do not fail cxl_configure_adapter() on tunnel bar setting error.
    Log an info message instead, and continue configuring capi mode.

v3,v4: No change.
---
 drivers/misc/cxl/pci.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/misc/cxl/pci.c b/drivers/misc/cxl/pci.c
index 83f1d08058fc..355c789406f7 100644
--- a/drivers/misc/cxl/pci.c
+++ b/drivers/misc/cxl/pci.c

@@ -1742,6 +1742,10 @@ static int cxl_configure_adapter(struct cxl *adapter, struct pci_dev *dev)
 	/* Required for devices using CAPP DMA mode, harmless for others */
 	pci_set_master(dev);
 
+	if (cxl_is_power9())
+		if (pnv_pci_set_tunnel_bar(dev, 0x00020000E0000000ull, 1))
+			dev_info(&dev->dev, "Tunneled operations unsupported\n");
+
 	if ((rc = pnv_phb_to_cxl_mode(dev, adapter->native->sl_ops->capi_mode)))
 		goto err;
 

@@ -1768,6 +1772,8 @@ static void cxl_deconfigure_adapter(struct cxl *adapter)
 {
 	struct pci_dev *pdev = to_pci_dev(adapter->dev.parent);
 
+	if (cxl_is_power9())
+		pnv_pci_set_tunnel_bar(pdev, 0x00020000E0000000ull, 0);
 	cxl_native_release_psl_err_irq(adapter);
 	cxl_unmap_adapter_regs(adapter);
 

-- 
2.16.3

Failure to synchronize the tunneled operations does not prevent
the initialization of the cxl card. This patch reports the tunneled
operations status via /sys.

Signed-off-by: Philippe Bergheaud <redacted>
---
v3: Added this patch to report the tunneled operations status.

v4: Updated Documentation/ABI/testing/sysfs-class-cxl.
---
 Documentation/ABI/testing/sysfs-class-cxl |  8 ++++++++
 drivers/misc/cxl/cxl.h                    |  1 +
 drivers/misc/cxl/pci.c                    |  7 ++++++-
 drivers/misc/cxl/sysfs.c                  | 10 ++++++++++
 4 files changed, 25 insertions(+), 1 deletion(-)

diff --git a/Documentation/ABI/testing/sysfs-class-cxl b/Documentation/ABI/testing/sysfs-class-cxl
index 640f65e79ef1..8e69345c37cc 100644
--- a/Documentation/ABI/testing/sysfs-class-cxl
+++ b/Documentation/ABI/testing/sysfs-class-cxl

@@ -244,3 +244,11 @@ Description:    read only
                 Returns 1 if the psl timebase register is synchronized
                 with the core timebase register, 0 otherwise.
 Users:          https://github.com/ibm-capi/libcxl
+
+What:           /sys/class/cxl/<card>/tunneled_ops_supported
+Date:           May 2018
+Contact:        linuxppc-dev@lists.ozlabs.org
+Description:    read only
+                Returns 1 if tunneled operations are supported in capi mode,
+                0 otherwise.
+Users:          https://github.com/ibm-capi/libcxl

diff --git a/drivers/misc/cxl/cxl.h b/drivers/misc/cxl/cxl.h
index a4c9c8297a6d..918d4fb742d1 100644
--- a/drivers/misc/cxl/cxl.h
+++ b/drivers/misc/cxl/cxl.h

@@ -717,6 +717,7 @@ struct cxl {
 	bool perst_select_user;
 	bool perst_same_image;
 	bool psl_timebase_synced;
+	bool tunneled_ops_supported;
 
 	/*
 	 * number of contexts mapped on to this card. Possible values are:

diff --git a/drivers/misc/cxl/pci.c b/drivers/misc/cxl/pci.c
index 355c789406f7..008f50a0c465 100644
--- a/drivers/misc/cxl/pci.c
+++ b/drivers/misc/cxl/pci.c

@@ -1742,9 +1742,14 @@ static int cxl_configure_adapter(struct cxl *adapter, struct pci_dev *dev)
 	/* Required for devices using CAPP DMA mode, harmless for others */
 	pci_set_master(dev);
 
-	if (cxl_is_power9())
+	adapter->tunneled_ops_supported = false;
+
+	if (cxl_is_power9()) {
 		if (pnv_pci_set_tunnel_bar(dev, 0x00020000E0000000ull, 1))
 			dev_info(&dev->dev, "Tunneled operations unsupported\n");
+		else
+			adapter->tunneled_ops_supported = true;
+	}
 
 	if ((rc = pnv_phb_to_cxl_mode(dev, adapter->native->sl_ops->capi_mode)))
 		goto err;

diff --git a/drivers/misc/cxl/sysfs.c b/drivers/misc/cxl/sysfs.c
index 95285b7f636f..4b5a4c5d3c01 100644
--- a/drivers/misc/cxl/sysfs.c
+++ b/drivers/misc/cxl/sysfs.c

@@ -78,6 +78,15 @@ static ssize_t psl_timebase_synced_show(struct device *device,
 	return scnprintf(buf, PAGE_SIZE, "%i\n", adapter->psl_timebase_synced);
 }
 
+static ssize_t tunneled_ops_supported_show(struct device *device,
+					struct device_attribute *attr,
+					char *buf)
+{
+	struct cxl *adapter = to_cxl_adapter(device);
+
+	return scnprintf(buf, PAGE_SIZE, "%i\n", adapter->tunneled_ops_supported);
+}
+
 static ssize_t reset_adapter_store(struct device *device,
 				   struct device_attribute *attr,
 				   const char *buf, size_t count)

@@ -183,6 +192,7 @@ static struct device_attribute adapter_attrs[] = {
 	__ATTR_RO(base_image),
 	__ATTR_RO(image_loaded),
 	__ATTR_RO(psl_timebase_synced),
+	__ATTR_RO(tunneled_ops_supported),
 	__ATTR_RW(load_image_on_perst),
 	__ATTR_RW(perst_reloads_same_image),
 	__ATTR(reset, S_IWUSR, NULL, reset_adapter_store),

-- 
2.16.3

Le 14/05/2018 à 10:27, Philippe Bergheaud a écrit :

Failure to synchronize the tunneled operations does not prevent
the initialization of the cxl card. This patch reports the tunneled
operations status via /sys.

Signed-off-by: Philippe Bergheaud <redacted>
---

Thanks for adding the sysfs documentation.

Acked-by: Frederic Barrat <redacted>

Philippe Bergheaud [off-list ref] writes:

Skiboot used to set the default Tunnel BAR register value when capi mode
was enabled. This approach was ok for the cxl driver, but prevented other
drivers from choosing different values.

Skiboot versions > 5.11 will not set the default value any longer. This
patch modifies the cxl driver to set/reset the Tunnel BAR register when
entering/exiting the cxl mode, with pnv_pci_set_tunnel_bar().

That should work with old skiboot (since we are re-writing the value
already set) and new skiboot.

But doesn't that mean new skiboot can't boot any old kernel? That seems
undesirable.

cheers

On 14/05/2018 12:51, Michael Ellerman wrote:

Philippe Bergheaud [off-list ref] writes:

quoted

Skiboot used to set the default Tunnel BAR register value when capi mode
was enabled. This approach was ok for the cxl driver, but prevented other
drivers from choosing different values.

Skiboot versions > 5.11 will not set the default value any longer. This
patch modifies the cxl driver to set/reset the Tunnel BAR register when
entering/exiting the cxl mode, with pnv_pci_set_tunnel_bar().

That should work with old skiboot (since we are re-writing the value
already set) and new skiboot.

But doesn't that mean new skiboot can't boot any old kernel? That seems
undesirable.

cheers

Yes, with new skiboot, all kernels will boot.

Capi mode tunnelled operations did not work until linux-4.17-rc1 (Apr 
15, 2008). With new skiboot, kernels since linux-4.17-rc1 will loose 
tunneled operations in capi mode, until this patch set is merged.

Capi mode tunneled operations are not used in any IBM or partner GA 
products. The feature will be fully supported after this patch set and 
the libcxl PR https://github.com/ibm-capi/libcxl/pull/29 are merged.

Philippe

Philippe Bergheaud [off-list ref] writes:

On 14/05/2018 12:51, Michael Ellerman wrote:

quoted

Philippe Bergheaud [off-list ref] writes:

quoted

Skiboot used to set the default Tunnel BAR register value when capi mode
was enabled. This approach was ok for the cxl driver, but prevented other
drivers from choosing different values.

Skiboot versions > 5.11 will not set the default value any longer. This
patch modifies the cxl driver to set/reset the Tunnel BAR register when
entering/exiting the cxl mode, with pnv_pci_set_tunnel_bar().

That should work with old skiboot (since we are re-writing the value
already set) and new skiboot.

But doesn't that mean new skiboot can't boot any old kernel? That seems
undesirable.

Yes, with new skiboot, all kernels will boot.

OK.

Capi mode tunnelled operations did not work until linux-4.17-rc1 (Apr 
15, 2008). With new skiboot, kernels since linux-4.17-rc1 will loose 

        ^
        1

tunneled operations in capi mode, until this patch set is merged.

So it would be preferable if I put this patch into 4.17, so that the
4.17 release works. Am I right?

Capi mode tunneled operations are not used in any IBM or partner GA 
products. The feature will be fully supported after this patch set and 
the libcxl PR https://github.com/ibm-capi/libcxl/pull/29 are merged.

OK. In general I ignore that sort of stuff because people also ignore it
when they're filing bugs against the kernel. So if it's in the tree we
want it to work (as much as possible), regardless of what's been GA'ed,
or is supported officially etc.

cheers

On 15/05/2018 07:30, Michael Ellerman wrote:

Philippe Bergheaud [off-list ref] writes:

quoted

On 14/05/2018 12:51, Michael Ellerman wrote:

quoted

Philippe Bergheaud [off-list ref] writes:

quoted

Skiboot used to set the default Tunnel BAR register value when capi mode
was enabled. This approach was ok for the cxl driver, but prevented other
drivers from choosing different values.

Skiboot versions > 5.11 will not set the default value any longer. This
patch modifies the cxl driver to set/reset the Tunnel BAR register when
entering/exiting the cxl mode, with pnv_pci_set_tunnel_bar().

That should work with old skiboot (since we are re-writing the value
already set) and new skiboot.

But doesn't that mean new skiboot can't boot any old kernel? That seems
undesirable.

Yes, with new skiboot, all kernels will boot.

OK.

quoted

Capi mode tunnelled operations did not work until linux-4.17-rc1 (Apr
15, 2008). With new skiboot, kernels since linux-4.17-rc1 will loose

         ^
         1

quoted

tunneled operations in capi mode, until this patch set is merged.

So it would be preferable if I put this patch into 4.17, so that the
4.17 release works. Am I right?

Yes.

quoted

Capi mode tunneled operations are not used in any IBM or partner GA
products. The feature will be fully supported after this patch set and
the libcxl PR https://github.com/ibm-capi/libcxl/pull/29 are merged.

OK. In general I ignore that sort of stuff because people also ignore it
when they're filing bugs against the kernel. So if it's in the tree we
want it to work (as much as possible), regardless of what's been GA'ed,
or is supported officially etc.

cheers

On Mon, 2018-05-14 at 08:27:35 UTC, Philippe Bergheaud wrote:

Skiboot used to set the default Tunnel BAR register value when capi mode
was enabled. This approach was ok for the cxl driver, but prevented other
drivers from choosing different values.

Skiboot versions > 5.11 will not set the default value any longer. This
patch modifies the cxl driver to set/reset the Tunnel BAR register when
entering/exiting the cxl mode, with pnv_pci_set_tunnel_bar().

That should work with old skiboot (since we are re-writing the value
already set) and new skiboot.

Signed-off-by: Philippe Bergheaud <redacted>
Reviewed-by: Christophe Lombard <redacted>
Acked-by: Frederic Barrat <redacted>

Series applied to powerpc fixes, thanks.

https://git.kernel.org/powerpc/c/401dca8cbd14fc4b32d93499dcd12a

cheers