[PATCH v2 0/4] iommu/arm-smmu: misc fixes for SMMUv1/SMMUv2

The intention behind this patchset is to allow usage of IOMMU based
DMA mappings for 32bit devices. It is based on '4.5-rc3' tag of linux
mainline tree and Robin's SMMU patchset "[PATCH 0/4] Miscellaneous
ARM SMMU patches".

This patchset and its dependent patches are available in smmu_v2
branch of https://github.com/Broadcom/arm64-linux.git

All patches have been tested on Broadcom SoCs having SMMU-500.

Anup Patel (3):
  iommu/arm-smmu: Invoke DT probe from arm_smmu_of_setup()
  of: iommu: Increment DT node refcount in of_iommu_set_ops()
  iommu/arm-smmu: Add stub of_xlate() operation in SMMUv1/SMMUv2 driver

Sricharan R (1):
  iommu/arm-smmu: Init driver using IOMMU_OF_DECLARE

 drivers/iommu/arm-smmu.c | 52 ++++++++++++++++++++++++++++++++++++------------
 drivers/iommu/of_iommu.c |  1 +
 2 files changed, 40 insertions(+), 13 deletions(-)

-- 
1.9.1

The SMMUv1/SMMUv2 driver is initialized very early using the
IOMMU_OF_DECLARE() but the actual platform device is probed
via normal DT probing.

This patch uses of_platform_device_create() from arm_smmu_of_setup()
to ensure that SMMU platform device is probed immediately.

Signed-off-by: Anup Patel <redacted>
---
 drivers/iommu/arm-smmu.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index 2c8f871..02cd67d 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c

@@ -40,6 +40,7 @@
 #include <linux/of.h>
 #include <linux/of_address.h>
 #include <linux/of_iommu.h>
+#include <linux/of_platform.h>
 #include <linux/pci.h>
 #include <linux/platform_device.h>
 #include <linux/slab.h>

@@ -1956,10 +1957,15 @@ static int __init arm_smmu_init(void)
 
 static int __init arm_smmu_of_setup(struct device_node *np)
 {
+	struct platform_device *pdev;
 
 	if (!init_done)
 		arm_smmu_init();
 
+	pdev = of_platform_device_create(np, NULL, NULL);
+	if (IS_ERR(pdev))
+		return PTR_ERR(pdev);
+
 	of_iommu_set_ops(np, &arm_smmu_ops);
 
 	return 0;

-- 
1.9.1

Hi,

-----Original Message-----
From: linux-arm-kernel [mailto:linux-arm-kernel-
bounces at lists.infradead.org] On Behalf Of Anup Patel
Sent: Monday, February 08, 2016 10:48 AM
To: Catalin Marinas; Joerg Roedel; Will Deacon; Robin Murphy; Sricharan R;
Linux IOMMU; Linux ARM Kernel
Cc: Mark Rutland; Device Tree; Scott Branden; Pawel Moll; Ian Campbell;

Ray

 A question here is,  There was a probe deferral series [1], to take care of
deferred
 probing of devices behind iommus. With that this sort of early device
probing during setup
 should not be required and also it clears other dependencies of iommus on
clocks, etc, if any.
 My intention was to check whats the right way to do this  ? (or) point me
to any updates 
 on the probe deferral series that I miss ?

[1]  http://lkml.iu.edu/hypermail/linux/kernel/1505.3/03280.html
 
Regards,
 Sricharan

On Thu, Feb 18, 2016 at 5:30 PM, Sricharan [off-list ref] wrote:

Hi,

quoted

-----Original Message-----
From: linux-arm-kernel [mailto:linux-arm-kernel-
bounces at lists.infradead.org] On Behalf Of Anup Patel
Sent: Monday, February 08, 2016 10:48 AM
To: Catalin Marinas; Joerg Roedel; Will Deacon; Robin Murphy; Sricharan R;
Linux IOMMU; Linux ARM Kernel
Cc: Mark Rutland; Device Tree; Scott Branden; Pawel Moll; Ian Campbell;

Ray

quoted

Jui; Linux Kernel; Vikram Prakash; Rob Herring; BCM Kernel Feedback; Kumar
Gala; Anup Patel
Subject: [PATCH v2 2/4] iommu/arm-smmu: Invoke DT probe from
arm_smmu_of_setup()

The SMMUv1/SMMUv2 driver is initialized very early using the
IOMMU_OF_DECLARE() but the actual platform device is probed via normal
DT probing.

This patch uses of_platform_device_create() from arm_smmu_of_setup() to
ensure that SMMU platform device is probed immediately.

Signed-off-by: Anup Patel <redacted>
---
 drivers/iommu/arm-smmu.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c index

2c8f871..02cd67d 100644

--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c

@@ -40,6 +40,7 @@
 #include <linux/of.h>
 #include <linux/of_address.h>
 #include <linux/of_iommu.h>
+#include <linux/of_platform.h>
 #include <linux/pci.h>
 #include <linux/platform_device.h>
 #include <linux/slab.h>

@@ -1956,10 +1957,15 @@ static int __init arm_smmu_init(void)

 static int __init arm_smmu_of_setup(struct device_node *np)  {
+     struct platform_device *pdev;

      if (!init_done)
              arm_smmu_init();

+     pdev = of_platform_device_create(np, NULL, NULL);
+     if (IS_ERR(pdev))
+             return PTR_ERR(pdev);
+
      of_iommu_set_ops(np, &arm_smmu_ops);

 A question here is,  There was a probe deferral series [1], to take care of
deferred
 probing of devices behind iommus. With that this sort of early device
probing during setup
 should not be required and also it clears other dependencies of iommus on
clocks, etc, if any.
 My intention was to check whats the right way to do this  ? (or) point me
to any updates
 on the probe deferral series that I miss ?

[1]  http://lkml.iu.edu/hypermail/linux/kernel/1505.3/03280.html

I think probe deferral is better way of handling this. I will drop this
patch.

Thanks,
Anup

To allow use of large memory (> 4Gb) with 32bit devices we need to use
IOMMU based DMA mappings for such 32bit devices. The IOMMU dt-bindings
allows us do this by specifying 'iommus' attribute in 32bit device DT
node. Unfortunately, specifying 'iommus' attribute does not work with
current SMMUv1/SMMUv2 driver because it requires of_xlate() operation
to be implemented by the driver.

This patch adds a stub implementation of of_xlate() in SMMUv1/SMMUv2
driver to allow usage of 'iommus' attribute in DT for 32bit devices.

Signed-off-by: Anup Patel <redacted>
Reviewed-by: Ray Jui <rjui@broadcom.com>
Reviewed-by: Scott Branden <sbranden@broadcom.com>
---
 drivers/iommu/arm-smmu.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index 02cd67d..8e090d8 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c

@@ -1398,6 +1398,16 @@ static int arm_smmu_init_platform_device(struct device *dev,
 	return 0;
 }
 
+int arm_smmu_of_xlate(struct device *dev, struct of_phandle_args *args)
+{
+	/*
+	 * Nothing to do here because SMMU is already aware of all
+	 * MMU masters and their stream IDs using mmu-master attibute
+	 * SMMU DT node.
+	 */
+	return 0;
+}
+
 static int arm_smmu_add_device(struct device *dev)
 {
 	struct iommu_group *group;

@@ -1495,6 +1505,7 @@ static struct iommu_ops arm_smmu_ops = {
 	.unmap			= arm_smmu_unmap,
 	.map_sg			= default_iommu_map_sg,
 	.iova_to_phys		= arm_smmu_iova_to_phys,
+	.of_xlate		= arm_smmu_of_xlate,
 	.add_device		= arm_smmu_add_device,
 	.remove_device		= arm_smmu_remove_device,
 	.device_group		= arm_smmu_device_group,

-- 
1.9.1

On Mon, Feb 08, 2016 at 10:47:32AM +0530, Anup Patel wrote:

NAK to this.

As previously mentioned by others [1], this is an abuse of the generic
iommu binding support code.

The SMMU binding currently does not define its implementation of the
generic IOMMU binding. This series did not define what an SMMU's
#iommu-cells would be nor what the contained values would represent.
Therefore it is not valid to use an SMMU node with an iommus property as
the SMMu doesn't follwo the generic IOMMU binding.

There is ongoing work to have generic iommu binding support for the
SMMU. In the absence of documentation for what this means for the
binding, I am worried that this hack harms that effort.

To use features of the generic IOMMU binding, we must properly implement
the generic IOMMU binding for the SMMU rather than bodging it onto the
side of the existing binding.

Thanks,
Mark.

[1] http://lists.infradead.org/pipermail/linux-arm-kernel/2016-January/402976.html

On Mon, Feb 8, 2016 at 4:48 PM, Mark Rutland [off-list ref] wrote:

On Mon, Feb 08, 2016 at 10:47:32AM +0530, Anup Patel wrote:

quoted

To allow use of large memory (> 4Gb) with 32bit devices we need to use
IOMMU based DMA mappings for such 32bit devices. The IOMMU dt-bindings
allows us do this by specifying 'iommus' attribute in 32bit device DT
node. Unfortunately, specifying 'iommus' attribute does not work with
current SMMUv1/SMMUv2 driver because it requires of_xlate() operation
to be implemented by the driver.

This patch adds a stub implementation of of_xlate() in SMMUv1/SMMUv2
driver to allow usage of 'iommus' attribute in DT for 32bit devices.

Signed-off-by: Anup Patel <redacted>
Reviewed-by: Ray Jui <rjui@broadcom.com>
Reviewed-by: Scott Branden <sbranden@broadcom.com>
---
 drivers/iommu/arm-smmu.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index 02cd67d..8e090d8 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c

@@ -1398,6 +1398,16 @@ static int arm_smmu_init_platform_device(struct device *dev,
      return 0;
 }

+int arm_smmu_of_xlate(struct device *dev, struct of_phandle_args *args)
+{
+     /*
+      * Nothing to do here because SMMU is already aware of all
+      * MMU masters and their stream IDs using mmu-master attibute
+      * SMMU DT node.
+      */
+     return 0;
+}

NAK to this.

I had no intention in continuing this change if I knew some work
on generic IOMMU binding was in-progress. In fact, I had asked
about alternate options previously. (Refer,
http://lists.infradead.org/pipermail/linux-arm-kernel/2016-January/403128.html)

As previously mentioned by others [1], this is an abuse of the generic
iommu binding support code.

The SMMU binding currently does not define its implementation of the
generic IOMMU binding. This series did not define what an SMMU's
#iommu-cells would be nor what the contained values would represent.
Therefore it is not valid to use an SMMU node with an iommus property as
the SMMu doesn't follwo the generic IOMMU binding.

There is ongoing work to have generic iommu binding support for the
SMMU. In the absence of documentation for what this means for the
binding, I am worried that this hack harms that effort.

Thanks for the info, I would like to try this on Broadcom SoCs.

Whats the ETA of patches for generic IOMMU binding for SMMU?

To use features of the generic IOMMU binding, we must properly implement
the generic IOMMU binding for the SMMU rather than bodging it onto the
side of the existing binding.

Thanks,
Mark.

Regards,
Anup

On Mon, Feb 08, 2016 at 06:13:11PM +0530, Anup Patel wrote:

On Mon, Feb 8, 2016 at 4:48 PM, Mark Rutland [off-list ref] wrote:

quoted

On Mon, Feb 08, 2016 at 10:47:32AM +0530, Anup Patel wrote:

quoted

To allow use of large memory (> 4Gb) with 32bit devices we need to use
IOMMU based DMA mappings for such 32bit devices. The IOMMU dt-bindings
allows us do this by specifying 'iommus' attribute in 32bit device DT
node. Unfortunately, specifying 'iommus' attribute does not work with
current SMMUv1/SMMUv2 driver because it requires of_xlate() operation
to be implemented by the driver.

This patch adds a stub implementation of of_xlate() in SMMUv1/SMMUv2
driver to allow usage of 'iommus' attribute in DT for 32bit devices.

Signed-off-by: Anup Patel <redacted>
Reviewed-by: Ray Jui <rjui@broadcom.com>
Reviewed-by: Scott Branden <sbranden@broadcom.com>
---
 drivers/iommu/arm-smmu.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index 02cd67d..8e090d8 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c

@@ -1398,6 +1398,16 @@ static int arm_smmu_init_platform_device(struct device *dev,
      return 0;
 }

+int arm_smmu_of_xlate(struct device *dev, struct of_phandle_args *args)
+{
+     /*
+      * Nothing to do here because SMMU is already aware of all
+      * MMU masters and their stream IDs using mmu-master attibute
+      * SMMU DT node.
+      */
+     return 0;
+}

NAK to this.

I had no intention in continuing this change if I knew some work
on generic IOMMU binding was in-progress. In fact, I had asked
about alternate options previously. (Refer,
http://lists.infradead.org/pipermail/linux-arm-kernel/2016-January/403128.html)

Ok.

To be clear, I expect the generic binding alone to be used for the case
you described, regardless of who implements that or how long it takes to
appear.

quoted

As previously mentioned by others [1], this is an abuse of the generic
iommu binding support code.

The SMMU binding currently does not define its implementation of the
generic IOMMU binding. This series did not define what an SMMU's
#iommu-cells would be nor what the contained values would represent.
Therefore it is not valid to use an SMMU node with an iommus property as
the SMMu doesn't follwo the generic IOMMU binding.

There is ongoing work to have generic iommu binding support for the
SMMU. In the absence of documentation for what this means for the
binding, I am worried that this hack harms that effort.

Thanks for the info, I would like to try this on Broadcom SoCs.

Whats the ETA of patches for generic IOMMU binding for SMMU?

That I do not know.

Robin, what's the state of the generic IOMMU binding support?

Thanks,
Mark.

We are saving pointer to iommu DT node in of_iommu_set_ops()
hence we should increment DT node ref count.

Signed-off-by: Anup Patel <redacted>
Reviewed-by: Ray Jui <rjui@broadcom.com>
Reviewed-by: Scott Branden <sbranden@broadcom.com>
Reviewed-by: Robin Murphy <Robin.Murphy@arm.com>
---
 drivers/iommu/of_iommu.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/iommu/of_iommu.c b/drivers/iommu/of_iommu.c
index 60ba238..5fea665 100644
--- a/drivers/iommu/of_iommu.c
+++ b/drivers/iommu/of_iommu.c

@@ -110,6 +110,7 @@ void of_iommu_set_ops(struct device_node *np, struct iommu_ops *ops)
 	if (WARN_ON(!iommu))
 		return;
 
+	of_node_get(np);
 	INIT_LIST_HEAD(&iommu->list);
 	iommu->np = np;
 	iommu->ops = ops;

-- 
1.9.1

From: Sricharan R <redacted>

This patch uses IOMMU_OF_DECLARE to register the driver
and the iommu_ops. So when master devices of the iommu are
registered, of_xlate callback can be used to add the master
configurations to the smmu driver.

Signed-off-by: Sricharan R <redacted>
Signed-off-by: Anup Patel <redacted>
Tested-by: Anup Patel <redacted>
---
 drivers/iommu/arm-smmu.c | 35 ++++++++++++++++++++++-------------
 1 file changed, 22 insertions(+), 13 deletions(-)

diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
index bf1895c..2c8f871 100644
--- a/drivers/iommu/arm-smmu.c
+++ b/drivers/iommu/arm-smmu.c

@@ -39,6 +39,7 @@
 #include <linux/module.h>
 #include <linux/of.h>
 #include <linux/of_address.h>
+#include <linux/of_iommu.h>
 #include <linux/pci.h>
 #include <linux/platform_device.h>
 #include <linux/slab.h>

@@ -265,6 +266,8 @@
 #define FSYNR0_WNR			(1 << 4)
 
 static int force_stage;
+static bool init_done;
+
 module_param_named(force_stage, force_stage, int, S_IRUGO);
 MODULE_PARM_DESC(force_stage,
 	"Force SMMU mappings to be installed@a particular stage of translation. A value of '1' or '2' forces the corresponding stage. All other values are ignored (i.e. no stage is forced). Note that selecting a specific stage will disable support for nested translation.");

@@ -1926,20 +1929,8 @@ static struct platform_driver arm_smmu_driver = {
 
 static int __init arm_smmu_init(void)
 {
-	struct device_node *np;
 	int ret;
 
-	/*
-	 * Play nice with systems that don't have an ARM SMMU by checking that
-	 * an ARM SMMU exists in the system before proceeding with the driver
-	 * and IOMMU bus operation registration.
-	 */
-	np = of_find_matching_node(NULL, arm_smmu_of_match);
-	if (!np)
-		return 0;
-
-	of_node_put(np);
-
 	ret = platform_driver_register(&arm_smmu_driver);
 	if (ret)
 		return ret;

@@ -1958,15 +1949,33 @@ static int __init arm_smmu_init(void)
 		bus_set_iommu(&pci_bus_type, &arm_smmu_ops);
 #endif
 
+	init_done = true;
+
 	return 0;
 }
 
+static int __init arm_smmu_of_setup(struct device_node *np)
+{
+
+	if (!init_done)
+		arm_smmu_init();
+
+	of_iommu_set_ops(np, &arm_smmu_ops);
+
+	return 0;
+}
+
+IOMMU_OF_DECLARE(arm_smmu_v1, "arm,smmu-v1", arm_smmu_of_setup);
+IOMMU_OF_DECLARE(arm_smmu_v2, "arm,smmu-v2", arm_smmu_of_setup);
+IOMMU_OF_DECLARE(arm_smmu_400, "arm,mmu-400", arm_smmu_of_setup);
+IOMMU_OF_DECLARE(arm_smmu_401, "arm,mmu-401", arm_smmu_of_setup);
+IOMMU_OF_DECLARE(arm_smmu_500, "arm,mmu-500", arm_smmu_of_setup);
+
 static void __exit arm_smmu_exit(void)
 {
 	return platform_driver_unregister(&arm_smmu_driver);
 }
 
-subsys_initcall(arm_smmu_init);
 module_exit(arm_smmu_exit);
 
 MODULE_DESCRIPTION("IOMMU API for ARM architected SMMU implementations");

-- 
1.9.1