-----Original Message-----
From: Yoder Stuart-B08248
Sent: Tuesday, June 17, 2014 12:24 AM
To: Will Deacon
Cc: Sethi Varun-B16395; Thierry Reding; Mark Rutland;
devicetree at vger.kernel.org; linux-samsung-soc at vger.kernel.org; Pawel
Moll; Arnd Bergmann; Ian Campbell; Grant Grundler; Stephen Warren; linux-
kernel at vger.kernel.org; Marc Zyngier; Linux IOMMU; Rob Herring; Kumar
Gala; linux-tegra at vger.kernel.org; Cho KyongHo; Dave P Martin; linux-arm-
kernel at lists.infradead.org
Subject: RE: [PATCH v2] devicetree: Add generic IOMMU device tree
bindings

quoted

-----Original Message-----
From: Will Deacon [mailto:will.deacon at arm.com]
Sent: Monday, June 16, 2014 12:04 PM
To: Yoder Stuart-B08248
Cc: Sethi Varun-B16395; Thierry Reding; Mark Rutland;
devicetree at vger.kernel.org; linux-samsung-soc at vger.kernel.org; Pawel
Moll; Arnd Bergmann; Ian Campbell; Grant Grundler; Stephen Warren;
linux- kernel at vger.kernel.org; Marc Zyngier; Linux IOMMU; Rob Herring;
Kumar Gala; linux-tegra at vger.kernel.org; Cho KyongHo; Dave P Martin;
linux-arm- kernel at lists.infradead.org
Subject: Re: [PATCH v2] devicetree: Add generic IOMMU device tree
bindings

Hi Stuart,

On Mon, Jun 16, 2014 at 05:56:32PM +0100, Stuart Yoder wrote:

quoted

quoted

Do you have use-cases where you really need to change these
mappings dynamically?

Yes.  In the case of a PCI bus-- you may not know in advance how many
PCI devices there are until you probe the bus.   We have another FSL
proprietary bus we call the "fsl-mc" bus that is similar.

For that case, though, you could still describe an algorithmic
transformation from RequesterID to StreamID which corresponds to a
fixed mapping.

quoted

Another thing to consider-- starting with SMMUv2, as you know, there
is a new distributed architecture with multiple TBUs and a
centralized TCU that walks the SMMU page tables.  So instead of
sprinkling multiple SMMUs all over an SoC you now have the option a
1 central TCU and

sprinkling

quoted

multiple TBUs around.   However, this means that the stream ID

namespace

quoted

is now global and can be pretty limited.  In the SMMU implementation
we have there are only 64 stream ID total for our Soc.  But we have
many

more

quoted

masters than that.

So we look at stream IDs as really corresponding to an 'isolation

context'

quoted

and not to a bus master.  An isolation context is the domain you are
trying to isolate with the SMMU.  Devices that all belong to the
same 'isolation context' can share the same stream ID, since they
share the same domain and page tables.

Ok, this is more compelling.

quoted

So, perhaps by default some/most SMMU masters may have a default
stream

ID

quoted

of 0x0 that is used by the host...and that could be represented
statically in the device tree.

But, we absolutely will need to dynamically set new stream IDs into
masters when a new IOMMU 'domain' is created and devices
are added to it.   All the devices in a domain will share
the same stream ID.

So whatever we do, let's please have an architecture flexible enough
to allow for this.

What is the software interface to the logic that assigns the StreamIDs?
Is
it part of the SMMU, or a separate device (or set of devices)?

For us at the hardware level there are a few different ways that the
streamIDs can be set.  It is not part of the SMMU.  In the cases where
there is simply
1 device to 1 streamID (e.g. USB controller) there is an SoC register
where
you just program the stream ID.   In the case of PCI, our PCI controller
has a RequesterID-to-streamID table that you set via some PCI controller
registers.

The way we generally thought it would work was something like
this:
   -u-boot/bootloader makes any static streamID allocation if needed,
    sets a default streamID  (e.g. 0x0) in device and expresses
    that in the device tree
   -device tree would express relationship between devices
    (including bus controllers) and the SMMU through mmu-masters
    property
   -u-boot would express the range of unused (or used) streamIDs via a
new
    device tree property so the kernel SMMU driver knows what streamIDs
are
    free
   -in the SMMU driver a different vendor specific 'add_device' callback
    could be used to handle our special cases where we need to set/change
    the stream ID for devices added to a domain

Another possibility, could be to program the stream Id in the device registers (reference for the stream ID register can be obtained from the device tree) during device attach. This could be relevant in case of VFIO, when we are assigning multiple devices to a single VM. All the devices can share the same stream ID.

-Varun