[RFC] Describing arbitrary bus mastering relationships in DT

On Fri, May 09, 2014 at 03:16:33PM +0100, Dave Martin wrote:

On Fri, May 02, 2014 at 12:17:50PM -0600, Jason Gunthorpe wrote:

quoted

I wonder if this might be a better naming scheme, I actually don't
really like 'slave' for this, it really only applies well to AXI style
unidirectional busses, and any sort of message-based bus architectures
(HT, PCI, QPI, etc) just have the concept of an initiator and target.

Since initiator/target applies equally well to master/slave buses,
that seems like better, clearer, naming.

Sure, I wouldn't have a problem with such a suggestion.  A more neutral
naming is less likely to cause confusion.

quoted

Using a nomenclature where
  'reg' describes a target reachable from the CPU initiator via the
        natural DT hierarchy

I would say, reachable from the parent device node (which implies your
statement).  This is consistent with the way ePAPR describes device-to-
device DMA (even if Linux doesn't usually make a lot of use of that).

Trying to simplify, but yes, that is right..

quoted

  'initiator' describes a non-CPU (eg 'DMA') source of ops, and
        travels via the path described to memory (which is the
	target).

CPUs are initiators only; non-mastering devices are targets only.

We might want some terminology to distinguish between mastering 
devices and bridges, both of which act as initiators and targets.

I was hoping to simplify a bit. What the kernel needs, really, is the
node that initates a transaction, the node that is the ultimate
completing target of that transaction and the path through all
intervening (transformative or not) components.

The fact a bridge is bus-level slave on one side and a bus-level
master on another is not relevant to the above - a bridge is not an
initiator and it is not a completing target.

We could have a concept of a "forwarder" or "gateway".  But a bus
may still be a target as well as forwarder: if the bus contains some
control registers for example.  There is nothing to stop "reg" and
"ranges" being present on the same node.

Then the node is a 'target', and a 'bridge'. The key is to carefully
define how the DT properties are used for each view-point.

quoted

  'upstream' path direction toward the target, typically memory.

I'm not keen on that, because we would describe the hop between /
and /memory as downstream or upstream depending on who initiates the
transaction.  (I appreciate you weren't including CPUs in your
discussion, but if the termology works for the whole system it
would be a bonus).

I'm just using the word 'upstream' as meaning 'moving closer to the
completing target'.

It isn't a great word, 'forward path' would do better to borrow
a networking term.

Then we have the 'return path' which, on message based busses is the
path a completion message from 'completing target' travels to reach
the 'initiator'.

They actually can be asymmetric in some situations, but I'm not sure
that is worth considering in DT, we can just assume completions travel
a reverse path that hits every transformative bridge.

quoted

  'upstream-bridge' The next hop on a path between an initiator/target

Maybe.  I'm still not sure quite why this is considered different
from the downward path through the DT, except that you consider
the cross-links in the DT to be "upward", but I considered them
"downward" (which I think are mostly equivalent approaches).

Can you elaborate?

Let us not worry about upstream/downstream and just talk about the
next bridge on the 'forward path' toward the 'completing target'.

quoted

But I would encourage you to think about the various limitations this
still has
 - NUMA systems. How does one describe the path from each
   CPU to a target regs, and target memory? This is important for
   automatically setting affinities.

This is a good point.

Currently I had only been considering visibility, not affinity.
We actually have a similar problem with GIC, where there may
be multiple MSI mailboxes visible to a device, but one that is
preferred (due to being fewer hops away in the silicon, even though
the routing may be transparent).

Really, the MSI affinity isn't handled by the architecture like x86?
Funky.
 

We could describe a whole separate bus for each CPU, with links
to common interconnect subtrees downstream.  But that might involve
a lot of duplication.  Your example below doesn't look too bad
though.

Unfortunately my example ignores the ePAPR scheme of having a /cpu
node, I didn't think too hard to fix that though.

quoted

 - Peer-to-Peer DMA, this is where a non-CPU initiator speaks to a
   non-memory target, possibly through IOMMUs and what not. ie
   a graphics card in a PCI-E slot DMA'ing through a QPI bus to
   a graphics card in a PCI-E slot attached to a different socket.

Actually, I do intend to describe that and I think I achieved it :)

To try to keep the length of this mail down a bit I won't try to
give an example here, but I'm happy to follow up later if this is
still not answered elsewhere in the thread.

I think you got most of it, if I understand properly. The tricky bit I
was concerned with, is where the CPU and DMA paths are not the same.

quoted

                intiator1 {
                        ranges = < ... >;
                        // View from this DMA initiator back to memory
                        upstream-bridge = <&interconnect0>;
                };
		/* For some reason this peripheral has two DMA
		   initiation ports. */
                intiator2 {
                        ranges = < ... >;
                        upstream-bridge = <&interconnect0>;
                };

Describing separate masters within a device in this way looks quite nice.

Understanding what to do with them can still be left up to the driver
for the parent node (peripheral at 0 in this case).

I was thinking the DMA API could learn to have a handle to the
initiator, with no handle it assumes the device node is itself the
initiator (eg dma-rages case)

quoted

             peripheral at 1 {
                ranges = < ... >;
   		regs = <>;
                intiator {
                        ranges = < ... >;
                        // View from this DMA initiator back to memory
                        upstream-bridge = <&interconnect1>;
                };
                target {
		        reg = <..>
                        /* This peripheral has integrated memory!
                           But notice the CPU path is
                             smp_system -> socket1 -> interconnect1_control -> target
			   While a DMA path is
                             intiator1 -> interconnect0 -> interconnect1 -> target
			 */
                };

By hiding slaves (as opposed to masters) inside subnodes, can DT do
generic reachability analysis?  Maybe the answer is "yes".  I know
devices hanging of buses whose compatible string is not "simple-bus" are
not automatically probed, but there are other reasons for that, such as
bus-specific power-on and probing methods.

Again, in this instance, it becomes up to the driver for peripheral at 1
to do something sensible with the buried nodes.

The generic DT machinery will happily convert the reg of target into a
CPU address for you.

quoted

            };
            peripheral2 at 0 {
   		regs = <>;

		// Or we can write the simplest case like this.
		dma-ranges = <>;
		upstream-bridge = <&interconnect1>;
                /* if upstream-bridge is omitted then it defaults to
	           &parent, eg interconnect1_control */

This doesn't seem so different from my approach, though I need to
think about it a bit more.

This is how I was thinking to unify the language with the existing
syntax.
 - dma-ranges alone in an initiator context is equivalent to using an
   implicit buried node:
    initiator {
       ranges == dma_ranges
       upstream-bridge = <&parent>;
    }
 - While in a bridge context it it attached to the
   'forward path edge'.

Now we have a very precise definition for dma-ranges in the same
language as the rest, and we can identify every involved node as
'initiator', 'target' or 'bridge'

quoted

It is computable that ops from initator2 -> target flow through
interconnect0, interconnect1, and then are delivered to target.

It has a fair symmetry with the interrupt-parent mechanism..

Although that language is rather different from mine, I think my
proposal could describe this.  

Yes, I think it did, I was mostly thinking to tighten up the language.

It doesn't preclude multi-rooted trees etc.; we could give a CPU a
"slaves" property to override the default child for transaction
rooting (which for CPUs is / -- somewhat illogical, but that's the
way ePAPR has it).

Right..

There's no reason why buses can't be cross-connected using slaves
properties.  I'd avoided such things so far, because it introduces
new cycle risks, such as
socket at 0 -> cross -> socket at 1 -> cross -> socket at 0 in the following.

That is actually really how hardware works though. The socket-routers
are configured to not have cycles on an address-by-address basis, but
the actual high level topology is cyclic.

/ {
	cpus {
		cpu at 0 {
			slaves = <&socket0_interconnect>;
		};
		cpu at 1 {
			slaves = <&socket0_interconnect>;
		};
		cpu at 2 {
			slaves = <&socket1_interconnect>;
		};
		cpu at 3 {
			slaves = <&socket1_interconnect>;
		};
	};

So, this has dis-aggregated the sockets, which looses the coherent
address view.

I feel it is important to have a single top level node that represents
the start point for *any* CPU issued transaction. This is the coherent
memory space of the system. The DT tree follows a representative
physical topology, eg from the cpu0, socket 0 view.

Knowing affinity should be computable later on top of that.

Of course, nothing about this tells an OS anything about affinity,
except what it can guess from the number of nodes that must be traversed
between two points -- which may be misleading, particular if extra nodes
are inserted in order to describe mappings and linkages.

Right, now you have to start adding a 'cost' to edges in the graph :)

Or maybe this is wrong headed and nodes should simply have an
 affinity = <&cpu0 &cpu1 &memory0 &memory1>

Cycles could be avoided via the cross-connector ranges properties -- I
would sincerely hope that the hardware really does something
equivalent -- but then you cannot answer questions like "is the path
from X to Y cycle-free" without also specifying an address.

Correct, and that is how HW works.

The DT description for a bridge might actually need to include address
based routing :(

next-hop = <BASE SIZE &target
            BASE SIZE &memory>

The downside of this approach is that the DT is unparseable to any
parser that doesn't understand the new concepts.

This is why I feel the top level 'coherent view' node is so
important. It retains the compatability.

Or go back to the suggestion I gave last time - keep the DT tree
basically as-is today and store a graph edge list separately.

Jason