Re: Help needed to describe a custom bus in the device tree

Hi Dave,

On Tuesday 11 March 2008 23:54, David Gibson wrote:

On Tue, Mar 11, 2008 at 03:27:26PM +0100, Laurent Pinchart wrote:

quoted

Hi everybody,

the migration process from ARCH=ppc to ARCH=powerpc is easier than I
thought in some parts, but a few devices are still giving me
headaches. This should hopefully be one of my last major requests
for help (I'm sure most of you will be happy to see traffic on this
list going down when I'll be done :-))

I'm having trouble describing a custom bus named MS bus (completely
unrelated to a well-known software company) in the device tree. The
hardware is MPC8248-based and has the following hardware topology.

MPC8248 <-- localbus --> FPGA <-- ms bus --> Custom peripherals

The bus interrupt controller, serial access (SPI) controller and
status registers are accessed through memory-mapped registers in the
FPGA. Parallel access to the MS bus is handled transparently by the
FPGA which handles address mapping.

The FPGA is mapped on the locabus at address 0xf4000000. Bus control
registers are at 0xf4002000 - 0xf4003000. The parallel bus memory
window on the localbus is located at 0xf5000000.

My current dts draft describes that topology as follows (unrelated
devices on the local bus such as flash memory are removed for
clarity).

localbus@f0010100 {
	compatible = "fsl,pq2-localbus";
	#address-cells = <2>;
	#size-cells = <1>;
	reg = <f0010100 40>;

	ranges = <0 0 40000000 01000000
		  2 0 f2000000 00100000
		  3 0 f3000000 00100000
		  4 0 f4000000 00100000
		  5 0 f5000000 00100000>;

	fpga@4,0 {
		#address-cells = <1>;
		#size-cells = <1>;
		ranges = <4 0 0 00010000>;

		msbus-arbitrer@2000 {
			compatible = "tbox,cp11-msbus-arbitrer";
			reg = <2000 4>;
		};

		msbus_pic: interrupt-controller@2100 {
			compatible = "tbox,cp11-msbus-pic";
			reg = <2100 8>;
			interrupts = <17 2>;
			interrupt-parent = <&cpm_pic>;
			#interrupt-cells = <1>;
			interrupt-controller;
		};

		msbus-spi@2200 {
			compatible = "tbox,cp11-msbus-spi";
			reg = <2200 100>;
			interrupts = <18 8>;
			interrupt-parent = <&cpm_pic>;
		};

		sdhc@5000 {
			compatible = "tbox,sdhci";
			reg = <5000 1000>;
			interrupts = <16 8>;
			interrupt-parent = <&cpm_pic>;
		};
	};

	msbus@5,0 {
		compatible = "tbox,cp11-msbus";
		#address-cells = <1>;
		#size-cells = <1>;
		#interrupt-cells = <1>;
		reg = <5 0 0 00000400>;
		interrupt-parent = <&msbus_pic>;
	};
};

The device tree reflects the physical topology but makes driver
access to the bus quite complex. An OF platform device driver
matching on compatible = "tbox,cp11-msbus" will not have the bus
FPGA registers described in its device node.

Having a look at the various device trees included in the kernel
sources, it seems platforms with a PCI bus experience a similar
problem. To solve it the PCI bus node address and registers describe
the configuration registers, and the memory window to access PCI
devices is described by the ranges property.  Applying that to my
custom bus would lead to the following tree.

localbus@f0010100 {
	compatible = "fsl,pq2-localbus";
	#address-cells = <2>;
	#size-cells = <1>;
	reg = <f0010100 40>;

	ranges = <0 0 40000000 01000000
		  2 0 f2000000 00100000
		  3 0 f3000000 00100000
		  4 0 f4000000 00100000
		  4 1 f4002000 00000100
		  5 0 f5000000 00100000>;

	fpga@4,0 {
		#address-cells = <1>;
		#size-cells = <1>;
		ranges = <4 0 0 00010000>;

		msbus_pic: interrupt-controller@2100 {
			compatible = "tbox,cp11-msbus-pic";
			reg = <2100 8>;
			interrupts = <17 2>;
			interrupt-parent = <&cpm_pic>;
			#interrupt-cells = <1>;
			interrupt-controller;
		};

		msbus-spi@2200 {
			compatible = "tbox,cp11-msbus-spi";
			reg = <2200 100>;
			interrupts = <18 8>;
			interrupt-parent = <&cpm_pic>;
		};

		sdhc@5000 {
			compatible = "tbox,sdhci";
			reg = <5000 1000>;
			interrupts = <16 8>;
			interrupt-parent = <&cpm_pic>;
		};
	};

	msbus@4,1 {
		compatible = "tbox,cp11-msbus";
		#address-cells = <1>;
		#size-cells = <1>;
		#interrupt-cells = <1>;
		reg = <4 1 4>;
		interrupt-parent = <&msbus_pic>;
		ranges = <5 0 0 00000400>;
	};
};

Is this correct ? Is that the best way to describe my custom bus in
the device tree ?

Your second example looks closer to right.  Certainly you should use
'reg' only for bus control registers, and 'ranges' for windows into
the bus address space itself.

Ok.

The device tree describes hardware from a functional point of view, so
I don't know that it's relevant that all the bus control functions are
implemented in an FPGA.  Each of the subnodes are more-or-less
independent devices, so they could just have separate nodes.

Or, if this seems more sensible, you could decide that they're
sufficiently closely related to put them all as one node, with
multiple register blocks listed in the 'reg' property.  That would
probably get messy for your PIC at the very least though.

I suppose I could implement PIC support in the bus driver itself, but having 
separate nodes with separate OF devices and separate drivers seems cleaner to 
me (although it can make dependencies a bit more difficult to handle).

quoted

How would the relationships between the bus and
its PIC and SPI controller be handled in the drivers ?

If the msbus driver needs to work with the associated PIC and SPI
controllers, then you should put properties in the msbus node giving
their phandles.

The PIC can live pretty much by itself, but the SPI controller is used to 
enumerate devices on the bus (and perform some other tasks at runtime). I 
plan to have the SPI device and driver be self-contained, and have the bus 
node reference the SPI device through its phandle. I will just have to make 
sure the bus driver is initialised after the SPI driver.

quoted

I also don't
understand how interrupt mappings are supposed to be handled. PCI
busses have two CPM interrupt lines, one for the PCI PIC and one for
the PCI bus, with the PCI bus having the CPM PIC as its interrupt
controller. My bus PIC uses a single interrupt line. Is there some
documentation explaining how PICs and interrupt mappings should be
described ?

Are interrupts from devices on the msbus routed over the msbus, or are
they routed independently to the mspic or the cpm PIC?

There is a single active low interrupt line on the msbus. When the mspic 
detects an interrupt condition, it will read the interrupt source registers 
from all devices on the bus and generate a host interrupt to the cpm PIC. The 
mspic driver then process the CPM interrupt in its demux handler, reads the 
interrupt sources from the mspic registers and dispatch the interrupts to the 
msbus device drivers.

If I understand things correctly, the mspic node should have an 'interrupts' 
attribute describing the cascaded interrupt line (mspic -> cpm PIC irq), and 
the msbus node should have an 'interrupts' attribute describing the interrupt 
line used to report bus-related events (hotplug events for instance). Is that 
right ?

To make things a bit more complex, the msbus interrupt and the bus-related 
events interrupt share the same CPM irq line. Can I use the same virq number 
in both nodes, or do I have to demux the interrupts in a separate driver ? If 
I have to demux the interrupts in a separate PIC driver, how do I know what 
virtual irq number will be assigned to each device so that I can reference 
them in the device tree ?

Thanks a lot for your help.

Best regards,

-- 
Laurent Pinchart
CSE Semaphore Belgium

Chaussée de Bruxelles, 732A
B-1410 Waterloo
Belgium

T +32 (2) 387 42 59
F +32 (2) 387 42 75