Neophyte questions about PCIe

On 13/03/17 10:59, Mason wrote:

On 10/03/2017 17:00, Robin Murphy wrote:

quoted

On 10/03/17 15:35, David Laight wrote:

quoted

Robin Murphy wrote:

quoted

The appropriate DT property would be "dma-ranges", i.e.

pci at ... {
	...
	dma-ranges = <(PCI bus address) (CPU phys address) (size)>;
}

Isn't that just saying which physical addresses the cpu can assign
for buffers for those devices?
There is also an offset between the 'cpu physical address' and the
'dma address'.

That offset is inherent in what "dma-ranges" describes. Say (for ease of
calculation) that BAR0 has been put at a mem space address of 0x20000000
and maps the first 1GB of physical DRAM. That would give us:

	dma-ranges = <0x20000000 0x80000000 0x40000000>;

Then a "virt = dma_alloc_coherent(..., &handle, ...)", borrowing the
numbers from earlier in the thread, would automatically end up with:

	virt == 0xd0855000;
	handle == 0x2e07e000;

(with the physical address of 0x8e07e000 in between being irrelevant to
the consuming driver)

It is true that the device's DMA mask assignment is also part and parcel
of this, whereby we will limit what physical addresses the kernel
considers valid for DMA involving devices behind this range to the lower
3GB (i.e. 0x80000000 + 0x40000000 - 1). With a bit of luck,
CONFIG_DMABOUNCE should do the rest of the job of working around that
where necessary.

AFAICT, the parser for the "dma-ranges" property is implemented
in of_dma_get_range() in drivers/of/address.c

http://lxr.free-electrons.com/source/drivers/of/address.c#L808

/**
 * of_dma_get_range - Get DMA range info
 * @np:		device node to get DMA range info
 * @dma_addr:	pointer to store initial DMA address of DMA range
 * @paddr:	pointer to store initial CPU address of DMA range
 * @size:	pointer to store size of DMA range
 *
 * Look in bottom up direction for the first "dma-ranges" property
 * and parse it.
 *  dma-ranges format:
 *	DMA addr (dma_addr)	: naddr cells
 *	CPU addr (phys_addr_t)	: pna cells
 *	size			: nsize cells
 *
 * It returns -ENODEV if "dma-ranges" property was not found
 * for this device in DT.
 */

I didn't find anything relevant in Documentation/devicetree/bindings
except Documentation/devicetree/bindings/iommu/iommu.txt but I'm not
sure this applies to my SoC.

It's not clear to me how ranges and dma-ranges interact...
Is it perhaps: ranges for cpu-to-bus, dma-ranges for bus-to-cpu?

That's it exactly. FWIW the in-kernel description can be found in
Documentation/devicetree/booting-without-of.txt, but the
www.devicetree.org spec (which succeeds ePAPR) should be the
authoritative reference now. Note that the property itself is fully
capable of describing multiple discontiguous ranges which might look
like the perfect solution to your multiple-DRAM-banks problem, but Linux
can't actually cope with that at all :(

Robin.

ePAPR (Version 1.1 -- 08 April) provides a formal definition

2.3.9 dma-ranges

Property: dma-ranges

Value type: <empty> or <prop-encoded-array> encoded as arbitrary number
of triplets of (child-bus-address, parent-bus-address, length).

Description:

The dma-ranges property is used to describe the direct memory access (DMA) structure of a
memory-mapped bus whose device tree parent can be accessed from DMA operations
originating from the bus. It provides a means of defining a mapping or translation between the
physical address space of the bus and the physical address space of the parent of the bus.
The format of the value of the dma-ranges property is an arbitrary number of triplets of
(child-bus-address, parent-bus-address, length). Each triplet specified describes a contiguous
DMA address range.

o The child-bus-address is a physical address within the child bus' address space. The
number of cells to represent the address depends on the bus and can be determined
from the #address-cells of this node (the node in which the dma-ranges property
appears).

o The parent-bus-address is a physical address within the parent bus' address space.
The number of cells to represent the parent address is bus dependent and can be
determined from the #address-cells property of the node that defines the parent's
address space.

o The length specifies the size of the range in the child's address space. The number of
cells to represent the size can be determined from the #size-cells of this node (the
node in which the dma-ranges property appears).


I'm still digging :-)

Regards.