Re: Ethernet over PCIe driver for Inter-Processor Communication
From: David Hawkins <hidden>
Date: 2013-08-30 18:07:07
Hi S.Saravanan,
I successfully mapped the Programmable Interrupt Controller registers in the EP to the PCI space. Thus now I can write the shared message interrupt registers in the EP from the RC over PCI.
Excellent.
But I am facing the following problems now. 1) In my driver at EP, to register for this interrupt I need to know the hardware irq number but I can't find any interrupt number assigned by the PIC for the messages interrupt sources(Page 451 , MPC8641DRM manual). 2) Otherwise i need to get the virtual irq number assigned by kernel corresponding to the message interrupt . I am unable to find a method to get this also.
I recall having to ask a similar question when trying to map a GPIO interrupt into a Linux interrupt number. I forget the convention (I'm "the hardware guy"). It may be a device tree thing, or an offset, I'll let someone more knowledgeable comment.
In the RC side driver i get the virtual irq number after calling pci_enable_msi() which is straightforward. I studied the RC code which sets up shared message interrupts (Page 481, MPC manual) for PCI MSI interrupts . When msi is enabled the "arch_setup_msi_irqs()" is called leading to the fsl_setup_msi_irqs() (http://lxr.free-electrons.com/source/arch/powerpc/sysdev/fsl_msi.c?v=3.7#L151) . In this function the virtual irq no is obtained as below: /virq = irq_create_mapping(msi_data->irqhost, hwirq);/ In the above function the hardware irq number is same as the value written into the Shared Message Signaled Interrupt Index Register (Page 482) which is strange. Further these functions are called in the RC during pci_probe at boot time or when pci_enable_msi() is called . Thus there is a always a PCI slave device context to it. However I require to do it in the EP which has no pci probing nor any pci device reference whatsoever as it a slave. Is this approach right ?
I'm not sure.
You'll have two drivers;
* The root-complex.
This is a standard PCIe driver, so you'll just follow convention
there
* The end-point driver.
This driver needs to use the PCIe bus, but its not responsible
for the PCIe bus in the way a root-complex is. The driver needs
to know what the root-complex is interrupting it for, eg.,
"transmitter empty" (I've read your last message) or "receiver
ready" (there is a message from me, waiting for you).
So you need at least two unique interrupts or messages from the
root-complex to the end-point.
quoted
Its always a good idea to discuss different options, and to stub out drivers or create minimal (but functional) drivers. That way you'll be able to see how similar your new driver is to other drivers, and you'll quickly discover if there is a hardware feature in the existing driver that you cannot emulate (eg., some SRIO feature used by the rionet driver).Right now I am trying a very primitive driver just to check the feasibility of bi-directional communication between the RC and the EP. Once this is established I will be in a better position to get inputs on making it a more effective one.
You're on the right track. When I looked at using the messaging registers on the PLX PCI device, I started by simply creating what was effectively a serial port (one char at a time). Section 4 explains the interlocking required between two processors http://www.ovro.caltech.edu/~dwh/correlator/pdf/cobra_driver.pdf The mailbox/interrupt registers are effectively being used to implement a mutex between the two processors. I think at one point Ira took similar code to this and hooked it into the actual serial layer, so that you had a tty over PCI. You could always start with a simplification like that too. Cheers, Dave