On Thu, Mar 22, 2018 at 10:57:32PM +0000, Stephen  Bates wrote:

quoted

 I've seen the response that peers directly below a Root Port could not
DMA to each other through the Root Port because of the "route to self"
issue, and I'm not disputing that.  

Bjorn 

You asked me for a reference to RTS in the PCIe specification. As
luck would have it I ended up in an Irish bar with Peter Onufryk
this week at OCP Summit. We discussed the topic. It is not
explicitly referred to as "Route to Self" and it's certainly not
explicit (or obvious) but r6.2.8.1 of the PCIe 4.0 specification
discusses error conditions for virtual PCI bridges. One of these
conditions (given in the very first bullet in that section) applies
to a request that is destined for the same port it came in on. When
this occurs the request must be terminated as a UR.

Thanks for that reference!

I suspect figure 10-3 in sec 10.1.1 might also be relevant, although
it's buried in the ATS section.  It shows internal routing between
functions of a multifunction device.  That suggests that the functions
should be able to DMA to each other without those transactions ever
appearing on the link.

Popping way up the stack, my original point was that I'm trying to
remove restrictions on what devices can participate in peer-to-peer
DMA.  I think it's fairly clear that in conventional PCI, any devices
in the same PCI hierarchy, i.e., below the same host-to-PCI bridge,
should be able to DMA to each other.

The routing behavior of PCIe is supposed to be compatible with
conventional PCI, and I would argue that this effectively requires
multi-function PCIe devices to have the internal routing required to
avoid the route-to-self issue.

Bjorn