On Fri, Oct 14, 2022 at 11:18:36AM +0000, Radovanovic, Aleksandar wrote:

And that still does not imply lack of ordering or sharing of MSI
target addresses between devices.

Either the end point generates the MSI, and maybe the bridge mangles
it, or it raises a lot of suspicion that this is not right. If the end
point generates the MSI then it raises the question why do we need to
tolerate these limits?

This is a highly programmable IP block, at the core of which is an
interconnect interfacing to programmable logic (PL), a number of
PCIe controllers (either endpoint or root-port), DMA engines,
offload engines, the embedded processor subsystem (PSX), etc. DMA
and interrupts can be routed across it in almost any (meaningful)
direction. The datapath 'endpoints' request DMA and interrupts, but
don't concern themselves with the mechanics of delivering that in
the target domain. It is the responsibility of the egress bridges to
the target domains to convert the interconnect interrupt
transactions to whatever the interrupt delivery mechanism for that
domain is. E.g. for PCIe controllers in endpoint mode, that would be
through PCIe MSI-X tables internal to the controller (and managed by
the PCIe host), for PSX that would be the PSX bridge (partially
managed by the PSX OS, mediated through firmware, i.e. through CDX
bus driver) and so on. It is the responsibility of the interconnect
to maintain transaction ordering (including DMA vs. interrupts). It
is the responsibility of the firmware to manage the bridges
according to the implemented use-case, so everything works as
expected.

Again, this all just seems wrongly designed. MSI should not be part
of an interconnect bridge. We did that already 20 years ago, it was
called IOAPICs on x86 and I think everyone is happy to see it gone.

If you want to build IOAPICs again, I guess you can, but that is a
slightly different SW setup than the MSI you are trying to use here,
and even that didn't have the same limitations you are proposing.

So, yes, the hardware that translates interrupt transactions to GIC
AXI writes is shared between endpoints, but what I said above still
applies. And that doesn't necessarily make it weird/wrong, it's just
more complex than you might think.

If it doesn't fit the architecture, then I think it must be considered
wrong. Mis-using platform architected components like MSI in HW is
problematic.

You should design the HW properly so you don't have these
problems. Involving FW in the MSI setup is also a bad idea, POWER did
this and it made a big mess of their arch code :(

Jason