RE: Virtualizing MSI-X on IMS via VFIO

On Fri, Jun 25 2021 at 10:43, Thomas Gleixner wrote:

On Fri, Jun 25 2021 at 05:21, Kevin Tian wrote:

quoted

quoted

From: Alex Williamson <redacted>
So caching/latching occurs on unmask for MSI-X, but I can't find
similar statements for MSI.  If you have, please note them.  It's
possible MSI is per interrupt.

I checked PCI Local Bus Specification rev3.0. At that time MSI and
MSI-X were described/compared together in almost every paragraph 
in 6.8.3.4 (Per-vector Masking and Function Masking). The paragraph
that you cited is the last one in that section. It's a pity that MSI is
not clarified in this paragraph but it gives me the impression that 
MSI function is not permitted to cache address and data values. 
Later after MSI and MSI-X descriptions were split into separate 
sections in PCIe spec, this impression is definitely weakened a lot.

If true, this even implies that software is free to change data/addr
when MSI is unmasked, which is sort of counter-intuitive to most
people.

Yes, software is free to do that and it has to deal with the
consequences. See arch/x86/kernel/apic/msi.c::msi_set_affinity().

Well, it's actually forced to do so when the MSI implementation does not
provide masking. If masking is available then it should be used also for
MSI as it prevents the nasty update problem where the hardware can
observe inconsistent state... Which x86 does correctly except for that
startup issue you spotted, but I'm not at all sure about the rest.

The startup issue is halfways trivial to fix I think, but there are
other issues with the PCI/MSI-X handling which I discovered while
staring at that code for a while. Still working on it.

Thanks,

        tglx