From: Naman Jain <redacted> Sent: Tuesday, November 3, 2026 12:43 AM
On 11/1/2026 7:52 AM, Michael Kelley wrote:
quoted
Add a section to the documentation describing the IRQ domain and IRQ chip
hierarchy that is created for virtual PCI devices, including the
differences on x86 and arm64, and the reasons for those differences.
In the context of the IRQ domain and chip hierarchies, describe how the
CPU selected for effective affinity is spread across CPUs in the affinity
masks. Again, account for the differences in x86 and arm64.
Finally, rework existing text on how MSI messages are composed. Provide
an expanded section on the overall approach that Hyper-V takes to MSI
messages. The section provides more context on the handling of MSI messages
in the Hyper-V vPCI driver, and why it is structured the way it is.
Signed-off-by: Michael Kelley <redacted>
---
<snip>
quoted
+MSI Messages
+------------
+In a typical guest on KVM, the VMM traps guest writes to a vPCI
+device's MSI/MSI-X registers/tables. The guest writes a virutal
+message (with a guest APIC address and a guest vector), and VMM
+emulation translates the message to program the physical device and
+physical device interrupt remapping (IR) tables on the guest's
+behalf. The guest never sees or handles the host-level MSI message.
+
+Hyper-V takes a different approach in which MSI/MSI-X register or
+table writes are not trapped. The guest writes the device's MSI/MSI-X
+registers/tables directly with a host-valid MSI message. The physical
+IOMMU and its remapping table are still owned by the host, so the
+guest must ask the host for the proper message.
+
Could we distinguish MSI-X table writes from MSI configuration-space
writes here?
pci_write_msg_msi() programs the MSI address/data through
pci_write_config_*(), which goes through the Hyper-V config-space
access mechanism. As described in the existing "PCI Device Setup"
and "PCI config space access in a CoCo VM" sections, those accesses
trap to Hyper-V in normal VMs, or use explicit hypercalls in CoCo VMs.
The distinction seems to be that Hyper-V requires the guest to obtain
a host-valid MSI message explicitly over VMBus, rather than relying
on interception of guest MSI programming to establish the interrupt
mapping. That is separate from whether the programming accesses
themselves are trapped. We should probably make that distinction instead
of saying that both MSI and MSI-X writes are untrapped?
Yes, you are right. The current text is imprecise and conflates the MSI
and MSI-X cases. Both cases require the guest to request the MSI
message from the Hyper-V host, but only for MSI-X is the address/data
written to an address that is not trapped. My assumption is that Hyper-V
takes this approach for MSI so that the same mechanism is used for
both MSI and MSI-X, even though in the MSI case the address/data is
written to a PCI config space address that is trapped (or uses a hypercall
in the CoCo VM case).
I'll reword this section to make the distinction clear.
FWIW, the code comment for hv_pci_restore_msi_state() conflates
MSI and MSI-X in the same way. I'll see about a patch to tweak that.
quoted
+The "asking" is done by the .irq_compose_msi_msg function for the
+"Hyper-V PCIe MSI" chip, which is hv_compose_msi_msg(). The
+implementation requires sending a VMBus message to the Hyper-V host
+and awaiting an interrupt indicating receipt of a reply message.
+Since .irq_compose_msi_msg can be called with IRQ spin locks held, it
+doesn't work to do the normal sleep until awakened by the interrupt.
+Instead hv_compose_msi_msg() must send the VMBus message, and
+poll for the completion message. As further complexity, the vPCI
+device could be ejected/rescinded while the polling is in progress,
+so this scenario must be detected as well. See comments in the code
+regarding this very tricky area. A synchronous hypercall to get the
+MSI message would be a much nicer interface, but the code makes do
+with what's available.
+
+Once the host-valid MSI message is obtained, the guest must treat it
+as a black-box that the guest does not interpret or modify. The only
+valid guest action is to write it to the MSI/MSI-X registers/tables
+using the .irq_chip_write_msi_msg function on the HV-PCI-MSIX-<PCIaddr>
+chip. That function is pci_msi_domain_write_msg(), which calls
+__pci_write_msi_msg().
+
+If the affinity of an IRQ changes, the steps must be
+repeated. A new host-valid MSI message is requested via the
+.irq_compose_msi_msg function and that message is written
+to the MSI/MSI-X registers using the .irq_chip_write_msi_msg
+function.
I don't think ordinary affinity changes require a new MSI message,
including with the companion arm64 spreading patch applied.
hv_pcie_init_dev_msi_info() installs irq_chip_set_affinity_parent
instead of the generic compose-and-write affinity handler.
On arm64, the new hv_pci_arm64_set_affinity() selects a CPU and passes
a single-CPU mask to the parent. gic_set_affinity() then updates the
SPI's GICD_IROUTER and effective affinity, without recomposing or
rewriting the device's MSI message.
Similarly, on x86, the new CPU/vector is communicated through the
Hyper-V unmask/retarget path using the existing MSI descriptor.
Could this paragraph describe updating the routing of the existing
message, rather than obtaining and programming a new one?
Indeed, I agree. The .irq_set_affinity function is originally set
to msi_domain_set_affinity() by msi_lib_init_dev_msi_info().
msi_domain_set_affinity() is the compose-and-write affinity
handler you referred to.
But hv_pcie_init_dev_msi_info() overwrites msi_domain_set_affinity()
with irq_chip_set_affinity_parent(). Evidently the mechanisms you
describe above for changing the target guest CPU are sufficient.
Just to be sure, I ran a test on an arm64 VM in Azure with a Mellanox
VF, and indeed there are no calls to hv_compose_msi_msg() when
manually changing the affinity of one of the Mellanox IRQs. I wish
I knew more about what exactly Hyper-V is doing under-the-hood
so could better interpret the Linux-side code!
I'll update this portion of the text as well.
Thanks for the careful review. Let me know if there are other
aspects of this topic that you've figured out the hard way that
should be documented. Hopefully getting this written down in
one place will help someone new coming along in the future.
Michael