On Thu, Aug 30, 2018 at 7:23 PM Arnd Bergmann [off-list ref] wrote:

On Tue, Aug 28, 2018 at 3:17 PM Sunil Kovvuri [off-list ref] wrote:

quoted

On Tue, Aug 28, 2018 at 6:27 PM Arnd Bergmann [off-list ref] wrote:

quoted

On Tue, Aug 28, 2018 at 2:42 PM Sunil Kovvuri [off-list ref] wrote:

quoted

On Tue, Aug 28, 2018 at 5:39 PM Arnd Bergmann [off-list ref] wrote:

quoted

On Tue, Aug 28, 2018 at 12:58 PM [off-list ref] wrote:

This admin function is a PCI device which is capable of provisioning
HW blocks to other PCIe SRIOV devices in the system. Each HW block
(eg memory buffer pools, NIC dewscriptors, crypto engines e.t.c) needs
certain no of MSIX vectors. Admin function has a set of 32K MSIX vectors
in memory (not on-chip) which based on HW block provisioning to a PCI device
attaches the required number of vectors to that device. Some part of this
configuration is done by low level firmware.

RVU_AF_MSIXTR_BASE points to the memory region allocated for 32K MSIX
vectors. If kernel is booted with IOMMU enabled and admin function device
is attached to SMMU, HW will go through translation to access this MSIX
vector memory region. Hence the mapping done in this patch.

Do you mean this is not a regular PCIe MSI-X interrupt to the GIC, but
something internal to your device that gets routed through the IOMMU
back into the device?

This is a regular PCIe MSI-X interrupt, the difference is that the
bunch of PCI devices
here doesn't have a fixed set of MSIX vectors. Admin function has a
memory region with
32K MSIX vectors which it provisions to PCI devices based on the HW
functional blocks
attached to them. A PCI device which works as a ethernet device needs
X number of vectors
and a crypto device needs Y number of vectors.

Since the admin function owns the whole MSIX vector region, HW uses
this device's stream ID
to access the vectors. Hence the IOMMU mapping. Once MSIX vectors are
provisioned to
a PCI device they work as normal MSIX interrupt like any other device.

Ok, I think I got it now, just to confirm: the MSIX vectors you allocate
in the admin device refer to memory backing the BAR that contains
the MSI-X entries of the other functions, right?

Yes, that's correct.

I was a bit confused here and assumed that you were mapping
the MMIO area of an interrupt controller that receives the interupt
transactions.

         Arnd