Hi Alex,

-----Original Message-----
From: Alex Williamson [mailto:alex.williamson at redhat.com]
Sent: Tuesday, July 28, 2015 9:52 PM
To: Pranavkumar Sawargaonkar
Cc: kvm at vger.kernel.org; kvmarm at lists.cs.columbia.edu; linux-arm-
kernel at lists.infradead.org; linux-kernel at vger.kernel.org;
christoffer.dall at linaro.org; marc.zyngier at arm.com; will.deacon at arm.com;
bhelgaas at google.com; arnd at arndb.de; rob.herring at linaro.org;
eric.auger at linaro.org; patches at apm.com; Bhushan Bharat-R65777; Yoder
Stuart-B08248
Subject: Re: [RFC 0/2] VFIO: Add virtual MSI doorbell support.

On Fri, 2015-07-24 at 14:33 +0530, Pranavkumar Sawargaonkar wrote:

quoted

In current VFIO MSI/MSI-X implementation, linux host kernel allocates
MSI/MSI-X vectors when userspace requests through vfio ioctls.
Vfio creates irqfd mappings to notify MSI/MSI-X interrupts to the
userspace when raised.
Guest OS will see emulated MSI/MSI-X controller and receives an
interrupt when kernel notifies the same via irqfd.

Host kernel allocates MSI/MSI-X using standard linux routines like
pci_enable_msix_range() and pci_enable_msi_range().
These routines along with requset_irq() in host kernel sets up
MSI/MSI-X vectors with Physical MSI/MSI-X addresses provided by
interrupt controller driver in host kernel.

This means when a device is assigned with the guest OS, MSI/MSI-X
addresses present in PCIe EP are the PAs programmed by the host linux

kernel.

quoted

In x86 MSI/MSI-X physical address range is reserved and iommu is aware
about these addreses and transalation is bypassed for these address range.

Unlike x86, ARM/ARM64 does not reserve MSI/MSI-X Physical address
range and all the transactions including MSI go through iommu/smmu

without bypass.

quoted

This requires extending current vfio MSI layer with additional
functionality for ARM/ARM64 by 1. Programing IOVA (referred as a MSI
virtual doorbell address)
   in device's MSI vector as a MSI address.
   This IOVA will be provided by the userspace based on the
   MSI/MSI-X addresses reserved for the guest.
2. Create an IOMMU mapping between this IOVA and
   Physical address (PA) assigned to the MSI vector.

This RFC is proposing a solution for MSI/MSI-X passthrough for

ARM/ARM64.


Hi Pranavkumar,

Freescale has the same, or very similar, need, so any solution in this space
will need to work for both ARM and powerpc.  I'm not a big fan of this
approach as it seems to require the user to configure MSI/X via ioctl and then
call a separate ioctl mapping the doorbells.  That's more code for the user,
more code to get wrong and potentially a gap between configuring MSI/X
and enabling mappings where we could see IOMMU faults.

If we know that doorbell mappings are required, why can't we set aside a
bank of IOVA space and have them mapped automatically as MSI/X is being
configured?  Then the user's need for special knowledge and handling of this
case is limited to setup.  The IOVA space will be mapped and used as needed,
we only need the user to specify the IOVA space reserved for this.  Thanks,

We probably need a mix of both to support Freescale PowerPC and ARM based machines.
In this mix mode kernel vfio driver will reserve some IOVA for mapping MSI page/s. If any other iova mapping will overlap with this then it will return error and user-space. Ideally this should be choosen in such a way that it never overlap, which is easy on some systems but can be tricky on some other system like Freescale PowerPC. This is not sufficient for at-least Freescale PowerPC based SOC. This is because of hardware limitation, where we need to fit this reserved iova address within aperture decided by user-space. So if we allow user-space to change this reserved iova address to a value decided by user-spece itself then we can support both ARM/PowerPC based solutions.

I have some implementation ready/tested with this approach and if this approach looks good then I can submit a RFC patch.

Thanks
-Bharat

Alex