-----Original Message-----
From: Christoffer Dall [mailto:christoffer.dall at linaro.org]
Sent: Friday, September 25, 2015 10:42 PM
To: Marc Zyngier <redacted>
Cc: Bhushan Bharat-R65777 <redacted>; Pranavkumar
Sawargaonkar [off-list ref]; kvm at vger.kernel.org; Alex
Williamson [off-list ref]; kvmarm at lists.cs.columbia.edu;
linux-arm-kernel at lists.infradead.org; linux-kernel at vger.kernel.org; Will
Deacon [off-list ref]; bhelgaas at google.com; arnd at arndb.de;
rob.herring at linaro.org; eric.auger at linaro.org; patches at apm.com; Yoder
Stuart-B08248 [off-list ref]
Subject: Re: [RFC 0/2] VFIO: Add virtual MSI doorbell support.

On Tue, Sep 22, 2015 at 11:09:14PM +0100, Marc Zyngier wrote:

quoted

On Tue, 4 Aug 2015 06:52:01 +0100
Bhushan Bharat [off-list ref] wrote:

quoted

quoted

-----Original Message-----
From: Pranavkumar Sawargaonkar [mailto:pranavkumar at linaro.org]
Sent: Tuesday, August 04, 2015 11:18 AM
To: Bhushan Bharat-R65777
Cc: kvm at vger.kernel.org; Alex Williamson;
kvmarm at lists.cs.columbia.edu;
linux-arm-kernel at lists.infradead.org;
linux-kernel at vger.kernel.org; christoffer.dall at linaro.org;
marc.zyngier at arm.com; will.deacon at arm.com; bhelgaas at google.com;
arnd at arndb.de; rob.herring at linaro.org; eric.auger at linaro.org;
patches at apm.com; Yoder Stuart-B08248
Subject: Re: [RFC 0/2] VFIO: Add virtual MSI doorbell support.

Hi Bharat,

On 28 July 2015 at 23:28, Alex Williamson
[off-list ref]
wrote:

quoted

On Tue, 2015-07-28 at 17:23 +0000, Bhushan Bharat wrote:

quoted

Hi Alex,

quoted

-----Original Message-----
From: Alex Williamson [mailto:alex.williamson at redhat.com]
Sent: Tuesday, July 28, 2015 9:52 PM
To: Pranavkumar Sawargaonkar
Cc: kvm at vger.kernel.org; kvmarm at lists.cs.columbia.edu;
linux-arm- kernel at lists.infradead.org;
linux-kernel at vger.kernel.org; christoffer.dall at linaro.org;
marc.zyngier at arm.com; will.deacon at arm.com;
bhelgaas at google.com; arnd at arndb.de; rob.herring at linaro.org;
eric.auger at linaro.org; patches at apm.com; Bhushan
Bharat-R65777; Yoder
Stuart-B08248
Subject: Re: [RFC 0/2] VFIO: Add virtual MSI doorbell support.

On Fri, 2015-07-24 at 14:33 +0530, Pranavkumar Sawargaonkar

wrote:

quoted

quoted

quoted

quoted

quoted

quoted

quoted

In current VFIO MSI/MSI-X implementation, linux host kernel
allocates MSI/MSI-X vectors when userspace requests through
vfio

ioctls.

quoted

quoted

quoted

quoted

Vfio creates irqfd mappings to notify MSI/MSI-X interrupts
to the userspace when raised.
Guest OS will see emulated MSI/MSI-X controller and
receives an interrupt when kernel notifies the same via irqfd.

Host kernel allocates MSI/MSI-X using standard linux
routines like
pci_enable_msix_range() and pci_enable_msi_range().
These routines along with requset_irq() in host kernel sets
up MSI/MSI-X vectors with Physical MSI/MSI-X addresses
provided by interrupt controller driver in host kernel.

This means when a device is assigned with the guest OS,
MSI/MSI-X addresses present in PCIe EP are the PAs
programmed by the host linux

kernel.

quoted

In x86 MSI/MSI-X physical address range is reserved and
iommu is aware about these addreses and transalation is
bypassed for these

address range.

quoted

quoted

quoted

quoted

Unlike x86, ARM/ARM64 does not reserve MSI/MSI-X Physical
address range and all the transactions including MSI go
through iommu/smmu

without bypass.

quoted

This requires extending current vfio MSI layer with
additional functionality for ARM/ARM64 by 1. Programing
IOVA (referred as a MSI virtual doorbell address)
   in device's MSI vector as a MSI address.
   This IOVA will be provided by the userspace based on the
   MSI/MSI-X addresses reserved for the guest.
2. Create an IOMMU mapping between this IOVA and
   Physical address (PA) assigned to the MSI vector.

This RFC is proposing a solution for MSI/MSI-X passthrough
for

ARM/ARM64.


Hi Pranavkumar,

Freescale has the same, or very similar, need, so any
solution in this space will need to work for both ARM and
powerpc.  I'm not a big fan of this approach as it seems to
require the user to configure MSI/X via ioctl and then call a
separate ioctl mapping the doorbells.  That's more code for
the user, more code to get wrong and potentially a gap
between configuring MSI/X and enabling

mappings where we could see IOMMU faults.

quoted

quoted

quoted

If we know that doorbell mappings are required, why can't we
set aside a bank of IOVA space and have them mapped
automatically as MSI/X is being configured?  Then the user's
need for special knowledge and handling of this case is
limited to setup.  The IOVA space will be mapped and used as
needed, we only need the user to specify the IOVA space
reserved for this.  Thanks,

We probably need a mix of both to support Freescale PowerPC and
ARM based machines.
In this mix mode kernel vfio driver will reserve some IOVA for
mapping MSI page/s.

If vfio is reserving pages independently from the user, this
becomes what Marc called "shaping" the VM and what x86
effectively does.  An interface extension should expose these
implicit regions so the user can avoid them for DMA memory

mapping.

quoted

quoted

quoted

quoted

quoted

 If any other iova mapping will overlap with this then it will
return error and user-space. Ideally this should be choosen in
such a way that it never overlap, which is easy on some systems
but can be tricky on some other system like Freescale PowerPC.
This is not sufficient for at-least Freescale PowerPC based
SOC. This is because of hardware limitation, where we need to
fit this reserved iova address within aperture decided by
user-space. So if we allow user-space to change this reserved
iova address to a value decided by user-spece itself then we
can support both ARM/PowerPC based

solutions.

quoted

Yes, that's my intention, to allow userspace to specify the
reserved region.  I believe you have some additional
restrictions on the number of MSI banks available and whether
MSI banks can be shared, but I would hope that doesn't preclude a

shared interface with ARM.

quoted

quoted

quoted

quoted

quoted

I have some implementation ready/tested with this approach and
if this approach looks good then I can submit a RFC patch.

Yes, please post.  Thanks,

Could you please share a tentative timeline by which you will be
posting your patches ?

I have not touched that code for a while, I am planning to send the
patch in couple of weeks.

Have we made any progress on this subject? It looks like a lot of time
has passed, but I haven't seen anything. Did I miss it?

Pranav is going to respin his series, because we are clarly not making progress
on this front.

Sorry for long long delay, some unplanned leaves delays this work.
Finally I sent of the RFC patches, those patches are not tested with PCI devices but tested with some freescale specific platform/devices.

Thanks
-Bharat

Thanks,
-Christoffer