From: Marc Zyngier <hidden> Date: 2014-12-08 20:12:35
MSI-like interrupts are starting to creep out of the PCI world, and
can now be seen into a number of "platform"-type busses. The MSI
domain patches recognise that fact, and start providing a way to
implement this.
Another problem we have to solve is to identify which MSI domain a
device is "connected" to. Currently, PCI gets away with a mixture of
arch-specific callbacks, and a msi_controller structure that can
optionally carry a pointer to an MSI domain. As we add new bus types
and start dealing with topologies that do not map to what PCI does,
this doesn't scale anymore.
This patch series tries to address some of it by providing a basic
link between 'struct device' and an MSI domain. It also adds (yet
another) way for PCI to propagate the domain pointer through the PCI
device hierarchy, provides a method for OF to kickstart the
propagation process, and finally allows the PCI/MSI layer to use that
information. Hopefully this can serve as a model to implement support
for different but types.
Additionally, the last two patches use all the above to remove any
trace of the msi_controller structure from the two GIC interrupt
controllers we use on arm64, so that they solely rely on the above
infrastructure.
This has been tested on arm64 with GICv2m (AMD Seattle) and GICv3 ITS
(FVP model).
Patches are on top of 3.18-rc7 + tip/irq/irq-domain-arm, and available at:
git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git irq/msi_domain
As always, comments most welcome.
M.
Marc Zyngier (6):
device core: Introduce per-device MSI domain pointer
PCI/MSI: add hooks to populate the msi_domain field
PCI/MSI: of: add support for OF-provided msi_domain
PCI/MSI: Let pci_msi_get_domain use struct device's msi_domain
irqchip: GICv2m: Get rid of struct msi_controller
irqchip: gicv3-its: Get rid of struct msi_controller
drivers/irqchip/irq-gic-v2m.c | 26 +++++++++-----------------
drivers/irqchip/irq-gic-v3-its.c | 29 ++++++++++++-----------------
drivers/pci/msi.c | 3 ++-
drivers/pci/of.c | 15 +++++++++++++++
drivers/pci/probe.c | 25 +++++++++++++++++++++++++
include/linux/device.h | 20 ++++++++++++++++++++
include/linux/pci.h | 3 +++
7 files changed, 86 insertions(+), 35 deletions(-)
--
2.1.3
From: Marc Zyngier <hidden> Date: 2014-12-08 20:12:37
Now that we can easily find which MSI domain a PCI device is
using, use dev_get_msi_domain as a way to retrieve the information.
The original code is still used as a fallback.
Signed-off-by: Marc Zyngier <redacted>
---
drivers/pci/msi.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
From: Marc Zyngier <hidden> Date: 2014-12-08 20:12:42
GICv2m only uses the msi_controller structure as a way to match
the PHB with its MSI HW, and thus the msi_domain. But now that
we can directly associate an msi_domain with a device, there is
no use keeping this msi_controller around.
Just remove all traces of msi_controller from the driver.
Signed-off-by: Marc Zyngier <redacted>
---
drivers/irqchip/irq-gic-v2m.c | 26 +++++++++-----------------
1 file changed, 9 insertions(+), 17 deletions(-)
From: Marc Zyngier <hidden> Date: 2014-12-08 20:12:48
The GICv3 ITS only uses the msi_controller structure as a way
to match the PHB with its MSI HW, and thus the msi_domain.
But now that we can directly associate an msi_domain with a device,
there is no use keeping this msi_controller around.
Just remove all traces of msi_controller from the driver.
Signed-off-by: Marc Zyngier <redacted>
---
drivers/irqchip/irq-gic-v3-its.c | 29 ++++++++++++-----------------
1 file changed, 12 insertions(+), 17 deletions(-)
From: Marc Zyngier <hidden> Date: 2014-12-08 20:13:25
In order to populate the PHB msi_domain, use the "msi-parent"
attribute to lookup a corresponding irq domain. If found,
this is our MSI domain.
This gets plugged into the core PCI code.
Signed-off-by: Marc Zyngier <redacted>
---
drivers/pci/of.c | 15 +++++++++++++++
drivers/pci/probe.c | 1 +
include/linux/pci.h | 2 ++
3 files changed, 18 insertions(+)
From: Marc Zyngier <hidden> Date: 2014-12-08 20:13:44
In order to be able to populate the device msi_domain field,
add the necesary hooks to propagate the PHB msi_domain across
secondary busses to devices.
So far, nobody populates the initial msi_domain.
Signed-off-by: Marc Zyngier <redacted>
---
drivers/pci/probe.c | 24 ++++++++++++++++++++++++
include/linux/pci.h | 1 +
2 files changed, 25 insertions(+)
From: Marc Zyngier <hidden> Date: 2014-12-08 20:14:00
As MSI-type features are creeping into non-PCI devices, it is
starting to make sense to give our struct device some form of
support for this, by allowing a pointer to an MSI irq domain to
be set/retrieved.
Signed-off-by: Marc Zyngier <redacted>
---
include/linux/device.h | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)
From: Yijing Wang <hidden> Date: 2014-12-09 02:03:29
On 2014/12/9 4:12, Marc Zyngier wrote:
quoted hunk
In order to be able to populate the device msi_domain field,
add the necesary hooks to propagate the PHB msi_domain across
secondary busses to devices.
So far, nobody populates the initial msi_domain.
Signed-off-by: Marc Zyngier <redacted>
---
drivers/pci/probe.c | 24 ++++++++++++++++++++++++
include/linux/pci.h | 1 +
2 files changed, 25 insertions(+)
Hi Marc, we can not assume pci devices under same phb share the same msi irq domain,
now in x86, pci devices under the same phb may associate different msi irq domain.
@@ -1546,6 +1566,9 @@ void pci_device_add(struct pci_dev *dev, struct pci_bus *bus) /* Initialize various capabilities */ pci_init_capabilities(dev);+ /* Setup MSI irq domain */+ pci_set_msi_domain(dev);+ /* * Add the device to our list of discovered devices * and the bus list for fixup functions, etc.
From: Marc Zyngier <hidden> Date: 2014-12-09 10:02:52
Hi Yijing,
On 09/12/14 02:03, Yijing Wang wrote:
On 2014/12/9 4:12, Marc Zyngier wrote:
quoted
In order to be able to populate the device msi_domain field,
add the necesary hooks to propagate the PHB msi_domain across
secondary busses to devices.
So far, nobody populates the initial msi_domain.
Signed-off-by: Marc Zyngier <redacted>
---
drivers/pci/probe.c | 24 ++++++++++++++++++++++++
include/linux/pci.h | 1 +
2 files changed, 25 insertions(+)
Hi Marc, we can not assume pci devices under same phb share the same msi irq domain,
now in x86, pci devices under the same phb may associate different msi irq domain.
Well, this is not supposed to be a perfect solution yet, but instead a
basis for discussion. What I'd like to find out is:
- What is the minimum granularity for associating a device with its MSI
domain in existing platforms?
- What topology data structures do you use to find out what MSI
controller a device should be matched with?
- What in-tree platform already has this requirements?
Thanks,
M.
--
Jazz is not dead. It just smells funny...
Hi Marc, we can not assume pci devices under same phb share the same msi irq domain,
now in x86, pci devices under the same phb may associate different msi irq domain.
Hi Marc,
Well, this is not supposed to be a perfect solution yet, but instead a
basis for discussion. What I'd like to find out is:
- What is the minimum granularity for associating a device with its MSI
domain in existing platforms?
PCI device, after Gerry's msi irq domain patchset which now in linux-next,
in x86, we will find msi irq domain by pci_dev.
I generally agree your first patch which associate basic device with msi irq domain.
- What topology data structures do you use to find out what MSI
controller a device should be matched with?
Now only arm and arm64 use msi controller to setup/teardown msi irqs,
in arm, now msi controller saved in pci_sys_data, and for arm64, it seems
to be saved in pci_bus. For a more common method to find msi controller/irq domain,
I prefer pci_dev/device.
- What in-tree platform already has this requirements?
Hi Marc, we can not assume pci devices under same phb share the same msi irq domain,
now in x86, pci devices under the same phb may associate different msi irq domain.
Hi Marc,
quoted
Well, this is not supposed to be a perfect solution yet, but instead a
basis for discussion. What I'd like to find out is:
- What is the minimum granularity for associating a device with its MSI
domain in existing platforms?
PCI device, after Gerry's msi irq domain patchset which now in linux-next,
in x86, we will find msi irq domain by pci_dev.
Are you *really* associating the MSI domain on a per pci-device basis?
That is, you have devices on the same PCI bus talking to different MSI hw?
I generally agree your first patch which associate basic device with msi irq domain.
quoted
- What topology data structures do you use to find out what MSI
controller a device should be matched with?
Now only arm and arm64 use msi controller to setup/teardown msi irqs,
in arm, now msi controller saved in pci_sys_data, and for arm64, it seems
to be saved in pci_bus. For a more common method to find msi controller/irq domain,
I prefer pci_dev/device.
Forget about msi_controller, the whole goal of this series is to make it
obsolete. On your x86 platform, what how do you identify which MSI
domain should be associated with a given PCI device? Surely you must
have a set of data structures or ACPI tables which give you that
information.
quoted
- What in-tree platform already has this requirements?
As mentioned above, x86 does.
Let me rephrase that in a non-ambiguous manner: can you point me to a
file implementing this in mainline?
Thanks,
M.
--
Jazz is not dead. It just smells funny...
From: Yijing Wang <hidden> Date: 2014-12-09 12:24:45
quoted
quoted
Well, this is not supposed to be a perfect solution yet, but instead a
basis for discussion. What I'd like to find out is:
- What is the minimum granularity for associating a device with its MSI
domain in existing platforms?
PCI device, after Gerry's msi irq domain patchset which now in linux-next,
in x86, we will find msi irq domain by pci_dev.
Are you *really* associating the MSI domain on a per pci-device basis?
That is, you have devices on the same PCI bus talking to different MSI hw?
Yes.
quoted
I generally agree your first patch which associate basic device with msi irq domain.
quoted
- What topology data structures do you use to find out what MSI
controller a device should be matched with?
Now only arm and arm64 use msi controller to setup/teardown msi irqs,
in arm, now msi controller saved in pci_sys_data, and for arm64, it seems
to be saved in pci_bus. For a more common method to find msi controller/irq domain,
I prefer pci_dev/device.
Forget about msi_controller, the whole goal of this series is to make it
obsolete. On your x86 platform, what how do you identify which MSI
domain should be associated with a given PCI device? Surely you must
have a set of data structures or ACPI tables which give you that
information.
Yes, by ACPI DMAR table.
quoted
quoted
- What in-tree platform already has this requirements?
As mentioned above, x86 does.
Let me rephrase that in a non-ambiguous manner: can you point me to a
file implementing this in mainline?
Please refer to arch/x86/kernel/apic/msi.c native_setup_msi_irqs() in linux-next tree.
Thanks!
Yijing.
Hi Marc, we can not assume pci devices under same phb share the same msi irq domain,
now in x86, pci devices under the same phb may associate different msi irq domain.
Hi Marc,
quoted
Well, this is not supposed to be a perfect solution yet, but instead a
basis for discussion. What I'd like to find out is:
- What is the minimum granularity for associating a device with its MSI
domain in existing platforms?
PCI device, after Gerry's msi irq domain patchset which now in linux-next,
in x86, we will find msi irq domain by pci_dev.
Are you *really* associating the MSI domain on a per pci-device basis?
That is, you have devices on the same PCI bus talking to different MSI hw?
Hi Marc,
This is a little wild:(
On x86 platform with Intel VT-d(not the case for AMD-v),
interrupt remapping is tight to DMA remapping (IOMMU) unit.
For most common cases, IOMMU unit manages PCI bus and its sub-hierarchy.
But it may also manage a specific PCI device. This is typically used to
provide QoS for audio device by using dedicated IOMMU unit to avoid
resource contention on DMA remapping tables. BIOS uses ACPI table to
report PCI bus/device to IOMMU unit mapping relationship. (To be honest,
I have no really experience with such a hardware platform yet, just for
theoretical analysis)
On the other hand, we now support hierarchy irqdomain. So to
support per-PCI IOMMU unit case, we need maintain irqdomain at PCI
device level.
This piece of code from your [4/6] is flexible enough, which
retrieves msi_domain from PCI device, then fallback to PCI bus,
then fallback to platform specific method.
domain = dev_get_msi_domain(&dev->dev);
if (!domain && dev->bus->msi)
domain = dev->bus->msi->domain;
if (!domain)
domain = arch_get_pci_msi_domain(dev);
Thanks!
Gerry
quoted
I generally agree your first patch which associate basic device with msi irq domain.
quoted
- What topology data structures do you use to find out what MSI
controller a device should be matched with?
Now only arm and arm64 use msi controller to setup/teardown msi irqs,
in arm, now msi controller saved in pci_sys_data, and for arm64, it seems
to be saved in pci_bus. For a more common method to find msi controller/irq domain,
I prefer pci_dev/device.
Forget about msi_controller, the whole goal of this series is to make it
obsolete. On your x86 platform, what how do you identify which MSI
domain should be associated with a given PCI device? Surely you must
have a set of data structures or ACPI tables which give you that
information.
quoted
quoted
- What in-tree platform already has this requirements?
As mentioned above, x86 does.
Let me rephrase that in a non-ambiguous manner: can you point me to a
file implementing this in mainline?
Thanks,
M.
From: Jiang Liu <hidden> Date: 2014-12-09 12:58:04
On 2014/12/9 4:12, Marc Zyngier wrote:
quoted hunk
In order to be able to populate the device msi_domain field,
add the necesary hooks to propagate the PHB msi_domain across
secondary busses to devices.
So far, nobody populates the initial msi_domain.
Signed-off-by: Marc Zyngier <redacted>
---
drivers/pci/probe.c | 24 ++++++++++++++++++++++++
include/linux/pci.h | 1 +
2 files changed, 25 insertions(+)
Hi Marc,
The default implementation of pci_set_msi_domain() conflicts
with interrupt mapping based on x86 Intel VT-d:(
If pci_set_msi_domain() is weak, thing gets perfect.
Regards!
Gerry
quoted hunk
void pci_device_add(struct pci_dev *dev, struct pci_bus *bus)
{
int ret;
@@ -1546,6 +1566,9 @@ void pci_device_add(struct pci_dev *dev, struct pci_bus *bus) /* Initialize various capabilities */ pci_init_capabilities(dev);+ /* Setup MSI irq domain */+ pci_set_msi_domain(dev);+ /* * Add the device to our list of discovered devices * and the bus list for fixup functions, etc.
Hi Marc, we can not assume pci devices under same phb share the same msi irq domain,
now in x86, pci devices under the same phb may associate different msi irq domain.
Hi Marc,
quoted
Well, this is not supposed to be a perfect solution yet, but instead a
basis for discussion. What I'd like to find out is:
- What is the minimum granularity for associating a device with its MSI
domain in existing platforms?
PCI device, after Gerry's msi irq domain patchset which now in linux-next,
in x86, we will find msi irq domain by pci_dev.
Are you *really* associating the MSI domain on a per pci-device basis?
That is, you have devices on the same PCI bus talking to different MSI hw?
Hi Marc,
This is a little wild:(
On x86 platform with Intel VT-d(not the case for AMD-v),
interrupt remapping is tight to DMA remapping (IOMMU) unit.
For most common cases, IOMMU unit manages PCI bus and its sub-hierarchy.
But it may also manage a specific PCI device. This is typically used to
provide QoS for audio device by using dedicated IOMMU unit to avoid
resource contention on DMA remapping tables. BIOS uses ACPI table to
report PCI bus/device to IOMMU unit mapping relationship. (To be honest,
I have no really experience with such a hardware platform yet, just for
theoretical analysis)
On the other hand, we now support hierarchy irqdomain. So to
support per-PCI IOMMU unit case, we need maintain irqdomain at PCI
device level.
This piece of code from your [4/6] is flexible enough, which
retrieves msi_domain from PCI device, then fallback to PCI bus,
then fallback to platform specific method.
domain = dev_get_msi_domain(&dev->dev);
if (!domain && dev->bus->msi)
domain = dev->bus->msi->domain;
if (!domain)
domain = arch_get_pci_msi_domain(dev);
OK. But what I'd really like to see is a way to setup the
device<->domain binding as early as possible, without having to use more
conditional code in pci_msi_get_domain.
IOW, can we do something similar to what pci_set_bus_msi_domain and
pci_set_msi_domain do in this patch?
Thanks,
M.
--
Jazz is not dead. It just smells funny...
Hi Marc, we can not assume pci devices under same phb share the same msi irq domain,
now in x86, pci devices under the same phb may associate different msi irq domain.
Hi Marc,
quoted
Well, this is not supposed to be a perfect solution yet, but instead a
basis for discussion. What I'd like to find out is:
- What is the minimum granularity for associating a device with its MSI
domain in existing platforms?
PCI device, after Gerry's msi irq domain patchset which now in linux-next,
in x86, we will find msi irq domain by pci_dev.
Are you *really* associating the MSI domain on a per pci-device basis?
That is, you have devices on the same PCI bus talking to different MSI hw?
Hi Marc,
This is a little wild:(
On x86 platform with Intel VT-d(not the case for AMD-v),
interrupt remapping is tight to DMA remapping (IOMMU) unit.
For most common cases, IOMMU unit manages PCI bus and its sub-hierarchy.
But it may also manage a specific PCI device. This is typically used to
provide QoS for audio device by using dedicated IOMMU unit to avoid
resource contention on DMA remapping tables. BIOS uses ACPI table to
report PCI bus/device to IOMMU unit mapping relationship. (To be honest,
I have no really experience with such a hardware platform yet, just for
theoretical analysis)
On the other hand, we now support hierarchy irqdomain. So to
support per-PCI IOMMU unit case, we need maintain irqdomain at PCI
device level.
This piece of code from your [4/6] is flexible enough, which
retrieves msi_domain from PCI device, then fallback to PCI bus,
then fallback to platform specific method.
domain = dev_get_msi_domain(&dev->dev);
if (!domain && dev->bus->msi)
domain = dev->bus->msi->domain;
if (!domain)
domain = arch_get_pci_msi_domain(dev);
OK. But what I'd really like to see is a way to setup the
device<->domain binding as early as possible, without having to use more
conditional code in pci_msi_get_domain.
IOW, can we do something similar to what pci_set_bus_msi_domain and
pci_set_msi_domain do in this patch?
Hi Marc,
I have checked x86 code, we could set pci_dev->msi_domain
when creating PCI devices, just need to find some hook points
into PCI core next step. If arch code doesn't set pci_dev->msi_domain,
PCI MSI core may provide a default way to set pci_dev->msi_domain.
This may make the implementation simpler, I guess:)
Thanks!
Gerry
Hi Marc, we can not assume pci devices under same phb share the same msi irq domain,
now in x86, pci devices under the same phb may associate different msi irq domain.
Hi Marc,
quoted
Well, this is not supposed to be a perfect solution yet, but instead a
basis for discussion. What I'd like to find out is:
- What is the minimum granularity for associating a device with its MSI
domain in existing platforms?
PCI device, after Gerry's msi irq domain patchset which now in linux-next,
in x86, we will find msi irq domain by pci_dev.
Are you *really* associating the MSI domain on a per pci-device basis?
That is, you have devices on the same PCI bus talking to different MSI hw?
Hi Marc,
This is a little wild:(
On x86 platform with Intel VT-d(not the case for AMD-v),
interrupt remapping is tight to DMA remapping (IOMMU) unit.
For most common cases, IOMMU unit manages PCI bus and its sub-hierarchy.
But it may also manage a specific PCI device. This is typically used to
provide QoS for audio device by using dedicated IOMMU unit to avoid
resource contention on DMA remapping tables. BIOS uses ACPI table to
report PCI bus/device to IOMMU unit mapping relationship. (To be honest,
I have no really experience with such a hardware platform yet, just for
theoretical analysis)
On the other hand, we now support hierarchy irqdomain. So to
support per-PCI IOMMU unit case, we need maintain irqdomain at PCI
device level.
This piece of code from your [4/6] is flexible enough, which
retrieves msi_domain from PCI device, then fallback to PCI bus,
then fallback to platform specific method.
domain = dev_get_msi_domain(&dev->dev);
if (!domain && dev->bus->msi)
domain = dev->bus->msi->domain;
if (!domain)
domain = arch_get_pci_msi_domain(dev);
OK. But what I'd really like to see is a way to setup the
device<->domain binding as early as possible, without having to use more
conditional code in pci_msi_get_domain.
IOW, can we do something similar to what pci_set_bus_msi_domain and
pci_set_msi_domain do in this patch?
Hi Marc,
I have checked x86 code, we could set pci_dev->msi_domain
when creating PCI devices, just need to find some hook points
into PCI core next step. If arch code doesn't set pci_dev->msi_domain,
PCI MSI core may provide a default way to set pci_dev->msi_domain.
This may make the implementation simpler, I guess:)
Right. So following your earlier suggestion, I could make
pci_set_msi_domain a weak symbol and let arch code override this.
My preference would have been to have arch code to create a set of
arch-independent data structures describing the topology, and use that
for everything, but maybe that's a bit ambitious for a start.
I'll rework the series to make the symbols weak.
Thanks,
M.
--
Jazz is not dead. It just smells funny...
Hi Marc, we can not assume pci devices under same phb share the same msi irq domain,
now in x86, pci devices under the same phb may associate different msi irq domain.
Hi Marc,
quoted
Well, this is not supposed to be a perfect solution yet, but instead a
basis for discussion. What I'd like to find out is:
- What is the minimum granularity for associating a device with its MSI
domain in existing platforms?
PCI device, after Gerry's msi irq domain patchset which now in linux-next,
in x86, we will find msi irq domain by pci_dev.
Are you *really* associating the MSI domain on a per pci-device basis?
That is, you have devices on the same PCI bus talking to different MSI hw?
Hi Marc,
This is a little wild:(
On x86 platform with Intel VT-d(not the case for AMD-v),
interrupt remapping is tight to DMA remapping (IOMMU) unit.
For most common cases, IOMMU unit manages PCI bus and its sub-hierarchy.
But it may also manage a specific PCI device. This is typically used to
provide QoS for audio device by using dedicated IOMMU unit to avoid
resource contention on DMA remapping tables. BIOS uses ACPI table to
report PCI bus/device to IOMMU unit mapping relationship. (To be honest,
I have no really experience with such a hardware platform yet, just for
theoretical analysis)
On the other hand, we now support hierarchy irqdomain. So to
support per-PCI IOMMU unit case, we need maintain irqdomain at PCI
device level.
This piece of code from your [4/6] is flexible enough, which
retrieves msi_domain from PCI device, then fallback to PCI bus,
then fallback to platform specific method.
domain = dev_get_msi_domain(&dev->dev);
if (!domain && dev->bus->msi)
domain = dev->bus->msi->domain;
if (!domain)
domain = arch_get_pci_msi_domain(dev);
OK. But what I'd really like to see is a way to setup the
device<->domain binding as early as possible, without having to use more
conditional code in pci_msi_get_domain.
IOW, can we do something similar to what pci_set_bus_msi_domain and
pci_set_msi_domain do in this patch?
Hi Marc,
I have checked x86 code, we could set pci_dev->msi_domain
when creating PCI devices, just need to find some hook points
into PCI core next step. If arch code doesn't set pci_dev->msi_domain,
PCI MSI core may provide a default way to set pci_dev->msi_domain.
This may make the implementation simpler, I guess:)
Right. So following your earlier suggestion, I could make
pci_set_msi_domain a weak symbol and let arch code override this.
My preference would have been to have arch code to create a set of
arch-independent data structures describing the topology, and use that
for everything, but maybe that's a bit ambitious for a start.
I'll rework the series to make the symbols weak.
Hi Marc,
I think we may not need the weak symbol at all. With following
draft patch, the PCI MSI core may simply do:
if (pci_dev->dev.msi_domain == NULL)
dev_set_msi_domain(&dev->dev,
dev_get_msi_domain(&dev->bus->dev));
-----------------------------------------------------------------------
Note: the patch won't pass compilation, just to show the key idea:)
Hi Marc, we can not assume pci devices under same phb share the same msi irq domain,
now in x86, pci devices under the same phb may associate different msi irq domain.
Hi Marc,
quoted
Well, this is not supposed to be a perfect solution yet, but instead a
basis for discussion. What I'd like to find out is:
- What is the minimum granularity for associating a device with its MSI
domain in existing platforms?
PCI device, after Gerry's msi irq domain patchset which now in linux-next,
in x86, we will find msi irq domain by pci_dev.
Are you *really* associating the MSI domain on a per pci-device basis?
That is, you have devices on the same PCI bus talking to different MSI hw?
Hi Marc,
This is a little wild:(
On x86 platform with Intel VT-d(not the case for AMD-v),
interrupt remapping is tight to DMA remapping (IOMMU) unit.
For most common cases, IOMMU unit manages PCI bus and its sub-hierarchy.
But it may also manage a specific PCI device. This is typically used to
provide QoS for audio device by using dedicated IOMMU unit to avoid
resource contention on DMA remapping tables. BIOS uses ACPI table to
report PCI bus/device to IOMMU unit mapping relationship. (To be honest,
I have no really experience with such a hardware platform yet, just for
theoretical analysis)
On the other hand, we now support hierarchy irqdomain. So to
support per-PCI IOMMU unit case, we need maintain irqdomain at PCI
device level.
This piece of code from your [4/6] is flexible enough, which
retrieves msi_domain from PCI device, then fallback to PCI bus,
then fallback to platform specific method.
domain = dev_get_msi_domain(&dev->dev);
if (!domain && dev->bus->msi)
domain = dev->bus->msi->domain;
if (!domain)
domain = arch_get_pci_msi_domain(dev);
OK. But what I'd really like to see is a way to setup the
device<->domain binding as early as possible, without having to use more
conditional code in pci_msi_get_domain.
IOW, can we do something similar to what pci_set_bus_msi_domain and
pci_set_msi_domain do in this patch?
Hi Marc,
I have checked x86 code, we could set pci_dev->msi_domain
when creating PCI devices, just need to find some hook points
into PCI core next step. If arch code doesn't set pci_dev->msi_domain,
PCI MSI core may provide a default way to set pci_dev->msi_domain.
This may make the implementation simpler, I guess:)
Right. So following your earlier suggestion, I could make
pci_set_msi_domain a weak symbol and let arch code override this.
My preference would have been to have arch code to create a set of
arch-independent data structures describing the topology, and use that
for everything, but maybe that's a bit ambitious for a start.
I'll rework the series to make the symbols weak.
Hi Marc,
I think we may not need the weak symbol at all. With following
draft patch, the PCI MSI core may simply do:
if (pci_dev->dev.msi_domain == NULL)
dev_set_msi_domain(&dev->dev,
dev_get_msi_domain(&dev->bus->dev));
-----------------------------------------------------------------------
Note: the patch won't pass compilation, just to show the key idea:)
@@ -652,6 +652,9 @@ int pcibios_add_device(struct pci_dev *dev)pa_data=data->next;iounmap(data);}++dev->dev.msi_domain=x86_get_pci_msi_domain(dev);+return0;}
Right. So you set the msi_domain using the pcibios_add_device callback.
That will require some minimal surgery (the call to pci_set_msi_domain
happens before the pcibios call, so it needs to be relocated after), but
that seems like a sensible solution to me.
Thanks!
M.
--
Jazz is not dead. It just smells funny...
Hi Marc, we can not assume pci devices under same phb share the same msi irq domain,
now in x86, pci devices under the same phb may associate different msi irq domain.
Hi Marc,
quoted
Well, this is not supposed to be a perfect solution yet, but instead a
basis for discussion. What I'd like to find out is:
- What is the minimum granularity for associating a device with its MSI
domain in existing platforms?
PCI device, after Gerry's msi irq domain patchset which now in linux-next,
in x86, we will find msi irq domain by pci_dev.
Are you *really* associating the MSI domain on a per pci-device basis?
That is, you have devices on the same PCI bus talking to different MSI hw?
Hi Marc,
This is a little wild:(
On x86 platform with Intel VT-d(not the case for AMD-v),
interrupt remapping is tight to DMA remapping (IOMMU) unit.
For most common cases, IOMMU unit manages PCI bus and its sub-hierarchy.
But it may also manage a specific PCI device. This is typically used to
provide QoS for audio device by using dedicated IOMMU unit to avoid
resource contention on DMA remapping tables. BIOS uses ACPI table to
report PCI bus/device to IOMMU unit mapping relationship. (To be honest,
I have no really experience with such a hardware platform yet, just for
theoretical analysis)
On the other hand, we now support hierarchy irqdomain. So to
support per-PCI IOMMU unit case, we need maintain irqdomain at PCI
device level.
This piece of code from your [4/6] is flexible enough, which
retrieves msi_domain from PCI device, then fallback to PCI bus,
then fallback to platform specific method.
domain = dev_get_msi_domain(&dev->dev);
if (!domain && dev->bus->msi)
domain = dev->bus->msi->domain;
if (!domain)
domain = arch_get_pci_msi_domain(dev);
OK. But what I'd really like to see is a way to setup the
device<->domain binding as early as possible, without having to use more
conditional code in pci_msi_get_domain.
IOW, can we do something similar to what pci_set_bus_msi_domain and
pci_set_msi_domain do in this patch?
Hi Marc,
I have checked x86 code, we could set pci_dev->msi_domain
when creating PCI devices, just need to find some hook points
into PCI core next step. If arch code doesn't set pci_dev->msi_domain,
PCI MSI core may provide a default way to set pci_dev->msi_domain.
This may make the implementation simpler, I guess:)
Right. So following your earlier suggestion, I could make
pci_set_msi_domain a weak symbol and let arch code override this.
My preference would have been to have arch code to create a set of
arch-independent data structures describing the topology, and use that
for everything, but maybe that's a bit ambitious for a start.
I'll rework the series to make the symbols weak.
Hi Marc,
I think we may not need the weak symbol at all. With following
draft patch, the PCI MSI core may simply do:
if (pci_dev->dev.msi_domain == NULL)
dev_set_msi_domain(&dev->dev,
dev_get_msi_domain(&dev->bus->dev));
-----------------------------------------------------------------------
Note: the patch won't pass compilation, just to show the key idea:)
@@ -652,6 +652,9 @@ int pcibios_add_device(struct pci_dev *dev)pa_data=data->next;iounmap(data);}++dev->dev.msi_domain=x86_get_pci_msi_domain(dev);+return0;}
Right. So you set the msi_domain using the pcibios_add_device callback.
That will require some minimal surgery (the call to pci_set_msi_domain
happens before the pcibios call, so it needs to be relocated after), but
that seems like a sensible solution to me.
So the key point is clear now:
The PCI MSI core will try to set a default value for
pci_dev->dev.msi_domain if the arch code doesn't do that.
Seems like a solution:)
On Mon, Dec 08, 2014 at 08:12:19PM +0000, Marc Zyngier wrote:
quoted hunk
In order to be able to populate the device msi_domain field,
add the necesary hooks to propagate the PHB msi_domain across
secondary busses to devices.
So far, nobody populates the initial msi_domain.
Signed-off-by: Marc Zyngier <redacted>
---
drivers/pci/probe.c | 24 ++++++++++++++++++++++++
include/linux/pci.h | 1 +
2 files changed, 25 insertions(+)
We haven't previously used the "phb" abbreviation in the PCI core, so I'd
rather not start now. I'm not really sure how relevant it is anyway --
maybe the name doesn't need to include a reference to the host bridge at
all?
This is *really* a nitpick, but "setup" suggests that we're doing more than
just setting a pointer, i.e., it suggests that we're doing something more
involved to get it all ready for use. But here, I think we really are just
setting a pointer, so I think the comment is pointless since the function
name already says that.
quoted hunk
+ pci_set_msi_domain(dev);
+
/*
* Add the device to our list of discovered devices
* and the bus list for fixup functions, etc.
On Mon, Dec 08, 2014 at 08:12:20PM +0000, Marc Zyngier wrote:
quoted hunk
In order to populate the PHB msi_domain, use the "msi-parent"
attribute to lookup a corresponding irq domain. If found,
this is our MSI domain.
This gets plugged into the core PCI code.
Signed-off-by: Marc Zyngier <redacted>
---
drivers/pci/of.c | 15 +++++++++++++++
drivers/pci/probe.c | 1 +
include/linux/pci.h | 2 ++
3 files changed, 18 insertions(+)
On Mon, Jan 26, 2015 at 06:40:05PM -0600, Bjorn Helgaas wrote:
On Mon, Dec 08, 2014 at 08:12:19PM +0000, Marc Zyngier wrote:
quoted
In order to be able to populate the device msi_domain field,
add the necesary hooks to propagate the PHB msi_domain across
secondary busses to devices.
So far, nobody populates the initial msi_domain.
Signed-off-by: Marc Zyngier <redacted>
---
drivers/pci/probe.c | 24 ++++++++++++++++++++++++
include/linux/pci.h | 1 +
2 files changed, 25 insertions(+)
We haven't previously used the "phb" abbreviation in the PCI core, so I'd
rather not start now. I'm not really sure how relevant it is anyway --
maybe the name doesn't need to include a reference to the host bridge at
all?
Sorry, I meant to reply to the v2 patch, not the old one. But I think the
comment still applies there.
This is *really* a nitpick, but "setup" suggests that we're doing more than
just setting a pointer, i.e., it suggests that we're doing something more
involved to get it all ready for use. But here, I think we really are just
setting a pointer, so I think the comment is pointless since the function
name already says that.
quoted
+ pci_set_msi_domain(dev);
+
/*
* Add the device to our list of discovered devices
* and the bus list for fixup functions, etc.