From: Alex Williamson <hidden> Date: 2011-08-23 19:30:53
On Tue, 2011-08-23 at 07:01 +1000, Benjamin Herrenschmidt wrote:
On Mon, 2011-08-22 at 09:45 -0600, Alex Williamson wrote:
quoted
Yes, that's the idea. An open question I have towards the configuration
side is whether we might add iommu driver specific options to the
groups. For instance on x86 where we typically have B:D.F granularity,
should we have an option not to trust multi-function devices and use a
B:D granularity for grouping?
Or even B or range of busses... if you want to enforce strict isolation
you really can't trust anything below a bus level :-)
quoted
Right, we can also combine models. Binding a device to vfio
creates /dev/vfio$GROUP, which only allows a subset of ioctls and no
device access until all the group devices are also bound. I think
the /dev/vfio/$GROUP might help provide an enumeration interface as well
though, which could be useful.
Could be tho in what form ? returning sysfs pathes ?
I'm at a loss there, please suggest. I think we need an ioctl that
returns some kind of array of devices within the group and another that
maybe takes an index from that array and returns an fd for that device.
A sysfs path string might be a reasonable array element, but it sounds
like a pain to work with.
quoted
1:1 group<->process is probably too strong. Not allowing concurrent
open()s on the group file enforces a single userspace entity is
responsible for that group. Device fds can be passed to other
processes, but only retrieved via the group fd. I suppose we could even
branch off the dma interface into a different fd, but it seems like we
would logically want to serialize dma mappings at each iommu group
anyway. I'm open to alternatives, this just seemed an easy way to do
it. Restricting on UID implies that we require isolated qemu instances
to run as different UIDs. I know that's a goal, but I don't know if we
want to make it an assumption in the group security model.
1:1 process has the advantage of linking to an -mm which makes the whole
mmu notifier business doable. How do you want to track down mappings and
do the second level translation in the case of explicit map/unmap (like
on power) if you are not tied to an mm_struct ?
Right, I threw away the mmu notifier code that was originally part of
vfio because we can't do anything useful with it yet on x86. I
definitely don't want to prevent it where it makes sense though. Maybe
we just record current->mm on open and restrict subsequent opens to the
same.
quoted
Yes. I'm not sure there's a good ROI to prioritize that model. We have
to assume >1 device per guest is a typical model and that the iotlb is
large enough that we might improve thrashing to see both a resource and
performance benefit from it. I'm open to suggestions for how we could
include it though.
Sharing may or may not be possible depending on setups so yes, it's a
bit tricky.
My preference is to have a static interface (and that's actually where
your pet netlink might make some sense :-) to create "synthetic" groups
made of other groups if the arch allows it. But that might not be the
best approach. In another email I also proposed an option for a group to
"capture" another one...
I already made some comments on this in a different thread, so I won't
repeat here.
quoted
quoted
If that's
not what you're saying, how would the domains - now made up of a
user's selection of groups, rather than individual devices - be
configured?
quoted
Hope that captures it, feel free to jump in with corrections and
suggestions. Thanks,
Another aspect I don't see discussed is how we represent these things to
the guest.
On Power for example, I have a requirement that a given iommu domain is
represented by a single dma window property in the device-tree. What
that means is that that property needs to be either in the node of the
device itself if there's only one device in the group or in a parent
node (ie a bridge or host bridge) if there are multiple devices.
Now I do -not- want to go down the path of simulating P2P bridges,
besides we'll quickly run out of bus numbers if we go there.
For us the most simple and logical approach (which is also what pHyp
uses and what Linux handles well) is really to expose a given PCI host
bridge per group to the guest. Believe it or not, it makes things
easier :-)
I'm all for easier. Why does exposing the bridge use less bus numbers
than emulating a bridge?
On x86, I want to maintain that our default assignment is at the device
level. A user should be able to pick single or multiple devices from
across several groups and have them all show up as individual,
hotpluggable devices on bus 0 in the guest. Not surprisingly, we've
also seen cases where users try to attach a bridge to the guest,
assuming they'll get all the devices below the bridge, so I'd be in
favor of making this "just work" if possible too, though we may have to
prevent hotplug of those.
Given the device requirement on x86 and since everything is a PCI device
on x86, I'd like to keep a qemu command line something like -device
vfio,host=00:19.0. I assume that some of the iommu properties, such as
dma window size/address, will be query-able through an architecture
specific (or general if possible) ioctl on the vfio group fd. I hope
that will help the specification, but I don't fully understand what all
remains. Thanks,
Alex
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2011-08-23 23:36:33
On Tue, 2011-08-23 at 15:18 +0200, Roedel, Joerg wrote:
On Mon, Aug 22, 2011 at 05:03:53PM -0400, Benjamin Herrenschmidt wrote:
quoted
quoted
I am in favour of /dev/vfio/$GROUP. If multiple devices should be
assigned to a guest, there can also be an ioctl to bind a group to an
address-space of another group (certainly needs some care to not allow
that both groups belong to different processes).
Btw, a problem we havn't talked about yet entirely is
driver-deassignment. User space can decide to de-assign the device from
vfio while a fd is open on it. With PCI there is no way to let this fail
(the .release function returns void last time i checked). Is this a
problem, and yes, how we handle that?
We can treat it as a hard unplug (like a cardbus gone away).
IE. Dispose of the direct mappings (switch to MMIO emulation) and return
all ff's from reads (& ignore writes).
Then send an unplug event via whatever mechanism the platform provides
(ACPI hotplug controller on x86 for example, we haven't quite sorted out
what to do on power for hotplug yet).
Hmm, good idea. But as far as I know the hotplug-event needs to be in
the guest _before_ the device is actually unplugged (so that the guest
can unbind its driver first). That somehow brings back the sleep-idea
and the timeout in the .release function.
That's for normal assisted hotplug, but don't we support hard hotplug ?
I mean, things like cardbus, thunderbolt (if we ever support that)
etc... will need it and some platforms do support hard hotplug of PCIe
devices.
(That's why drivers should never spin on MMIO waiting for a 1 bit to
clear without a timeout :-)
Cheers,
Ben.
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2011-08-23 23:42:21
On Tue, 2011-08-23 at 10:23 -0600, Alex Williamson wrote:
Yeah. Joerg's idea of binding groups internally (pass the fd of one
group to another via ioctl) is one option. The tricky part will be
implementing it to support hot unplug of any group from the
supergroup.
I believe Ben had a suggestion that supergroups could be created in
sysfs, but I don't know what the mechanism to do that looks like. It
would also be an extra management step to dynamically bind and unbind
groups to the supergroup around hotplug. Thanks,
I don't really care that much what the method for creating them is, to
be honest, I just prefer this concept of "meta groups" or "super groups"
or "synthetic groups" (whatever you want to name them) to having a
separate uiommu file descriptor.
The one reason I have a slight preference for creating them "statically"
using some kind of separate interface (again, I don't care whether it's
sysfs, netlink, etc...) is that it means things like qemu don't have to
care about them.
In general, apps that want to use vfio can just get passed the path to
such a group or the /dev/ path or the group number (whatever we chose as
the way to identify a group), and don't need to know anything about
"super groups", how to manipulate them, create them, possible
constraints etc...
Now, libvirt might want to know about that other API in order to provide
control on the creation of these things, but that's a different issue.
By "static" I mean they persist, they aren't tied to the lifetime of an
fd.
Now that's purely a preference on my side because I believe it will make
life easier for actual programs wanting to use vfio to not have to care
about those super-groups, but as I said earlier, I don't actually care
that much :-)
Cheers,
Ben.
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2011-08-23 23:52:07
quoted
For us the most simple and logical approach (which is also what pHyp
uses and what Linux handles well) is really to expose a given PCI host
bridge per group to the guest. Believe it or not, it makes things
easier :-)
I'm all for easier. Why does exposing the bridge use less bus numbers
than emulating a bridge?
Because a host bridge doesn't look like a PCI to PCI bridge at all for
us. It's an entire separate domain with it's own bus number space
(unlike most x86 setups).
In fact we have some problems afaik in qemu today with the concept of
PCI domains, for example, I think qemu has assumptions about a single
shared IO space domain which isn't true for us (each PCI host bridge
provides a distinct IO space domain starting at 0). We'll have to fix
that, but it's not a huge deal.
So for each "group" we'd expose in the guest an entire separate PCI
domain space with its own IO, MMIO etc... spaces, handed off from a
single device-tree "host bridge" which doesn't itself appear in the
config space, doesn't need any emulation of any config space etc...
On x86, I want to maintain that our default assignment is at the device
level. A user should be able to pick single or multiple devices from
across several groups and have them all show up as individual,
hotpluggable devices on bus 0 in the guest. Not surprisingly, we've
also seen cases where users try to attach a bridge to the guest,
assuming they'll get all the devices below the bridge, so I'd be in
favor of making this "just work" if possible too, though we may have to
prevent hotplug of those.
Given the device requirement on x86 and since everything is a PCI device
on x86, I'd like to keep a qemu command line something like -device
vfio,host=00:19.0. I assume that some of the iommu properties, such as
dma window size/address, will be query-able through an architecture
specific (or general if possible) ioctl on the vfio group fd. I hope
that will help the specification, but I don't fully understand what all
remains. Thanks,
Well, for iommu there's a couple of different issues here but yes,
basically on one side we'll have some kind of ioctl to know what segment
of the device(s) DMA address space is assigned to the group and we'll
need to represent that to the guest via a device-tree property in some
kind of "parent" node of all the devices in that group.
We -might- be able to implement some kind of hotplug of individual
devices of a group under such a PHB (PCI Host Bridge), I don't know for
sure yet, some of that PAPR stuff is pretty arcane, but basically, for
all intend and purpose, we really want a group to be represented as a
PHB in the guest.
We cannot arbitrary have individual devices of separate groups be
represented in the guest as siblings on a single simulated PCI bus.
Cheers,
Ben.
From: Alexander Graf <hidden> Date: 2011-08-24 03:37:19
On 23.08.2011, at 18:41, Benjamin Herrenschmidt wrote:
On Tue, 2011-08-23 at 10:23 -0600, Alex Williamson wrote:
quoted
=20
Yeah. Joerg's idea of binding groups internally (pass the fd of one
group to another via ioctl) is one option. The tricky part will be
implementing it to support hot unplug of any group from the
supergroup.
I believe Ben had a suggestion that supergroups could be created in
sysfs, but I don't know what the mechanism to do that looks like. It
would also be an extra management step to dynamically bind and unbind
groups to the supergroup around hotplug. Thanks,=20
=20
I don't really care that much what the method for creating them is, to
be honest, I just prefer this concept of "meta groups" or "super =
groups"
or "synthetic groups" (whatever you want to name them) to having a
separate uiommu file descriptor.
=20
The one reason I have a slight preference for creating them =
"statically"
using some kind of separate interface (again, I don't care whether =
it's
sysfs, netlink, etc...) is that it means things like qemu don't have =
to
care about them.
=20
In general, apps that want to use vfio can just get passed the path to
such a group or the /dev/ path or the group number (whatever we chose =
as
the way to identify a group), and don't need to know anything about
"super groups", how to manipulate them, create them, possible
constraints etc...
=20
Now, libvirt might want to know about that other API in order to =
provide
control on the creation of these things, but that's a different issue.
=20
By "static" I mean they persist, they aren't tied to the lifetime of =
an
fd.
=20
Now that's purely a preference on my side because I believe it will =
make
life easier for actual programs wanting to use vfio to not have to =
care
about those super-groups, but as I said earlier, I don't actually care
that much :-)
Oh I think it's one of the building blocks we need for a sane user space =
device exposure API. If I want to pass user X a few devices that are all =
behind a single IOMMU, I just chown that device node to user X and be =
done with it.
The user space tool actually using the VFIO interface wouldn't be in =
configuration business then - and it really shouldn't. That's what =
system configuration is there for :).
But I'm fairly sure we managed to persuade Alex that this is the right =
path on the BOF :)
Alex
From: Alexander Graf <hidden> Date: 2011-08-24 03:40:49
On 23.08.2011, at 18:51, Benjamin Herrenschmidt wrote:
=20
quoted
quoted
For us the most simple and logical approach (which is also what pHyp
uses and what Linux handles well) is really to expose a given PCI =
host
quoted
quoted
bridge per group to the guest. Believe it or not, it makes things
easier :-)
=20
I'm all for easier. Why does exposing the bridge use less bus =
numbers
quoted
than emulating a bridge?
=20
Because a host bridge doesn't look like a PCI to PCI bridge at all for
us. It's an entire separate domain with it's own bus number space
(unlike most x86 setups).
=20
In fact we have some problems afaik in qemu today with the concept of
PCI domains, for example, I think qemu has assumptions about a single
shared IO space domain which isn't true for us (each PCI host bridge
provides a distinct IO space domain starting at 0). We'll have to fix
that, but it's not a huge deal.
=20
So for each "group" we'd expose in the guest an entire separate PCI
domain space with its own IO, MMIO etc... spaces, handed off from a
single device-tree "host bridge" which doesn't itself appear in the
config space, doesn't need any emulation of any config space etc...
=20
quoted
On x86, I want to maintain that our default assignment is at the =
device
quoted
level. A user should be able to pick single or multiple devices from
across several groups and have them all show up as individual,
hotpluggable devices on bus 0 in the guest. Not surprisingly, we've
also seen cases where users try to attach a bridge to the guest,
assuming they'll get all the devices below the bridge, so I'd be in
favor of making this "just work" if possible too, though we may have =
to
quoted
prevent hotplug of those.
=20
Given the device requirement on x86 and since everything is a PCI =
device
quoted
on x86, I'd like to keep a qemu command line something like -device
vfio,host=3D00:19.0. I assume that some of the iommu properties, =
such as
quoted
dma window size/address, will be query-able through an architecture
specific (or general if possible) ioctl on the vfio group fd. I hope
that will help the specification, but I don't fully understand what =
all
quoted
remains. Thanks,
=20
Well, for iommu there's a couple of different issues here but yes,
basically on one side we'll have some kind of ioctl to know what =
segment
of the device(s) DMA address space is assigned to the group and we'll
need to represent that to the guest via a device-tree property in some
kind of "parent" node of all the devices in that group.
=20
We -might- be able to implement some kind of hotplug of individual
devices of a group under such a PHB (PCI Host Bridge), I don't know =
for
sure yet, some of that PAPR stuff is pretty arcane, but basically, for
all intend and purpose, we really want a group to be represented as a
PHB in the guest.
=20
We cannot arbitrary have individual devices of separate groups be
represented in the guest as siblings on a single simulated PCI bus.
So would it make sense for you to go the same route that we need to go =
on embedded power, with a separate VFIO style interface that simply =
exports memory ranges and irq bindings, but doesn't know anything about =
PCI? For e500, we'll be using something like that to pass through a full =
PCI bus into the system.
Alex
On Tue, Aug 23, 2011 at 03:30:06PM -0400, Alex Williamson wrote:
On Tue, 2011-08-23 at 07:01 +1000, Benjamin Herrenschmidt wrote:
quoted
Could be tho in what form ? returning sysfs pathes ?
I'm at a loss there, please suggest. I think we need an ioctl that
returns some kind of array of devices within the group and another that
maybe takes an index from that array and returns an fd for that device.
A sysfs path string might be a reasonable array element, but it sounds
like a pain to work with.
Limiting to PCI we can just pass the BDF as the argument to optain the
device-fd. For a more generic solution we need a unique identifier in
some way which is unique across all 'struct device' instances in the
system. As far as I know we don't have that yet (besides the sysfs-path)
so we either add that or stick with bus-specific solutions.
quoted
1:1 process has the advantage of linking to an -mm which makes the whole
mmu notifier business doable. How do you want to track down mappings and
do the second level translation in the case of explicit map/unmap (like
on power) if you are not tied to an mm_struct ?
Right, I threw away the mmu notifier code that was originally part of
vfio because we can't do anything useful with it yet on x86. I
definitely don't want to prevent it where it makes sense though. Maybe
we just record current->mm on open and restrict subsequent opens to the
same.
Hmm, I think we need io-page-fault support in the iommu-api then.
quoted
Another aspect I don't see discussed is how we represent these things to
the guest.
On Power for example, I have a requirement that a given iommu domain is
represented by a single dma window property in the device-tree. What
that means is that that property needs to be either in the node of the
device itself if there's only one device in the group or in a parent
node (ie a bridge or host bridge) if there are multiple devices.
Now I do -not- want to go down the path of simulating P2P bridges,
besides we'll quickly run out of bus numbers if we go there.
For us the most simple and logical approach (which is also what pHyp
uses and what Linux handles well) is really to expose a given PCI host
bridge per group to the guest. Believe it or not, it makes things
easier :-)
I'm all for easier. Why does exposing the bridge use less bus numbers
than emulating a bridge?
On x86, I want to maintain that our default assignment is at the device
level. A user should be able to pick single or multiple devices from
across several groups and have them all show up as individual,
hotpluggable devices on bus 0 in the guest. Not surprisingly, we've
also seen cases where users try to attach a bridge to the guest,
assuming they'll get all the devices below the bridge, so I'd be in
favor of making this "just work" if possible too, though we may have to
prevent hotplug of those.
A side-note: Might it be better to expose assigned devices in a guest on
a seperate bus? This will make it easier to emulate an IOMMU for the
guest inside qemu.
Joerg
--
AMD Operating System Research Center
Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo, Andrew Bowd
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632
On Tue, Aug 23, 2011 at 01:08:29PM -0400, Alex Williamson wrote:
On Tue, 2011-08-23 at 15:14 +0200, Roedel, Joerg wrote:
quoted
Handling it through fds is a good idea. This makes sure that everything
belongs to one process. I am not really sure yet if we go the way to
just bind plain groups together or if we create meta-groups. The
meta-groups thing seems somewhat cleaner, though.
I'm leaning towards binding because we need to make it dynamic, but I
don't really have a good picture of the lifecycle of a meta-group.
In my view the life-cycle of the meta-group is a subrange of the
qemu-instance's life-cycle.
quoted
Putting the process to sleep (which would be uninterruptible) seems bad.
The process would sleep until the guest releases the device-group, which
can take days or months.
The best thing (and the most intrusive :-) ) is to change PCI core to
allow unbindings to fail, I think. But this probably further complicates
the way to upstream VFIO...
Yes, it's not ideal but I think it's sufficient for now and if we later
get support for returning an error from release, we can set a timeout
after notifying the user to make use of that. Thanks,
Ben had the idea of just forcing to hard-unplug this device from the
guest. Thats probably the best way to deal with that, I think. VFIO
sends a notification to qemu that the device is gone and qemu informs
the guest in some way about it.
Joerg
--
AMD Operating System Research Center
Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo, Andrew Bowd
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632
On Tue, Aug 23, 2011 at 07:35:37PM -0400, Benjamin Herrenschmidt wrote:
On Tue, 2011-08-23 at 15:18 +0200, Roedel, Joerg wrote:
quoted
Hmm, good idea. But as far as I know the hotplug-event needs to be in
the guest _before_ the device is actually unplugged (so that the guest
can unbind its driver first). That somehow brings back the sleep-idea
and the timeout in the .release function.
That's for normal assisted hotplug, but don't we support hard hotplug ?
I mean, things like cardbus, thunderbolt (if we ever support that)
etc... will need it and some platforms do support hard hotplug of PCIe
devices.
(That's why drivers should never spin on MMIO waiting for a 1 bit to
clear without a timeout :-)
Right, thats probably the best semantic for this issue then. The worst
thing that happens is that the admin crashed the guest.
Joerg
--
AMD Operating System Research Center
Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo, Andrew Bowd
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632
On Tue, Aug 23, 2011 at 01:33:14PM -0400, Aaron Fabbri wrote:
On 8/23/11 10:01 AM, "Alex Williamson" [off-list ref] wrote:
quoted
The iommu domain would probably be allocated when the first device is
bound to vfio. As each device is bound, it gets attached to the group.
DMAs are done via an ioctl on the group.
I think group + uiommu leads to effectively reliving most of the
problems with the current code. The only benefit is the group
assignment to enforce hardware restrictions. We still have the problem
that uiommu open() = iommu_domain_alloc(), whose properties are
meaningless without attached devices (groups). Which I think leads to
the same awkward model of attaching groups to define the domain, then we
end up doing mappings via the group to enforce ordering.
Is there a better way to allow groups to share an IOMMU domain?
Maybe, instead of having an ioctl to allow a group A to inherit the same
iommu domain as group B, we could have an ioctl to fully merge two groups
(could be what Ben was thinking):
A.ioctl(MERGE_TO_GROUP, B)
The group A now goes away and its devices join group B. If A ever had an
iommu domain assigned (and buffers mapped?) we fail.
Groups cannot get smaller (they are defined as minimum granularity of an
IOMMU, initially). They can get bigger if you want to share IOMMU
resources, though.
Any downsides to this approach?
As long as this is a 2-way road its fine. There must be a way to split
the groups again after the guest exits. But then we are again at the
super-groups (aka meta-groups, aka uiommu) point.
Joerg
--
AMD Operating System Research Center
Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo, Andrew Bowd
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632
On Tue, Aug 23, 2011 at 12:54:27PM -0400, aafabbri wrote:
On 8/23/11 4:04 AM, "Joerg Roedel" [off-list ref] wrote:
quoted
That is makes uiommu basically the same as the meta-groups, right?
Yes, functionality seems the same, thus my suggestion to keep uiommu
explicit. Is there some need for group-groups besides defining sets of
groups which share IOMMU resources?
I do all this stuff (bringing up sets of devices which may share IOMMU
domain) dynamically from C applications. I don't really want some static
(boot-time or sysfs fiddling) supergroup config unless there is a good
reason KVM/power needs it.
As you say in your next email, doing it all from ioctls is very easy,
programmatically.
I don't see a reason to make this meta-grouping static. It would harm
flexibility on x86. I think it makes things easier on power but there
are options on that platform to get the dynamic solution too.
Joerg
--
AMD Operating System Research Center
Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo, Andrew Bowd
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632
From: David Gibson <hidden> Date: 2011-08-24 09:33:16
On Wed, Aug 24, 2011 at 11:14:26AM +0200, Roedel, Joerg wrote:
On Tue, Aug 23, 2011 at 12:54:27PM -0400, aafabbri wrote:
quoted
On 8/23/11 4:04 AM, "Joerg Roedel" [off-list ref] wrote:
quoted
That is makes uiommu basically the same as the meta-groups, right?
Yes, functionality seems the same, thus my suggestion to keep uiommu
explicit. Is there some need for group-groups besides defining sets of
groups which share IOMMU resources?
I do all this stuff (bringing up sets of devices which may share IOMMU
domain) dynamically from C applications. I don't really want some static
(boot-time or sysfs fiddling) supergroup config unless there is a good
reason KVM/power needs it.
As you say in your next email, doing it all from ioctls is very easy,
programmatically.
I don't see a reason to make this meta-grouping static. It would harm
flexibility on x86. I think it makes things easier on power but there
are options on that platform to get the dynamic solution too.
I think several people are misreading what Ben means by "static". I
would prefer to say 'persistent', in that the meta-groups lifetime is
not tied to an fd, but they can be freely created, altered and removed
during runtime.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
On Wed, Aug 24, 2011 at 05:33:00AM -0400, David Gibson wrote:
On Wed, Aug 24, 2011 at 11:14:26AM +0200, Roedel, Joerg wrote:
quoted
I don't see a reason to make this meta-grouping static. It would harm
flexibility on x86. I think it makes things easier on power but there
are options on that platform to get the dynamic solution too.
I think several people are misreading what Ben means by "static". I
would prefer to say 'persistent', in that the meta-groups lifetime is
not tied to an fd, but they can be freely created, altered and removed
during runtime.
Even if it can be altered at runtime, from a usability perspective it is
certainly the best to handle these groups directly in qemu. Or are there
strong reasons to do it somewhere else?
Joerg
--
AMD Operating System Research Center
Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo, Andrew Bowd
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632
From: Alex Williamson <hidden> Date: 2011-08-24 14:48:17
On Wed, 2011-08-24 at 09:51 +1000, Benjamin Herrenschmidt wrote:
quoted
quoted
For us the most simple and logical approach (which is also what pHyp
uses and what Linux handles well) is really to expose a given PCI host
bridge per group to the guest. Believe it or not, it makes things
easier :-)
I'm all for easier. Why does exposing the bridge use less bus numbers
than emulating a bridge?
Because a host bridge doesn't look like a PCI to PCI bridge at all for
us. It's an entire separate domain with it's own bus number space
(unlike most x86 setups).
Ok, I missed the "host" bridge.
In fact we have some problems afaik in qemu today with the concept of
PCI domains, for example, I think qemu has assumptions about a single
shared IO space domain which isn't true for us (each PCI host bridge
provides a distinct IO space domain starting at 0). We'll have to fix
that, but it's not a huge deal.
Yep, I've seen similar on ia64 systems.
So for each "group" we'd expose in the guest an entire separate PCI
domain space with its own IO, MMIO etc... spaces, handed off from a
single device-tree "host bridge" which doesn't itself appear in the
config space, doesn't need any emulation of any config space etc...
quoted
On x86, I want to maintain that our default assignment is at the device
level. A user should be able to pick single or multiple devices from
across several groups and have them all show up as individual,
hotpluggable devices on bus 0 in the guest. Not surprisingly, we've
also seen cases where users try to attach a bridge to the guest,
assuming they'll get all the devices below the bridge, so I'd be in
favor of making this "just work" if possible too, though we may have to
prevent hotplug of those.
Given the device requirement on x86 and since everything is a PCI device
on x86, I'd like to keep a qemu command line something like -device
vfio,host=00:19.0. I assume that some of the iommu properties, such as
dma window size/address, will be query-able through an architecture
specific (or general if possible) ioctl on the vfio group fd. I hope
that will help the specification, but I don't fully understand what all
remains. Thanks,
Well, for iommu there's a couple of different issues here but yes,
basically on one side we'll have some kind of ioctl to know what segment
of the device(s) DMA address space is assigned to the group and we'll
need to represent that to the guest via a device-tree property in some
kind of "parent" node of all the devices in that group.
We -might- be able to implement some kind of hotplug of individual
devices of a group under such a PHB (PCI Host Bridge), I don't know for
sure yet, some of that PAPR stuff is pretty arcane, but basically, for
all intend and purpose, we really want a group to be represented as a
PHB in the guest.
We cannot arbitrary have individual devices of separate groups be
represented in the guest as siblings on a single simulated PCI bus.
I think the vfio kernel layer we're describing easily supports both.
This is just a matter of adding qemu-vfio code to expose different
topologies based on group iommu capabilities and mapping mode. Thanks,
Alex
From: Alex Williamson <hidden> Date: 2011-08-24 14:56:24
On Wed, 2011-08-24 at 10:43 +0200, Joerg Roedel wrote:
On Tue, Aug 23, 2011 at 03:30:06PM -0400, Alex Williamson wrote:
quoted
On Tue, 2011-08-23 at 07:01 +1000, Benjamin Herrenschmidt wrote:
quoted
quoted
Could be tho in what form ? returning sysfs pathes ?
I'm at a loss there, please suggest. I think we need an ioctl that
returns some kind of array of devices within the group and another that
maybe takes an index from that array and returns an fd for that device.
A sysfs path string might be a reasonable array element, but it sounds
like a pain to work with.
Limiting to PCI we can just pass the BDF as the argument to optain the
device-fd. For a more generic solution we need a unique identifier in
some way which is unique across all 'struct device' instances in the
system. As far as I know we don't have that yet (besides the sysfs-path)
so we either add that or stick with bus-specific solutions.
quoted
quoted
1:1 process has the advantage of linking to an -mm which makes the whole
mmu notifier business doable. How do you want to track down mappings and
do the second level translation in the case of explicit map/unmap (like
on power) if you are not tied to an mm_struct ?
Right, I threw away the mmu notifier code that was originally part of
vfio because we can't do anything useful with it yet on x86. I
definitely don't want to prevent it where it makes sense though. Maybe
we just record current->mm on open and restrict subsequent opens to the
same.
Hmm, I think we need io-page-fault support in the iommu-api then.
Yeah, when we can handle iommu page faults, this gets more interesting.
quoted
quoted
Another aspect I don't see discussed is how we represent these things to
the guest.
On Power for example, I have a requirement that a given iommu domain is
represented by a single dma window property in the device-tree. What
that means is that that property needs to be either in the node of the
device itself if there's only one device in the group or in a parent
node (ie a bridge or host bridge) if there are multiple devices.
Now I do -not- want to go down the path of simulating P2P bridges,
besides we'll quickly run out of bus numbers if we go there.
For us the most simple and logical approach (which is also what pHyp
uses and what Linux handles well) is really to expose a given PCI host
bridge per group to the guest. Believe it or not, it makes things
easier :-)
I'm all for easier. Why does exposing the bridge use less bus numbers
than emulating a bridge?
On x86, I want to maintain that our default assignment is at the device
level. A user should be able to pick single or multiple devices from
across several groups and have them all show up as individual,
hotpluggable devices on bus 0 in the guest. Not surprisingly, we've
also seen cases where users try to attach a bridge to the guest,
assuming they'll get all the devices below the bridge, so I'd be in
favor of making this "just work" if possible too, though we may have to
prevent hotplug of those.
A side-note: Might it be better to expose assigned devices in a guest on
a seperate bus? This will make it easier to emulate an IOMMU for the
guest inside qemu.
I think we want that option, sure. A lot of guests aren't going to
support hotplugging buses though, so I think our default, map the entire
guest model should still be using bus 0. The ACPI gets a lot more
complicated for that model too; dynamic SSDTs? Thanks,
Alex
From: Alex Williamson <hidden> Date: 2011-08-24 15:07:57
On Wed, 2011-08-24 at 10:52 +0200, Roedel, Joerg wrote:
On Tue, Aug 23, 2011 at 01:08:29PM -0400, Alex Williamson wrote:
quoted
On Tue, 2011-08-23 at 15:14 +0200, Roedel, Joerg wrote:
quoted
quoted
Handling it through fds is a good idea. This makes sure that everything
belongs to one process. I am not really sure yet if we go the way to
just bind plain groups together or if we create meta-groups. The
meta-groups thing seems somewhat cleaner, though.
I'm leaning towards binding because we need to make it dynamic, but I
don't really have a good picture of the lifecycle of a meta-group.
In my view the life-cycle of the meta-group is a subrange of the
qemu-instance's life-cycle.
I guess I mean the lifecycle of a super-group that's actually exposed as
a new group in sysfs. Who creates it? How? How are groups dynamically
added and removed from the super-group? The group merging makes sense
to me because it's largely just an optimization that qemu will try to
merge groups. If it works, great. If not, it manages them separately.
When all the devices from a group are unplugged, unmerge the group if
necessary.
quoted
quoted
Putting the process to sleep (which would be uninterruptible) seems bad.
The process would sleep until the guest releases the device-group, which
can take days or months.
The best thing (and the most intrusive :-) ) is to change PCI core to
allow unbindings to fail, I think. But this probably further complicates
the way to upstream VFIO...
Yes, it's not ideal but I think it's sufficient for now and if we later
get support for returning an error from release, we can set a timeout
after notifying the user to make use of that. Thanks,
Ben had the idea of just forcing to hard-unplug this device from the
guest. Thats probably the best way to deal with that, I think. VFIO
sends a notification to qemu that the device is gone and qemu informs
the guest in some way about it.
We need to try the polite method of attempting to hot unplug the device
from qemu first, which the current vfio code already implements. We can
then escalate if it doesn't respond. The current code calls abort in
qemu if the guest doesn't respond, but I agree we should also be
enforcing this at the kernel interface. I think the problem with the
hard-unplug is that we don't have a good revoke mechanism for the mmio
mmaps. Thanks,
Alex
From: Alex Williamson <hidden> Date: 2011-08-24 21:14:06
Joerg,
Is this roughly what you're thinking of for the iommu_group component?
Adding a dev_to_group iommu ops callback let's us consolidate the sysfs
support in the iommu base. Would AMD-Vi do something similar (or
exactly the same) for group #s? Thanks,
Alex
Signed-off-by: Alex Williamson <redacted>
@@ -438,6 +439,10 @@ static int __init intel_iommu_setup(char *str)printk(KERN_INFO"Intel-IOMMU: disable supported super page\n");intel_iommu_superpage=0;+}elseif(!strncmp(str,"no_mf_groups",12)){+printk(KERN_INFO+"Intel-IOMMU: disable separate groups for multifunction devices\n");+intel_iommu_no_mf_groups=1;}str+=strcspn(str,",");
@@ -3902,6 +3907,52 @@ static int intel_iommu_domain_has_cap(struct iommu_domain *domain,return0;}+/* Group numbers are arbitrary. Device with the same group number+*indicatetheiommucannotdifferentiatebetweenthem.Toavoid+*trackingusedgroupswejustusetheseg|bus|devfnofthelowest+*levelwe'reabletodifferentiatedevices*/+staticlongintel_iommu_dev_to_group(structdevice*dev)+{+structpci_dev*pdev=to_pci_dev(dev);+structpci_dev*bridge;+union{+struct{+u8devfn;+u8bus;+u16segment;+}pci;+u32group;+}id;++if(iommu_no_mapping(dev))+return-ENODEV;++id.pci.segment=pci_domain_nr(pdev->bus);+id.pci.bus=pdev->bus->number;+id.pci.devfn=pdev->devfn;++if(!device_to_iommu(id.pci.segment,id.pci.bus,id.pci.devfn))+return-ENODEV;++bridge=pci_find_upstream_pcie_bridge(pdev);+if(bridge){+if(pci_is_pcie(bridge)){+id.pci.bus=bridge->subordinate->number;+id.pci.devfn=0;+}else{+id.pci.bus=bridge->bus->number;+id.pci.devfn=bridge->devfn;+}+}++/* Virtual functions always get their own group */+if(!pdev->is_virtfn&&intel_iommu_no_mf_groups)+id.pci.devfn=PCI_DEVFN(PCI_SLOT(id.pci.devfn),0);++/* FIXME - seg # >= 0x8000 on 32b */+returnid.group;+}+staticstructiommu_opsintel_iommu_ops={.domain_init=intel_iommu_domain_init,.domain_destroy=intel_iommu_domain_destroy,
Hi Alex,
On Wed, Aug 24, 2011 at 05:13:49PM -0400, Alex Williamson wrote:
Is this roughly what you're thinking of for the iommu_group component?
Adding a dev_to_group iommu ops callback let's us consolidate the sysfs
support in the iommu base. Would AMD-Vi do something similar (or
exactly the same) for group #s? Thanks,
The concept looks good, I have some comments, though. On AMD-Vi the
implementation would look a bit different because there is a
data-structure were the information can be gathered from, so no need for
PCI bus scanning there.
We need to solve this differently. ARM is starting to use the iommu-api
too and this definitly does not work there. One possible solution might
be to make the iommu-ops per-bus.
quoted hunk
bool iommu_found(void)
@@ -94,6 +130,14 @@ int iommu_domain_has_cap(struct iommu_domain *domain, } EXPORT_SYMBOL_GPL(iommu_domain_has_cap);+long iommu_dev_to_group(struct device *dev)+{+ if (iommu_ops->dev_to_group)+ return iommu_ops->dev_to_group(dev);+ return -ENODEV;+}+EXPORT_SYMBOL_GPL(iommu_dev_to_group);
Please rename this to iommu_device_group(). The dev_to_group name
suggests a conversion but it is actually just a property of the device.
Also the return type should not be long but something that fits into
32bit on all platforms. Since you use -ENODEV, probably s32 is a good
choice.
quoted hunk
+
int iommu_map(struct iommu_domain *domain, unsigned long iova,
phys_addr_t paddr, int gfp_order, int prot)
{
@@ -438,6 +439,10 @@ static int __init intel_iommu_setup(char *str)printk(KERN_INFO"Intel-IOMMU: disable supported super page\n");intel_iommu_superpage=0;+}elseif(!strncmp(str,"no_mf_groups",12)){+printk(KERN_INFO+"Intel-IOMMU: disable separate groups for multifunction devices\n");+intel_iommu_no_mf_groups=1;
This should really be a global iommu option and not be VT-d specific.
quoted hunk
str += strcspn(str, ",");
@@ -3902,6 +3907,52 @@ static int intel_iommu_domain_has_cap(struct iommu_domain *domain, return 0; }+/* Group numbers are arbitrary. Device with the same group number+ * indicate the iommu cannot differentiate between them. To avoid+ * tracking used groups we just use the seg|bus|devfn of the lowest+ * level we're able to differentiate devices */+static long intel_iommu_dev_to_group(struct device *dev)+{+ struct pci_dev *pdev = to_pci_dev(dev);+ struct pci_dev *bridge;+ union {+ struct {+ u8 devfn;+ u8 bus;+ u16 segment;+ } pci;+ u32 group;+ } id;++ if (iommu_no_mapping(dev))+ return -ENODEV;++ id.pci.segment = pci_domain_nr(pdev->bus);+ id.pci.bus = pdev->bus->number;+ id.pci.devfn = pdev->devfn;++ if (!device_to_iommu(id.pci.segment, id.pci.bus, id.pci.devfn))+ return -ENODEV;++ bridge = pci_find_upstream_pcie_bridge(pdev);+ if (bridge) {+ if (pci_is_pcie(bridge)) {+ id.pci.bus = bridge->subordinate->number;+ id.pci.devfn = 0;+ } else {+ id.pci.bus = bridge->bus->number;+ id.pci.devfn = bridge->devfn;+ }+ }++ /* Virtual functions always get their own group */+ if (!pdev->is_virtfn && intel_iommu_no_mf_groups)+ id.pci.devfn = PCI_DEVFN(PCI_SLOT(id.pci.devfn), 0);++ /* FIXME - seg # >= 0x8000 on 32b */+ return id.group;+}
This looks like code duplication in the VT-d driver. It doesn't need to
be generalized now, but we should keep in mind to do a more general
solution later.
Maybe it is beneficial if the IOMMU drivers only setup the number in
dev->arch.iommu.groupid and the iommu-api fetches it from there then.
But as I said, this is some more work and does not need to be done for
this patch(-set).
--
AMD Operating System Research Center
Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo, Andrew Bowd
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632
On Wed, Aug 24, 2011 at 10:56:13AM -0400, Alex Williamson wrote:
On Wed, 2011-08-24 at 10:43 +0200, Joerg Roedel wrote:
quoted
A side-note: Might it be better to expose assigned devices in a guest on
a seperate bus? This will make it easier to emulate an IOMMU for the
guest inside qemu.
I think we want that option, sure. A lot of guests aren't going to
support hotplugging buses though, so I think our default, map the entire
guest model should still be using bus 0. The ACPI gets a lot more
complicated for that model too; dynamic SSDTs? Thanks,
Ok, if only AMD-Vi should be emulated then it is not strictly
necessary. For this IOMMU we can specify that devices on the same bus
belong to different IOMMUs. So we can implement an IOMMU that handles
internal qemu-devices and one that handles pass-through devices.
Not sure if this is possible with VT-d too. Okay VT-d emulation would
also require that the devices emulation of a PCIe bridge, no?
Joerg
--
AMD Operating System Research Center
Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo, Andrew Bowd
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632
On Wed, Aug 24, 2011 at 11:07:46AM -0400, Alex Williamson wrote:
On Wed, 2011-08-24 at 10:52 +0200, Roedel, Joerg wrote:
quoted
On Tue, Aug 23, 2011 at 01:08:29PM -0400, Alex Williamson wrote:
quoted
On Tue, 2011-08-23 at 15:14 +0200, Roedel, Joerg wrote:
quoted
quoted
Handling it through fds is a good idea. This makes sure that everything
belongs to one process. I am not really sure yet if we go the way to
just bind plain groups together or if we create meta-groups. The
meta-groups thing seems somewhat cleaner, though.
I'm leaning towards binding because we need to make it dynamic, but I
don't really have a good picture of the lifecycle of a meta-group.
In my view the life-cycle of the meta-group is a subrange of the
qemu-instance's life-cycle.
I guess I mean the lifecycle of a super-group that's actually exposed as
a new group in sysfs. Who creates it? How? How are groups dynamically
added and removed from the super-group? The group merging makes sense
to me because it's largely just an optimization that qemu will try to
merge groups. If it works, great. If not, it manages them separately.
When all the devices from a group are unplugged, unmerge the group if
necessary.
Right. The super-group thing is an optimization.
We need to try the polite method of attempting to hot unplug the device
from qemu first, which the current vfio code already implements. We can
then escalate if it doesn't respond. The current code calls abort in
qemu if the guest doesn't respond, but I agree we should also be
enforcing this at the kernel interface. I think the problem with the
hard-unplug is that we don't have a good revoke mechanism for the mmio
mmaps.
For mmio we could stop the guest and replace the mmio region with a
region that is filled with 0xff, no?
Joerg
--
AMD Operating System Research Center
Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo, Andrew Bowd
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632
From: Alexander Graf <hidden> Date: 2011-08-25 13:26:05
On 25.08.2011, at 07:31, Roedel, Joerg wrote:
On Wed, Aug 24, 2011 at 11:07:46AM -0400, Alex Williamson wrote:
quoted
On Wed, 2011-08-24 at 10:52 +0200, Roedel, Joerg wrote:
[...]
quoted
We need to try the polite method of attempting to hot unplug the device
from qemu first, which the current vfio code already implements. We can
then escalate if it doesn't respond. The current code calls abort in
qemu if the guest doesn't respond, but I agree we should also be
enforcing this at the kernel interface. I think the problem with the
hard-unplug is that we don't have a good revoke mechanism for the mmio
mmaps.
For mmio we could stop the guest and replace the mmio region with a
region that is filled with 0xff, no?
Sure, but that happens in user space. The question is how does kernel space enforce an MMIO region to not be mapped after the hotplug event occured? Keep in mind that user space is pretty much untrusted here - it doesn't have to be QEMU. It could just as well be a generic user space driver. And that can just ignore hotplug events.
Alex
From: Don Dutile <hidden> Date: 2011-08-25 15:39:13
On 08/25/2011 06:54 AM, Roedel, Joerg wrote:
Hi Alex,
On Wed, Aug 24, 2011 at 05:13:49PM -0400, Alex Williamson wrote:
quoted
Is this roughly what you're thinking of for the iommu_group component?
Adding a dev_to_group iommu ops callback let's us consolidate the sysfs
support in the iommu base. Would AMD-Vi do something similar (or
exactly the same) for group #s? Thanks,
The concept looks good, I have some comments, though. On AMD-Vi the
implementation would look a bit different because there is a
data-structure were the information can be gathered from, so no need for
PCI bus scanning there.
We need to solve this differently. ARM is starting to use the iommu-api
too and this definitly does not work there. One possible solution might
be to make the iommu-ops per-bus.
When you think of a system where there isn't just one bus-type
with iommu support, it makes more sense.
Additionally, it also allows the long-term architecture to use different types
of IOMMUs on each bus segment -- think per-PCIe-switch/bridge IOMMUs --
esp. 'tuned' IOMMUs -- ones better geared for networks, ones better geared
for direct-attach disk hba's.
quoted
bool iommu_found(void)
@@ -94,6 +130,14 @@ int iommu_domain_has_cap(struct iommu_domain *domain, } EXPORT_SYMBOL_GPL(iommu_domain_has_cap);+long iommu_dev_to_group(struct device *dev)+{+ if (iommu_ops->dev_to_group)+ return iommu_ops->dev_to_group(dev);+ return -ENODEV;+}+EXPORT_SYMBOL_GPL(iommu_dev_to_group);
Please rename this to iommu_device_group(). The dev_to_group name
suggests a conversion but it is actually just a property of the device.
Also the return type should not be long but something that fits into
32bit on all platforms. Since you use -ENODEV, probably s32 is a good
choice.
quoted
+
int iommu_map(struct iommu_domain *domain, unsigned long iova,
phys_addr_t paddr, int gfp_order, int prot)
{
@@ -438,6 +439,10 @@ static int __init intel_iommu_setup(char *str)printk(KERN_INFO"Intel-IOMMU: disable supported super page\n");intel_iommu_superpage=0;+}elseif(!strncmp(str,"no_mf_groups",12)){+printk(KERN_INFO+"Intel-IOMMU: disable separate groups for multifunction devices\n");+intel_iommu_no_mf_groups=1;
This should really be a global iommu option and not be VT-d specific.
quoted
str += strcspn(str, ",");
@@ -3902,6 +3907,52 @@ static int intel_iommu_domain_has_cap(struct iommu_domain *domain, return 0; }+/* Group numbers are arbitrary. Device with the same group number+ * indicate the iommu cannot differentiate between them. To avoid+ * tracking used groups we just use the seg|bus|devfn of the lowest+ * level we're able to differentiate devices */+static long intel_iommu_dev_to_group(struct device *dev)+{+ struct pci_dev *pdev = to_pci_dev(dev);+ struct pci_dev *bridge;+ union {+ struct {+ u8 devfn;+ u8 bus;+ u16 segment;+ } pci;+ u32 group;+ } id;++ if (iommu_no_mapping(dev))+ return -ENODEV;++ id.pci.segment = pci_domain_nr(pdev->bus);+ id.pci.bus = pdev->bus->number;+ id.pci.devfn = pdev->devfn;++ if (!device_to_iommu(id.pci.segment, id.pci.bus, id.pci.devfn))+ return -ENODEV;++ bridge = pci_find_upstream_pcie_bridge(pdev);+ if (bridge) {+ if (pci_is_pcie(bridge)) {+ id.pci.bus = bridge->subordinate->number;+ id.pci.devfn = 0;+ } else {+ id.pci.bus = bridge->bus->number;+ id.pci.devfn = bridge->devfn;+ }+ }++ /* Virtual functions always get their own group */+ if (!pdev->is_virtfn&& intel_iommu_no_mf_groups)+ id.pci.devfn = PCI_DEVFN(PCI_SLOT(id.pci.devfn), 0);++ /* FIXME - seg #>= 0x8000 on 32b */+ return id.group;+}
This looks like code duplication in the VT-d driver. It doesn't need to
be generalized now, but we should keep in mind to do a more general
solution later.
Maybe it is beneficial if the IOMMU drivers only setup the number in
dev->arch.iommu.groupid and the iommu-api fetches it from there then.
But as I said, this is some more work and does not need to be done for
this patch(-set).
On Thu, Aug 25, 2011 at 11:38:09AM -0400, Don Dutile wrote:
On 08/25/2011 06:54 AM, Roedel, Joerg wrote:
quoted
We need to solve this differently. ARM is starting to use the iommu-api
too and this definitly does not work there. One possible solution might
be to make the iommu-ops per-bus.
When you think of a system where there isn't just one bus-type
with iommu support, it makes more sense.
Additionally, it also allows the long-term architecture to use different types
of IOMMUs on each bus segment -- think per-PCIe-switch/bridge IOMMUs --
esp. 'tuned' IOMMUs -- ones better geared for networks, ones better geared
for direct-attach disk hba's.
Not sure how likely it is to have different types of IOMMUs within a
given bus-type. But if they become reality we can multiplex in the
iommu-api without much hassle :)
For now, something like bus_set_iommu() or bus_register_iommu() would
provide a nice way to do bus-specific setups for a given iommu
implementation.
Joerg
--
AMD Operating System Research Center
Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo, Andrew Bowd
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632
From: Alex Williamson <hidden> Date: 2011-08-25 17:21:13
On Thu, 2011-08-25 at 12:54 +0200, Roedel, Joerg wrote:
Hi Alex,
On Wed, Aug 24, 2011 at 05:13:49PM -0400, Alex Williamson wrote:
quoted
Is this roughly what you're thinking of for the iommu_group component?
Adding a dev_to_group iommu ops callback let's us consolidate the sysfs
support in the iommu base. Would AMD-Vi do something similar (or
exactly the same) for group #s? Thanks,
The concept looks good, I have some comments, though. On AMD-Vi the
implementation would look a bit different because there is a
data-structure were the information can be gathered from, so no need for
PCI bus scanning there.
Probably add a 0x prefix so userspace knows the format?
I think I'll probably change it to %u. Seems common to have decimal in
sysfs and doesn't get confusing if we cat it with a string. As a bonus,
it abstracts that vt-d is just stuffing a PCI device address in there,
which nobody should ever rely on.
We need to solve this differently. ARM is starting to use the iommu-api
too and this definitly does not work there. One possible solution might
be to make the iommu-ops per-bus.
That sounds good. Is anyone working on it? It seems like it doesn't
hurt to use this in the interim, we may just be watching the wrong bus
and never add any sysfs group info.
quoted
bool iommu_found(void)
@@ -94,6 +130,14 @@ int iommu_domain_has_cap(struct iommu_domain *domain, } EXPORT_SYMBOL_GPL(iommu_domain_has_cap);+long iommu_dev_to_group(struct device *dev)+{+ if (iommu_ops->dev_to_group)+ return iommu_ops->dev_to_group(dev);+ return -ENODEV;+}+EXPORT_SYMBOL_GPL(iommu_dev_to_group);
Please rename this to iommu_device_group(). The dev_to_group name
suggests a conversion but it is actually just a property of the device.
Ok.
Also the return type should not be long but something that fits into
32bit on all platforms. Since you use -ENODEV, probably s32 is a good
choice.
The convenience of using seg|bus|dev|fn was too much to resist, too bad
it requires a full 32bits. Maybe I'll change it to:
int iommu_device_group(struct device *dev, unsigned int *group)
quoted
+
int iommu_map(struct iommu_domain *domain, unsigned long iova,
phys_addr_t paddr, int gfp_order, int prot)
{
@@ -438,6 +439,10 @@ static int __init intel_iommu_setup(char *str)printk(KERN_INFO"Intel-IOMMU: disable supported super page\n");intel_iommu_superpage=0;+}elseif(!strncmp(str,"no_mf_groups",12)){+printk(KERN_INFO+"Intel-IOMMU: disable separate groups for multifunction devices\n");+intel_iommu_no_mf_groups=1;
This should really be a global iommu option and not be VT-d specific.
You think? It's meaningless on benh's power systems.
quoted
str += strcspn(str, ",");
@@ -3902,6 +3907,52 @@ static int intel_iommu_domain_has_cap(struct iommu_domain *domain, return 0; }+/* Group numbers are arbitrary. Device with the same group number+ * indicate the iommu cannot differentiate between them. To avoid+ * tracking used groups we just use the seg|bus|devfn of the lowest+ * level we're able to differentiate devices */+static long intel_iommu_dev_to_group(struct device *dev)+{+ struct pci_dev *pdev = to_pci_dev(dev);+ struct pci_dev *bridge;+ union {+ struct {+ u8 devfn;+ u8 bus;+ u16 segment;+ } pci;+ u32 group;+ } id;++ if (iommu_no_mapping(dev))+ return -ENODEV;++ id.pci.segment = pci_domain_nr(pdev->bus);+ id.pci.bus = pdev->bus->number;+ id.pci.devfn = pdev->devfn;++ if (!device_to_iommu(id.pci.segment, id.pci.bus, id.pci.devfn))+ return -ENODEV;++ bridge = pci_find_upstream_pcie_bridge(pdev);+ if (bridge) {+ if (pci_is_pcie(bridge)) {+ id.pci.bus = bridge->subordinate->number;+ id.pci.devfn = 0;+ } else {+ id.pci.bus = bridge->bus->number;+ id.pci.devfn = bridge->devfn;+ }+ }++ /* Virtual functions always get their own group */+ if (!pdev->is_virtfn && intel_iommu_no_mf_groups)+ id.pci.devfn = PCI_DEVFN(PCI_SLOT(id.pci.devfn), 0);++ /* FIXME - seg # >= 0x8000 on 32b */+ return id.group;+}
This looks like code duplication in the VT-d driver. It doesn't need to
be generalized now, but we should keep in mind to do a more general
solution later.
Maybe it is beneficial if the IOMMU drivers only setup the number in
dev->arch.iommu.groupid and the iommu-api fetches it from there then.
But as I said, this is some more work and does not need to be done for
this patch(-set).
The iommu-api reaches into dev->arch.iommu.groupid? I figured we should
at least start out with a lightweight, optional interface without the
overhead of predefining groupids setup by bus notification callbacks in
each iommu driver. Thanks,
Alex
On Thu, Aug 25, 2011 at 11:20:30AM -0600, Alex Williamson wrote:
On Thu, 2011-08-25 at 12:54 +0200, Roedel, Joerg wrote:
quoted
We need to solve this differently. ARM is starting to use the iommu-api
too and this definitly does not work there. One possible solution might
be to make the iommu-ops per-bus.
That sounds good. Is anyone working on it? It seems like it doesn't
hurt to use this in the interim, we may just be watching the wrong bus
and never add any sysfs group info.
I'll cook something up for RFC over the weekend.
quoted
Also the return type should not be long but something that fits into
32bit on all platforms. Since you use -ENODEV, probably s32 is a good
choice.
The convenience of using seg|bus|dev|fn was too much to resist, too bad
it requires a full 32bits. Maybe I'll change it to:
int iommu_device_group(struct device *dev, unsigned int *group)
If we really expect segment numbers that need the full 16 bit then this
would be the way to go. Otherwise I would prefer returning the group-id
directly and partition the group-id space for the error values (s32 with
negative numbers being errors).
quoted
quoted
@@ -438,6 +439,10 @@ static int __init intel_iommu_setup(char *str) printk(KERN_INFO "Intel-IOMMU: disable supported super page\n"); intel_iommu_superpage = 0;+ } else if (!strncmp(str, "no_mf_groups", 12)) {+ printk(KERN_INFO+ "Intel-IOMMU: disable separate groups for multifunction devices\n");+ intel_iommu_no_mf_groups = 1;
This should really be a global iommu option and not be VT-d specific.
You think? It's meaningless on benh's power systems.
But it is not meaningless on AMD-Vi systems :) There should be one
option for both.
On the other hand this requires an iommu= parameter on ia64, but thats
probably not that bad.
quoted
This looks like code duplication in the VT-d driver. It doesn't need to
be generalized now, but we should keep in mind to do a more general
solution later.
Maybe it is beneficial if the IOMMU drivers only setup the number in
dev->arch.iommu.groupid and the iommu-api fetches it from there then.
But as I said, this is some more work and does not need to be done for
this patch(-set).
The iommu-api reaches into dev->arch.iommu.groupid? I figured we should
at least start out with a lightweight, optional interface without the
overhead of predefining groupids setup by bus notification callbacks in
each iommu driver. Thanks,
As I said, this is just an idea for an later optimization. It is fine
for now as it is in this patch.
Joerg
From: David Gibson <hidden> Date: 2011-08-26 04:24:32
On Wed, Aug 24, 2011 at 01:03:32PM +0200, Roedel, Joerg wrote:
On Wed, Aug 24, 2011 at 05:33:00AM -0400, David Gibson wrote:
quoted
On Wed, Aug 24, 2011 at 11:14:26AM +0200, Roedel, Joerg wrote:
quoted
quoted
I don't see a reason to make this meta-grouping static. It would harm
flexibility on x86. I think it makes things easier on power but there
are options on that platform to get the dynamic solution too.
I think several people are misreading what Ben means by "static". I
would prefer to say 'persistent', in that the meta-groups lifetime is
not tied to an fd, but they can be freely created, altered and removed
during runtime.
Even if it can be altered at runtime, from a usability perspective it is
certainly the best to handle these groups directly in qemu. Or are there
strong reasons to do it somewhere else?
Funny, Ben and I think usability demands it be the other way around.
If the meta-groups are transient - that is lifetime tied to an fd -
then any program that wants to use meta-groups *must* know the
interfaces for creating one, whatever they are.
But if they're persistent, the admin can use other tools to create the
meta-group then just hand it to a program to use, since the interfaces
for _using_ a meta-group are identical to those for an atomic group.
This doesn't preclude a program from being meta-group aware, and
creating its own if it wants to, of course. My guess is that qemu
would not want to build its own meta-groups, but libvirt probably
would.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
From: David Gibson <hidden> Date: 2011-08-26 04:24:32
On Thu, Aug 25, 2011 at 08:25:45AM -0500, Alexander Graf wrote:
On 25.08.2011, at 07:31, Roedel, Joerg wrote:
quoted
On Wed, Aug 24, 2011 at 11:07:46AM -0400, Alex Williamson wrote:
quoted
On Wed, 2011-08-24 at 10:52 +0200, Roedel, Joerg wrote:
[...]
quoted
quoted
We need to try the polite method of attempting to hot unplug the device
from qemu first, which the current vfio code already implements. We can
then escalate if it doesn't respond. The current code calls abort in
qemu if the guest doesn't respond, but I agree we should also be
enforcing this at the kernel interface. I think the problem with the
hard-unplug is that we don't have a good revoke mechanism for the mmio
mmaps.
For mmio we could stop the guest and replace the mmio region with a
region that is filled with 0xff, no?
Sure, but that happens in user space. The question is how does
kernel space enforce an MMIO region to not be mapped after the
hotplug event occured? Keep in mind that user space is pretty much
untrusted here - it doesn't have to be QEMU. It could just as well
be a generic user space driver. And that can just ignore hotplug
events.
We're saying you hard yank the mapping from the userspace process.
That is, you invalidate all its PTEs mapping the MMIO space, and don't
let it fault them back in.
As I see it there are two options: (a) make subsequent accesses from
userspace or the guest result in either a SIGBUS that userspace must
either deal with or die, or (b) replace the mapping with a dummy RO
mapping containing 0xff, with any trapped writes emulated as nops.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
On Fri, Aug 26, 2011 at 12:24:23AM -0400, David Gibson wrote:
On Thu, Aug 25, 2011 at 08:25:45AM -0500, Alexander Graf wrote:
quoted
On 25.08.2011, at 07:31, Roedel, Joerg wrote:
quoted
quoted
For mmio we could stop the guest and replace the mmio region with a
region that is filled with 0xff, no?
Sure, but that happens in user space. The question is how does
kernel space enforce an MMIO region to not be mapped after the
hotplug event occured? Keep in mind that user space is pretty much
untrusted here - it doesn't have to be QEMU. It could just as well
be a generic user space driver. And that can just ignore hotplug
events.
We're saying you hard yank the mapping from the userspace process.
That is, you invalidate all its PTEs mapping the MMIO space, and don't
let it fault them back in.
As I see it there are two options: (a) make subsequent accesses from
userspace or the guest result in either a SIGBUS that userspace must
either deal with or die, or (b) replace the mapping with a dummy RO
mapping containing 0xff, with any trapped writes emulated as nops.
The biggest problem with this approach is that it has to happen in the
context of the given process. Linux can't really modify an mm which
which belong to another context in a safe way.
The more I think about this, I come to the conclusion that it would be
the best to just kill the process accessing the device if it is manually
de-assigned from vfio. It should be a non-standard path anyway so it
doesn't make a lot of sense to implement complicated handling semantics
for it, no?
Joerg
--
AMD Operating System Research Center
Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo, Andrew Bowd
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632
On Fri, Aug 26, 2011 at 12:20:00AM -0400, David Gibson wrote:
On Wed, Aug 24, 2011 at 01:03:32PM +0200, Roedel, Joerg wrote:
quoted
On Wed, Aug 24, 2011 at 05:33:00AM -0400, David Gibson wrote:
quoted
On Wed, Aug 24, 2011 at 11:14:26AM +0200, Roedel, Joerg wrote:
quoted
quoted
I don't see a reason to make this meta-grouping static. It would harm
flexibility on x86. I think it makes things easier on power but there
are options on that platform to get the dynamic solution too.
I think several people are misreading what Ben means by "static". I
would prefer to say 'persistent', in that the meta-groups lifetime is
not tied to an fd, but they can be freely created, altered and removed
during runtime.
Even if it can be altered at runtime, from a usability perspective it is
certainly the best to handle these groups directly in qemu. Or are there
strong reasons to do it somewhere else?
Funny, Ben and I think usability demands it be the other way around.
The reason is that you mean the usability for the programmer and I mean
it for the actual user of qemu :)
If the meta-groups are transient - that is lifetime tied to an fd -
then any program that wants to use meta-groups *must* know the
interfaces for creating one, whatever they are.
But if they're persistent, the admin can use other tools to create the
meta-group then just hand it to a program to use, since the interfaces
for _using_ a meta-group are identical to those for an atomic group.
This doesn't preclude a program from being meta-group aware, and
creating its own if it wants to, of course. My guess is that qemu
would not want to build its own meta-groups, but libvirt probably
would.
Doing it in libvirt makes it really hard for a plain user of qemu to
assign more than one device to a guest. What I want it that a user just
types
qemu -device vfio,host=00:01.0 -device vfio,host=00:02.0 ...
and it just works. Qemu creates the meta-groups and they are
automatically destroyed when qemu exits. That the programs are not aware
of meta-groups is not a big problem because all software using vfio
needs still to be written :)
Btw, with this concept the programmer can still decide to not use
meta-groups and just multiplex the mappings to all open device-fds it
uses.
Joerg
--
AMD Operating System Research Center
Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo, Andrew Bowd
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632
From: Alexander Graf <hidden> Date: 2011-08-26 14:07:59
On 26.08.2011, at 04:33, Roedel, Joerg wrote:
On Fri, Aug 26, 2011 at 12:20:00AM -0400, David Gibson wrote:
quoted
On Wed, Aug 24, 2011 at 01:03:32PM +0200, Roedel, Joerg wrote:
quoted
On Wed, Aug 24, 2011 at 05:33:00AM -0400, David Gibson wrote:
quoted
On Wed, Aug 24, 2011 at 11:14:26AM +0200, Roedel, Joerg wrote:
=20
quoted
quoted
I don't see a reason to make this meta-grouping static. It would =
harm
quoted
quoted
quoted
quoted
flexibility on x86. I think it makes things easier on power but =
there
quoted
quoted
quoted
quoted
are options on that platform to get the dynamic solution too.
=20
I think several people are misreading what Ben means by "static". =
I
quoted
quoted
quoted
would prefer to say 'persistent', in that the meta-groups lifetime =
is
quoted
quoted
quoted
not tied to an fd, but they can be freely created, altered and =
removed
quoted
quoted
quoted
during runtime.
=20
Even if it can be altered at runtime, from a usability perspective =
it is
quoted
quoted
certainly the best to handle these groups directly in qemu. Or are =
there
quoted
quoted
strong reasons to do it somewhere else?
=20
Funny, Ben and I think usability demands it be the other way around.
=20
The reason is that you mean the usability for the programmer and I =
mean
it for the actual user of qemu :)
No, we mean the actual user of qemu. The reason being that making a =
device available for any user space application is an administrative =
task.
Forget the KVM case for a moment and think of a user space device =
driver. I as a user am not root. But I as a user when having access to =
/dev/vfioX want to be able to access the device and manage it - and only =
it. The admin of that box needs to set it up properly for me to be able =
to access it.
So having two steps is really the correct way to go:
* create VFIO group
* use VFIO group
because the two are done by completely different users. It's similar to =
how tun/tap works in Linux too. Of course nothing keeps you from also =
creating a group on the fly, but it shouldn't be the only interface =
available. The persistent setup is definitely more useful.
=20
quoted
If the meta-groups are transient - that is lifetime tied to an fd -
then any program that wants to use meta-groups *must* know the
interfaces for creating one, whatever they are.
=20
But if they're persistent, the admin can use other tools to create =
the
quoted
meta-group then just hand it to a program to use, since the =
interfaces
quoted
for _using_ a meta-group are identical to those for an atomic group.
=20
This doesn't preclude a program from being meta-group aware, and
creating its own if it wants to, of course. My guess is that qemu
would not want to build its own meta-groups, but libvirt probably
would.
=20
Doing it in libvirt makes it really hard for a plain user of qemu to
assign more than one device to a guest. What I want it that a user =
just
types
=20
qemu -device vfio,host=3D00:01.0 -device vfio,host=3D00:02.0 ...
=20
and it just works. Qemu creates the meta-groups and they are
automatically destroyed when qemu exits. That the programs are not =
aware
of meta-groups is not a big problem because all software using vfio
needs still to be written :)
=20
Btw, with this concept the programmer can still decide to not use
meta-groups and just multiplex the mappings to all open device-fds it
uses.
What I want to see is:
# vfio-create 00:01.0
/dev/vfio0
# vftio-create -a /dev/vfio0 00:02.0
/dev/vfio0
$ qemu -vfio dev=3D/dev/vfio0,id=3Dvfio0 -device vfio,vfio=3Dvfio0.0 =
-device vfio,vfio=3Dvfio0.1
Alex
On Fri, Aug 26, 2011 at 09:07:35AM -0500, Alexander Graf wrote:
On 26.08.2011, at 04:33, Roedel, Joerg wrote:
quoted
The reason is that you mean the usability for the programmer and I mean
it for the actual user of qemu :)
No, we mean the actual user of qemu. The reason being that making a
device available for any user space application is an administrative
task.
Forget the KVM case for a moment and think of a user space device
driver. I as a user am not root. But I as a user when having access to
/dev/vfioX want to be able to access the device and manage it - and
only it. The admin of that box needs to set it up properly for me to
be able to access it.
Right, and that task is being performed by attaching the device(s) in
question to the vfio driver. The rights-management happens on the
/dev/vfio/$group file.
So having two steps is really the correct way to go:
* create VFIO group
* use VFIO group
because the two are done by completely different users. It's similar
to how tun/tap works in Linux too. Of course nothing keeps you from
also creating a group on the fly, but it shouldn't be the only
interface available. The persistent setup is definitely more useful.
I see the use-case. But to make it as easy as possible for the end-user
we can do both.
So the user of (qemu again) does this:
# vfio-ctl attach 00:01.0
vfio-ctl: attached to group 8
# vfio-ctl attach 00:02.0
vfio-ctl: attached to group 16
$ qemu -device vfio-pci,host=00:01.0 -device vfio,host=00:01.0 ...
which should cover the usecase you prefer. Qemu still creates the
meta-group that allow the devices to share the same page-table. But what
should also be possible is:
# qemu -device vfio-pci,host=00:01.0 -device vfio-pci,host=00:02.0
In that case qemu detects that the devices are not yet bound to vfio and
will do so and also unbinds them afterwards (essentially the developer
use-case).
Your interface which requires pre-binding of devices into one group by
the administrator only makes sense if you want to force userspace to
use certain devices (which do not belong to the same hw-group) only
together. But I don't see a usecase for defining such constraints (yet).
Joerg
From: Alexander Graf <hidden> Date: 2011-08-26 15:29:12
On 26.08.2011, at 10:24, Joerg Roedel wrote:
On Fri, Aug 26, 2011 at 09:07:35AM -0500, Alexander Graf wrote:
quoted
On 26.08.2011, at 04:33, Roedel, Joerg wrote:
quoted
=20
The reason is that you mean the usability for the programmer and I =
mean
quoted
quoted
it for the actual user of qemu :)
=20
No, we mean the actual user of qemu. The reason being that making a
device available for any user space application is an administrative
task.
=20
Forget the KVM case for a moment and think of a user space device
driver. I as a user am not root. But I as a user when having access =
to
quoted
/dev/vfioX want to be able to access the device and manage it - and
only it. The admin of that box needs to set it up properly for me to
be able to access it.
=20
Right, and that task is being performed by attaching the device(s) in
question to the vfio driver. The rights-management happens on the
/dev/vfio/$group file.
Yup :)
=20
quoted
So having two steps is really the correct way to go:
=20
* create VFIO group
* use VFIO group
=20
because the two are done by completely different users. It's similar
to how tun/tap works in Linux too. Of course nothing keeps you from
also creating a group on the fly, but it shouldn't be the only
interface available. The persistent setup is definitely more useful.
=20
I see the use-case. But to make it as easy as possible for the =
end-user
we can do both.
=20
So the user of (qemu again) does this:
=20
# vfio-ctl attach 00:01.0
vfio-ctl: attached to group 8
# vfio-ctl attach 00:02.0
vfio-ctl: attached to group 16
$ qemu -device vfio-pci,host=3D00:01.0 -device vfio,host=3D00:01.0 ...
=20
which should cover the usecase you prefer. Qemu still creates the
meta-group that allow the devices to share the same page-table. But =
what
should also be possible is:
=20
# qemu -device vfio-pci,host=3D00:01.0 -device vfio-pci,host=3D00:02.0
=20
In that case qemu detects that the devices are not yet bound to vfio =
and
will do so and also unbinds them afterwards (essentially the developer
use-case).
I agree. The same it works with tun today. You can either have qemu =
spawn a tun device dynamically or have a preallocated one you use. If =
you run qemu as a user (which I always do), I preallocate a tun device =
and attach qemu to it.
Your interface which requires pre-binding of devices into one group by
the administrator only makes sense if you want to force userspace to
use certain devices (which do not belong to the same hw-group) only
together. But I don't see a usecase for defining such constraints =
(yet).
Agreed. As long as the kernel backend can always figure out the =
hw-groups, we're good :)
Alex
On 8/26/11 7:07 AM, "Alexander Graf" [off-list ref] wrote:
<snip>
Forget the KVM case for a moment and think of a user space device driver. I as
a user am not root. But I as a user when having access to /dev/vfioX want to
be able to access the device and manage it - and only it. The admin of that
box needs to set it up properly for me to be able to access it.
So having two steps is really the correct way to go:
* create VFIO group
* use VFIO group
because the two are done by completely different users.
This is not the case for my userspace drivers using VFIO today.
Each process will open vfio devices on the fly, and they need to be able to
share IOMMU resources.
So I need the ability to dynamically bring up devices and assign them to a
group. The number of actual devices and how they map to iommu domains is
not known ahead of time. We have a single piece of silicon that can expose
hundreds of pci devices.
In my case, the only administrative task would be to give my processes/users
access to the vfio groups (which are initially singletons), and the
application actually opens them and needs the ability to merge groups
together to conserve IOMMU resources (assuming we're not going to expose
uiommu).
-Aaron
From: Alex Williamson <hidden> Date: 2011-08-26 18:04:37
On Thu, 2011-08-25 at 20:05 +0200, Joerg Roedel wrote:
On Thu, Aug 25, 2011 at 11:20:30AM -0600, Alex Williamson wrote:
quoted
On Thu, 2011-08-25 at 12:54 +0200, Roedel, Joerg wrote:
quoted
quoted
We need to solve this differently. ARM is starting to use the iommu-api
too and this definitly does not work there. One possible solution might
be to make the iommu-ops per-bus.
That sounds good. Is anyone working on it? It seems like it doesn't
hurt to use this in the interim, we may just be watching the wrong bus
and never add any sysfs group info.
I'll cook something up for RFC over the weekend.
quoted
quoted
Also the return type should not be long but something that fits into
32bit on all platforms. Since you use -ENODEV, probably s32 is a good
choice.
The convenience of using seg|bus|dev|fn was too much to resist, too bad
it requires a full 32bits. Maybe I'll change it to:
int iommu_device_group(struct device *dev, unsigned int *group)
If we really expect segment numbers that need the full 16 bit then this
would be the way to go. Otherwise I would prefer returning the group-id
directly and partition the group-id space for the error values (s32 with
negative numbers being errors).
It's unlikely to have segments using the top bit, but it would be broken
for an iommu driver to define it's group numbers using pci s:b:d.f if we
don't have that bit available. Ben/David, do PEs have an identifier of
a convenient size? I'd guess any hardware based identifier is going to
use a full unsigned bit width. Thanks,
Alex
On 8/26/11 12:35 PM, "Chris Wright" [off-list ref] wrote:
* Aaron Fabbri (aafabbri@cisco.com) wrote:
quoted
On 8/26/11 7:07 AM, "Alexander Graf" [off-list ref] wrote:
quoted
Forget the KVM case for a moment and think of a user space device driver. I
as
a user am not root. But I as a user when having access to /dev/vfioX want to
be able to access the device and manage it - and only it. The admin of that
box needs to set it up properly for me to be able to access it.
So having two steps is really the correct way to go:
* create VFIO group
* use VFIO group
because the two are done by completely different users.
This is not the case for my userspace drivers using VFIO today.
Each process will open vfio devices on the fly, and they need to be able to
share IOMMU resources.
How do you share IOMMU resources w/ multiple processes, are the processes
sharing memory?
Sorry, bad wording. I share IOMMU domains *within* each process.
E.g. If one process has 3 devices and another has 10, I can get by with two
iommu domains (and can share buffers among devices within each process).
If I ever need to share devices across processes, the shared memory case
might be interesting.
quoted
So I need the ability to dynamically bring up devices and assign them to a
group. The number of actual devices and how they map to iommu domains is
not known ahead of time. We have a single piece of silicon that can expose
hundreds of pci devices.
This does not seem fundamentally different from the KVM use case.
We have 2 kinds of groupings.
1) low-level system or topoolgy grouping
Some may have multiple devices in a single group
* the PCIe-PCI bridge example
* the POWER partitionable endpoint
Many will not
* singleton group, e.g. typical x86 PCIe function (majority of
assigned devices)
Not sure it makes sense to have these administratively defined as
opposed to system defined.
2) logical grouping
* multiple low-level groups (singleton or otherwise) attached to same
process, allowing things like single set of io page tables where
applicable.
These are nominally adminstratively defined. In the KVM case, there
is likely a privileged task (i.e. libvirtd) involved w/ making the
device available to the guest and can do things like group merging.
In your userspace case, perhaps it should be directly exposed.
Yes. In essence, I'd rather not have to run any other admin processes.
Doing things programmatically, on the fly, from each process, is the
cleanest model right now.
quoted
In my case, the only administrative task would be to give my processes/users
access to the vfio groups (which are initially singletons), and the
application actually opens them and needs the ability to merge groups
together to conserve IOMMU resources (assuming we're not going to expose
uiommu).
I agree, we definitely need to expose _some_ way to do this.
thanks,
-chris
From: Chris Wright <hidden> Date: 2011-08-26 20:35:25
* Aaron Fabbri (aafabbri@cisco.com) wrote:
On 8/26/11 7:07 AM, "Alexander Graf" [off-list ref] wrote:
quoted
Forget the KVM case for a moment and think of a user space device driver. I as
a user am not root. But I as a user when having access to /dev/vfioX want to
be able to access the device and manage it - and only it. The admin of that
box needs to set it up properly for me to be able to access it.
So having two steps is really the correct way to go:
* create VFIO group
* use VFIO group
because the two are done by completely different users.
This is not the case for my userspace drivers using VFIO today.
Each process will open vfio devices on the fly, and they need to be able to
share IOMMU resources.
How do you share IOMMU resources w/ multiple processes, are the processes
sharing memory?
So I need the ability to dynamically bring up devices and assign them to a
group. The number of actual devices and how they map to iommu domains is
not known ahead of time. We have a single piece of silicon that can expose
hundreds of pci devices.
This does not seem fundamentally different from the KVM use case.
We have 2 kinds of groupings.
1) low-level system or topoolgy grouping
Some may have multiple devices in a single group
* the PCIe-PCI bridge example
* the POWER partitionable endpoint
Many will not
* singleton group, e.g. typical x86 PCIe function (majority of
assigned devices)
Not sure it makes sense to have these administratively defined as
opposed to system defined.
2) logical grouping
* multiple low-level groups (singleton or otherwise) attached to same
process, allowing things like single set of io page tables where
applicable.
These are nominally adminstratively defined. In the KVM case, there
is likely a privileged task (i.e. libvirtd) involved w/ making the
device available to the guest and can do things like group merging.
In your userspace case, perhaps it should be directly exposed.
In my case, the only administrative task would be to give my processes/users
access to the vfio groups (which are initially singletons), and the
application actually opens them and needs the ability to merge groups
together to conserve IOMMU resources (assuming we're not going to expose
uiommu).
I agree, we definitely need to expose _some_ way to do this.
thanks,
-chris
From: Chris Wright <hidden> Date: 2011-08-26 21:07:11
* Aaron Fabbri (aafabbri@cisco.com) wrote:
On 8/26/11 12:35 PM, "Chris Wright" [off-list ref] wrote:
quoted
* Aaron Fabbri (aafabbri@cisco.com) wrote:
quoted
Each process will open vfio devices on the fly, and they need to be able to
share IOMMU resources.
How do you share IOMMU resources w/ multiple processes, are the processes
sharing memory?
Sorry, bad wording. I share IOMMU domains *within* each process.
Ah, got it. Thanks.
E.g. If one process has 3 devices and another has 10, I can get by with two
iommu domains (and can share buffers among devices within each process).
If I ever need to share devices across processes, the shared memory case
might be interesting.
quoted
quoted
So I need the ability to dynamically bring up devices and assign them to a
group. The number of actual devices and how they map to iommu domains is
not known ahead of time. We have a single piece of silicon that can expose
hundreds of pci devices.
This does not seem fundamentally different from the KVM use case.
We have 2 kinds of groupings.
1) low-level system or topoolgy grouping
Some may have multiple devices in a single group
* the PCIe-PCI bridge example
* the POWER partitionable endpoint
Many will not
* singleton group, e.g. typical x86 PCIe function (majority of
assigned devices)
Not sure it makes sense to have these administratively defined as
opposed to system defined.
2) logical grouping
* multiple low-level groups (singleton or otherwise) attached to same
process, allowing things like single set of io page tables where
applicable.
These are nominally adminstratively defined. In the KVM case, there
is likely a privileged task (i.e. libvirtd) involved w/ making the
device available to the guest and can do things like group merging.
In your userspace case, perhaps it should be directly exposed.
Yes. In essence, I'd rather not have to run any other admin processes.
Doing things programmatically, on the fly, from each process, is the
cleanest model right now.
I don't see an issue w/ this. As long it can not add devices to the
system defined groups, it's not a privileged operation. So we still
need the iommu domain concept exposed in some form to logically put
groups into a single iommu domain (if desired). In fact, I believe Alex
covered this in his most recent recap:
...The group fd will provide interfaces for enumerating the devices
in the group, returning a file descriptor for each device in the group
(the "device fd"), binding groups together, and returning a file
descriptor for iommu operations (the "iommu fd").
thanks,
-chris
From: Avi Kivity <hidden> Date: 2011-08-28 13:14:45
On 08/26/2011 12:24 PM, Roedel, Joerg wrote:
quoted
As I see it there are two options: (a) make subsequent accesses from
userspace or the guest result in either a SIGBUS that userspace must
either deal with or die, or (b) replace the mapping with a dummy RO
mapping containing 0xff, with any trapped writes emulated as nops.
The biggest problem with this approach is that it has to happen in the
context of the given process. Linux can't really modify an mm which
which belong to another context in a safe way.
Is use_mm() insufficient?
--
error compiling committee.c: too many arguments to function
On Sun, Aug 28, 2011 at 04:14:00PM +0300, Avi Kivity wrote:
On 08/26/2011 12:24 PM, Roedel, Joerg wrote:
quoted
The biggest problem with this approach is that it has to happen in the
context of the given process. Linux can't really modify an mm which
which belong to another context in a safe way.
Is use_mm() insufficient?
Yes, it introduces a set of race conditions when a process that already
has an mm wants to take over another processes mm temporarily (and when
use_mm is modified to actually provide this functionality). It is only
save when used from kernel-thread context.
One example:
Process A Process B Process C
. . .
. <-- takes A->mm .
. and assignes as B->mm .
. . --> Wants to take
. . B->mm, but gets
A->mm now
This can't be secured by a lock, because it introduces potential
A->B<-->B->A lock problem when two processes try to take each others mm.
It could probably be solved by a task->real_mm pointer, havn't thought
about this yet...
Joerg
From: Avi Kivity <hidden> Date: 2011-08-28 14:04:50
On 08/28/2011 04:56 PM, Joerg Roedel wrote:
On Sun, Aug 28, 2011 at 04:14:00PM +0300, Avi Kivity wrote:
quoted
On 08/26/2011 12:24 PM, Roedel, Joerg wrote:
quoted
quoted
The biggest problem with this approach is that it has to happen in the
context of the given process. Linux can't really modify an mm which
which belong to another context in a safe way.
Is use_mm() insufficient?
Yes, it introduces a set of race conditions when a process that already
has an mm wants to take over another processes mm temporarily (and when
use_mm is modified to actually provide this functionality). It is only
save when used from kernel-thread context.
One example:
Process A Process B Process C
. . .
. <-- takes A->mm .
. and assignes as B->mm .
. . --> Wants to take
. . B->mm, but gets
A->mm now
Good catch.
This can't be secured by a lock, because it introduces potential
A->B<-->B->A lock problem when two processes try to take each others mm.
It could probably be solved by a task->real_mm pointer, havn't thought
about this yet...
Or a workqueue - you get a kernel thread context with a bit of boilerplate.
--
error compiling committee.c: too many arguments to function
From: David Gibson <hidden> Date: 2011-08-30 01:34:58
eOn Fri, Aug 26, 2011 at 01:17:05PM -0700, Aaron Fabbri wrote:
[snip]
Yes. In essence, I'd rather not have to run any other admin processes.
Doing things programmatically, on the fly, from each process, is the
cleanest model right now.
The "persistent group" model doesn't necessarily prevent that.
There's no reason your program can't use the administrative interface
as well as the "use" interface, and I don't see that making the admin
interface separate and persistent makes this any harder.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
On Fri, Aug 26, 2011 at 12:04:22PM -0600, Alex Williamson wrote:
On Thu, 2011-08-25 at 20:05 +0200, Joerg Roedel wrote:
quoted
If we really expect segment numbers that need the full 16 bit then this
would be the way to go. Otherwise I would prefer returning the group-id
directly and partition the group-id space for the error values (s32 with
negative numbers being errors).
It's unlikely to have segments using the top bit, but it would be broken
for an iommu driver to define it's group numbers using pci s:b:d.f if we
don't have that bit available. Ben/David, do PEs have an identifier of
a convenient size? I'd guess any hardware based identifier is going to
use a full unsigned bit width.
Okay, if we want to go the secure way I am fine with the "int *group"
parameter. Another option is to just return u64 and use the extended
number space for errors. But that is even worse as an interface, I
think.
Joerg
On Sun, Aug 28, 2011 at 05:04:32PM +0300, Avi Kivity wrote:
On 08/28/2011 04:56 PM, Joerg Roedel wrote:
quoted
This can't be secured by a lock, because it introduces potential
A->B<-->B->A lock problem when two processes try to take each others mm.
It could probably be solved by a task->real_mm pointer, havn't thought
about this yet...
Or a workqueue - you get a kernel thread context with a bit of boilerplate.
Right, a workqueue might do the trick. We'll evaluate that. Thanks for
the idea :)
Joerg