From: Jason Gunthorpe <jgg@nvidia.com>
Sent: Wednesday, June 2, 2021 1:57 AM

On Tue, Jun 01, 2021 at 08:38:00AM +0000, Tian, Kevin wrote:

quoted

quoted

From: Jason Gunthorpe <jgg@nvidia.com>
Sent: Saturday, May 29, 2021 3:59 AM

On Thu, May 27, 2021 at 07:58:12AM +0000, Tian, Kevin wrote:

quoted

5. Use Cases and Flows

Here assume VFIO will support a new model where every bound device
is explicitly listed under /dev/vfio thus a device fd can be acquired w/o
going through legacy container/group interface. For illustration purpose
those devices are just called dev[1...N]:

	device_fd[1...N] = open("/dev/vfio/devices/dev[1...N]", mode);

As explained earlier, one IOASID fd is sufficient for all intended use

cases:

quoted

quoted

quoted

	ioasid_fd = open("/dev/ioasid", mode);

For simplicity below examples are all made for the virtualization story.
They are representative and could be easily adapted to a non-

virtualization

quoted

quoted

quoted

scenario.

For others, I don't think this is *strictly* necessary, we can
probably still get to the device_fd using the group_fd and fit in
/dev/ioasid. It does make the rest of this more readable though.

Jason, want to confirm here. Per earlier discussion we remain an
impression that you want VFIO to be a pure device driver thus
container/group are used only for legacy application.

Let me call this a "nice wish".

If you get to a point where you hard need this, then identify the hard
requirement and let's do it, but I wouldn't bloat this already large
project unnecessarily.

OK, got your point. So let's start by keeping this room. For new
sub-systems like vDPA,  they don't need inventing group fd uAPI
and just leave to their user to meet the group limitation. For existing
sub-system i.e. VFIO, it could keep a stronger group enforcement
uAPI like today. One day, we may revisit it if the simple policy works
well for all other new sub-systems.

Similarly I wouldn't depend on the group fd existing in this design
so it could be changed later.

Yes, this is guaranteed. /dev/ioasid uAPI has no group concept.

quoted

From this comment are you suggesting that VFIO can still keep
container/ group concepts and user just deprecates the use of vfio
iommu uAPI (e.g. VFIO_SET_IOMMU) by using /dev/ioasid (which has a
simple policy that an IOASID will reject cmd if partially-attached
group exists)?

I would say no on the container. /dev/ioasid == the container, having
two competing objects at once in a single process is just a mess.

If the group fd can be kept requires charting a path through the
ioctls where the container is not used and /dev/ioasid is sub'd in
using the same device FD specific IOCTLs you show here.

yes

I didn't try to chart this out carefully.

Also, ultimately, something need to be done about compatability with
the vfio container fd. It looks clear enough to me that the the VFIO
container FD is just a single IOASID using a special ioctl interface
so it would be quite rasonable to harmonize these somehow.

Possibly multiple IOASIDs as VFIO container cay hold incompatible devices
today. Suppose helper functions will be provided for VFIO container to
create IOASID and then use map/unmap to manage its I/O page table.
This is the shim iommu driver concept in previous discussion between
you and Alex.

This can be done at a later stage. Let's focus on /dev/ioasid uAPI, and
bear some code duplication between it and vfio type1 for now. 

But that is too complicated and far out for me at least to guess on at
this point..

We're working on a prototype in parallel with this discussion. Based on
this work we'll figure out what's the best way to start with.

Thanks
Kevin