From: Jason Gunthorpe <redacted>
Sent: Wednesday, May 5, 2021 1:13 AM

On Wed, Apr 28, 2021 at 06:58:19AM +0000, Tian, Kevin wrote:

quoted

quoted

From: Jason Gunthorpe <redacted>
Sent: Wednesday, April 28, 2021 1:12 AM

[...]

quoted

quoted

As Alex says, if this line fails because of the group restrictions,
that's not great because it's not very obvious what's gone wrong.

Okay, that is fair, but let's solve that problem directly. For
instance netlink has been going in the direction of adding a "extack"
from the kernel which is a descriptive error string. If the failing
ioctl returned the string:

  "cannot join this device to the IOASID because device XXX in the
   same group #10 is in use"

Would you agree it is now obvious what has gone wrong? In fact would
you agree this is a lot better user experience than what applications
do today even though they have the group FD?

Currently all the discussions are around implicit vs. explicit uAPI semantics
on the group restriction. However if we look beyond group the implicit
semantics might be inevitable when dealing with incompatible iommu
domains. An existing example of iommu incompatibility is IOMMU_
CACHE.

I still think we need to get rid of these incompatibilities
somehow. Having multiple HW incompatible IOASID in the same platform
is just bad all around.

When modeling in userspace IOMMU_CACHE sounds like it is a property of
each individual IOASID, not an attribute that requires a new domain.

sure. the iommu domain is an kernel-internal concept. The userspace 
should focus everything on IOASID.

People that want to create cache bypass IOASID's should just ask for
that that directly.

Yes, in earlier discussion we agreed on a scheme that ioasid module
will return an error to userspace indicating incompatibility detected
when binding a device to ioasid then the userspace should create 
a new IOASID for this device. This has to be done 'explicitly'. 

When I used it as the example for 'implicit semantics" is that the kernel 
won't create another group-like object to contain devices with compatible 
attributes and 'explicitly' manage it in uAPI like group_fd. If we anyway
rely on the userspace to have more intelligence on those hardware
restrictions, it's little sense to only explicitly handle group_fd in uAPI.

Thanks
Kevin