On Fri, Nov 26 2021, Jason Gunthorpe [off-list ref] wrote:

On Fri, Nov 26, 2021 at 01:56:26PM +0100, Cornelia Huck wrote:

quoted

On Thu, Nov 25 2021, Jason Gunthorpe [off-list ref] wrote:

quoted

On Thu, Nov 25, 2021 at 01:27:12PM +0100, Cornelia Huck wrote:

quoted

On Wed, Nov 24 2021, Jason Gunthorpe [off-list ref] wrote:

quoted

On Wed, Nov 24, 2021 at 05:55:49PM +0100, Cornelia Huck wrote:

quoted

quoted

What I meant to say: If we give userspace the flexibility to operate
this, we also must give different device types some flexibility. While
subchannels will follow the general flow, they'll probably condense/omit
some steps, as I/O is quite different to PCI there.

I would say no - migration is general, no device type should get to
violate this spec.  Did you have something specific in mind? There is
very little PCI specific here already

I'm not really thinking about violating the spec, but more omitting
things that do not really apply to the hardware. For example, it is
really easy to shut up a subchannel, we don't really need to wait until
nothing happens anymore, and it doesn't even have MMIO. 

I've never really looked closely at the s390 mdev drivers..

What does something like AP even do anyhow? The ioctl handler doesn't
do anything, there is no mmap hook, how does the VFIO userspace
interact with this thing?

For AP, the magic is in the hardware/firmware; the vfio parts are needed
to configure what is exposed to a given guest, not for operation. Once
it is up, the hardware will handle any instructions directly, the
hypervisor will not see them. (Unfortunately, none of the details have
public documentation.) I have no idea how this would play with migration.

That is kind of what I thought..

VFIO is all about exposing a device to userspace control, sounds like
the S390 drivers skipped that step.

Note that what I wrote above is about AP; CCW does indeed trigger
operations like start subchannel from userspace and relays interrupts
back to userspace. AP is just very dissimilar to basically anything
else.

KVM is all about taking what userspace can already control and giving
it to a guest, in an accelerated way.

Making a bypass where a KVM guest has more capability than the user
process because VFIO and KVM have been directly coupled completely
upends the whole logical model.

As we talked with Intel's wbinvd stuff you should have a mental model
where the VFIO userspace process can do anything the KVM guest can do
via ioctls on the mdev. KVM is just an accelerated way to do that same
stuff. Maybe S390 doesn't implement those ioctls, but they are
logically part of the model.

FWIW, AP had been a pain to model in a way that we could hand the
devices to the guest; if we are supposed to use vfio for this purpose,
the current design is probably the best we can get, at least nobody has
been able to come up with a better way to interact with the interfaces
that we have.

CCW needs a kernel part for translations, as it doesn't have an iommu,
and the I/O instructions are of course privileged (but so are the
instructions for s390 PCI); I think it is quite close to other devices
in other respects, only that it has a more transaction-based model.

So, for the migration doc, imagine some non-accelerated KVM that was
intercepting the guest operations and calling the logical ioctls on
the mdev instead. When we talk about MMIO/PIO/etc it also includes
mdev operation ioctls too, and by extension any ioctl accelerated
inside KVM.

I think only AP is the really odd one out here; CCW will likely differ
in some details... I just wanted to make sure that this will not run
counter to the documentation.