On Fri, Nov 26, 2021 at 01:56:26PM +0100, Cornelia Huck wrote:

On Thu, Nov 25 2021, Jason Gunthorpe [off-list ref] wrote:

quoted

On Thu, Nov 25, 2021 at 01:27:12PM +0100, Cornelia Huck wrote:

quoted

On Wed, Nov 24 2021, Jason Gunthorpe [off-list ref] wrote:

quoted

On Wed, Nov 24, 2021 at 05:55:49PM +0100, Cornelia Huck wrote:

quoted

quoted

What I meant to say: If we give userspace the flexibility to operate
this, we also must give different device types some flexibility. While
subchannels will follow the general flow, they'll probably condense/omit
some steps, as I/O is quite different to PCI there.

I would say no - migration is general, no device type should get to
violate this spec.  Did you have something specific in mind? There is
very little PCI specific here already

I'm not really thinking about violating the spec, but more omitting
things that do not really apply to the hardware. For example, it is
really easy to shut up a subchannel, we don't really need to wait until
nothing happens anymore, and it doesn't even have MMIO. 

I've never really looked closely at the s390 mdev drivers..

What does something like AP even do anyhow? The ioctl handler doesn't
do anything, there is no mmap hook, how does the VFIO userspace
interact with this thing?

For AP, the magic is in the hardware/firmware; the vfio parts are needed
to configure what is exposed to a given guest, not for operation. Once
it is up, the hardware will handle any instructions directly, the
hypervisor will not see them. (Unfortunately, none of the details have
public documentation.) I have no idea how this would play with migration.

That is kind of what I thought..

VFIO is all about exposing a device to userspace control, sounds like
the S390 drivers skipped that step.

KVM is all about taking what userspace can already control and giving
it to a guest, in an accelerated way.

Making a bypass where a KVM guest has more capability than the user
process because VFIO and KVM have been directly coupled completely
upends the whole logical model.

As we talked with Intel's wbinvd stuff you should have a mental model
where the VFIO userspace process can do anything the KVM guest can do
via ioctls on the mdev. KVM is just an accelerated way to do that same
stuff. Maybe S390 doesn't implement those ioctls, but they are
logically part of the model.

So, for the migration doc, imagine some non-accelerated KVM that was
intercepting the guest operations and calling the logical ioctls on
the mdev instead. When we talk about MMIO/PIO/etc it also includes
mdev operation ioctls too, and by extension any ioctl accelerated
inside KVM.

Jason