On Tue, Oct 26, 2021 at 08:52:10AM -0600, Alex Williamson wrote:

On Tue, 26 Oct 2021 09:13:53 -0300
Jason Gunthorpe [off-list ref] wrote:

quoted

On Tue, Oct 26, 2021 at 09:40:34AM +0100, Dr. David Alan Gilbert wrote:

quoted

* Jason Gunthorpe (jgg@nvidia.com) wrote:  

quoted

On Mon, Oct 25, 2021 at 07:47:29PM +0100, Dr. David Alan Gilbert wrote:
  

quoted

It may need some further refinement; for example in that quiesed state
do counters still tick? will a NIC still respond to packets that don't
get forwarded to the host?  

At least for the mlx5 NIC the two states are 'able to issue outbound
DMA' and 'all internal memories and state are frozen and unchanging'.  

Yeh, so my point was just that if you're adding a new state to this
process, you need to define the details like that.  

We are not planning to propose any patches/uAPI specification for this
problem until after the mlx5 vfio driver is merged..

I'm not super comfortable with that.  If we're expecting to add a new
bit to define a quiescent state prior to clearing the running flag and
this is an optional device feature that userspace migration needs to be
aware of and it's really not clear from a hypervisor when p2p DMA might
be in use, I think that leaves userspace in a pickle how and when
they'd impose restrictions on assignment with multiple assigned
devices.  It's likely that the majority of initial use cases wouldn't
need this feature, which would make it difficult to arbitrarily impose
later.

Either supporting a freeze/quiesce split is an optional feature, and
current mlx5_vfio is an example of a driver that does not implement
it

Or, freeze/quiesce is a mandatory HW feature and no VFIO driver will
be merged that doesn't implement it - even if the HW cannot support
it.

Frankly, I think we will see HW that doesn't support this and VFIO
will be required to have it as an optional feature

What does userspace do? I don't know - but it does need to be aware if
the optional HW feature is available and it does need to make choices
if the VM will be migratable or not.

I don't see any path where userspace truly cannot care unless HW
support for this is mandatory.

Jason