From: Jason Wang <redacted>
Sent: Wednesday, June 1, 2022 11:54 PM

On Thu, Jun 2, 2022 at 10:59 AM Parav Pandit [off-list ref] wrote:

quoted

quoted

From: Jason Wang <redacted>
Sent: Wednesday, June 1, 2022 10:00 PM

On Thu, Jun 2, 2022 at 2:58 AM Parav Pandit [off-list ref] wrote:

quoted

quoted

From: Jason Wang <redacted>
Sent: Tuesday, May 31, 2022 10:42 PM

Well, the ability to query the virtqueue state was proposed as
another feature (Eugenio, please correct me). This should be
sufficient for making virtio-net to be live migrated.

The device is stopped, it won't answer to this special vq config done

here.

quoted

quoted

This depends on the definition of the stop. Any query to the device
state should be allowed otherwise it's meaningless for us.

quoted

Programming all of these using cfg registers doesn't scale for
on-chip

memory and for the speed.

Well, they are orthogonal and what I want to say is, we should first
define the semantics of stop and state of the virtqueue.

Such a facility could be accessed by either transport specific
method or admin virtqueue, it totally depends on the hardware

architecture of the vendor.

quoted

quoted

I find it hard to believe that a vendor can implement a CVQ but not AQ and

chose to expose tens of hundreds of registers.

quoted

But maybe, it fits some specific hw.

You can have a look at the ifcvf dpdk driver as an example.

Ifcvf is an example of using registers.
It is not an answer why AQ is hard for it. :)
virtio spec has definition of queue now and implementing yet another queue shouldn't be a problem.

So far no one seem to have problem with the additional queue.
So I take it as AQ is ok.

But another thing that is unrelated to hardware architecture is the nesting
support. Having admin virtqueue in a nesting environment looks like an
overkill. Presenting a register in L1 and map it to L0's admin should be good
enough.

So may be a optimized interface can be added that fits nested env.
At this point in time real users that we heard are interested in non-nested use cases. Let's enable them first.

quoted

I like to learn the advantages of such method other than simplicity.

We can clearly that we are shifting away from such PCI registers with SIOV,

IMS and other scalable solutions.

quoted

virtio drifting in reverse direction by introducing more registers as

transport.

quoted

I expect it to an optional transport like AQ.

Actually, I had a proposal of using admin virtqueue as a transport, it's
designed to be SIOV/IMS capable. And it's not hard to extend it with the
state/stop support etc.

quoted

quoted

quoted

Next would be to program hundreds of statistics of the 64 VQs
through a

giant PCI config space register in some busy polling scheme.

We don't need giant config space, and this method has been
implemented by some vDPA vendors.

There are tens of 64-bit counters per VQs. These needs to programmed on

destination side.

quoted

Programming these via registers requires exposing them on the registers.
In one of the proposals, I see them being queried via CVQ from the device.

I didn't see a proposal like this. And I don't think querying general virtio state
like idx with a device specific CVQ is a good design.

My example was not for the idx. But for VQ statistics that is queried via CVQ.

quoted

Programming them via cfg registers requires large cfg space or synchronous

programming until receiving ACK from it.

quoted

This means one entry at a time...

Programming them via CVQ needs replicate and align cmd values etc on all

device types. All duplicate and hard to maintain.

quoted

quoted

quoted

I can clearly see how all these are inefficient for faster LM.
We need an efficient AQ to proceed with at minimum.

I'm fine with admin virtqueue, but the stop and state are orthogonal to

that.

quoted

quoted

And using admin virtqueue for stop/state will be more natural if we
use admin virtqueue as a transport.

Ok.
We should have defined it bit earlier that all vendors can use. :(

I agree.

I remember few months back, you acked in the weekly meeting that TC has approved the AQ direction.
And we are still in this circle of debating the AQ.