Re: [PATCH vhost 20/23] vdpa/mlx5: Pre-create hardware VQs at vdpa .dev_add time

On Mon, Jul 08, 2024 at 11:17:06AM +0000, Dragos Tatulea wrote:

On Mon, 2024-07-08 at 07:11 -0400, Michael S. Tsirkin wrote:

quoted

On Mon, Jul 08, 2024 at 11:01:39AM +0000, Dragos Tatulea wrote:

quoted

On Wed, 2024-07-03 at 18:01 +0200, Eugenio Perez Martin wrote:

quoted

On Wed, Jun 26, 2024 at 11:27 AM Dragos Tatulea [off-list ref] wrote:

quoted

On Wed, 2024-06-19 at 17:54 +0200, Eugenio Perez Martin wrote:

quoted

On Mon, Jun 17, 2024 at 5:09 PM Dragos Tatulea [off-list ref] wrote:

quoted

Currently, hardware VQs are created right when the vdpa device gets into
DRIVER_OK state. That is easier because most of the VQ state is known by
then.

This patch switches to creating all VQs and their associated resources
at device creation time. The motivation is to reduce the vdpa device
live migration downtime by moving the expensive operation of creating
all the hardware VQs and their associated resources out of downtime on
the destination VM.

The VQs are now created in a blank state. The VQ configuration will
happen later, on DRIVER_OK. Then the configuration will be applied when
the VQs are moved to the Ready state.

When .set_vq_ready() is called on a VQ before DRIVER_OK, special care is
needed: now that the VQ is already created a resume_vq() will be
triggered too early when no mr has been configured yet. Skip calling
resume_vq() in this case, let it be handled during DRIVER_OK.

For virtio-vdpa, the device configuration is done earlier during
.vdpa_dev_add() by vdpa_register_device(). Avoid calling
setup_vq_resources() a second time in that case.

I guess this happens if virtio_vdpa is already loaded, but I cannot
see how this is different here. Apart from the IOTLB, what else does
it change from the mlx5_vdpa POV?

I don't understand your question, could you rephrase or provide more context
please?

My main point is that the vdpa parent driver should not be able to
tell the difference between vhost_vdpa and virtio_vdpa. The only
difference I can think of is because of the vhost IOTLB handling.

Do you also observe this behavior if you add the device with "vdpa
add" without the virtio_vdpa module loaded, and then modprobe
virtio_vdpa?

Aah, now I understand what you mean. Indeed in my tests I was loading the
virtio_vdpa module before adding the device. When doing it the other way around
the device doesn't get configured during probe.
 

quoted

At least the comment should be something in the line of "If we have
all the information to initialize the device, pre-warm it here" or
similar.

Makes sense. I will send a v3 with the commit + comment message update.

Is commit update the only change then?

I was planning to drop the paragraph in the commit message (it is confusing) and
edit the comment below (scroll down to see which).

Let me know if I should send the v3 or not. I have it prepared.

You can do this but pls document that the only change is in commit log.

quoted

quoted

quoted

quoted

Thanks,
Dragos

quoted

quoted

Signed-off-by: Dragos Tatulea <dtatulea@nvidia.com>
Reviewed-by: Cosmin Ratiu <redacted>
---
 drivers/vdpa/mlx5/net/mlx5_vnet.c | 37 ++++++++++++++++++++++++++++++++-----
 1 file changed, 32 insertions(+), 5 deletions(-)

diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c
index 249b5afbe34a..b2836fd3d1dd 100644
--- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
+++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c

@@ -2444,7 +2444,7 @@ static void mlx5_vdpa_set_vq_ready(struct vdpa_device *vdev, u16 idx, bool ready
        mvq = &ndev->vqs[idx];
        if (!ready) {
                suspend_vq(ndev, mvq);
-       } else {
+       } else if (mvdev->status & VIRTIO_CONFIG_S_DRIVER_OK) {
                if (resume_vq(ndev, mvq))
                        ready = false;
        }

@@ -3078,10 +3078,18 @@ static void mlx5_vdpa_set_status(struct vdpa_device *vdev, u8 status)
                                goto err_setup;
                        }
                        register_link_notifier(ndev);
-                       err = setup_vq_resources(ndev, true);
-                       if (err) {
-                               mlx5_vdpa_warn(mvdev, "failed to setup driver\n");
-                               goto err_driver;
+                       if (ndev->setup) {
+                               err = resume_vqs(ndev);
+                               if (err) {
+                                       mlx5_vdpa_warn(mvdev, "failed to resume VQs\n");
+                                       goto err_driver;
+                               }
+                       } else {
+                               err = setup_vq_resources(ndev, true);
+                               if (err) {
+                                       mlx5_vdpa_warn(mvdev, "failed to setup driver\n");
+                                       goto err_driver;
+                               }
                        }
                } else {
                        mlx5_vdpa_warn(mvdev, "did not expect DRIVER_OK to be cleared\n");

@@ -3142,6 +3150,7 @@ static int mlx5_vdpa_compat_reset(struct vdpa_device *vdev, u32 flags)
                if (mlx5_vdpa_create_dma_mr(mvdev))
                        mlx5_vdpa_warn(mvdev, "create MR failed\n");
        }
+       setup_vq_resources(ndev, false);
        up_write(&ndev->reslock);

        return 0;

@@ -3836,8 +3845,21 @@ static int mlx5_vdpa_dev_add(struct vdpa_mgmt_dev *v_mdev, const char *name,
                goto err_reg;

        mgtdev->ndev = ndev;
+
+       /* For virtio-vdpa, the device was set up during device register. */
+       if (ndev->setup)
+               return 0;
+

This comment updated to:

/* The VQs might have been pre-created during device register.
 * This happens when virtio_vdpa is loaded before the vdpa device is added.
 */

quoted

quoted

quoted

quoted

quoted

quoted

+       down_write(&ndev->reslock);
+       err = setup_vq_resources(ndev, false);
+       up_write(&ndev->reslock);
+       if (err)
+               goto err_setup_vq_res;
+
        return 0;

+err_setup_vq_res:
+       _vdpa_unregister_device(&mvdev->vdev);
 err_reg:
        destroy_workqueue(mvdev->wq);
 err_res2:

@@ -3863,6 +3885,11 @@ static void mlx5_vdpa_dev_del(struct vdpa_mgmt_dev *v_mdev, struct vdpa_device *

        unregister_link_notifier(ndev);
        _vdpa_unregister_device(dev);
+
+       down_write(&ndev->reslock);
+       teardown_vq_resources(ndev);
+       up_write(&ndev->reslock);
+
        wq = mvdev->wq;
        mvdev->wq = NULL;
        destroy_workqueue(wq);

--

2.45.1

Thanks,
Dragos