virtio-blk: add num_io_queues module parameter

Started by Max Gurtovoy on Aug 30, 2021 · 4 messages from 3 authors · Last reply Aug 30, 2021

Subsystems (4)

Show 1 more

Maintainers (5)

Show 1 more
STALE1902d

open the first message on its own page

Max Gurtovoy

Message details and related content
Open message details and related content
Sometimes a user would like to control the amount of IO queues to be
created for a block device. For example, for limiting the memory
footprint of virtio-blk devices.

Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
---
 drivers/block/virtio_blk.c | 26 +++++++++++++++++++++++++-
 1 file changed, 25 insertions(+), 1 deletion(-)

diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
index e574fbf5e6df..77e8468e8593 100644
--- a/drivers/block/virtio_blk.c
+++ b/drivers/block/virtio_blk.c
@@ -24,6 +24,28 @@
 /* The maximum number of sg elements that fit into a virtqueue */
 #define VIRTIO_BLK_MAX_SG_ELEMS 32768
 
+static int virtblk_queue_count_set(const char *val,
+		const struct kernel_param *kp)
+{
+	unsigned int n;
+	int ret;
+
+	ret = kstrtouint(val, 10, &n);
+	if (ret != 0 || n > nr_cpu_ids)
+		return -EINVAL;
+	return param_set_uint(val, kp);
+}
+
+static const struct kernel_param_ops queue_count_ops = {
+	.set = virtblk_queue_count_set,
+	.get = param_get_uint,
+};
+
+static unsigned int num_io_queues;
+module_param_cb(num_io_queues, &queue_count_ops, &num_io_queues, 0644);
+MODULE_PARM_DESC(num_io_queues,
+		 "Number of IO virt queues to use for blk device.");
+
 static int major;
 static DEFINE_IDA(vd_index_ida);
 
@@ -501,7 +523,9 @@ static int init_vq(struct virtio_blk *vblk)
 	if (err)
 		num_vqs = 1;
 
-	num_vqs = min_t(unsigned int, nr_cpu_ids, num_vqs);
+	num_vqs = min_t(unsigned int,
+			min_not_zero(num_io_queues, nr_cpu_ids),
+			num_vqs);
 
 	vblk->vqs = kmalloc_array(num_vqs, sizeof(*vblk->vqs), GFP_KERNEL);
 	if (!vblk->vqs)
-- 
2.18.1

Christoph Hellwig replied to Max Gurtovoy

Message details and related content
Open message details and related content
On Mon, Aug 30, 2021 at 03:00:23PM +0300, Max Gurtovoy wrote:
quoted hunk
+static int virtblk_queue_count_set(const char *val,
+		const struct kernel_param *kp)
+{
+	unsigned int n;
+	int ret;
+
+	ret = kstrtouint(val, 10, &n);
+	if (ret != 0 || n > nr_cpu_ids)
+		return -EINVAL;
+	return param_set_uint(val, kp);
+}

Thi can use param_set_uint_minmax.

Michael S. Tsirkin replied to Max Gurtovoy

Message details and related content
Open message details and related content
On Mon, Aug 30, 2021 at 03:00:23PM +0300, Max Gurtovoy wrote:
Sometimes a user would like to control the amount of IO queues to be
created for a block device. For example, for limiting the memory
footprint of virtio-blk devices.

Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>

Hmm. It's already limited by # of CPUs... Why not just limit
from the hypervisor side? What's the actual use-case here?
quoted hunk
---
 drivers/block/virtio_blk.c | 26 +++++++++++++++++++++++++-
 1 file changed, 25 insertions(+), 1 deletion(-)

diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
index e574fbf5e6df..77e8468e8593 100644
--- a/drivers/block/virtio_blk.c
+++ b/drivers/block/virtio_blk.c
@@ -24,6 +24,28 @@
 /* The maximum number of sg elements that fit into a virtqueue */
 #define VIRTIO_BLK_MAX_SG_ELEMS 32768
 
+static int virtblk_queue_count_set(const char *val,
+		const struct kernel_param *kp)
+{
+	unsigned int n;
+	int ret;
+
+	ret = kstrtouint(val, 10, &n);
+	if (ret != 0 || n > nr_cpu_ids)
+		return -EINVAL;
+	return param_set_uint(val, kp);
+}
+
+static const struct kernel_param_ops queue_count_ops = {
+	.set = virtblk_queue_count_set,
+	.get = param_get_uint,
+};
+
+static unsigned int num_io_queues;
+module_param_cb(num_io_queues, &queue_count_ops, &num_io_queues, 0644);
+MODULE_PARM_DESC(num_io_queues,
+		 "Number of IO virt queues to use for blk device.");
+
 static int major;
 static DEFINE_IDA(vd_index_ida);
 
@@ -501,7 +523,9 @@ static int init_vq(struct virtio_blk *vblk)
 	if (err)
 		num_vqs = 1;

10 more quoted lines






  

    

      
      Max Gurtovoy      replied to Michael S. Tsirkin      
      
    

  

  

    Message details and related content
    
Open message details and related content

  

  
On 8/31/2021 12:48 AM, Michael S. Tsirkin wrote:
On Mon, Aug 30, 2021 at 03:00:23PM +0300, Max Gurtovoy wrote:
quoted
Sometimes a user would like to control the amount of IO queues to be
created for a block device. For example, for limiting the memory
footprint of virtio-blk devices.

Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
Hmm. It's already limited by # of CPUs... Why not just limit
from the hypervisor side? What's the actual use-case here?
Limiting and minimizing resource allocation is a real use case.

# of CPUs today might be 64 or 128. HW virtio-blk device might have this amount of queues (or at least 32).

But it's a waste to use all the queues since the device may reach to max IOPs with less queues. Multiply this by 16 or 32 devices we get a lot of memory wasted without a real need.

It's a common configuration we do in NVMf connect command and it can also be seen in other drivers in some variation (null_blk.submit_queues, ib_srp.ch_count and more).

Another use case is to add some flexibility for QOS.

Also if we can set the queue depth, it's a good idea to control the queue count as well.

If no objections, I'll take the comment from Christoph and send v2.

quoted
---
  drivers/block/virtio_blk.c | 26 +++++++++++++++++++++++++-
  1 file changed, 25 insertions(+), 1 deletion(-)

diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c
index e574fbf5e6df..77e8468e8593 100644
--- a/drivers/block/virtio_blk.c
+++ b/drivers/block/virtio_blk.c
@@ -24,6 +24,28 @@
  /* The maximum number of sg elements that fit into a virtqueue */
  #define VIRTIO_BLK_MAX_SG_ELEMS 32768
  
+static int virtblk_queue_count_set(const char *val,
+		const struct kernel_param *kp)
+{
+	unsigned int n;
+	int ret;
+
+	ret = kstrtouint(val, 10, &n);
+	if (ret != 0 || n > nr_cpu_ids)
+		return -EINVAL;
+	return param_set_uint(val, kp);
+}
+
+static const struct kernel_param_ops queue_count_ops = {
+	.set = virtblk_queue_count_set,
+	.get = param_get_uint,
+};
+
+static unsigned int num_io_queues;
+module_param_cb(num_io_queues, &queue_count_ops, &num_io_queues, 0644);
+MODULE_PARM_DESC(num_io_queues,
+		 "Number of IO virt queues to use for blk device.");
+
  static int major;
  static DEFINE_IDA(vd_index_ida);
  
@@ -501,7 +523,9 @@ static int init_vq(struct virtio_blk *vblk)
  	if (err)
11 more quoted lines






    
  

  
    

      

        Keyboard shortcuts
      

      

        
          
hback out one level

          
jnext message in thread / next series-diff position

          
kprevious message in thread / previous series-diff position

          
ldrill in

          
Escclose help / fold thread tree

          
?toggle this help