[RESEND PATCH v2 0/4] nvme-tcp: NIC topology aware I/O queue scaling and queue info export

Hi,

This is a resend of the previous series to include the networking maintainers
and mailing list. There are no code or commit message changes since the
previous posting.

This series has been updated based on the feedback received during LSFMM.
The changelog is updated accordingly.

The NVMe/TCP host driver currently provisions I/O queues primarily based
on CPU availability rather than the capabilities and topology of the
underlying network interface.

On modern systems with many CPUs but fewer NIC hardware queues, this can
lead to multiple NVMe/TCP I/O workers contending for the same TX/RX queue,
resulting in increased lock contention, cacheline bouncing, and degraded
throughput.

This RFC proposes a set of changes to better align NVMe/TCP I/O queues
with NIC queue resources, and to expose queue/flow information to enable
more effective system-level tuning.

Key ideas
---------

1. Scale NVMe/TCP I/O queues based on NIC queue count
   Instead of relying solely on CPU count, limit the number of I/O workers
   to:
       min(num_online_cpus, netdev->real_num_{tx,rx}_queues)

2. Improve CPU locality
   Align NVMe/TCP I/O workers with CPUs associated with NIC IRQ affinity
   to reduce cross-CPU traffic and improve cache locality.

3. Expose queue and flow information via debugfs
   Export per-I/O queue information including:
       - queue id (qid)
       - CPU affinity
       - TCP flow (src/dst IP and ports)

   This enables userspace tools to configure:
       - IRQ affinity
       - RPS/XPS
       - ntuple steering
       - or any other scaling as deemed feasible

4. Provide infrastructure for extensible debugfs support in NVMe

Together, these changes allow better alignment of:
    flow -> NIC queue -> IRQ -> CPU -> NVMe/TCP I/O worker

Performance Evaluation
----------------------
Tests were conducted using fio over NVMe/TCP with the following parameters:
    ioengine=io_uring
    direct=1
    bs=4k
    numjobs=<#nic-queues>
    iodepth=64
System:
    CPUs: 72
    NIC: 100G mlx5

Two configurations were evaluated.

Scenario 1: NIC queues < CPU count
----------------------------------
- CPUs: 72
- NIC queues: 32

                Baseline        Patched        Patched + tuning
randread        3141 MB/s       3228 MB/s      7509 MB/s
                (767k IOPS)     (788k IOPS)    (1833k IOPS)

randwrite       4510 MB/s       6172 MB/s      7518 MB/s
                (1101k IOPS)    (1507k IOPS)   (1836k IOPS)

randrw (read)   2156 MB/s       2560 MB/s      3932 MB/s
                (526k IOPS)     (625k IOPS)    (960k IOPS)

randrw (write)  2155 MB/s       2560 MB/s      3932 MB/s
                (526k IOPS)     (625k IOPS)    (960k IOPS)

Observation:
When CPU count exceeds NIC queue count, the baseline configuration
suffers from queue contention. The proposed changes provide modest
improvements on their own, and when combined with queue-aware tuning
(IRQ affinity, ntuple steering, and CPU alignment), enable up to 
~1.5x–2.5x throughput improvement.

Scenario 2: NIC queues == CPU count
-----------------------------------

- CPUs: 72
- NIC queues: 72

                Baseline                Patched + tuning
randread        4310 MB/s               7987 MB/s
                (1052k IOPS)            (1950k IOPS)

randwrite       7947 MB/s               7972 MB/s
                (1940k IOPS)            (1946k IOPS)

randrw (read)   3583 MB/s               4030 MB/s
                (875k IOPS)             (984k IOPS)

randrw (write)  3583 MB/s               4029 MB/s
                (875k IOPS)             (984k IOPS)

Observation:
When NIC queues are already aligned with CPU count, the baseline performs
well. The proposed changes maintain write performance (no regression) and
still improve read and mixed workloads due to better flow-to-CPU locality.

Notes on tuning
---------------
The "patched + tuning" configuration includes:
    - aligning NVMe/TCP I/O workers with NIC queue count
    - IRQ affinity configuration per RX queue
    - ntuple-based flow steering
    - CPU/queue affinity alignment

These tuning steps are enabled by the queue/flow information exposed through
this patchset.

As usual, feedback/comment/suggestions are most welcome!

Changes from v1:
  - remove the "match-hw-queues" fabric option; always limit the number
    of NVMe/TCP I/O queues to min(num_online_cpus, num_nic_queues)
  - drop the diagnostic patch reporting NIC queue underutilization
  - split the netdev helper into a separate patch
  - move the netdev helper implementation to net/core/dev.c
 
Link to v1: https://lore.kernel.org/all/20260420115716.3071293-1-nilay@linux.ibm.com/

Nilay Shroff (4):
  net: add helper for device lookup by destination address
  nvme-tcp: limit I/O queue count based on NIC queue count
  nvme: add debugfs helpers for NVMe drivers
  nvme: expose queue information via debugfs

 drivers/nvme/host/Makefile  |   2 +-
 drivers/nvme/host/core.c    |   3 +
 drivers/nvme/host/debugfs.c | 191 ++++++++++++++++++++++++++++++++++++
 drivers/nvme/host/nvme.h    |  12 +++
 drivers/nvme/host/tcp.c     | 115 ++++++++++++++++++++++
 include/linux/netdevice.h   |   5 +
 net/core/dev.c              |  84 ++++++++++++++++
 7 files changed, 411 insertions(+), 1 deletion(-)
 create mode 100644 drivers/nvme/host/debugfs.c

-- 
2.53.0

Add netdev_get_by_addr(), a helper that looks up the routing table to
retrieve the network device associated with a destination address. The
helper also supports netdev reference tracking.

Signed-off-by: Nilay Shroff <redacted>
---
 include/linux/netdevice.h |  5 +++
 net/core/dev.c            | 84 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 89 insertions(+)

diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 9981d637f8b5..f43c3aa8486c 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h

@@ -3487,6 +3487,11 @@ struct net_device *netdev_get_by_index(struct net *net, int ifindex,
 struct net_device *netdev_get_by_index_lock(struct net *net, int ifindex);
 struct net_device *netdev_get_by_name(struct net *net, const char *name,
 				      netdevice_tracker *tracker, gfp_t gfp);
+struct net_device *netdev_get_by_addr(struct net *net,
+				      struct sockaddr_storage *src,
+				      struct sockaddr_storage *dest,
+				      netdevice_tracker *tracker,
+				      gfp_t gfp);
 struct net_device *netdev_get_by_flags_rcu(struct net *net, netdevice_tracker *tracker,
 					   unsigned short flags, unsigned short mask);
 struct net_device *dev_get_by_index_rcu(struct net *net, int ifindex);

diff --git a/net/core/dev.c b/net/core/dev.c
index 5933c5dab09e..fd8afcd26de4 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c

@@ -163,6 +163,8 @@
 #include <net/page_pool/memory_provider.h>
 #include <net/rps.h>
 #include <linux/phy_link_topology.h>
+#include <net/route.h>
+#include <net/ip6_route.h>
 
 #include "dev.h"
 #include "devmem.h"

@@ -944,6 +946,88 @@ struct net_device *netdev_get_by_name(struct net *net, const char *name,
 }
 EXPORT_SYMBOL(netdev_get_by_name);
 
+/**
+ *	netdev_get_by_addr() - find an egress device by destination address
+ *	@net: the applicable net namespace
+ *	@src: optional source address
+ *	@dest: destination address
+ *	@tracker: tracking onject for the acquired reference
+ *	@gfp: allocation flag for the tracker
+ *
+ *	Find an interface by looking up route table entry based on dest address
+ *	and an optional src address. The returned handle has the usage count
+ *	incremented and the caller must use netdev_put() to release it when it
+ *	is no longer needed. %NULL is returned if dest is not specified, or src
+ *	and dest belong to different address families, or no matching device is
+ *	found.
+ */
+struct net_device *netdev_get_by_addr(struct net *net,
+				      struct sockaddr_storage *src,
+				      struct sockaddr_storage *dest,
+				      netdevice_tracker *tracker,
+				      gfp_t gfp)
+{
+	struct net_device *dev = NULL;
+
+	if (!dest)
+		return NULL;
+
+	if (src && src->ss_family != dest->ss_family)
+		return NULL;
+
+	if (dest->ss_family == AF_INET) {
+		struct rtable *rt;
+		struct flowi4 fl4 = {};
+
+		fl4.daddr = ((struct sockaddr_in *)dest)->sin_addr.s_addr;
+		if (src)
+			fl4.saddr = ((struct sockaddr_in *)src)->sin_addr.s_addr;
+
+		rt = ip_route_output_key(net, &fl4);
+		if (IS_ERR(rt))
+			return NULL;
+
+		dev = dst_dev(&rt->dst);
+		/*
+		 * Get reference to netdev as ip_rt_put() will release netdev
+		 * reference.
+		 */
+		if (dev)
+			netdev_hold(dev, tracker, gfp);
+
+		ip_rt_put(rt);
+
+#if IS_ENABLED(CONFIG_IPV6)
+	} else if (dest->ss_family == AF_INET6) {
+		struct dst_entry *dst;
+		struct flowi6 fl6 = {};
+
+		fl6.daddr = ((struct sockaddr_in6 *)dest)->sin6_addr;
+		if (src)
+			fl6.saddr = ((struct sockaddr_in6 *)src)->sin6_addr;
+
+		dst = ip6_route_output(net, NULL, &fl6);
+		if (dst->error) {
+			dst_release(dst);
+			return NULL;
+		}
+
+		dev = dst_dev(dst);
+		/*
+		 * Get reference to netdev as dst_release() will
+		 * release the netdev reference.
+		 */
+		if (dev)
+			netdev_hold(dev, tracker, gfp);
+
+		dst_release(dst);
+#endif
+	}
+
+	return dev;
+}
+EXPORT_SYMBOL(netdev_get_by_addr);
+
 /**
  *	__dev_get_by_index - find a device by its ifindex
  *	@net: the applicable net namespace

-- 
2.53.0

NVMe-TCP currently provisions I/O queues based primarily on the number
of online CPUs. On systems where the CPU count significantly exceeds the
number of NIC hardware queues, multiple NVMe-TCP I/O queues end up
sharing the same NIC TX/RX queues. This increases lock contention,
cacheline bouncing, and inter-processor interrupts (IPIs), reducing I/O
efficiency.

Limit the number of NVMe-TCP default I/O queues to the smaller of the
number of online CPUs and the number of NIC hardware queues. Aligning
the number of NVMe-TCP I/O queues with the NIC queue topology reduces
queue sharing, improves locality, and can improve throughput while
reducing tail latency.

The number of NVMe-TCP I/O queues is now limited to:

    min(num_online_cpus, num_nic_queues)

Signed-off-by: Nilay Shroff <redacted>
---
 drivers/nvme/host/tcp.c | 61 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 61 insertions(+)

diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index ba5c7b3e2a7c..a2110287099e 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c

@@ -1774,6 +1774,50 @@ static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl,
 	return ret;
 }
 
+static struct net_device *nvme_tcp_get_netdev(struct nvme_ctrl *ctrl,
+		netdevice_tracker *tracker, gfp_t gfp)
+{
+	struct net_device *dev = NULL;
+
+	if (ctrl->opts->mask & NVMF_OPT_HOST_IFACE)
+		dev = netdev_get_by_name(&init_net, ctrl->opts->host_iface,
+				tracker, gfp);
+	else {
+		struct nvme_tcp_ctrl *tctrl = to_tcp_ctrl(ctrl);
+		struct sockaddr_storage *src = NULL, *dest = NULL;
+
+		if (ctrl->opts->mask & NVMF_OPT_HOST_TRADDR)
+			src = &tctrl->src_addr;
+
+		dest = &tctrl->addr;
+
+		dev = netdev_get_by_addr(&init_net, src, dest, tracker, gfp);
+	}
+	return dev;
+}
+
+/*
+ * Returns number of active NIC queues (min of TX/RX), or 0 if device cannot
+ * be determined.
+ */
+static int nvme_tcp_get_netdev_current_queue_count(struct nvme_ctrl *ctrl)
+{
+	struct net_device *dev;
+	int tx_queues, rx_queues;
+	netdevice_tracker tracker;
+
+	dev = nvme_tcp_get_netdev(ctrl, &tracker, GFP_KERNEL);
+	if (!dev)
+		return 0;
+
+	tx_queues = dev->real_num_tx_queues;
+	rx_queues = dev->real_num_rx_queues;
+
+	netdev_put(dev, &tracker);
+
+	return min(tx_queues, rx_queues);
+}
+
 static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid,
 				key_serial_t pskid)
 {

@@ -2165,6 +2209,23 @@ static int nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl)
 	unsigned int nr_io_queues;
 	int ret;
 
+	if (!(ctrl->opts->mask & NVMF_OPT_NR_IO_QUEUES)) {
+		int nr_hw_queues;
+
+		nr_hw_queues = nvme_tcp_get_netdev_current_queue_count(ctrl);
+		if (nr_hw_queues <= 0)
+			goto init_queue;
+
+		ctrl->opts->nr_io_queues = min(nr_hw_queues, num_online_cpus());
+
+		if (ctrl->opts->nr_io_queues < num_online_cpus())
+			dev_info(ctrl->device,
+				"limiting I/O queues to %u (NIC queues %d, CPUs %u)\n",
+				ctrl->opts->nr_io_queues, nr_hw_queues,
+				num_online_cpus());
+	}
+
+init_queue:
 	nr_io_queues = nvmf_nr_io_queues(ctrl->opts);
 	ret = nvme_set_queue_count(ctrl, &nr_io_queues);
 	if (ret)

-- 
2.53.0

Introduce helper APIs that allow NVMe drivers to register and unregister
debugfs entries, along with a reusable attribute structure for defining
new debugfs files.

The implementation uses seq_file interfaces to safely expose per-
namespace or per-path statistics, while supporting both simple show
callbacks and full seq_operations.

This will be used by subsequent patches to expose NVMe-TCP queue
and flow information for tuning NVMe TCP I/O workqueue and network stack
components.

Signed-off-by: Nilay Shroff <redacted>
---
 drivers/nvme/host/Makefile  |   2 +-
 drivers/nvme/host/debugfs.c | 129 ++++++++++++++++++++++++++++++++++++
 drivers/nvme/host/nvme.h    |  10 +++
 3 files changed, 140 insertions(+), 1 deletion(-)
 create mode 100644 drivers/nvme/host/debugfs.c

diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile
index 6414ec968f99..7962dfc3b2ad 100644
--- a/drivers/nvme/host/Makefile
+++ b/drivers/nvme/host/Makefile

@@ -10,7 +10,7 @@ obj-$(CONFIG_NVME_FC)			+= nvme-fc.o
 obj-$(CONFIG_NVME_TCP)			+= nvme-tcp.o
 obj-$(CONFIG_NVME_APPLE)		+= nvme-apple.o
 
-nvme-core-y				+= core.o ioctl.o sysfs.o pr.o
+nvme-core-y				+= core.o ioctl.o sysfs.o pr.o debugfs.o
 nvme-core-$(CONFIG_NVME_VERBOSE_ERRORS)	+= constants.o
 nvme-core-$(CONFIG_TRACING)		+= trace.o
 nvme-core-$(CONFIG_NVME_MULTIPATH)	+= multipath.o

diff --git a/drivers/nvme/host/debugfs.c b/drivers/nvme/host/debugfs.c
new file mode 100644
index 000000000000..b4802ea3bd48
--- /dev/null
+++ b/drivers/nvme/host/debugfs.c

@@ -0,0 +1,129 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2026 IBM Corporation
+ *	Nilay Shroff <nilay@linux.ibm.com>
+ */
+
+#include <linux/debugfs.h>
+#include <linux/seq_file.h>
+#include <linux/slab.h>
+
+#include "nvme.h"
+
+struct nvme_debugfs_attr {
+	const char *name;
+	umode_t mode;
+	int (*show)(void *data, struct seq_file *m);
+	const struct seq_operations *seq_ops;
+	bool (*get)(void *data);
+	void (*put)(void *data);
+};
+
+struct nvme_debugfs_ctx {
+	void *data;
+	struct nvme_debugfs_attr *attr;
+};
+
+static int nvme_debugfs_show(struct seq_file *m, void *v)
+{
+	struct nvme_debugfs_ctx *ctx = m->private;
+	void *data = ctx->data;
+	struct nvme_debugfs_attr *attr = ctx->attr;
+
+	return attr->show(data, m);
+}
+
+static int nvme_debugfs_open(struct inode *inode, struct file *file)
+{
+	void *data = inode->i_private;
+	struct nvme_debugfs_attr *attr = debugfs_get_aux(file);
+	struct nvme_debugfs_ctx *ctx;
+	struct seq_file *m;
+	int ret;
+
+	if (attr->get && !attr->get(data))
+		return -ENODEV;
+
+	ctx = kzalloc_obj(*ctx);
+	if (WARN_ON_ONCE(!ctx)) {
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	ctx->data = data;
+	ctx->attr = attr;
+
+	if (attr->seq_ops) {
+		ret = seq_open(file, attr->seq_ops);
+		if (ret)
+			goto out_free;
+
+		m = file->private_data;
+		m->private = ctx;
+		return ret;
+	}
+
+	if (WARN_ON_ONCE(!attr->show)) {
+		ret = -EPERM;
+		goto out_free;
+	}
+
+	ret = single_open(file, nvme_debugfs_show, ctx);
+	if (!ret)
+		return ret;
+
+out_free:
+	kfree(ctx);
+out:
+	if (attr->put)
+		attr->put(data);
+	return ret;
+}
+
+static int nvme_debugfs_release(struct inode *inode, struct file *file)
+{
+	struct seq_file *m = file->private_data;
+	struct nvme_debugfs_ctx *ctx = m->private;
+	struct nvme_debugfs_attr *attr = ctx->attr;
+	int ret;
+
+	if (attr->seq_ops)
+		ret = seq_release(inode, file);
+	else
+		ret = single_release(inode, file);
+
+	if (attr->put)
+		attr->put(ctx->data);
+
+	kfree(ctx);
+	return ret;
+}
+
+static const struct file_operations nvme_debugfs_fops = {
+	.owner   = THIS_MODULE,
+	.open    = nvme_debugfs_open,
+	.read    = seq_read,
+	.llseek  = seq_lseek,
+	.release = nvme_debugfs_release,
+};
+
+static const struct nvme_debugfs_attr nvme_ns_debugfs_attrs[] = {
+	{},
+};
+
+static void nvme_debugfs_create_files(struct request_queue *q,
+		const struct nvme_debugfs_attr *attr, void *data)
+{
+	if (WARN_ON_ONCE(!q->debugfs_dir))
+		return;
+
+	for (; attr->name; attr++)
+		debugfs_create_file_aux(attr->name, attr->mode, q->debugfs_dir,
+				data, (void *)attr, &nvme_debugfs_fops);
+}
+
+void nvme_debugfs_register(struct gendisk *disk)
+{
+	nvme_debugfs_create_files(disk->queue, nvme_ns_debugfs_attrs,
+			disk->private_data);
+}

diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 824651cc898d..72f9d9c668a1 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h

@@ -474,6 +474,16 @@ struct nvme_ctrl {
 	u16			awupf; /* 0's based value. */
 };
 
+void nvme_debugfs_register(struct gendisk *disk);
+static inline void nvme_debugfs_unregister(struct gendisk *disk)
+{
+	/*
+	 * Nothing to do for now. When the request queue is unregistered,
+	 * all files under q->debugfs_dir are recursively deleted.
+	 * This is just a placeholder; the compiler will optimize it out.
+	 */
+}
+
 static inline enum nvme_ctrl_state nvme_ctrl_state(struct nvme_ctrl *ctrl)
 {
 	return READ_ONCE(ctrl->state);

-- 
2.53.0

Add a new debugfs attribute "io_queue_info" to expose per-queue
information for NVMe controllers. For NVMe-TCP, this includes the
CPU handling each I/O queue and the associated TCP flow (source and
destination address/port).

This information can be useful for understanding and tuning the
interaction between NVMe-TCP I/O queues and network stack components,
such as IRQ affinity, RPS/RFS, XPS, or NIC flow steering (ntuple).

The data is exported using seq_file interfaces to allow iteration
over all controller queues.

Signed-off-by: Nilay Shroff <redacted>
---
 drivers/nvme/host/core.c    |  3 ++
 drivers/nvme/host/debugfs.c | 64 ++++++++++++++++++++++++++++++++++++-
 drivers/nvme/host/nvme.h    |  2 ++
 drivers/nvme/host/tcp.c     | 54 +++++++++++++++++++++++++++++++
 4 files changed, 122 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 453c1f0b2dd0..50e81892a9d6 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c

@@ -4264,6 +4264,8 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)
 	if (device_add_disk(ctrl->device, ns->disk, nvme_ns_attr_groups))
 		goto out_cleanup_ns_from_list;
 
+	nvme_debugfs_register(ns->disk);
+
 	if (!nvme_ns_head_multipath(ns->head))
 		nvme_add_ns_cdev(ns);
 

@@ -4344,6 +4346,7 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 
 	nvme_mpath_remove_sysfs_link(ns);
 
+	nvme_debugfs_unregister(ns->disk);
 	del_gendisk(ns->disk);
 
 	mutex_lock(&ns->ctrl->namespaces_lock);

diff --git a/drivers/nvme/host/debugfs.c b/drivers/nvme/host/debugfs.c
index b4802ea3bd48..6146f33bc586 100644
--- a/drivers/nvme/host/debugfs.c
+++ b/drivers/nvme/host/debugfs.c

@@ -24,6 +24,61 @@ struct nvme_debugfs_ctx {
 	struct nvme_debugfs_attr *attr;
 };
 
+static void *nvme_io_queue_info_start(struct seq_file *m, loff_t *pos)
+{
+	struct nvme_debugfs_ctx *ctx = m->private;
+	struct nvme_ns *ns = ctx->data;
+	struct nvme_ctrl *ctrl = ns->ctrl;
+
+	/*
+	 * IO queues starts at offset 1.
+	 */
+	return (++*pos < ctrl->queue_count) ? pos : NULL;
+}
+
+static void *nvme_io_queue_info_next(struct seq_file *m, void *v, loff_t *pos)
+{
+	struct nvme_debugfs_ctx *ctx = m->private;
+	struct nvme_ns *ns = ctx->data;
+	struct nvme_ctrl *ctrl = ns->ctrl;
+
+	return (++*pos < ctrl->queue_count) ? pos : NULL;
+}
+
+static void nvme_io_queue_info_stop(struct seq_file *m, void *v)
+{
+	/* nothing to be done */
+}
+
+static int nvme_io_queue_info_show(struct seq_file *m, void *v)
+{
+	struct nvme_debugfs_ctx *ctx = m->private;
+	struct nvme_ns *ns = ctx->data;
+	struct nvme_ctrl *ctrl = ns->ctrl;
+
+	if (ctrl->ops->print_io_queue_info)
+		return ctrl->ops->print_io_queue_info(m, ctrl, *(loff_t *)v);
+
+	return 0;
+}
+
+const struct seq_operations nvme_io_queue_info_seq_ops = {
+	.start = nvme_io_queue_info_start,
+	.next = nvme_io_queue_info_next,
+	.stop = nvme_io_queue_info_stop,
+	.show = nvme_io_queue_info_show
+};
+
+static bool nvme_debugfs_get_ns(void *data)
+{
+	return nvme_get_ns(data);
+}
+
+static void nvme_debugfs_put_ns(void *data)
+{
+	nvme_put_ns(data);
+}
+
 static int nvme_debugfs_show(struct seq_file *m, void *v)
 {
 	struct nvme_debugfs_ctx *ctx = m->private;

@@ -108,7 +163,14 @@ static const struct file_operations nvme_debugfs_fops = {
 };
 
 static const struct nvme_debugfs_attr nvme_ns_debugfs_attrs[] = {
-	{},
+	{
+		.name    = "io_queue_info",
+		.mode    = 0400,
+		.seq_ops = &nvme_io_queue_info_seq_ops,
+		.get     = nvme_debugfs_get_ns,
+		.put     = nvme_debugfs_put_ns
+	},
+	{}
 };
 
 static void nvme_debugfs_create_files(struct request_queue *q,

diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 72f9d9c668a1..430971c56406 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h

@@ -663,6 +663,8 @@ struct nvme_ctrl_ops {
 	void (*print_device_info)(struct nvme_ctrl *ctrl);
 	bool (*supports_pci_p2pdma)(struct nvme_ctrl *ctrl);
 	unsigned long (*get_virt_boundary)(struct nvme_ctrl *ctrl, bool is_admin);
+	int (*print_io_queue_info)(struct seq_file *m, struct nvme_ctrl *ctrl,
+			int qid);
 };
 
 /*

diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index a2110287099e..e98f025ce99f 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c

@@ -2651,6 +2651,59 @@ static void nvme_tcp_free_ctrl(struct nvme_ctrl *nctrl)
 	kfree(ctrl);
 }
 
+static int nvme_tcp_print_io_queue_info(struct seq_file *m,
+		struct nvme_ctrl *ctrl, int qid)
+{
+	int cpu;
+	struct sockaddr_storage src, dst;
+	struct nvme_tcp_ctrl *tctrl = to_tcp_ctrl(ctrl);
+	struct nvme_tcp_queue *queue = &tctrl->queues[qid];
+	int ret = -EINVAL;
+
+	if (!qid || qid >= ctrl->queue_count)
+		return -EINVAL;
+
+	if (!test_bit(NVME_TCP_Q_LIVE, &queue->flags))
+		return 0;
+
+	mutex_lock(&queue->queue_lock);
+	if (!queue->sock)
+		goto unlock;
+
+	ret = kernel_getsockname(queue->sock, (struct sockaddr *)&src);
+	if (ret <= 0)
+		goto unlock;
+
+	ret = kernel_getpeername(queue->sock, (struct sockaddr *)&dst);
+	if (ret <= 0)
+		goto unlock;
+
+	cpu = (queue->io_cpu == WORK_CPU_UNBOUND) ? -1 : queue->io_cpu;
+
+	if (src.ss_family == AF_INET) {
+		struct sockaddr_in *sip = (struct sockaddr_in *)&src;
+		struct sockaddr_in *dip = (struct sockaddr_in *)&dst;
+
+		seq_printf(m, "qid=%d cpu=%d src_ip=%pI4 src_port=%u dst_ip=%pI4 dst_port=%u\n",
+				qid, cpu,
+				&sip->sin_addr.s_addr, ntohs(sip->sin_port),
+				&dip->sin_addr.s_addr, ntohs(dip->sin_port));
+		ret = 0;
+	} else if (src.ss_family == AF_INET6) {
+		struct sockaddr_in6 *sip6 = (struct sockaddr_in6 *)&src;
+		struct sockaddr_in6 *dip6 = (struct sockaddr_in6 *)&dst;
+
+		seq_printf(m, "qid=%d cpu=%d src_ip=%pI6c src_port=%u dst_ip=%pI6c dst_port=%u\n",
+				qid, cpu,
+				&sip6->sin6_addr, ntohs(sip6->sin6_port),
+				&dip6->sin6_addr, ntohs(dip6->sin6_port));
+		ret = 0;
+	}
+unlock:
+	mutex_unlock(&queue->queue_lock);
+	return ret;
+}
+
 static void nvme_tcp_set_sg_null(struct nvme_command *c)
 {
 	struct nvme_sgl_desc *sg = &c->common.dptr.sgl;

@@ -2951,6 +3004,7 @@ static const struct nvme_ctrl_ops nvme_tcp_ctrl_ops = {
 	.get_address		= nvme_tcp_get_address,
 	.stop_ctrl		= nvme_tcp_stop_ctrl,
 	.get_virt_boundary	= nvmf_get_virt_boundary,
+	.print_io_queue_info	= nvme_tcp_print_io_queue_info,
 };
 
 static bool

-- 
2.53.0

On Fri Jul 31, 2026 at 9:39 AM CEST, Nilay Shroff wrote:

+static int nvme_debugfs_open(struct inode *inode, struct file *file)
+{
+	void *data = inode->i_private;
+	struct nvme_debugfs_attr *attr = debugfs_get_aux(file);
+	struct nvme_debugfs_ctx *ctx;
+	struct seq_file *m;
+	int ret;
+
+	if (attr->get && !attr->get(data))
+		return -ENODEV;
+
+	ctx = kzalloc_obj(*ctx);
+	if (WARN_ON_ONCE(!ctx)) {
+		ret = -ENOMEM;
+		goto out;
+	}

Just a nit, I don't think that raising a warning just because
kzalloc() returned NULL is a good idea as failures can naturally
occur under OOM conditions and aren't bugs.

Maurizio

On 07/31, Nilay Shroff wrote:

Hi,

This is a resend of the previous series to include the networking maintainers
and mailing list. There are no code or commit message changes since the
previous posting.

This series has been updated based on the feedback received during LSFMM.
The changelog is updated accordingly.

The NVMe/TCP host driver currently provisions I/O queues primarily based
on CPU availability rather than the capabilities and topology of the
underlying network interface.

On modern systems with many CPUs but fewer NIC hardware queues, this can
lead to multiple NVMe/TCP I/O workers contending for the same TX/RX queue,
resulting in increased lock contention, cacheline bouncing, and degraded
throughput.

This RFC proposes a set of changes to better align NVMe/TCP I/O queues
with NIC queue resources, and to expose queue/flow information to enable
more effective system-level tuning.

Key ideas
---------

1. Scale NVMe/TCP I/O queues based on NIC queue count
   Instead of relying solely on CPU count, limit the number of I/O workers
   to:
       min(num_online_cpus, netdev->real_num_{tx,rx}_queues)

2. Improve CPU locality
   Align NVMe/TCP I/O workers with CPUs associated with NIC IRQ affinity
   to reduce cross-CPU traffic and improve cache locality.

3. Expose queue and flow information via debugfs
   Export per-I/O queue information including:
       - queue id (qid)
       - CPU affinity
       - TCP flow (src/dst IP and ports)

[..]

   This enables userspace tools to configure:
       - IRQ affinity
       - RPS/XPS
       - ntuple steering
       - or any other scaling as deemed feasible

Can you expand on this a bit? What specifically helped the most for your
tuned case?

On 07/31, Nilay Shroff wrote:

[..]
 

+	if (!(ctrl->opts->mask & NVMF_OPT_NR_IO_QUEUES)) {
+		int nr_hw_queues;

Looks like the userspace can already pass the preferred number of queues,
so in this case, why not do all this netdev resolution and queue
estimation in the userspace? Presumably most or the users you care
about always go through nvme-cli, right?

On 7/31/26 10:11 PM, Stanislav Fomichev wrote:

On 07/31, Nilay Shroff wrote:

quoted

NVMe-TCP currently provisions I/O queues based primarily on the number
of online CPUs. On systems where the CPU count significantly exceeds the
number of NIC hardware queues, multiple NVMe-TCP I/O queues end up
sharing the same NIC TX/RX queues. This increases lock contention,
cacheline bouncing, and inter-processor interrupts (IPIs), reducing I/O
efficiency.

Limit the number of NVMe-TCP default I/O queues to the smaller of the
number of online CPUs and the number of NIC hardware queues. Aligning
the number of NVMe-TCP I/O queues with the NIC queue topology reduces
queue sharing, improves locality, and can improve throughput while
reducing tail latency.

The number of NVMe-TCP I/O queues is now limited to:

     min(num_online_cpus, num_nic_queues)

Signed-off-by: Nilay Shroff <redacted>
---
  drivers/nvme/host/tcp.c | 61 +++++++++++++++++++++++++++++++++++++++++
  1 file changed, 61 insertions(+)

diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index ba5c7b3e2a7c..a2110287099e 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c

@@ -1774,6 +1774,50 @@ static int nvme_tcp_start_tls(struct nvme_ctrl *nctrl,
  	return ret;
  }
  
+static struct net_device *nvme_tcp_get_netdev(struct nvme_ctrl *ctrl,
+		netdevice_tracker *tracker, gfp_t gfp)
+{
+	struct net_device *dev = NULL;
+
+	if (ctrl->opts->mask & NVMF_OPT_HOST_IFACE)
+		dev = netdev_get_by_name(&init_net, ctrl->opts->host_iface,
+				tracker, gfp);
+	else {
+		struct nvme_tcp_ctrl *tctrl = to_tcp_ctrl(ctrl);
+		struct sockaddr_storage *src = NULL, *dest = NULL;
+
+		if (ctrl->opts->mask & NVMF_OPT_HOST_TRADDR)
+			src = &tctrl->src_addr;
+
+		dest = &tctrl->addr;
+
+		dev = netdev_get_by_addr(&init_net, src, dest, tracker, gfp);
+	}
+	return dev;
+}
+
+/*
+ * Returns number of active NIC queues (min of TX/RX), or 0 if device cannot
+ * be determined.
+ */
+static int nvme_tcp_get_netdev_current_queue_count(struct nvme_ctrl *ctrl)
+{
+	struct net_device *dev;
+	int tx_queues, rx_queues;
+	netdevice_tracker tracker;
+
+	dev = nvme_tcp_get_netdev(ctrl, &tracker, GFP_KERNEL);
+	if (!dev)
+		return 0;
+
+	tx_queues = dev->real_num_tx_queues;
+	rx_queues = dev->real_num_rx_queues;
+
+	netdev_put(dev, &tracker);
+
+	return min(tx_queues, rx_queues);
+}
+
  static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid,
  				key_serial_t pskid)
  {

@@ -2165,6 +2209,23 @@ static int nvme_tcp_alloc_io_queues(struct nvme_ctrl *ctrl)
  	unsigned int nr_io_queues;
  	int ret;

[..]
  

quoted

+	if (!(ctrl->opts->mask & NVMF_OPT_NR_IO_QUEUES)) {
+		int nr_hw_queues;

Looks like the userspace can already pass the preferred number of queues,
so in this case, why not do all this netdev resolution and queue
estimation in the userspace? Presumably most or the users you care
about always go through nvme-cli, right?

Yes, userspace can already specify the preferred number of I/O queues, and nvme-cli
provides an option to do so when creating an NVMe/TCP connection. However, choosing
an appropriate value requires userspace to know both the number of online CPUs and
the number of active TX/RX queues on the NIC used for the connection. Determining
the latter also requires identifying the correct netdevice. That may involve a route
lookup to determine the egress interface, particularly when the NVMe/TCP host and
target are not on the same subnet.

So while this could be implemented in nvme-cli, it would require userspace to duplicate
the logic needed to determine the actual netdevice and its current queue configuration.
The intent of this change is to make the default queue selection automatic and avoid
requiring users to determine and specify this topology manually.

An explicitly specified "nr_io_queues" would still take precedence, so userspace can
override the default when desired.

Just for the note, this change also follows the general approach used by nvme-pci, where
the default number of I/O queues is constrained by both the number of possible CPUs and the
queue resources available from the controller.

Thanks,
--Nilay

On 7/31/26 10:09 PM, Stanislav Fomichev wrote:

On 07/31, Nilay Shroff wrote:

quoted

Hi,

This is a resend of the previous series to include the networking maintainers
and mailing list. There are no code or commit message changes since the
previous posting.

This series has been updated based on the feedback received during LSFMM.
The changelog is updated accordingly.

The NVMe/TCP host driver currently provisions I/O queues primarily based
on CPU availability rather than the capabilities and topology of the
underlying network interface.

On modern systems with many CPUs but fewer NIC hardware queues, this can
lead to multiple NVMe/TCP I/O workers contending for the same TX/RX queue,
resulting in increased lock contention, cacheline bouncing, and degraded
throughput.

This RFC proposes a set of changes to better align NVMe/TCP I/O queues
with NIC queue resources, and to expose queue/flow information to enable
more effective system-level tuning.

Key ideas
---------

1. Scale NVMe/TCP I/O queues based on NIC queue count
    Instead of relying solely on CPU count, limit the number of I/O workers
    to:
        min(num_online_cpus, netdev->real_num_{tx,rx}_queues)

2. Improve CPU locality
    Align NVMe/TCP I/O workers with CPUs associated with NIC IRQ affinity
    to reduce cross-CPU traffic and improve cache locality.

3. Expose queue and flow information via debugfs
    Export per-I/O queue information including:
        - queue id (qid)
        - CPU affinity
        - TCP flow (src/dst IP and ports)

[..]

quoted

    This enables userspace tools to configure:
        - IRQ affinity
        - RPS/XPS
        - ntuple steering
        - or any other scaling as deemed feasible

Can you expand on this a bit? What specifically helped the most for your
tuned case?

For my network topology and workload, configuring ntuple steering together with
IRQ affinity and XPS provided the most benefit. The NIC used for testing supports
ntuple filters, which allowed me to steer the TCP flow associated with an NVMe/TCP
I/O queue to a specific NIC RX queue. I then configured the IRQ affinity for that
RX queue and XPS for the corresponding TX path so that the NVMe/TCP I/O queue
processing, RX interrupt processing, and TX/RX packet processing were aligned to
the same CPU as much as possible. This reduced cross-CPU processing for an I/O
flow and provided the best improvement among the tuning combinations I tested.

Thanks,
--Nilay

On 7/31/26 1:44 PM, Maurizio Lombardi wrote:

On Fri Jul 31, 2026 at 9:39 AM CEST, Nilay Shroff wrote:

quoted

+static int nvme_debugfs_open(struct inode *inode, struct file *file)
+{
+	void *data = inode->i_private;
+	struct nvme_debugfs_attr *attr = debugfs_get_aux(file);
+	struct nvme_debugfs_ctx *ctx;
+	struct seq_file *m;
+	int ret;
+
+	if (attr->get && !attr->get(data))
+		return -ENODEV;
+
+	ctx = kzalloc_obj(*ctx);
+	if (WARN_ON_ONCE(!ctx)) {
+		ret = -ENOMEM;
+		goto out;
+	}

Just a nit, I don't think that raising a warning just because
kzalloc() returned NULL is a good idea as failures can naturally
occur under OOM conditions and aren't bugs.

Alright, I'll change it to just return -ENOMEM without producing
a warning, in case kzalloc fails.

Thanks,
--Nilay

On Sat, 1 Aug 2026 19:08:05 +0530 Nilay Shroff wrote:

quoted

Looks like the userspace can already pass the preferred number of queues,
so in this case, why not do all this netdev resolution and queue
estimation in the userspace? Presumably most or the users you care
about always go through nvme-cli, right?  

Yes, userspace can already specify the preferred number of I/O queues, and nvme-cli
provides an option to do so when creating an NVMe/TCP connection. However, choosing
an appropriate value requires userspace to know both the number of online CPUs and
the number of active TX/RX queues on the NIC used for the connection. Determining
the latter also requires identifying the correct netdevice. That may involve a route
lookup to determine the egress interface, particularly when the NVMe/TCP host and
target are not on the same subnet.

So while this could be implemented in nvme-cli, it would require userspace to duplicate
the logic needed to determine the actual netdevice and its current queue configuration.
The intent of this change is to make the default queue selection automatic and avoid
requiring users to determine and specify this topology manually.

In another message you said you add ntuple filters. So you _are_ doing
what you describe here as a problem. User space will know something we
don't know sooner or later, so you should just add the uAPI instead of
guessing in the kernel. BTW the queue count is likely to change after
all of user space boots, so if you run before whatever configures
queues for the machine in userspace you'll be using wrong counts.

An explicitly specified "nr_io_queues" would still take precedence, so userspace can
override the default when desired.

Just for the note, this change also follows the general approach used by nvme-pci, where
the default number of I/O queues is constrained by both the number of possible CPUs and the
queue resources available from the controller.

Not sure that maps well to networking. For TCP at least there will be 
a protocol stack that runs between the device queues and your queues. 
I guess that will depend on the network and the details of the
benchmark. But again, better to let the user tune to their workload
and machine.

Consider patch 1 nacked.

On 8/8/26 4:39 AM, Jakub Kicinski wrote:

On Sat, 1 Aug 2026 19:08:05 +0530 Nilay Shroff wrote:

quoted

quoted

Looks like the userspace can already pass the preferred number of queues,
so in this case, why not do all this netdev resolution and queue
estimation in the userspace? Presumably most or the users you care
about always go through nvme-cli, right?

Yes, userspace can already specify the preferred number of I/O queues, and nvme-cli
provides an option to do so when creating an NVMe/TCP connection. However, choosing
an appropriate value requires userspace to know both the number of online CPUs and
the number of active TX/RX queues on the NIC used for the connection. Determining
the latter also requires identifying the correct netdevice. That may involve a route
lookup to determine the egress interface, particularly when the NVMe/TCP host and
target are not on the same subnet.

So while this could be implemented in nvme-cli, it would require userspace to duplicate
the logic needed to determine the actual netdevice and its current queue configuration.
The intent of this change is to make the default queue selection automatic and avoid
requiring users to determine and specify this topology manually.

In another message you said you add ntuple filters. So you _are_ doing
what you describe here as a problem. User space will know something we
don't know sooner or later, so you should just add the uAPI instead of
guessing in the kernel. BTW the queue count is likely to change after
all of user space boots, so if you run before whatever configures
queues for the machine in userspace you'll be using wrong counts.

Well, that ntuple filter configuration was done looking at the debugfs
output which is produced in patch 4/4. The debugfs generates the
enough information including queue count and per queue flow information
which is then programmed into ntuple filter.

quoted

An explicitly specified "nr_io_queues" would still take precedence, so userspace can
override the default when desired.

Just for the note, this change also follows the general approach used by nvme-pci, where
the default number of I/O queues is constrained by both the number of possible CPUs and the
queue resources available from the controller.

Not sure that maps well to networking. For TCP at least there will be
a protocol stack that runs between the device queues and your queues.
I guess that will depend on the network and the details of the
benchmark. But again, better to let the user tune to their workload
and machine.

Consider patch 1 nacked.

My motivation here was to improve the default behavior for the common case
where nr_io_queues is not explicitly specified. So if the preference is to
keep this in userspace, would you be open to exposing the required information
through a kernel interface (if something is still missing) and implementing
the queue selection logic in nvme-cli instead? That would still allow us to
automate the default queue selection without embedding this change in the kernel.

Thanks,
--Nilay