Re: [RESEND PATCH v2 2/4] nvme-tcp: limit I/O queue count based on NIC queue count

On 8/8/26 4:39 AM, Jakub Kicinski wrote:

On Sat, 1 Aug 2026 19:08:05 +0530 Nilay Shroff wrote:

quoted

quoted

Looks like the userspace can already pass the preferred number of queues,
so in this case, why not do all this netdev resolution and queue
estimation in the userspace? Presumably most or the users you care
about always go through nvme-cli, right?

Yes, userspace can already specify the preferred number of I/O queues, and nvme-cli
provides an option to do so when creating an NVMe/TCP connection. However, choosing
an appropriate value requires userspace to know both the number of online CPUs and
the number of active TX/RX queues on the NIC used for the connection. Determining
the latter also requires identifying the correct netdevice. That may involve a route
lookup to determine the egress interface, particularly when the NVMe/TCP host and
target are not on the same subnet.

So while this could be implemented in nvme-cli, it would require userspace to duplicate
the logic needed to determine the actual netdevice and its current queue configuration.
The intent of this change is to make the default queue selection automatic and avoid
requiring users to determine and specify this topology manually.

In another message you said you add ntuple filters. So you _are_ doing
what you describe here as a problem. User space will know something we
don't know sooner or later, so you should just add the uAPI instead of
guessing in the kernel. BTW the queue count is likely to change after
all of user space boots, so if you run before whatever configures
queues for the machine in userspace you'll be using wrong counts.

Well, that ntuple filter configuration was done looking at the debugfs
output which is produced in patch 4/4. The debugfs generates the
enough information including queue count and per queue flow information
which is then programmed into ntuple filter.

quoted

An explicitly specified "nr_io_queues" would still take precedence, so userspace can
override the default when desired.

Just for the note, this change also follows the general approach used by nvme-pci, where
the default number of I/O queues is constrained by both the number of possible CPUs and the
queue resources available from the controller.

Not sure that maps well to networking. For TCP at least there will be
a protocol stack that runs between the device queues and your queues.
I guess that will depend on the network and the details of the
benchmark. But again, better to let the user tune to their workload
and machine.

Consider patch 1 nacked.

My motivation here was to improve the default behavior for the common case
where nr_io_queues is not explicitly specified. So if the preference is to
keep this in userspace, would you be open to exposing the required information
through a kernel interface (if something is still missing) and implementing
the queue selection logic in nvme-cli instead? That would still allow us to
automate the default queue selection without embedding this change in the kernel.

Thanks,
--Nilay