Re: [RESEND PATCH v2 0/4] nvme-tcp: NIC topology aware I/O queue scaling and queue info export

On 7/31/26 10:09 PM, Stanislav Fomichev wrote:

On 07/31, Nilay Shroff wrote:

quoted

Hi,

This is a resend of the previous series to include the networking maintainers
and mailing list. There are no code or commit message changes since the
previous posting.

This series has been updated based on the feedback received during LSFMM.
The changelog is updated accordingly.

The NVMe/TCP host driver currently provisions I/O queues primarily based
on CPU availability rather than the capabilities and topology of the
underlying network interface.

On modern systems with many CPUs but fewer NIC hardware queues, this can
lead to multiple NVMe/TCP I/O workers contending for the same TX/RX queue,
resulting in increased lock contention, cacheline bouncing, and degraded
throughput.

This RFC proposes a set of changes to better align NVMe/TCP I/O queues
with NIC queue resources, and to expose queue/flow information to enable
more effective system-level tuning.

Key ideas
---------

1. Scale NVMe/TCP I/O queues based on NIC queue count
    Instead of relying solely on CPU count, limit the number of I/O workers
    to:
        min(num_online_cpus, netdev->real_num_{tx,rx}_queues)

2. Improve CPU locality
    Align NVMe/TCP I/O workers with CPUs associated with NIC IRQ affinity
    to reduce cross-CPU traffic and improve cache locality.

3. Expose queue and flow information via debugfs
    Export per-I/O queue information including:
        - queue id (qid)
        - CPU affinity
        - TCP flow (src/dst IP and ports)

[..]

quoted

    This enables userspace tools to configure:
        - IRQ affinity
        - RPS/XPS
        - ntuple steering
        - or any other scaling as deemed feasible

Can you expand on this a bit? What specifically helped the most for your
tuned case?

For my network topology and workload, configuring ntuple steering together with
IRQ affinity and XPS provided the most benefit. The NIC used for testing supports
ntuple filters, which allowed me to steer the TCP flow associated with an NVMe/TCP
I/O queue to a specific NIC RX queue. I then configured the IRQ affinity for that
RX queue and XPS for the corresponding TX path so that the NVMe/TCP I/O queue
processing, RX interrupt processing, and TX/RX packet processing were aligned to
the same CPU as much as possible. This reduced cross-CPU processing for an I/O
flow and provided the best improvement among the tuning combinations I tested.

Thanks,
--Nilay