Re: Question: net/mlx5e kTLS TX TIS reuse vs in-flight WQEs

From
Rishikesh Jethwani <hidden>
Date
2026-11-09 23:21:41
Reply to
Tariq Toukan, 2026-11-08
On Sun, Nov 8, 2026 at 11:45 AM Tariq Toukan [off-list ref] wrote:

On 06/11/2026 21:58, Rishikesh Jethwani wrote:
 > Thanks. Agreed on (3a), (i) and the net fixes, with one condition on
 > (i) below.

Hi Rishikesh,
Hi Tariq,

Thanks. I've now reviewed the seven-patch RFC [1]. The held-skb,
full-resync and pass-through changes cover the points I raised. I've
replied on patches 7 and 4 with two remaining holes in the one-SQ
guarantee: the selection-time ownership transfer, and the PTP SQs in
the drain.
Thanks. I took your points into the patches, details inline. I'm
re-running the ConnectX-6 Dx tests with the changes, and will share the
RFCs once they pass.

 > Thanks. Until then, the two hardware questions remain open, including
 > whether DESTROY_SQ can fail; the destroy wrapper discards the status.

DESTROY_SQ doesn't fail while the function is healthy. If it does,
something went badly wrong in the firmware.
Thanks for clarifying the healthy-function case. I'll continue to use
CQEs or successful SQ destruction as the retirement evidence.

The DEK question is still open: once all old-key WQEs have retired, may
the old DEK be destroyed or returned for reuse while an idle TIS still
names it? This needs to cover both standalone DEKs and pooled IDs reused
by another connection. A flushed replacement need not have installed
the new key, and deletion can return the still-bound TIS to its pool.
The resync fix restores the context on its next use, but does not by
itself establish that releasing the old DEK before then is safe.

Please keep that HW/FW answer explicit. Until it is settled, I cannot
treat a flush/close alone as sufficient to release a still-named DEK;
the conservative alternative is to retain the possible bindings until
a successful replacement or TIS destruction, including through pooling.
 > One correction to my last mail: recovery clears RECOVERING even on a
 > flush timeout, so the queue can wake once stop_room frees.

Thanks. WQEs posted to that SQ then complete with error CQEs, which mark
their connections for a full resync, as in the first error.

 > Agreed. What rekey needs from that resync fix: a lost initialization
 > or replacement always triggers a full resync on the next skb that
 > needs the TLS context, and a connection marked again while the driver
 > is consuming an earlier mark stays marked (test-and-clear, not
 > clear-after).

Both hold now. The mark stays until an skb that uses the context
consumes it, and it is taken with a test-and-clear before the resync is
posted, so a loss reported meanwhile marks the connection again.

 > An explicit bypass right after the
 > datalen check in mlx5e_ktls_handle_tx_skb() closes that without a
 > sequence comparison:
 >
 >         if (!skb_is_decrypted(skb))
 >                 return true;

Agreed, it's in the pass-through patch, with credit.

 > I'm happy to use (i), but warning/counting alone does not cover the
 > forced-queue case: a detected conflict has to prevent the conflicting
 > hardware access, since old TIS accesses must have retired before
 > another SQ can modify the context. Dropping before any TLS WQE or
 > context-state change would preserve safety without SW crypto.

Agreed, the counter is now a drop. Each connection records the TX queue
that it posts on. An skb on another TX queue moves the connection only
if no other skb of its socket is in flight: an skb is freed only once
the WQEs posted for it completed, or were destroyed with their SQ, so
nothing of the connection is then pending on any SQ. Otherwise the skb
is dropped before anything is posted, and counted, and TCP retransmits
it.

The move is made under the old TX queue's lock, taken with a trylock,
under which the connection posts there. That serializes the check and
the publication with that path, and leaves the connection's datapath
with a compare of the queue index.

This covers your points 1 and 2 where the WQEs are posted, wherever the
queue came from. For 2, the mlx5e pin now also re-checks wmem at
selection as you suggest, so a stale ooo_okay doesn't lead to drops in
the first place. Does that match what you had in mind?

 > 3. The drain has to cover the QoS/HTB SQs too, and reap deactivated
 >    SQs although mlx5e_poll_tx_cq() returns early once ENABLED is
 >    clear; a missing or replaced txq2sq entry isn't evidence of
 >    retirement. On a drain timeout, the fallback must close the old
 >    SQs first (which keeps the guarantee, modulo DESTROY_SQ), never
 >    activate with them alive.

That's how the drain works: it covers the QoS SQs, reaps the deactivated
SQs directly, and on a timeout closes the old channels before activating
the new ones. The drop above doesn't rely on txq2sq either: an SQ that
is deactivated alone, e.g. on an HTB leaf delete, keeps its skbs until
it is destroyed, so their socket keeps its wmem until then.

 > On the price you mention for priority changes, agreed. Please
 > document in Documentation/networking/tls-offload.rst that pinning
 > delays priority/DSCP/HTB-driven TX queue changes until migration is
 > safe, potentially indefinitely under sustained traffic.

Done.

 > Nice, that's simpler. On the core side, the op's contract: the
 > callback prepares the new key and pending parameters but posts no TLS
 > WQEs for this connection.

[...]

 > Callbacks are serialized per connection, but another may arrive
 > before the previous replacement is posted. On success, the driver
 > must clean up any superseded, never-posted key.

Agreed on the contract.
Agreed. I'll replace v18's live TX del+add with the new callback and
implement mlx5e_ktls_rekey_tx() on the revised fixes. That includes
cleanup of superseded, never-posted keys, and explicit ownership of
retiring DEKs through WQE completion and deferred destruction. Devices
without the callback will stay in software after a KeyUpdate.

The delayed decrypted-clone case remains a separate core follow-up.
The pass-through bypass closes the unmarked-skb case, but the existing
32-bit record lookup can still alias a later record, even without
rekey. Rekey adds the cross-key case; the floor only covers delays
within its comparison window. The fix I'm considering needs TCP
support and a per-skb check in the driver's in-order TX path too; I'll
Cc you when I post it.
 > Record lookup also uses 32-bit TCP sequence numbers
 > today without rekey, so a sufficiently delayed clone can alias a live
 > record there too. The floor catches old-key clones within its
 > comparison window, but does not remove that ambiguity. I'll look at
 > closing this in the core separately.

Thanks. On the driver side, the bypass above keeps plaintext skbs out of
the record lookup, so it's down to the decrypted clones.

 > Please Cc me on the RFCs, and let me know which fixes the rekey
 > patches should depend on.

Will do. The rekey patches should go on top of these net fixes:

   net/mlx5e: kTLS, Hold a dropped skb until its posted WQEs complete
   net/mlx5e: kTLS, Resync TX fully after its WQEs were lost
   net/mlx5e: kTLS, Post TX context params only for skbs that use them
   net/mlx5e: Drain old TX SQs before activating new channels
   net: tls: keep the TX queue of a device-offloaded socket stable
   net/mlx5e: kTLS, Keep the TX queue of an offloaded socket stable

The drop goes to net-next separately, as "net/mlx5e: kTLS, Keep a TX
connection on one SQ at a time"; the rekey patches don't depend on it.
I do need patch 7, with the selection-time fix from my reply to it, as
a rekey dependency. Patches 1-6 alone do not cover egress skbedit or a
cached queue overwritten by another device on the route. A fenced
replacement on SQ B cannot order itself against old-key WQEs still on
SQ A. Keeping patch 7 in net-next is fine; I'll base rekey on it as
well as the net fixes. The RFC cover letter's "builds on this series"
matches that dependency.
Thanks,
Tariq
Thanks,
Rishikesh

[1] https://lore.kernel.org/netdev/20261109114734.2087878-1-tariqt@nvidia.com/ (local)
Thread (9 messages) 9 messages, 3 authors, last reply 6d ago
Keyboard shortcuts
hback out one level
jnext message in thread / next series-diff position
kprevious message in thread / previous series-diff position
ldrill in
Escclose help / fold thread tree
?toggle this help