On Tue, Nov 3, 2026 at 6:45 AM Tariq Toukan [off-list ref] wrote:
On 22/10/2026 21:17, Rishikesh Jethwani wrote:
quoted
Hi Tariq,
Hi Tariq,
Thanks, this answers the cross-SQ ordering question. Short version:
I agree with (1), and (3a) fits the series; I prefer it. For (2), I
agree with the goal, but wmem-gated reselection alone does not keep a
connection's work on one SQ in mlx5e; the holes are listed below,
with two ways to close them, and I'd like your preference.
Two hardware questions here, and a third on initial setup under (1):
- Does a completed DESTROY_SQ, or finding an SQ in ERR, guarantee
that none of its WQEs can still access a TIS or DEK? And can
DESTROY_SQ fail while the function is healthy?
In mlx5e_close_txqsq(), the void mlx5e_destroy_sq() wrapper
discards the destroy status, and mlx5e_free_txqsq_descs() then
frees descriptors, advancing sq->cc without CQEs, so the "moved
out of RDY" argument depends on the answer.
- DEK release in mlx5_crypto_dek_destroy() has two paths. Without
a DEK pool (dek_priv == NULL) the object is destroyed right away;
with one, the DEK is pushed back to the pool, its id can be
handed to another connection after the batched SYNC_CRYPTO, and
the object itself is destroyed later from destroy_work. After
all WQEs referencing a DEK have retired, is either path, including
reuse of a pooled id by another connection, safe while an idle
TIS still names that DEK, before replacement static parameters
are posted? Or is today's release only safe because of the
destruct-time ordering?
Hi Rishikesh,
Good analysis, and thanks for the detailed write-up.
quoted
Could a DUMP/SEND WQE from an old owner on SQ A read or update TIS
state after SQ B has reinitialised that TIS for a new owner?
Short answer: Yes, in principle it could, once del runs on a live socket
as in your series. Today, only parameter/DUMP WQEs of a dropped skb can
be left behind (item 1 below): a pending SEND holds sk_wmem_alloc, so
the socket is not destructed before it completes.
The TIS is a function-global object, not bound to an SQ: a WQE that
references its TISN reads/updates its state whenever that WQE executes,
from whichever SQ it was posted on, and the hardware provides no
cross-SQ ordering for such accesses. Details below, and at the end a
proposal for the way forward, on which I'd appreciate your feedback.
quoted
The existing TIS pool predates live rekey. My TLS 1.3 TX KeyUpdate
series (v17 [1]; v18 in preparation) calls tls_dev_del() and then
tls_dev_add() on a live socket, and the question arises through this
sequence:
1. A spurious RTO retransmit of old-key data is posted on SQ A.
2. While that data remains unacknowledged, setsockopt(TLS_TX) starts
rekey and installs the rekey validator, which stops admitting new
plaintext to hardware offload. An ACK covering all remaining
old-key data marks rekey ready before the driver observes the
retransmit's SQ completion. A subsequent sendmsg() enters rekey
completion.
3. Rekey completion calls synchronize_net(), which waits for CPU
transmit paths, not SQ completions, so a WQE on a stalled or
backlogged SQ A can still be pending. mlx5e_ktls_del_tx() then
releases the DEK and immediately calls pool_push(). DEK
reclamation uses batched SYNC_CRYPTO; returning the TIS does not
wait for synchronization covering its old DEK.
4. Before re-adding the offload, TLS allocates the new SW-fallback
AEAD. In that interval another connection's mlx5e_ktls_add_tx()
can pop the returned TIS and use it on SQ B. The same socket can
also reach SQ B on re-add: with the retransmit queue empty, TCP
sets ooo_okay even while the retransmitted skb still holds
sk_wmem_alloc, so queue selection may move the flow.
mlx5e_ktls_handle_tx_skb() posts the new owner's initial
parameters with skip_static_post=false, fence_first_post=false:
static parameters unfenced, progress parameters fenced, then a
fence NOP.
Your reading of the driver is accurate on all points. In particular,
mlx5e_ktls_del_tx() returns the TIS to the pool with no completion
tracking. The same goes for the DEK: the batched SYNC_CRYPTO guards
against stale cached keys when a DEK object is reused, but nothing
waits for pending WQEs before that, and without bulk DEK support the
DEK is destroyed right away. So the DEK needs the same lifetime
guarantee as the TIS.
The same-socket exposure is also broader than the ooo_okay disjunct:
the RTO and PLB rehash paths reset the dst, which clears the cached
TX queue, and txq_reselection_ms lets a busy flow re-pick its queue
periodically, both regardless of ooo_okay. So one connection's WQEs
can be in flight on two SQs today, even without rekey.
quoted
An unobserved completion does not by itself mean the WQE still accesses
the TIS; that is the hardware ordering question. The same question
applies on SQ A: resync fences its first parameter update, initial
setup does not, and I do not know what either guarantees against
earlier WQEs that reference the TIS.
The fence modes in the WQE ctrl segment (FM field; the parameter WQEs
use Initiator_Small_Fence) are all defined in terms of previous WQEs
on the same SQ: a fenced WQE waits until earlier WQEs on that SQ are
no longer in execution. Even the strongest mode (Strong_Ordering)
orders only against previous WQEs of the same SQ. No fence mode, and
no other WQE-level primitive, orders against outstanding work on
another SQ.
So within one SQ, the fenced first parameter WQE of a resync does
guarantee that earlier WQEs referencing the TIS are done executing
before the context is modified. The unfenced static parameters of the
initial setup are safe only because, under the current lifetime
contract, the previous owner's WQEs are known to be complete by the
time the TIS leaves the pool. Across SQs, nothing is ordered.
quoted
The normal close path has additional ordering. tls_dev_del() runs
from work queued by tls_device_sk_destruct(), which itself runs from
__sk_free(), i.e. only once sk_wmem_alloc reaches zero. For
transmitted mlx5 skbs, SQ completion calls napi_consume_skb(), which
invokes tcp_wfree() to release that accounting, and parameter and DUMP
WQEs preceding a posted SEND on the same SQ are retired by that SEND's
completion. For successful transmissions, this gives close-time
ordering without an explicit drain in del_tx. WQEs posted without a
subsequent SEND, and SQ error/reset paths, need separate lifetime
guarantees. Live rekey does not wait for socket destruction.
Correct, including the completion detail: parameter and DUMP WQEs do
not request CQEs of their own, so they are retired only when a later
CQE-generating WQE on the same SQ completes. And you have a point
about WQEs posted without a subsequent SEND: the error paths of
mlx5e_ktls_handle_tx_skb(), and a DMA mapping failure of the SEND
itself in mlx5e_sq_xmit_wqe(), drop the skb after parameter/DUMP WQEs
were posted for it. Its wmem is then released while those WQEs are
still pending. That is a narrow, pre-existing hole in the
destruct-time argument, independent of rekey. SQ error/reset is
covered: mlx5e_free_txqsq_descs() runs only once the SQ was moved out
of RDY, when the hardware is done with it.
quoted
Was the recycling fence relying on this socket-lifetime ordering, or
does it independently protect against outstanding work on another SQ?
c4dfe704f53f ("net/mlx5e: kTLS, Recycle objects of device-offloaded
TLS TX connections") says the fence ensures context modification does
not interfere with existing traffic. Does that cover old SEND, DUMP
and parameter WQEs on another SQ, including work the device has not
yet executed?
It relies on the socket-lifetime ordering. The fence in c4dfe704f53f
addresses the same-SQ pipelining: the new owner's data must not be
transmitted before the context update has completed, and the update
must not interfere with WQEs in execution on that SQ. It does not, and
cannot, cover WQEs pending on another SQ; those were assumed complete
by the destruct-time contract you describe, or stopped by the channel
teardown. Live rekey via del+add breaks that assumption.
quoted
If reuse must wait, would retaining the TIS and DEK until all
referencing work has completed or been definitively stopped suffice,
including during SQ error/reset?
Yes, that is the right condition: every WQE referencing the TIS (or
the DEK) has either completed - a CQE of a later WQE on the same SQ
retires it, as the completions of an SQ are in order - or its SQ was
moved out of RDY and its descriptors were reclaimed in
mlx5e_free_txqsq_descs(), which covers the error/recovery and
teardown paths.
As for how to get there, here is what I have in mind, in three parts.
Feedback is welcome on all of them, and in particular on the third,
as it affects your series.
1. Pre-existing, independent of rekey: the stranded WQEs above.
Rather than adding completion tracking for the TIS, I'd restore the
socket-lifetime contract in the driver: an skb dropped after WQEs
were posted for it is not freed right away, but held by a NOP that
requests a completion, so that the skb and its wmem are released
only once all WQEs posted ahead of it are retired (or the SQ is
reset). mlx5e_ktls_del_tx() stays as is. I intend to send this as a
fix to net.
Agreed. Three spots for it: on SEND DMA failure, mlx5e_sq_xmit_wqe() and
mlx5e_sq_xmit_mpwqe() already post a NOP through mlx5e_tx_flush() but
free the skb first, so the skb should be held instead; that NOP asks for
a completion because mlx5e_notify_hw() sets MLX5_WQE_CTRL_CQ_UPDATE, not
mlx5e_post_nop(). mlx5e_ktls_handle_tx_skb()'s err_out needs the same
hold-NOP and a doorbell, since it can leave parameter/DUMP WQEs posted
without one. The MPWQE case is reachable too: when TX MPWQE is enabled,
mlx5e_sq_xmit_mpwqe() takes any linear, untagged, non-GSO skb with
inline mode NONE, which includes a pass-through skb that posted pending
parameters and took SKIP_NO_DATA, since it leaves with no TISN.
One correction to "SQ error/reset is covered": error-CQE recovery never
runs mlx5e_free_txqsq_descs(). mlx5e_tx_reporter_err_cqe_recover()
waits in mlx5e_wait_for_sq_flush() for cc to reach pc, then zeroes both
counters, so on success every WQE has been retired by an error CQE and
the held skb is released normally. If that wait times out, the SQ stays
in ERR with its queue stopped but NAPI still polling, so unretired skbs
stay held until a later completion or close. Timeout recovery is
different: mlx5e_safe_reopen_channels() closes the old SQs and reclaims
their descriptors in mlx5e_free_txqsq_descs(). In all of them the
hold-NOP from (1) eventually gets its skb released; whether the device
has stopped touching the TIS by then is the first question above.
A related recovery gap exists today without rekey: if a connection's
initial static parameters are flushed without executing, recovery does
not re-arm ctx_post_pending. A retransmit of the same record can also
skip static parameters, so software does not guarantee that the new
context gets installed. What does the hardware do with a SEND naming a
TIS whose static parameters never executed, recycled or freshly created?
The rekey contract below handles the corresponding replacement case;
initial setup may need the same treatment.
When do you expect to post (1)? I'll base the series on it and can
test on ConnectX-6 Dx before you post.
2. Pre-existing, and a dependency of rekey: TX queue stability. I'd
like to address the queue moves above in the stack, opt-in: a
driver whose TX offload keeps per-connection state updated by queue
execution declares it with a tlsdev_ops flag, and for sockets
offloaded to it, the TX queue is re-selected only once
sk_wmem_alloc has drained. That is, ooo_okay keeps only its wmem
condition, and while wmem is held, a dst reset does not clear the
cached queue and txq_reselection_ms does not expire it. With (1), a
drained wmem means that no work referencing the connection state is
pending on any SQ. txhash rethink and PLB still fire, and their
network-level repathing (e.g. the IPv6 flowlabel) is unaffected;
only the local queue switch waits until the driver drains. Other
drivers, and non-offloaded sockets, keep today's behavior.
Gating ooo_okay, the dst-reset queue clear and txq_reselection_ms
on wmem covers every path through sk_tx_queue_mapping. In mlx5e the
SQ can still change while that mapping stays put, or bypass it:
- Channel replacement with tis_tir_td_order destroys the old
channels only after the new ones are active (3b88a535a8e1).
mlx5e_deactivate_txqsq() posts a final NOP, but nothing waits for
its CQE, and mlx5e_poll_tx_cq() returns early once ENABLED is
cleared. The deactivated SQ can still hold old-key WQEs (e.g.
under PFC pause) while the same txq index already names the new
SQ, so (3a)'s "same SQ as all old-key WQEs" holds only once the
old SQ is destroyed, which is the DESTROY_SQ question above.
- With num_tcs > 1, mlx5e_select_queue() adds a priority offset
after netdev_pick_tx() (DSCP under MLX5_QPTS_TRUST_DSCP, else VLAN
prio; HTB by skb->priority), so IP_TOS/IPV6_TCLASS or SO_PRIORITY
changes move the SQ.
- Egress skbedit queue_mapping bypasses ndo_select_queue(), and
SKBEDIT_F_TXQ_SKBHASH can move a flow after a txhash rethink.
- A queue-count change: netdev_pick_tx() ignores a cached index
>= real_num_tx_queues whatever wmem says.
Separately, "a drained wmem means that no work referencing the
connection state is pending on any SQ" has a pre-existing hole:
netem with delay, jitter or rate calls skb_orphan_partial(), which
keeps skb->sk but swaps tcp_wfree for sock_efree, releasing the wmem
charge while the skb is parked. can_skb_orphan_partial() refuses
decrypted skbs and anything not owned through sock_wfree/tcp_wfree,
so two kinds of tcp_wfree skb get through:
- plain TCP data queued before setsockopt(TLS_TX); at dequeue it
ends before the start marker, and mlx5e_ktls_handle_tx_skb()
posts the pending parameter WQEs naming the new TIS and takes
the MLX5E_KTLS_SYNC_SKIP_NO_DATA path;
- new-key ciphertext the SW context sent during rekey, an ordinary
tcp_wfree skb not marked decrypted; dequeued after rekey
completion, it passes through tls_validate_xmit_skb() on the
device or on a bond master, and ends before the completion's
marker.
The socket reference held by skb_set_owner_sk_safe() still delays
destruct, so your destruct-time argument in (1) is unaffected; only
the wmem gate in (2) can open early. Both orphan paths release the
charge, but (2) reads a drained charge as no outstanding work; the
partial orphan keeps the reference and drops the charge, so the two
come apart.
Both kinds of skb above can be handled in the driver alone: mlx5e can
skip the context WQEs for an skb with skb->destructor != tcp_wfree (the
test tcp_gso_segment() already uses), which covers the pre-install
plaintext, whose orphan decision netem_enqueue() made before the socket
was offloaded, and the parked rekey ciphertext alike. Both already go
out as they are; the pending parameters just wait for the next skb. No
record data awaiting HW encryption reaches that path: a decrypted skb
is never partially orphaned, and a fully orphaned decrypted one is
dropped in validate_xmit_skb() before any driver sees it. Refusing
partial orphan in can_skb_orphan_partial() would also close the rekey
case, but needs a core-to-TLS hook and still misses the pre-install
one.
So (2) as proposed is a good stability hint, but not proof. I see
two ways to get correctness:
i. Close the holes individually, without a new per-packet lock: the
driver check above, keeping the priority offset and skbedit on
the pinned queue for offloaded sockets, and, while kTLS TX
connections are live, closing the old channels before the
switch; mlx5e can tell that today only by subtracting tx_tls_del
from tx_tls_ctx, so it needs a proper count. That last part
gives back the downtime 3b88a535a8e1 saved, and still relies on
your answer to the first question; (i) only moves that
dependency earlier. It also spreads the invariant across the
stack and the driver, and a future queue-selection path can
silently break it.
ii. A driver-side owner check where WQEs are posted: priv_tx
records its owner SQ as (txq index, generation) and the last WQE
counter it posted there; a post on another SQ is admitted only
once the owner's completions cover that counter or its
generation is retired (by a completed DESTROY_SQ or recovery,
if the first question above confirms that suffices), and is
otherwise SW-encrypted with tls_encrypt_skb(), as chcr, nfp and
funeth do. Completion progress stays per SQ, so this is not
per-connection completion accounting, but it does need a
per-connection lock on the kTLS xmit path, which is uncontended
unless the flow moves. I'd measure it before posting.
I lean towards (ii) for robustness, with (2) as the hint that keeps
it off the slow path, but I'd rather hear your view on the hot-path
cost first.
3. Rekey. Releasing the TIS of a live connection to the pool, as
del+add does, requires that no old-key work is still pending, which
the driver cannot cheaply tell. I see two directions:
a. Keep the TIS, and replace the key in place through a new tlsdev
op (say, tls_dev_rekey()). With (2), the new parameters are
posted on the same SQ as all old-key WQEs, fenced, so they take
effect only after them, and the pool is not involved. The driver
keeps the replaced key until it is no longer in use. For that,
the op would be called only once all data sent with the current
key was acknowledged, which, as I understand it, your rekey
completion already waits for, and which is needed anyway, as the
device cannot retransmit old-key data after the switch.
b. Keep del+add, and have the TLS core call tls_dev_del() only once
sk_wmem_alloc has drained. With (1), that guarantees no pending
work references the TIS or the DEK. The cost is a potentially
long wait on a backlogged or paused queue, during which new data
is held back or sent in SW.
Tracking SEND completions per connection in the driver would also
work, but means per-packet accounting for a rare event.
I lean towards (a). Would it fit your series, or do you see reasons
to keep del+add?
(3a) fits, and I prefer it. sendmsg keeps sending SW ciphertext during
rekey, which holds wmem, so (3b) can wait indefinitely unless the
sender stalls. With this series, mlx5e is the only in-tree driver
accepting TLS 1.3 crypto_info, so the core can require the new op for
TX rekey and leave other devices in SW.
Proposed contract:
int (*tls_dev_rekey)(struct net_device *netdev, struct sock *sk,
enum tls_offload_ctx_dir direction,
struct tls_crypto_info *crypto_info,
u32 start_offload_tcp_sn);
- direction is kept for symmetry with tls_dev_add()/tls_dev_del();
only TX uses it for now, RX stays on del+add.
- Called from tls_device_complete_rekey() in process context, under
tx_lock, the socket lock and device_offload_lock (read), where
tls_dev_del() is called today. All old-key records are ACKed and
synchronize_net() has drained in-progress transmissions; skbs
still in a qdisc are revalidated at dequeue by the rekey
validator. crypto_info carries the new key and the live record
sequence; start_offload_tcp_sn is write_seq.
- The core builds the new fallback AEAD before the call (today it
is built between del and add) and commits it, with the record
marker, before restoring the HW validator. Until then the driver
sees no new offload plaintext.
- The driver keeps the TIS, allocates the new DEK and does all
sleeping firmware work first, then publishes key, expected
sequence and pending parameters. The re-post covers both static
and progress parameters, with the fence on the first WQE (unlike
today's initial post, where the static params go out unfenced).
- The old DEK is freed from process context once a successful
CQE covers the fenced replacement, subject to your answer to the
second question above.
- Retirement of the SQ only proves the old work stopped, not that
the replacement took effect. If the SQ errors or closes before
a successful CQE covers the replacement, the driver forces a
resync on the next offloaded skb regardless of expected_seq:
fenced static and progress params for that skb's record, with
the skip-static shortcut bypassed, and a DUMP of any record
prefix before the SEND. A plain repost is not enough, since
expected_seq has already advanced and the next skb can match
it while starting mid-record.
- On failure, the old context and key stay intact, nothing is
posted, and traffic stays in SW until teardown or a later KeyUpdate.
If this direction works for you, I'll write mlx5e_ktls_rekey_tx()
and whichever (2) option you prefer on top of your fix, and test
them on the same hardware.
Thanks,
Rishikesh
quoted
I have not reproduced a failure; the reuse is visible in the driver,
but I have not established the hardware ordering guarantee. A PRM
reference, or a statement of which completion must precede reuse,
would settle it.
Thanks,
Tariq
quoted
Thanks,
Rishikesh
[1] https://lore.kernel.org/netdev/20260917224355.2288021-1-rjethwani@purestorage.com/ (local)