On 6/4/26 1:19 AM, Mike Fara wrote:

Hi,

Software-kTLS (TLS_SW) TX over sendfile()/splice() drops to the TCP
persist-timer cadence (tens of KB/s, with individual sendfile() calls blocking
for tens of seconds) when the path MSS is large -- e.g. loopback (MSS 65483) or
jumbo frames. At a typical 1448-byte MSS it does not occur. Plain TCP
sendfile() on the same path is unaffected, and kTLS write() (no splice) is
unaffected, so it is specific to TLS_SW + the splice/sendfile path.

It triggers only on large-MSS paths with software kTLS (no NIC TLS offload), so
it is a niche path -- but it is a clean, reproducible multi-order-of-magnitude
cliff, so it seems worth a look. Reproduces on current mainline. CCing David
Howells as the author of the 2023 sendpage->MSG_SPLICE_PAGES splice_to_socket()
rework referenced below, and Eric/Paolo as this is as much a TCP-corking
interaction as a TLS one.

Environment
-----------
- net/tls TLS_SW (no NIC offload; ethtool tls-hw-tx-offload: off [fixed]).
- AES-GCM; gcm(aes) resolves to generic-gcm-vaes-avx512.

Reproducer (no OpenSSL/handshake; TLS_TX programmed with a fixed key, the
receiver discards ciphertext, like tools/testing/selftests/net/tls.c):

     cc -O2 -Wall -o ktls_sendfile_stall ktls_sendfile_stall.c
     ./ktls_sendfile_stall          # default loopback MSS (65483)
     ./ktls_sendfile_stall 1448     # clamp sender MSS via TCP_MAXSEG

Observed (loopback, single box):

     MSS=default    sent=     4.0 MiB in  52.08s  =>  0.0001 GiB/s   (stalled)
     MSS=1448       sent=  2048.0 MiB in   1.65s  =>  1.2106 GiB/s

i.e. ~four orders of magnitude; at the default MSS a single sendfile() blocks
for tens of seconds. For contrast, on the same loopback path:

     plain TCP sendfile() (no TLS ULP):           7.87  GiB/s
     kTLS write() (TLS_SW, no splice, 2 GiB):      1.99  GiB/s

I tested your ktls_sendfile_stall.c under stable 6.6 and upstream, but 
both of them works correctly.

~/code/tmp$ ./ktls_sendfile_stall 1448 MSS=1448 sent= 2048.0 MiB in 
2.32s => 0.8603 GiB/s ~/code/tmp$ ./ktls_sendfile_stall 1448 MSS=1448 
sent= 2048.0 MiB in 2.37s => 0.8439 GiB/s ~/code/tmp$ 
./ktls_sendfile_stall MSS=default sent= 2048.0 MiB in 1.64s => 1.2204 
GiB/s :~/code/tmp$ ./ktls_sendfile_stall MSS=default sent= 2048.0 MiB in 
1.70s => 1.1737 GiB/s ~/code/tmp$ ./ktls_sendfile_stall 1448 MSS=1448 
sent= 2048.0 MiB in 2.33s => 0.8570 GiB/s

~/code/tmp$ ethtool -k lo | grep "tls-hw-tx-offload" tls-hw-tx-offload: 
off [fixed]