[BUG] reason=TCP_ACK_UNSENT_DATA

For one particular workload, under load TCP streams grind to a halt, 
throughput falls to zero and never recovers.

On Power 11 systems, two asymmetric interfaces are bonded in an 
Active-Backup configuration. The primary interface is used with IPsec 
offload. The secondary interface only does software encryption (using 
aes_gcm_p10_crypto).

We are using iperf for the tests, and when we fail-over to the backup 
interface, all the iperf threads grind to a halt.

Once we are in this state, on the iperf client, the Send-Q is full. The 
iperf server is sending ACKs, but it appears that TCP (on the iperf 
client) is dropping them. Here is an example:

probe__tcp_rcv_established_L179    12 21012.316236112    19162 
kworker/12:2         __probe_ip=0xc0000000011b39f0, snd_una=0x535b271, 
snd_nxt=0x535b7f7, ack_seq=0x536420d, write_seq=0x57526df
         [c0000000011b39f0] tcp_rcv_established+0x3d0 ([kernel.kallsyms])
         [c0000000011b39f0] tcp_rcv_established+0x3d0 ([kernel.kallsyms])
         [c0000000011ca69c] tcp_v4_do_rcv+0x1ec ([kernel.kallsyms])
         [c0000000011cc9f0] tcp_v4_rcv+0xc30 ([kernel.kallsyms])
         [c000000001180ef0] ip_protocol_deliver_rcu+0x60 ([kernel.kallsyms])
         [c0000000011811d4] ip_local_deliver_finish+0xb4 ([kernel.kallsyms])
         [c000000001181338] ip_local_deliver+0x88 ([kernel.kallsyms])
         [c0000000011809d8] ip_sublist_rcv_finish+0x98 ([kernel.kallsyms])
         [c000000001180c0c] ip_sublist_rcv+0x1fc ([kernel.kallsyms])
         [c00000000118170c] ip_list_rcv+0x16c ([kernel.kallsyms])
         [c000000001068164] __netif_receive_skb_list_core+0x244 
([kernel.kallsyms])
         [c000000001068a88] netif_receive_skb_list_internal+0x248 
([kernel.kallsyms])
         [c000000001068eac] napi_complete_done+0x9c ([kernel.kallsyms])
         [c0000000010f940c] gro_cell_poll+0xcc ([kernel.kallsyms])
         [c0000000010690e0] __napi_poll+0x60 ([kernel.kallsyms])
         [c00000000106974c] net_rx_action+0x39c ([kernel.kallsyms])
         [c000000000241b68] handle_softirqs+0x158 ([kernel.kallsyms])
         [c000000000018998] do_softirq_own_stack+0x38 ([kernel.kallsyms])
         [c000000000242958] do_softirq+0x78 ([kernel.kallsyms])
         [c000000000242a44] __local_bh_enable_ip+0xc4 ([kernel.kallsyms])
         [c00800000a1c0bd0]
         [c00800000a1c07f4]
         [c000000000268a14] process_one_work+0x1e4 ([kernel.kallsyms])
         [c0000000002697ec] worker_thread+0x1ec ([kernel.kallsyms])
         [c0000000002762c4] kthread+0x194 ([kernel.kallsyms])
         [c00000000000ded8] start_kernel_thread+0x14 ([kernel.kallsyms])

skb__kfree_skb          12 21012.316237054    19162 kworker/12:2 
skbaddr=13835058059510742784, location=13835058055300675500, 
rx_sk=13835058058125504512, protocol=2048, reason=TCP_ACK_UNSENT_DATA
         [c0000000010d9924] __traceiter_kfree_skb+0x74 ([kernel.kallsyms])
         [0]
         [c0000000010372f4] sk_skb_reason_drop+0xf4 ([kernel.kallsyms])
         [c0000000011a77ac] tcp_drop_reason+0x6c ([kernel.kallsyms])
         [c0000000011ca69c] tcp_v4_do_rcv+0x1ec ([kernel.kallsyms])
         [c0000000011cc9f0] tcp_v4_rcv+0xc30 ([kernel.kallsyms])
         [c000000001180ef0] ip_protocol_deliver_rcu+0x60 ([kernel.kallsyms])
         [c0000000011811d4] ip_local_deliver_finish+0xb4 ([kernel.kallsyms])
         [c000000001181338] ip_local_deliver+0x88 ([kernel.kallsyms])
         [c0000000011809d8] ip_sublist_rcv_finish+0x98 ([kernel.kallsyms])
         [c000000001180c0c] ip_sublist_rcv+0x1fc ([kernel.kallsyms])
         [c00000000118170c] ip_list_rcv+0x16c ([kernel.kallsyms])
         [c000000001068164] __netif_receive_skb_list_core+0x244 
([kernel.kallsyms])
         [c000000001068a88] netif_receive_skb_list_internal+0x248 
([kernel.kallsyms])
         [c000000001068eac] napi_complete_done+0x9c ([kernel.kallsyms])
         [c0000000010f940c] gro_cell_poll+0xcc ([kernel.kallsyms])
         [c0000000010690e0] __napi_poll+0x60 ([kernel.kallsyms])
         [c00000000106974c] net_rx_action+0x39c ([kernel.kallsyms])
         [c000000000241b68] handle_softirqs+0x158 ([kernel.kallsyms])
         [c000000000018998] do_softirq_own_stack+0x38 ([kernel.kallsyms])
         [c000000000242958] do_softirq+0x78 ([kernel.kallsyms])
         [c000000000242a44] __local_bh_enable_ip+0xc4 ([kernel.kallsyms])
         [c00800000a1c0bd0]
         [c00800000a1c07f4]
         [c000000000268a14] process_one_work+0x1e4 ([kernel.kallsyms])
         [c0000000002697ec] worker_thread+0x1ec ([kernel.kallsyms])
         [c0000000002762c4] kthread+0x194 ([kernel.kallsyms])
         [c00000000000ded8] start_kernel_thread+0x14 ([kernel.kallsyms])

This has been recreated with kernel 7.2-rc5. This kernel does have the 
patch in:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=0c9de092ef8c50a7ee9612811566f0aa81d8d7b6

We tried turning off TSO as an experiment, but that didn't help.

This is quite easy to recreate (on Power 11), but it does appear that we 
do need specific combinations like a 100G crypto offload on the primary, 
and aes_gcm_p10_crypto for software encryption on the backup device. We 
require moderate to heavy traffic in both directions.

Bonding two interfaces, both using aes_gcm_p10_crypto and failing over 
between them doesn't recreate the problem.

Pradeep