[PATCH net-next v2] net/smc: Reduce overflow of smc clcsock listen queue
flat view
STALE1700d
LANDED: 1 (1M)
From: "D. Wythe" <alibuda@linux.alibaba.com>
Date: 2022-01-04 13:12:47
Also in:
linux-rdma, linux-s390
Subsystem:
networking [general], shared memory communications (smc) sockets, the rest · Maintainers:
"David S. Miller", Eric Dumazet, Jakub Kicinski, Paolo Abeni, D. Wythe, Dust Li, Sidraya Jayagond, Mahanta Jambigi, Linus Torvalds
1 review trailer (1 from subsystem maintainers); landed in mainline as 48b6190a0042 on 2022-02-11.
From: "D. Wythe" <alibuda@linux.alibaba.com> In nginx/wrk multithread and 10K connections benchmark, the backend TCP connection established very slowly, and lots of TCP connections stay in SYN_SENT state. Server: smc_run nginx Client: smc_run wrk -c 10000 -t 4 http://server Socket state in client host (wrk) shows like: ss -t | wc -l 10000 ss -t | grep "SYN-SENT" | wc -l 6248 While the socket state in server host (nginx) shows like: ss -t | wc -l 3752 Furthermore, the netstate of server host shows like: 145042 times the listen queue of a socket overflowed 145042 SYNs to LISTEN sockets dropped This issue caused by smc_listen_work(), since the smc_tcp_listen_work() shared the same workqueue (smc_hs_wq) with smc_listen_work(), while smc_listen_work() do blocking wait for smc connection established, which meanwhile block the accept() from TCP listen queue. This patch creates a independent workqueue(smc_tcp_ls_wq) for smc_tcp_listen_work(), separate it from smc_listen_work(), which is quite acceptable considering that smc_tcp_listen_work() runs very fast. After this patch, the smc 10K connections benchmark in my case is 5 times faster than before. Before patch : smc_run ./wrk -c 10000 -t 3 -d 20 http://server 3 threads and 10000 connections 143300 requests in 20.04s, 94.29MB read Requests/sec: 7150.33 Transfer/sec: 4.70MB After patch: smc_run ./wrk -c 10000 -t 3 -d 20 http://server 3 threads and 10000 connections 902091 requests in 21.99s, 593.56MB read Requests/sec: 41017.52 Transfer/sec: 26.99MB Signed-off-by: D. Wythe <alibuda@linux.alibaba.com> Reviewed-by: Tony Lu <tonylu@linux.alibaba.com> --- changelog: v2: code format --- net/smc/af_smc.c | 13 +++++++++++-- net/smc/smc.h | 1 + 2 files changed, 12 insertions(+), 2 deletions(-)
diff --git a/net/smc/af_smc.c b/net/smc/af_smc.c
index 0bb614e..08722c0 100644
--- a/net/smc/af_smc.c
+++ b/net/smc/af_smc.c@@ -62,6 +62,7 @@ * creation on client */ +struct workqueue_struct *smc_tcp_ls_wq; /* wq for tcp listen work */ struct workqueue_struct *smc_hs_wq; /* wq for handshake work */ struct workqueue_struct *smc_close_wq; /* wq for close work */
@@ -1872,7 +1873,7 @@ static void smc_clcsock_data_ready(struct sock *listen_clcsock) lsmc->clcsk_data_ready(listen_clcsock); if (lsmc->sk.sk_state == SMC_LISTEN) { sock_hold(&lsmc->sk); /* sock_put in smc_tcp_listen_work() */ - if (!queue_work(smc_hs_wq, &lsmc->tcp_listen_work)) + if (!queue_work(smc_tcp_ls_wq, &lsmc->tcp_listen_work)) sock_put(&lsmc->sk); } }
@@ -2610,9 +2611,14 @@ static int __init smc_init(void) goto out_nl; rc = -ENOMEM; + + smc_tcp_ls_wq = alloc_workqueue("smc_tcp_ls_wq", 0, 0); + if (!smc_tcp_ls_wq) + goto out_pnet; + smc_hs_wq = alloc_workqueue("smc_hs_wq", 0, 0); if (!smc_hs_wq) - goto out_pnet; + goto out_alloc_tcp_ls_wq; smc_close_wq = alloc_workqueue("smc_close_wq", 0, 0); if (!smc_close_wq)
@@ -2709,6 +2715,8 @@ static int __init smc_init(void) destroy_workqueue(smc_close_wq); out_alloc_hs_wq: destroy_workqueue(smc_hs_wq); +out_alloc_tcp_ls_wq: + destroy_workqueue(smc_tcp_ls_wq); out_pnet: smc_pnet_exit(); out_nl:
@@ -2728,6 +2736,7 @@ static void __exit smc_exit(void) smc_core_exit(); smc_ib_unregister_client(); destroy_workqueue(smc_close_wq); + destroy_workqueue(smc_tcp_ls_wq); destroy_workqueue(smc_hs_wq); proto_unregister(&smc_proto6); proto_unregister(&smc_proto);
diff --git a/net/smc/smc.h b/net/smc/smc.h
index b1d6625..18fa803 100644
--- a/net/smc/smc.h
+++ b/net/smc/smc.h@@ -256,6 +256,7 @@ static inline struct smc_sock *smc_sk(const struct sock *sk) return (struct smc_sock *)sk; } +extern struct workqueue_struct *smc_tcp_ls_wq; /* wq for tcp listen work */ extern struct workqueue_struct *smc_hs_wq; /* wq for handshake work */ extern struct workqueue_struct *smc_close_wq; /* wq for close work */
--
1.8.3.1