Julian

what we want is if RS weight is 0, then no new connections should be
served even if conn_reuse_mode is 0, just as commit dc7b3eb900aa
("ipvs: Fix reuse connection if real server is
dead") trying to do

Pls let me know if there are any other issues of concern

On Tue, Oct 26, 2021 at 2:13 PM yangxingwu [off-list ref] wrote:

thanks Julian

yes, I know that the one-second delay issue has been fixed by commit
f0a5e4d7a594e0fe237d3dfafb069bb82f80f42f if we set conn_reuse_mode to
1

BUT  it's still NOT what we expected with sysctl settings
(conn_reuse_mode == 0 && expire_nodest_conn == 1).

We run kubernetes in extremely diverse environments and this issue
happens a lot.

On Tue, Oct 26, 2021 at 1:44 PM Julian Anastasov [off-list ref] wrote:

quoted

        Hello,

On Tue, 26 Oct 2021, yangxingwu wrote:

quoted

thanks julian

What happens in this situation is that if we set the wait of the
realserver to 0 and do NOT remove the weight zero realserver with
sysctl settings (conn_reuse_mode == 0 && expire_nodest_conn == 1), and
the client reuses its source ports, the kernel will constantly
reuse connections and send the traffic to the weight 0 realserver.

        Yes, this is expected when conn_reuse_mode=0.

quoted

you may check the details from
https://github.com/kubernetes/kubernetes/issues/81775

        What happens if you try conn_reuse_mode=1? The
one-second delay in previous kernels should be corrected with

commit f0a5e4d7a594e0fe237d3dfafb069bb82f80f42f
Date:   Wed Jul 1 18:17:19 2020 +0300

    ipvs: allow connection reuse for unconfirmed conntrack

quoted

On Tue, Oct 26, 2021 at 2:12 AM Julian Anastasov [off-list ref] wrote:

quoted

On Mon, 25 Oct 2021, yangxingwu wrote:

quoted

Since commit dc7b3eb900aa ("ipvs: Fix reuse connection if real server is
dead"), new connections to dead servers are redistributed immediately to
new servers.

Then commit d752c3645717 ("ipvs: allow rescheduling of new connections when
port reuse is detected") disable expire_nodest_conn if conn_reuse_mode is
0. And new connection may be distributed to a real server with weight 0.

        Your change does not look correct to me. At the time
expire_nodest_conn was created, it was not checked when
weight is 0. At different places different terms are used
but in short, we have two independent states for real server:

- inhibited: weight=0 and no new connections should be served,
        packets for existing connections can be routed to server
        if it is still available and packets are not dropped
        by expire_nodest_conn.
        The new feature is that port reuse detection can
        redirect the new TCP connection into a new IPVS conn and
        to expire the existing cp/ct.

- unavailable (!IP_VS_DEST_F_AVAILABLE): server is removed,
        can be temporary, drop traffic for existing connections
        but on expire_nodest_conn we can select different server

        The new conn_reuse_mode flag allows port reuse to
be detected. Only then expire_nodest_conn has the
opportunity with commit dc7b3eb900aa to check weight=0
and to consider the old traffic as finished. If a new
server is selected, any retrans from previous connection
would be considered as part from the new connection. It
is a rapid way to switch server without checking with
is_new_conn_expected() because we can not have many
conns/conntracks to different servers.

Regards

--
Julian Anastasov [off-list ref]