[Discuss]Why enable individual port in bonding 8023ad?
From: <hidden>
Date: 2024-11-04 11:33:17
Also in:
lkml
Why is individual aggregator's port enabled in function ad_agg_selection_logic ? I have found no basis for this in the IEEE 802.3ad standard. In fact, I had the same problem as chengyechun [off-list ref] and Thomas Bogendoerfer [off-list ref]. https://lore.kernel.org/netdev/c464627d07434469b363134ad10e3b4c@huawei.com/ (local) https://lore.kernel.org/netdev/20240404114908.134034-1-tbogendoerfer@suse.de/T/ (local) I use port 1 and port 2 form a bond interface and use nftables to discard LACP packets received by port 1. The bond configuration is as follows: BONDING_OPTS='mode=4 miimon=100 lacp_rate=fast xmit_hash_policy=layer3+4' TYPE=Bond BONDING_MASTER=yes BOOTPROTO=static NM_CONTROLLED=no IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no IPV6_ADDR_GEN_MODE=stable-privacy NAME=bond0 DEVICE=bond0 ONBOOT=yes IPADDR=1.1.1.38 NETMASK=255.255.0.0 IPV6ADDR=1:1:1::39/64 The slave configuration is as follows: and I have four similar slaves enp13s0 NAME=enp12s0 DEVICE=enp12s0 BOOTPROTO=none ONBOOT=yes USERCTL=no NM_CONTROLLED=no MASTER=bond0 SLAVE=yes IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no The nftables configuration is as follows: # cat /etc/nftables.conf table netdev filter { chain ingress { type filter hook ingress device enp13s0 priority 0; policy accept; meta protocol 0x8809 drop } } Then nft -f /etc/nftables.conf to apply this conf. During aggregation, the time sequence is as follows: 1. When bond0 receives the NETDEV_PRE_UP event, port 1 chooses as the active LAG. Since port 1 has not received the LACPDU, port 1 is considered as an individual port and is enabled by __enable_port in function ad_agg_selection_logic. [37.643701] bond0: bond_netdev_event received NETDEV_PRE_UP [37.643740] bond0: (slave enp13s0): LAG 1 chosen as the active LAG 2. The MUX state machine of port 2 enters the AD_MUX_WAITING state. [37.643763] bond0: (slave enp14s0): Mux Machine: Port=2, Last State=0, Curr State=1 [37.748705] bond0: (slave enp14s0): Mux Machine: Port=2, Last State=1, Curr State=2 3. Port 2 receives the LACPDU, since port 2 has partner but port 1 has no partner, port 2 is elected as the best aggregator by ad_agg_selection_logic, then __disable_ports port 1. Port 2 is not enabled just like port 1 because port 2 has partner. At the same time, the MUX state machine of port 2 is still in AD_MUX_WAITING (it takes about 2s AD_WAIT_WHILE_TIMER). At this time, the system does not have any enabled port (or usable slave). [37.960715] bond0: (slave enp14s0): LAG 2 chosen as the active LAG 4. Two seconds later, the MUX state machine of port 2 enters the AD_MUX_COLLECTING_DISTRIBUTING state and enabled by ad_mux_machine. The system finally has an enabled port. [39.976696] bond0: (slave enp14s0): Mux Machine: Port=2, Last State=2, Curr State=3 [40.084710] bond0: (slave enp14s0): Mux Machine: Port=2, Last State=3, Curr State=4 Within the range from [37.960715] to [40.084710], the system does not have any available port. The bond_xmit_3ad_xor_slave_get cannot obtain an available slave port. The bond_3ad_xor_xmit return drop, and the bond port cannot send packets. But if port 2 does not receive LACPDU, then almost all the time the bond can send packets on port 1. (except that the MUX state machine of port 1 changes from AD_MUX_ATTACHED to AD_MUX_COLLECTING_DISTRIBUTING, which is about 100 ms) In the scenario where port 1 cannot receive LACPDUs and port 2 cannot receive LACPDUs, the behavior of the bond interface should be the same. That is, whatever port 1 or port 2 cannot receive LACPDU, packets cannot be transmitted within an equal time, 2s or 100ms. But Port 1 cannot receive LACPDUs leads to 2s packet loss and port 2 cannot receive LACPDUs leads to 100 ms packet loss. Therefore, it is critical to understand why the individuals aggregator's port is enabled in the function ad_agg_selection_logic. The status of the MUX state machine of port 1 is not verified. Actually, when port 1 is enabled, the MUX state machine of port 1 is not processed. According to the IEEE 802.3ad standard, the status of port 1 should be disabled.