M Muhammad Zuhayr al-Tayyar #
Message details and related content
Hi,
When a flowtable with "flags offload" is deleted (nft flush ruleset,
deleting the table/flowtable, or loading a ruleset that starts with
"flush ruleset"), the hardware entries of connections that are offloaded
at that moment are never removed from the NIC. The NIC keeps forwarding
those connections with no ruleset loaded. For a firewall this means that
replacing or removing the ruleset does not stop already offloaded
connections.
Environment
-----------
- NIC: NVIDIA ConnectX-6 Dx, in-tree mlx5_core, firmware 22.36.1010
(DEL0000000027), eswitch switchdev, flow_steering_mode smfs
- Kernel: AlmaLinux 10.2, 6.12.0-211.69.1.el10_2.x86_64 (distribution
kernel with backports; behaves like current mainline here)
- nft: flowtable "flags offload" on two VLAN subinterfaces of the
uplink (indirect block, mlx5e_rep_indr_setup_ft_cb); traffic
routed between the two VLANs
- Note: on mlx5 a separate driver bug keeps flowtable rules from being
hit until a chain-0 tc rule with prio >= 2 exists (reported
separately to netdev). For this test I added and removed such a
rule first:
tc qdisc add dev <uplink> ingress
tc filter add dev <uplink> ingress protocol ip prio 2 flower \
skip_sw dst_ip 192.0.2.254 action drop
tc qdisc del dev <uplink> ingress
Symptom
-------
With a TCP transfer offloaded ([HW_OFFLOAD], uplink rx_packets ~20 pps vs
rx_packets_phy ~19k pps), I ran "nft flush ruleset". Afterwards:
- the transfer continues and is still hardware-forwarded (rx_packets
~20 pps, rx_packets_phy ~19k pps), with NO ruleset loaded;
- the conntrack entry shows [ASSURED] (no longer offloaded), its counters
are frozen and its timeout only counts down, because no packet of the
connection reaches the kernel;
- "devlink dev eswitch set pci/<BDF> mode legacy" fails with
"Can't change mode, E-Switch is busy" (mlx5 still counts the leaked
rules in esw->user_count);