[BUG] netfilter: nf_tables: hw-offloaded flowtable entries leak when the flowtable is deleted (UNBIND at commit, DESTROY at release)

Started by Muhammad Zuhayr al-Tayyar on Nov 4, 2026 · 2 messages from 2 authors · Last reply Nov 4, 2026

open the first message on its own page

Message details and related content
Hi,
 
When a flowtable with "flags offload" is deleted (nft flush ruleset,
deleting the table/flowtable, or loading a ruleset that starts with
"flush ruleset"), the hardware entries of connections that are offloaded
at that moment are never removed from the NIC. The NIC keeps forwarding
those connections with no ruleset loaded. For a firewall this means that
replacing or removing the ruleset does not stop already offloaded
connections.

Environment
-----------
- NIC:     NVIDIA ConnectX-6 Dx, in-tree mlx5_core, firmware 22.36.1010
           (DEL0000000027), eswitch switchdev, flow_steering_mode smfs
- Kernel:  AlmaLinux 10.2, 6.12.0-211.69.1.el10_2.x86_64 (distribution
           kernel with backports; behaves like current mainline here)
- nft:     flowtable "flags offload" on two VLAN subinterfaces of the
           uplink (indirect block, mlx5e_rep_indr_setup_ft_cb); traffic
           routed between the two VLANs
- Note:    on mlx5 a separate driver bug keeps flowtable rules from being
           hit until a chain-0 tc rule with prio >= 2 exists (reported
           separately to netdev). For this test I added and removed such a
           rule first:
             tc qdisc add dev <uplink> ingress
             tc filter add dev <uplink> ingress protocol ip prio 2 flower \
                 skip_sw dst_ip 192.0.2.254 action drop
             tc qdisc del dev <uplink> ingress

Symptom
-------
With a TCP transfer offloaded ([HW_OFFLOAD], uplink rx_packets ~20 pps vs
rx_packets_phy ~19k pps), I ran "nft flush ruleset". Afterwards:
- the transfer continues and is still hardware-forwarded (rx_packets
  ~20 pps, rx_packets_phy ~19k pps), with NO ruleset loaded;
- the conntrack entry shows [ASSURED] (no longer offloaded), its counters
  are frozen and its timeout only counts down, because no packet of the
  connection reaches the kernel;
- "devlink dev eswitch set pci/<BDF> mode legacy" fails with
  "Can't change mode, E-Switch is busy" (mlx5 still counts the leaked
  rules in esw->user_count);

162 more lines

Message details and related content
Hi,

On Wed, Nov 04, 2026 at 10:07:51AM +0000, Muhammad Zuhayr al-Tayyar wrote:
Hi,
 
When a flowtable with "flags offload" is deleted (nft flush ruleset,
deleting the table/flowtable, or loading a ruleset that starts with
"flush ruleset"), the hardware entries of connections that are offloaded
at that moment are never removed from the NIC. The NIC keeps forwarding
those connections with no ruleset loaded. For a firewall this means that
replacing or removing the ruleset does not stop already offloaded
connections.
Thanks, this bugs has been already reported 3 times in the last two
weeks.

I posted an alternative patch which is not yet covering all cases.

This is on my radar.
Environment
-----------
- NIC:     NVIDIA ConnectX-6 Dx, in-tree mlx5_core, firmware 22.36.1010
           (DEL0000000027), eswitch switchdev, flow_steering_mode smfs
- Kernel:  AlmaLinux 10.2, 6.12.0-211.69.1.el10_2.x86_64 (distribution
           kernel with backports; behaves like current mainline here)
- nft:     flowtable "flags offload" on two VLAN subinterfaces of the
           uplink (indirect block, mlx5e_rep_indr_setup_ft_cb); traffic
           routed between the two VLANs
- Note:    on mlx5 a separate driver bug keeps flowtable rules from being
           hit until a chain-0 tc rule with prio >= 2 exists (reported
           separately to netdev). For this test I added and removed such a
           rule first:
             tc qdisc add dev <uplink> ingress
             tc filter add dev <uplink> ingress protocol ip prio 2 flower \
                 skip_sw dst_ip 192.0.2.254 action drop
             tc qdisc del dev <uplink> ingress

Symptom
-------
With a TCP transfer offloaded ([HW_OFFLOAD], uplink rx_packets ~20 pps vs
rx_packets_phy ~19k pps), I ran "nft flush ruleset". Afterwards:
- the transfer continues and is still hardware-forwarded (rx_packets
  ~20 pps, rx_packets_phy ~19k pps), with NO ruleset loaded;
- the conntrack entry shows [ASSURED] (no longer offloaded), its counters
  are frozen and its timeout only counts down, because no packet of the
  connection reaches the kernel;
- "devlink dev eswitch set pci/<BDF> mode legacy" fails with
  "Can't change mode, E-Switch is busy" (mlx5 still counts the leaked
  rules in esw->user_count);
- the entries stay until reboot. After "devlink dev reload ... action
  driver_reinit" the rules are gone from the NIC, but the eswitch still
  refuses mode changes ("busy") until a reboot;
- the mlx5 SW steering dump (/sys/kernel/debug/mlx5/<BDF>/steering/fdb/)
  still shows the two flow rules of the connection after the flush.

  Dump taken after "nft flush ruleset" (nft list ruleset is empty),
  decoded with a small local script:
    192.168.1.10:22 -> 192.168.2.10:41234 tcp  pop-vlan, set-dmac,
        set-smac, counter, -> FW table 0xf (VLAN push + out)

152 more quoted lines

Keyboard shortcuts
hback out one level
jnext message in thread / next series-diff position
kprevious message in thread / previous series-diff position
ldrill in
Escclose help / fold thread tree
?toggle this help