# Re: \[BUG\] netfilter: nf\_tables: hw-offloaded flowtable entries leak when the flowtable is deleted (UNBIND at commit, DESTROY at release)

- From: Pablo Neira Ayuso \<pablo@netfilter.org\>
- Date: 2026-11-04 10:14:39 
- List: netdev
- Page: https://ratatoskr.run/netdev/2026/11/17859688
- Thread: https://ratatoskr.run/netfilter-devel/2026/11/17859664/t.md

```text
Hi,

On Wed, Nov 04, 2026 at 10:07:51AM +0000, Muhammad Zuhayr al-Tayyar wrote:
> Hi,
>  
> When a flowtable with "flags offload" is deleted (nft flush ruleset,
> deleting the table/flowtable, or loading a ruleset that starts with
> "flush ruleset"), the hardware entries of connections that are offloaded
> at that moment are never removed from the NIC. The NIC keeps forwarding
> those connections with no ruleset loaded. For a firewall this means that
> replacing or removing the ruleset does not stop already offloaded
> connections.

Thanks, this bugs has been already reported 3 times in the last two
weeks.

I posted an alternative patch which is not yet covering all cases.

This is on my radar.

> Environment
> -----------
> - NIC:     NVIDIA ConnectX-6 Dx, in-tree mlx5_core, firmware 22.36.1010
>            (DEL0000000027), eswitch switchdev, flow_steering_mode smfs
> - Kernel:  AlmaLinux 10.2, 6.12.0-211.69.1.el10_2.x86_64 (distribution
>            kernel with backports; behaves like current mainline here)
> - nft:     flowtable "flags offload" on two VLAN subinterfaces of the
>            uplink (indirect block, mlx5e_rep_indr_setup_ft_cb); traffic
>            routed between the two VLANs
> - Note:    on mlx5 a separate driver bug keeps flowtable rules from being
>            hit until a chain-0 tc rule with prio >= 2 exists (reported
>            separately to netdev). For this test I added and removed such a
>            rule first:
>              tc qdisc add dev <uplink> ingress
>              tc filter add dev <uplink> ingress protocol ip prio 2 flower \
>                  skip_sw dst_ip 192.0.2.254 action drop
>              tc qdisc del dev <uplink> ingress
> 
> Symptom
> -------
> With a TCP transfer offloaded ([HW_OFFLOAD], uplink rx_packets ~20 pps vs
> rx_packets_phy ~19k pps), I ran "nft flush ruleset". Afterwards:
> - the transfer continues and is still hardware-forwarded (rx_packets
>   ~20 pps, rx_packets_phy ~19k pps), with NO ruleset loaded;
> - the conntrack entry shows [ASSURED] (no longer offloaded), its counters
>   are frozen and its timeout only counts down, because no packet of the
>   connection reaches the kernel;
> - "devlink dev eswitch set pci/<BDF> mode legacy" fails with
>   "Can't change mode, E-Switch is busy" (mlx5 still counts the leaked
>   rules in esw->user_count);
> - the entries stay until reboot. After "devlink dev reload ... action
>   driver_reinit" the rules are gone from the NIC, but the eswitch still
>   refuses mode changes ("busy") until a reboot;
> - the mlx5 SW steering dump (/sys/kernel/debug/mlx5/<BDF>/steering/fdb/)
>   still shows the two flow rules of the connection after the flush.
> 
>   Dump taken after "nft flush ruleset" (nft list ruleset is empty),
>   decoded with a small local script:
>     192.168.1.10:22 -> 192.168.2.10:41234 tcp  pop-vlan, set-dmac,
>         set-smac, counter, -> FW table 0xf (VLAN push + out)
>     192.168.2.10:41234 -> 192.168.1.10:22 tcp  pop-vlan, set-dmac,
>         set-smac, counter, -> FW table 0x10 (VLAN push + out)
>   Raw rule entries (match on 192.168.1.10/192.168.2.10, ports 22/41234,
>   tcp):
>     3303,0xf000004c,0x6ac5e1e0,...c0a8010ac0a8020a0016a11200000006
>     3303,0xf000004f,0x6ac5e660,...c0a8020ac0a8010aa112001600000006
> 
> Trace
> -----
> From an earlier run of the same setup: function_graph on
> nf_tables_flowtable_destroy, nf_flow_table_free,
> nf_flow_table_offload_setup and flow_offload_work_handler
> (funcgraph-retval, funcgraph-abstime), plus the mlx5:mlx5e_delete_flower
> event.
> 
> At load, BIND succeeds for both devices, and FLOW_CLS_REPLACE reaches the
> driver callbacks:
>   841.635738 | 15)  nf_flow_table_offload_setup [nf_flow_table]() {
>   841.635751 | 15)    } /* flow_indr_dev_setup_offload = 0x1 */
>   841.636970 |  3)      nf_flow_offload_tuple [nf_flow_table]() {
>   841.636972 |  3)        mlx5e_rep_indr_setup_ft_cb [mlx5_core]() {
>   841.637029 |  3)        } /* mlx5e_rep_indr_setup_ft_cb = -95 */  (other device)
>   841.637030 |  3)        mlx5e_rep_indr_setup_ft_cb [mlx5_core]() {
>   841.637729 |  3)        } /* mlx5e_rep_indr_setup_ft_cb = 0x0 */
>   841.637730 |  3)      } /* nf_flow_offload_tuple [nf_flow_table] = 0x1 */
> 
> At "nft flush ruleset", the release path runs and the DESTROY work finds
> no callbacks:
>   847.435723 | 23)  nf_tables_flowtable_destroy [nf_tables]() {
>   847.435724 | 23)    nf_flow_table_free [nf_flow_table]() {
>   847.435852 | 16)  flow_offload_work_handler [nf_flow_table]() {
>   847.435853 | 16)    nf_flow_offload_tuple [nf_flow_table]() {
>   847.435855 | 16)    } /* nf_flow_offload_tuple = 0x0 */  <- no driver callback
>   847.435855 | 16)    nf_flow_offload_tuple [nf_flow_table]() {
>   847.435857 | 16)    } /* nf_flow_offload_tuple = 0x0 */
>   847.435859 | 16)  } /* flow_offload_work_handler [nf_flow_table] = 0x0 */
>   847.435926 | 23)    } /* nf_flow_table_free [nf_flow_table] = ... */
>   847.435938 | 23)  } /* nf_tables_flowtable_destroy [nf_tables] = 0x0 */
> - No nf_flow_table_offload_setup(UNBIND) call inside
>   nf_tables_flowtable_destroy(); the unbind had already happened at commit.
> - mlx5e_delete_flower events: 0.
> 
> Control experiment: with the flowtable left in place, "conntrack -D" on the
> offloaded connection makes mlx5e_delete_flower fire twice (original +
> reply), and the hardware entries are removed correctly.
> 
> Analysis (mainline)
> -------------------
> - When NFT_MSG_DELFLOWTABLE is committed, the kernel calls
>   nft_unregister_flowtable_net_hooks() -> nft_unregister_flowtable_ops(),
>   which does nf_unregister_net_hook() and then
>   data.type->setup(..., FLOW_BLOCK_UNBIND).
> - FLOW_BLOCK_UNBIND only frees the flow_block_cb entries
>   (nf_flow_table_block_setup()). It does not remove offloaded flows.
> - Later the release path calls nf_tables_flowtable_destroy() ->
>   data.type->free() = nf_flow_table_free(). That tears the flows down and
>   queues FLOW_CLS_DESTROY through nf_flow_offload_tuple() over
>   flowtable->flow_block.cb_list, which is empty by then. The hardware
>   entries leak.
> 
> Before 13210fc63f35 ("netfilter: nf_tables: imbalance in flowtable
> binding", v6.13), nf_tables_flowtable_destroy() called free() first and
> only then FLOW_BLOCK_UNBIND for each hook, so the DESTROYs were delivered
> while the callbacks were still registered. That commit moved the UNBIND
> into __nft_unregister_flowtable_net_hooks() to fix BIND/UNBIND imbalances,
> but the hardware flows are now deleted only after the callbacks are gone.
> 
> The commit was backported to the stable branches. In the stable trees,
> 6.12.10+ (6.12.9 is the last without it), 6.6.y, 6.1.y, 5.15.y and 5.10.y
> all have the UNBIND in the commit path and none in
> nf_tables_flowtable_destroy(). So every maintained kernel appears to be
> affected, including the distribution kernel I tested. The code is
> unchanged as of v7.3 and v7.4-rc1.
> 
> The same leak has been reported independently on a different driver:
> Qualcomm IPQ9570 PPE, kernel 6.18.55 (ImmortalWrt/OpenWrt,
> https://github.com/perceival/openwrt-flint3/issues/115, 2026-10-22). The
> reporter's PPE flow table filled up after a few firewall reloads and gave
> the same root cause (hooks unbound at commit before nf_flow_table_free()
> sends the DESTROYs). Their fix is local to that driver.
> 
> Proposed tag:
>   Fixes: 13210fc63f35 ("netfilter: nf_tables: imbalance in flowtable binding")
> The fix should keep that commit's balancing and only make sure the
> hardware entries are deleted before FLOW_BLOCK_UNBIND.
> 
> Possible directions:
>  (a) tear down and flush the flowtable's hardware entries before
>      FLOW_BLOCK_UNBIND in the commit path, or
>  (b) for a full flowtable deletion, keep the UNBIND in
>      nf_tables_flowtable_destroy() after free(), as in v6.12.
> 
> One way to do (a) that covers every UNBIND caller (flowtable deletion,
> removing a device from a flowtable, the NETDEV_UNREGISTER event): in
> nf_flow_table_offload_setup(), before unbinding, tear down the flows that
> use the device and wait for their DESTROY work. This is UNTESTED:
> 
> 	if (!nf_flowtable_hw_offload(flowtable))
> 		return nf_flow_offload_xdp_setup(flowtable, dev, cmd);
> 
> +	/* Remove this device's hardware entries while the driver
> +	 * callbacks are still registered.
> +	 */
> +	if (cmd == FLOW_BLOCK_UNBIND)
> +		nf_flow_table_gc_cleanup(flowtable, dev);
> +
> 	if (dev->netdev_ops->ndo_setup_tc)
> 
> nf_flow_table_gc_cleanup() already does teardown -> flush_delayed_work(gc)
> -> nf_flow_table_offload_flush(), the order established by 91bfaa15a379
> ("netfilter: flowtable: Fix hardware flush order on nf_flow_table_cleanup")
> for the netdev-down case. This only works if no caller unbinds after
> nf_flow_table_free() has destroyed the hashtable, which is true for the
> current mainline destroy path. Removing one device from an existing
> flowtable needs the same care; I have not tested that case.
> 
> Reproduction (control vs. bug, same setup, only the teardown order differs)
> --------------------------------------------------------------------------
> Minimal ruleset:
> 
>   table inet leak {
>       flowtable ft {
>           hook ingress priority 0
>           devices = { "<uplink>.2", "<uplink>.3" }
>           flags offload
>           counter
>       }
>       chain forward {
>           type filter hook forward priority 0; policy accept;
>           meta l4proto tcp ct state established flow add @ft
>       }
>   }
> 
> Control: "nft flush chain inet leak forward" (stop new offloads),
> "conntrack -D" for the offloaded connection, wait 4 s, then
> "nft flush ruleset": mlx5e_delete_flower = 2, the kernel forwards again
> (rx_packets == rx_packets_phy), eswitch mode change works.
> 
> Bug: "nft flush ruleset" directly: mlx5e_delete_flower = 0, the NIC keeps
> forwarding, eswitch busy.
> 
> Steps for the bug case:
> 1. Load the ruleset above; start a TCP transfer between the two VLANs and
>    wait until it is [HW_OFFLOAD] with rx_packets << rx_packets_phy.
> 2. nft flush ruleset
> 3. The transfer keeps running with rx_packets << rx_packets_phy. The
>    conntrack entry is [ASSURED] with frozen counters. The flow rules are
>    still in the mlx5 steering dump. devlink eswitch mode change -> EBUSY.
> 
> 
> Thanks,
> Mohammed Zuhair, Earthlink Telecommunications
```
