Hi,
When a flowtable with "flags offload" is deleted (nft flush ruleset,
deleting the table/flowtable, or loading a ruleset that starts with
"flush ruleset"), the hardware entries of connections that are offloaded
at that moment are never removed from the NIC. The NIC keeps forwarding
those connections with no ruleset loaded. For a firewall this means that
replacing or removing the ruleset does not stop already offloaded
connections.
Environment
-----------
- NIC: NVIDIA ConnectX-6 Dx, in-tree mlx5_core, firmware 22.36.1010
(DEL0000000027), eswitch switchdev, flow_steering_mode smfs
- Kernel: AlmaLinux 10.2, 6.12.0-211.69.1.el10_2.x86_64 (distribution
kernel with backports; behaves like current mainline here)
- nft: flowtable "flags offload" on two VLAN subinterfaces of the
uplink (indirect block, mlx5e_rep_indr_setup_ft_cb); traffic
routed between the two VLANs
- Note: on mlx5 a separate driver bug keeps flowtable rules from being
hit until a chain-0 tc rule with prio >= 2 exists (reported
separately to netdev). For this test I added and removed such a
rule first:
tc qdisc add dev <uplink> ingress
tc filter add dev <uplink> ingress protocol ip prio 2 flower \
skip_sw dst_ip 192.0.2.254 action drop
tc qdisc del dev <uplink> ingress
Symptom
-------
With a TCP transfer offloaded ([HW_OFFLOAD], uplink rx_packets ~20 pps vs
rx_packets_phy ~19k pps), I ran "nft flush ruleset". Afterwards:
- the transfer continues and is still hardware-forwarded (rx_packets
~20 pps, rx_packets_phy ~19k pps), with NO ruleset loaded;
- the conntrack entry shows [ASSURED] (no longer offloaded), its counters
are frozen and its timeout only counts down, because no packet of the
connection reaches the kernel;
- "devlink dev eswitch set pci/<BDF> mode legacy" fails with
"Can't change mode, E-Switch is busy" (mlx5 still counts the leaked
rules in esw->user_count);
- the entries stay until reboot. After "devlink dev reload ... action
driver_reinit" the rules are gone from the NIC, but the eswitch still
refuses mode changes ("busy") until a reboot;
- the mlx5 SW steering dump (/sys/kernel/debug/mlx5/<BDF>/steering/fdb/)
still shows the two flow rules of the connection after the flush.
Dump taken after "nft flush ruleset" (nft list ruleset is empty),
decoded with a small local script:
192.168.1.10:22 -> 192.168.2.10:41234 tcp pop-vlan, set-dmac,
set-smac, counter, -> FW table 0xf (VLAN push + out)
192.168.2.10:41234 -> 192.168.1.10:22 tcp pop-vlan, set-dmac,
set-smac, counter, -> FW table 0x10 (VLAN push + out)
Raw rule entries (match on 192.168.1.10/192.168.2.10, ports 22/41234,
tcp):
3303,0xf000004c,0x6ac5e1e0,...c0a8010ac0a8020a0016a11200000006
3303,0xf000004f,0x6ac5e660,...c0a8020ac0a8010aa112001600000006
Trace
-----
From an earlier run of the same setup: function_graph on
nf_tables_flowtable_destroy, nf_flow_table_free,
nf_flow_table_offload_setup and flow_offload_work_handler
(funcgraph-retval, funcgraph-abstime), plus the mlx5:mlx5e_delete_flower
event.
At load, BIND succeeds for both devices, and FLOW_CLS_REPLACE reaches the
driver callbacks:
841.635738 | 15) nf_flow_table_offload_setup [nf_flow_table]() {
841.635751 | 15) } /* flow_indr_dev_setup_offload = 0x1 */
841.636970 | 3) nf_flow_offload_tuple [nf_flow_table]() {
841.636972 | 3) mlx5e_rep_indr_setup_ft_cb [mlx5_core]() {
841.637029 | 3) } /* mlx5e_rep_indr_setup_ft_cb = -95 */ (other device)
841.637030 | 3) mlx5e_rep_indr_setup_ft_cb [mlx5_core]() {
841.637729 | 3) } /* mlx5e_rep_indr_setup_ft_cb = 0x0 */
841.637730 | 3) } /* nf_flow_offload_tuple [nf_flow_table] = 0x1 */
At "nft flush ruleset", the release path runs and the DESTROY work finds
no callbacks:
847.435723 | 23) nf_tables_flowtable_destroy [nf_tables]() {
847.435724 | 23) nf_flow_table_free [nf_flow_table]() {
847.435852 | 16) flow_offload_work_handler [nf_flow_table]() {
847.435853 | 16) nf_flow_offload_tuple [nf_flow_table]() {
847.435855 | 16) } /* nf_flow_offload_tuple = 0x0 */ <- no driver callback
847.435855 | 16) nf_flow_offload_tuple [nf_flow_table]() {
847.435857 | 16) } /* nf_flow_offload_tuple = 0x0 */
847.435859 | 16) } /* flow_offload_work_handler [nf_flow_table] = 0x0 */
847.435926 | 23) } /* nf_flow_table_free [nf_flow_table] = ... */
847.435938 | 23) } /* nf_tables_flowtable_destroy [nf_tables] = 0x0 */
- No nf_flow_table_offload_setup(UNBIND) call inside
nf_tables_flowtable_destroy(); the unbind had already happened at commit.
- mlx5e_delete_flower events: 0.
Control experiment: with the flowtable left in place, "conntrack -D" on the
offloaded connection makes mlx5e_delete_flower fire twice (original +
reply), and the hardware entries are removed correctly.
Analysis (mainline)
-------------------
- When NFT_MSG_DELFLOWTABLE is committed, the kernel calls
nft_unregister_flowtable_net_hooks() -> nft_unregister_flowtable_ops(),
which does nf_unregister_net_hook() and then
data.type->setup(..., FLOW_BLOCK_UNBIND).
- FLOW_BLOCK_UNBIND only frees the flow_block_cb entries
(nf_flow_table_block_setup()). It does not remove offloaded flows.
- Later the release path calls nf_tables_flowtable_destroy() ->
data.type->free() = nf_flow_table_free(). That tears the flows down and
queues FLOW_CLS_DESTROY through nf_flow_offload_tuple() over
flowtable->flow_block.cb_list, which is empty by then. The hardware
entries leak.
Before 13210fc63f35 ("netfilter: nf_tables: imbalance in flowtable
binding", v6.13), nf_tables_flowtable_destroy() called free() first and
only then FLOW_BLOCK_UNBIND for each hook, so the DESTROYs were delivered
while the callbacks were still registered. That commit moved the UNBIND
into __nft_unregister_flowtable_net_hooks() to fix BIND/UNBIND imbalances,
but the hardware flows are now deleted only after the callbacks are gone.
The commit was backported to the stable branches. In the stable trees,
6.12.10+ (6.12.9 is the last without it), 6.6.y, 6.1.y, 5.15.y and 5.10.y
all have the UNBIND in the commit path and none in
nf_tables_flowtable_destroy(). So every maintained kernel appears to be
affected, including the distribution kernel I tested. The code is
unchanged as of v7.3 and v7.4-rc1.
The same leak has been reported independently on a different driver:
Qualcomm IPQ9570 PPE, kernel 6.18.55 (ImmortalWrt/OpenWrt,
https://github.com/perceival/openwrt-flint3/issues/115, 2026-10-22). The
reporter's PPE flow table filled up after a few firewall reloads and gave
the same root cause (hooks unbound at commit before nf_flow_table_free()
sends the DESTROYs). Their fix is local to that driver.
Proposed tag:
Fixes: 13210fc63f35 ("netfilter: nf_tables: imbalance in flowtable binding")
The fix should keep that commit's balancing and only make sure the
hardware entries are deleted before FLOW_BLOCK_UNBIND.
Possible directions:
(a) tear down and flush the flowtable's hardware entries before
FLOW_BLOCK_UNBIND in the commit path, or
(b) for a full flowtable deletion, keep the UNBIND in
nf_tables_flowtable_destroy() after free(), as in v6.12.
One way to do (a) that covers every UNBIND caller (flowtable deletion,
removing a device from a flowtable, the NETDEV_UNREGISTER event): in
nf_flow_table_offload_setup(), before unbinding, tear down the flows that
use the device and wait for their DESTROY work. This is UNTESTED:
if (!nf_flowtable_hw_offload(flowtable))
return nf_flow_offload_xdp_setup(flowtable, dev, cmd);
+ /* Remove this device's hardware entries while the driver
+ * callbacks are still registered.
+ */
+ if (cmd == FLOW_BLOCK_UNBIND)
+ nf_flow_table_gc_cleanup(flowtable, dev);
+
if (dev->netdev_ops->ndo_setup_tc)
nf_flow_table_gc_cleanup() already does teardown -> flush_delayed_work(gc)
-> nf_flow_table_offload_flush(), the order established by 91bfaa15a379
("netfilter: flowtable: Fix hardware flush order on nf_flow_table_cleanup")
for the netdev-down case. This only works if no caller unbinds after
nf_flow_table_free() has destroyed the hashtable, which is true for the
current mainline destroy path. Removing one device from an existing
flowtable needs the same care; I have not tested that case.
Reproduction (control vs. bug, same setup, only the teardown order differs)
--------------------------------------------------------------------------
Minimal ruleset:
table inet leak {
flowtable ft {
hook ingress priority 0
devices = { "<uplink>.2", "<uplink>.3" }
flags offload
counter
}
chain forward {
type filter hook forward priority 0; policy accept;
meta l4proto tcp ct state established flow add @ft
}
}
Control: "nft flush chain inet leak forward" (stop new offloads),
"conntrack -D" for the offloaded connection, wait 4 s, then
"nft flush ruleset": mlx5e_delete_flower = 2, the kernel forwards again
(rx_packets == rx_packets_phy), eswitch mode change works.
Bug: "nft flush ruleset" directly: mlx5e_delete_flower = 0, the NIC keeps
forwarding, eswitch busy.
Steps for the bug case:
1. Load the ruleset above; start a TCP transfer between the two VLANs and
wait until it is [HW_OFFLOAD] with rx_packets << rx_packets_phy.
2. nft flush ruleset
3. The transfer keeps running with rx_packets << rx_packets_phy. The
conntrack entry is [ASSURED] with frozen counters. The flow rules are
still in the mlx5 steering dump. devlink eswitch mode change -> EBUSY.
Thanks,
Mohammed Zuhair, Earthlink Telecommunications