net: airoha: PPE SRAM stale L2 entries survive TC flower destroy on station roam

Hi,

I spent the last few days debugging a WiFi throughput regression on an
Airoha AN7581 router (Quantum Fiber W1700K, OpenWrt, acting as a dumb AP
with bridger [1] doing TC flower-based L2 hardware offload). The symptom
is that when a WiFi station switches from one radio to another, throughput
drops to ~1 Mbit/s until something nudges the PPE out of its stale state.

I want to share what I found in case it helps identify a driver or firmware
issue, and ask a few questions for the Airoha folks.

The setup: bridger installs skip_sw TC flower filters on bridge member
interfaces to push L2 forwarding into the PPE.  For a station on phy0.2-ap0:

    phy0.2-ap0 ingress: src=STATION dst=GW  ->  redirect lan2
    lan2 ingress:       src=GW dst=STATION  ->  redirect phy0.2-ap0

When the station moves to phy0.1-ap0, bridger tears down the old filters
(RTM_DELTFILTER) and installs new ones (RTM_NEWTFILTER) for phy0.1-ap0.
The kernel and driver both report success. But the PPE keeps sending
downlink traffic to phy0.2-ap0, where the station is no longer associated,
so every frame is dropped. iw dev phy0.1-ap0 station get a few minutes
after the band switch:

    tx retries: 3429
    tx failed:  3435
    rx bitrate: 6.0 MBit/s (link rate degraded by retransmission backoff)

Meanwhile tc filter show lan2 ingress correctly shows redirect to phy0.1-ap0.
The kernel's view of the world is right; the PPE's isn't.

Bridger improvements along the way
==================================

During the investigation I made several improvements to bridger - better
handling of port mismatches, FDB migration ordering, and the DELNEIGH/NEWNEIGH
race during band switch - all in main-bridger-v4 [2]. These helped narrow
down the problem but in the end were not enough to fully resolve the issue on
their own, which is what led us to the PPE SRAM behaviour described below.

Root issue - PPE SRAM consistency
=================================

Even after the bridger fixes, there's a residual window where the PPE SRAM
has the old redirect in place. The sequence from the kernel's side looks
correct:

    RTM_DELTFILTER  ->  flow_offload_destroy()  ->  "flow_table entry
removed OK"
    RTM_NEWTFILTER  ->  flow_offload_add()      ->  success

But the hardware continues forwarding on the old SRAM entry.

While adding debug logging to bridger's nl.c to trace every RTM_DELTFILTER
and its outcome (patch c1a5f49 in the same branch), I observed two things
that suggest the SRAM management is not entirely consistent.

First, RTM_DELTFILTER frequently returns ENOENT even for filters that were
successfully installed earlier:

    flow offload del on ifindex 8 handle 0xf521a056: FAILED kernel error -2

The reason is that the PPE firmware proactively evicts idle flows from SRAM
on its own schedule, notifying the driver via flow_offload_destroy(), which
causes the kernel to remove the TC flower filter before bridger's idle timer
fires. So bridger's delete arrives late, after the filter is already gone.
In isolation this is harmless - ENOENT just means the hardware already did the
cleanup - but it does mean there's a window between the PPE evicting the entry
and bridger learning about it where the state is unclear.

Second, dmesg occasionally shows the same cookie receiving multiple DESTROY
notifications in rapid succession:

    airoha_eth: PPE: TC DESTROY cookie ffffff8017ee1000: flow_table
entry removed OK (TC filter will be deleted)
    airoha_eth: PPE: TC DESTROY cookie ffffff8017ee1000 NOT in
flow_table (already removed or never offloaded)
    airoha_eth: PPE: TC DESTROY cookie ffffff8017ee1000 NOT in
flow_table (already removed or never offloaded)

Same cookie, three notifications within milliseconds. The first removes the
entry cleanly; the next two find nothing. This looks like the NPU firmware
can fire DESTROY more than once for the same SRAM slot, possibly because the
invalidation commit to the hardware is not atomic from the firmware's
perspective.

The practical consequence: when bridger installs a new TC flower for the same
MAC pair on a different port (RTM_NEWTFILTER for phy0.1-ap0 after
RTM_DELTFILTER for phy0.2-ap0), there is apparently a window where the old
SRAM entry is not yet fully invalidated. If the new flow_offload_add() runs
into that window, the new SRAM entry either doesn't land or coexists with the
old one in a way that lets the hardware continue using the old redirect.

Workaround
==========

I tried a kernel-side workaround [3]: register a switchdev notifier for
SWITCHDEV_FDB_ADD_TO_DEVICE / SWITCHDEV_FDB_DEL_TO_DEVICE and, on each
event, walk the driver's flow_table and l2_flows and flush all PPE entries
matching the affected MAC.

Two things that flush handles:

  - Removing the flow_table entry makes airoha_ppe_flow_offload_destroy()
    return -ENOENT on RTM_DELTFILTER, which causes the TC layer to remove
    the software filter state cleanly.

  - Removing the l2_flows entry prevents airoha_ppe_foe_insert_entry() from
    re-learning the stale redirect direction after the BIND entry ages out.
    Without this, even after the SRAM entry expires, the old l2_flows parent
    is still around and the PPE re-establishes hardware offload in the wrong
    direction the next time traffic flows.

This fixed the symptom completely. I held off upstreaming it because it
felt like compensating for something the driver or firmware should handle
correctly - if RTM_DELTFILTER reliably invalidated the SRAM entry and
l2_flows were tied to the lifetime of the station's bridge port association,
the switchdev flush wouldn't be needed.

Questions for Airoha
====================

Is the NPU firmware's SRAM invalidation atomic? When flow_offload_destroy()
returns, can the driver guarantee the hardware has stopped using that entry,
or is there a commit pipeline that takes additional time?

Is it expected that the firmware can send multiple DESTROY notifications for
the same slot? The double-NOT-in-flow_table pattern in dmesg suggests so.

When flow_offload_add() is called for a MAC pair that may still have a
partially-invalidated SRAM entry from a previous flow, does the NPU guarantee
the new entry takes precedence immediately? Or is there a recommended
sequence (e.g. explicit invalidate, barrier, then add)?

Is there a flush or barrier call in the NPU firmware interface that the driver
could expose, so a switchdev handler or similar could ensure SRAM consistency
without needing to walk the entire flow_table?

Happy to run more tests, capture more dmesg, or share the full debug log
from bridger if it would help.

Regards,
Dan


Some info:
Kernel: 6.18.37, arm64
Platform: Airoha AN7581 (tested on Quantum Fiber W1700K)

[1] https://github.com/nbd168/bridger
[2] https://github.com/danpawlik/bridger/commits/main-bridger-v4
[3] Local branch -
https://github.com/danpawlik/openwrt/tree/airoha-npu-kernel-mailinglist
    commit: https://github.com/openwrt/openwrt/commit/7373faf11f4ad14a3cb16f33e51fbc4dccc85ca4