Re: [REGRESSION][BISECTED] r8169: TX stall with checksum offload on VLAN frames with inline tag (REORDER_HDR off) since 1517d1996b52

On 10/6/26 13:45, Jan Čermák wrote:

the Home Assistant OS recently shipped kernel update from 6.18.39 to
6.18.52 which was followed by a bunch of community reports about fatal
network breakages [1] which had one thing in common - r8169 driver and
configured VLANs. This manifested as network stall early in the setup
of the system with dmesg events like this:

quoted

r8169 0000:03:00.0 enp3s0: NETDEV WATCHDOG: CPU: 7: transmit queue 0 timed out 6125 ms
r8169 0000:03:00.0 enp3s0: rtl_rxtx_empty_cond == 0 (loop: 42, delay: 100).

As there was no change in the r8169 driver itself, I focused on the
vlan subsystem, where AI pointed me shortly to this suspect commit in
the range: 1517d1996b5236fe69eccd9d253f725e06996eb1 ("vlan: fix
skb_under_panic and races when toggling HW VLAN offload"), added in
6.18.51. I'm not referring to the mainline counterpart below for
regzbot (447cbe95ebb9) as I can't confirm whether it triggers the bug
there as well. FWIW there was another issue [2] with a similar
combination recently in the regressions ML but that one seems
unrelated, as disabling EEE fixed that issue, and it doesn't fix it
here.

Not my area of expertise, so take the following carefully, as it might
send you in the wrong direction:

There is another issue known for 447cbe95ebb9:
Commit 447cbe95ebb9 causes IOMMU DMA faults on macvlan/vlan with bnxt_en
https://lore.kernel.org/all/20261004122616.56714cbd@nargothrond/ (local)

A fix for it is under review:
bnxt_en: fix DMA mapping length for padded small packets
https://lore.kernel.org/all/20261006042153.199444-1-edumazet@kernel.org/ (local)

Is that maybe related?

Ciao, Thorsten

I don't have the hardware myself, I asked for testing with this commit
reverted, which confirmed that this change indeed started to cause
trouble. However, it's obvious that the change itself is not bad, as
it fixes another issue and the regression is scoped to very specific
hardware/setup combo.

Besides the revert, following workarounds were reported to fix the
issue as well:
- Disabling TX checksum offload with `ethtool.feature-tx off` in
NetworkManager for the connection (HAOS doesn't have ethtool binary,
hence this option instead of direct ethtool command)
- Enabling REORDER_HDR with `ip link set enp1s0.100 type vlan reorder_hdr on`

Clearly, this needs rather esoteric setup to trigger the bug - I guess
most systems set the REORDER_HDR flag by default, however, due to some
legacy in the DBus interface that HAOS stack uses to configure the
network [3], it was disabled. Anyway, I think that disabling it should
not lead to driver breakage as we're seeing.

So far it appears that this only affects RTL8168h/8111h, XID 541 -
there isn't any report of another chip/XID combination yet.
Unfortunately, as I said above, I don't have the hardware available
for testing but I believe I'll find some community members who'll be
willing to test a proposed fix for the issue if needed.

[1] https://github.com/home-assistant/operating-system/issues/5019
[2] https://lore.kernel.org/regressions/353419280.954808.1790290283530@mail.yahoo.com/ (local)
[3] https://github.com/home-assistant/supervisor/issues/7248

#regzbot introduced: 1517d1996b5236fe69eccd9d253f725e06996eb1
#regzbot link: https://github.com/home-assistant/operating-system/issues/5019

Cheers,
Jan