Re: [Cake] [PATCH net-next] net/sched: fq_codel, cake: widen backlogs to u64

On Fri, Sep 25, 2026 at 10:03 AM 'Sebastian Moeller' via
Hyper-Yielding Back-end Review & Insight System [off-list ref]
wrote:

Hi Eric,

quoted

On Sep 25, 2026, at 15:43, Eric Dumazet [off-list ref] wrote:

On Fri, Sep 25, 2026 at 3:30 PM Sebastian Moeller [off-list ref] wrote:

quoted

Hi Eric,

quoted

On Sep 25, 2026, at 15:03, Eric Dumazet via Cake [off-list ref] wrote:

On Fri, Sep 25, 2026 at 1:28 PM Jamal Hadi Salim [off-list ref] wrote:

quoted

On Fri, Sep 25, 2026 at 5:13 AM Eric Dumazet [off-list ref] wrote:

quoted

On Fri, Sep 25, 2026 at 10:54 AM Jamal Hadi Salim [off-list ref] wrote:

quoted

This is a follow-up to commit 8f735d64382d ("net/sched: bound
qdisc_pkt_len to prevent qdisc soft lockup"), which capped
qdisc_pkt_len() at QDISC_PKT_LEN_MAX (1 MiB). That cap bounds the stab
amplifier but leaves the per-flow backlog counter u32:
fq_codel_enqueue() accumulates qdisc_pkt_len(skb) into q->backlogs[idx],
so a flow can still accumulate 4096 packets of 1 MiB each and wrap the
counter mod 2^32. After a wrap, fq_codel_drop() sees a tiny maxbacklog
and drops from an almost-empty flow, and the dequeue-side subtractions
corrupt the counter further.

Widen the fq_codel backlogs table, the fat-flow scan (maxbacklog/len) and
the drop threshold to u64. fq_codel is not lockless: every writer runs
under the root qdisc lock, so plain u64 arithmetic keeps the WRITE_ONCE
publish / READ_ONCE-consume pattern. The dump path
(fq_codel_dump_class_stats) stays a lockless stat-only read.

CAKE accumulates the same generic qdisc_pkt_len(skb) into its per-flow
b->backlogs[] and per-tin b->tin_backlog and consumes the values for
longest-flow pruning (cake_heapify/cake_heapify_up) and for the shaper
staleness check, so it shares the bug. Widen those counters and the heap
comparison locals to u64; the class/tin stats keep exporting the low 32
bits through the unchanged uAPI fields.

Conditions to recreate the bug: CAP_NET_ADMIN in a user namespace;
CONFIG_NET_SCH_FQ_CODEL=y.

ip tuntap add tun0 mode tun
ip link set tun0 txqueuelen 32 up
ip addr add 10.99.0.1/24 dev tun0
tc qdisc add dev tun0 root handle 1: stab overhead 2000000000 \
    fq_codel flows 1 limit 4200 ecn drop_batch 4096
# hold the tun fd open without reading (IFF_BACKPRESSURE) so the qdisc
# backlog persists, then send at least 4300 packets (the wrap starts
# at 4096 resident; the over-limit drop that reads the wrapped
# threshold fires past the 4200 limit): backlogs[0] wraps at 4096 x
# 1 MiB and the fat-flow threshold reads the wrapped value.

With a 1 MiB qdisc_pkt_len cap the counter wraps at 4096 resident
packets. At limit 4200 the first over-limit enqueue (the 4201st) sees a
wrapped 105 MiB (half-backlog threshold 52 MiB, a ~52 packet drop burst),
where the u64 counter sees 4201 MiB (threshold 2100 MiB, a ~2100 packet
drop burst).

Reported-by: Sashiko (nipa) <sashiko-bot@kernel.org>
Closes: https://netdev-ai.bots.linux.dev/sashiko/#/patchset/20260818101130.16203-1-jhs@mojatatu.com
Link: https://lore.kernel.org/netdev/20260818101130.16203-1-jhs@mojatatu.com/ (local)
Tested-by: hybris <redacted>
Signed-off-by: Jamal Hadi Salim <jhs@mojatatu.com>
---

This makes no sense.

As absurd as it looks that code is reachable ;->

quoted

These qdisc have been developped to address bufferbloat issues.

Storing 4GB in a qdisc is absolutely insane.

The counter wrap is not because we stored 4GB, rather it is because
"tc .. stab overhead ..." inflates qdisc_pkt_len() for a 64B pkt to
1MB. So ~4K packets (put in other words a few "real" KB) makes that
backlog[0] cross 2^32.

Kill this stuff ? Who is still using this, for what reason ?

tc-stab? Same as before, if you need want to model a remote bottleneck with a local traffic shaper tc-stab is the generic solution (off the top of my head I only can enumerate cake as having its own traffic shaper that handles overheads).
One could argue that the "virtual" length should be accounted against cake's memlimit or fq-codel's memory_limit somehow...
In sane?/typical configurations overhead is expected to stay relatively small, so this would not limit the actual queue size too much, while potentially silencing this issue.

linux qdisc are in the fast path, for nearly all packets sent over this planet.

Tip of the head to the linux network experts that enabled that and that take care of it being efficient!

quoted

They aleady consume GW of energy.

Modeling / network emulation should incur zero cost on these
production grade qdisc.

It is not modeling in a scientific sense (sorry I used inappropriate terminology here), but simply the fact that to shape traffic to avoid filling remote queues one needs to be able to take the properties of that remote queue/interface/link-layer into account.
And that can be surprisingly close by. Last time I looked the kernel on say eth0 added the 14 bytes of ethernet related overhead it handled itself to the packet size on top of the payload (no complaints that makes sense) but that is insufficient to properly traffic shape e.g. the same ethernet interface... (as the relevant ethernet L1-frame overhead contains more bytes, like the FCS, preamble and SFD). And that gets worse with stuff like DSL, cable or fibre modems that are connected via ethernet to a linux router.

quoted

netem could be one answer, I do not know, or a special
CONFIG_NET_SCHED_EXPENSIVE_EMULATION

Well, tc-stab (or cake's overhead accounting) is already optional, so only those incur the cost that actually use it (and only those users are affected by the reported issue, it took tc-stab to cause the problem and arguably in a configuration that is "insane").
I might be trying to explain the internet here to people that make the internet work, so apologies in advance but I want to make this as explicit as I can.
TTraffic shaping without proper overhead accounting will not work as expected (at least if the expectation is that the traffic shaper will honor the set gross shaper rate). For a fixed packet size the lack over overhead accounting can be papered over with reducing the shaper rate, but the amount of the required "over-shaping" depends on packet size (smaller packets require more over-shaping), so generally that is sub optimal, because either the shaper's guarantee is brittle or the over-shaping quite extreme (think the header to payload ration of minimally- and MTU-sized packets).

All I am saying is, we do still have use cases for tc-stab and proper overhead accounting, at least on the leafs of the network like home internet links.

I am working on a v2 that drops at enqueue once the accounted backlog
crosses  (U32_MAX - QDISC_PKT_LEN_MAX), so no counter can wrap.
That is Eric's original suggestion which preserves the point you raised.

While looking at this closely for this update i noticed more qdiscs
which for different reasons also suffer from ((u64)backlog + len <=
limit idiom.
So far it's clear from bfifo, gred and plug are the only ones safe
because their limit is in bytes (as opposed to others which are packet
counting)

Eric: Unless i hear otherwise from you, the configurable byte ceiling
you mentioned (a good noun seems to be CONFIG_NET_SCH_BACKLOG_CEILING)
is a follow-up;

cheers,
jamal

quoted

quoted

I might be off my rocker, in which has ignore (or preferably enlighten me).