Re: [PATCH net-next v3 2/3] ipv6: add ip6_dst_lookup_flow_cookie

From
Ralf Lici <ralf@mandelbit.com>
Date
2026-11-18 14:39:28
Also in
lkml
Reply to
Xia Yan, 2026-11-18
Hi Xia,

On Wed, 18 Nov 2026 16:48:34 +0800, Xia Yan [off-list ref] wrote:
I traced the race and the fix against current net-next.

On the writer side, xfrm_policy_insert() links the policy and then calls
rt_genid_bump_ipv6(), which resolves to fib6_flush_trees(). A fresh serial
is allocated from the namespace counter first, then every node is
restamped under tb6_lock. A late rt6_get_cookie() can pick up that
post-insertion serial for a dst whose lookup ran before the insertion,
and the pair keeps passing dst_check until the next flush. Sampling the
cookie between two generation reads closes the in-lookup window. A
policy inserted after the helper returns is still caught later, because
the restamp changes fn_sernum itself and the stored pre-insertion cookie
stops matching. fib6_new_sernum() never returns 0, so the fallback
cookie always fails fib6_check().
Yes, that's a pretty good summary of the change with the small
correction that a newly linked node holds fn_sernum 0 until fib6_add
stamps it. So a cookie-0 dst can pass a check in that window, and fails
as soon as the stamp lands.
1/3 checks out as well. The old code kept writing -1 into the node, so a
second failure changed nothing and the namespace generation never moved.
The fresh allocation under tb6_lock matches the discipline the
fib6_ifup()/fib6_ifdown() paths already get from __fib6_clean_all().
The conversion boundary in 3/3 also matches the tree: rawv6 does not
cache the dst, and the accepted-socket and refill stores in tcp_ipv6.c
are listed as follow-up.
In fact there are many users of cached IPv6 dsts that will need
conversion and this series is mainly here to introduce the new API.
One thing I could not fully work out is the ordering behind the
generation checks. When the lookup misses a change, the reader relies on
the two generation reads to notice it. My understanding is that this
needs the change visible before the counter advance, and the counter
advance visible before the node restamps. I could not establish either
hop, though I may well be missing something.

In xfrm_policy_insert(), the policy is linked into the hash and
rt_genid_bump_ipv6() runs within the same critical section. As far as I
can tell, the linking stores and the cmpxchg on fib6_sernum are all
relaxed, with no barrier between them. Could a reader read the advanced
generation in gen0, miss the policy during the lookup, sample a
restamped fn_sernum, and still read the same generation in gen1?
Both hops are covered by fib6_new_sernum. It allocates the serial with
atomic_try_cmpxchg, which is fully ordered when it succeeds, so it acts
as an smp_mb before and after the update of fib6_sernum.

For the first hop, the policy (or route) is linked before
fib6_new_sernum runs, which gives message passing against the smp_rmb
after gen0: a reader that reads the advanced generation in gen0 also
finds the policy.
In fib6_flush_trees(), the serial is allocated before __fib6_clean_all()
takes tb6_lock, and fib6_clean_node() stamps the nodes with a plain
WRITE_ONCE(). spin_lock() is an ACQUIRE, and my reading of
memory-barriers.txt is that operations issued before an ACQUIRE may
complete after it. So a reader could sample the restamped fn_sernum
For the second hop, every fn_sernum write follows a successful
fib6_new_sernum on the same CPU: fib6_update_sernum,
__fib6_update_sernum_upto_root, and fib6_clean_node in the flush case.
The full barrier orders the counter update before the tb6_lock
acquisition and the stamps, so the ACQUIRE semantics of spin_lock don't
matter here. A reader that samples a restamped fn_sernum reads the
advanced generation in gen1 after the smp_rmb, and the helper returns
cookie 0.
between two reads that both still return the old generation. The only
pairing I found written down here is fib6_get_cookie_safe()'s smp_rmb()
against the smp_wmb() in __fib6_update_sernum_upto_root().
Right, this isn't documented. The barrier comments in
ip6_dst_lookup_flow_cookie should name fib6_new_sernum as the pairing
side, and fib6_new_sernum should state that its full ordering is relied
upon.

I also checked both hops with an LKMM litmus test: the outcome you
describe is forbidden with atomic_cmpxchg and becomes reachable with
atomic_cmpxchg_relaxed.

While going through the writers I also noticed that xfrm_policy_insert
unlinks a replaced policy only after rt_genid_bump_ipv6, so a lookup
after the bump can still match the old policy. That is a writer-side
ordering issue independent of this series, and it affects the current
late cookie sampling in the same way. I'll send a separate fix.

Does this explanation address your concern? If so, I'll carry your
Reviewed-by. I'm waiting for the Sashiko review of v3: if it reports
anything that needs a respin, I'll add the barrier comments in v4,
otherwise I'll send them as a follow-up.

Thanks for the careful review!

-- 
Ralf Lici
Mandelbit Srl
Thread (14 messages) 14 messages, 4 authors, last reply 1d ago
Keyboard shortcuts
hback out one level
jnext message in thread / next series-diff position
kprevious message in thread / previous series-diff position
ldrill in
Escclose help / fold thread tree
?toggle this help