Re: Problem with IPv6 privacy addresses in 7.0

On 5/28/26 1:12 PM, Stefano Brivio wrote:

On Thu, 28 May 2026 12:46:05 +0200
Fernando Fernandez Mancera [off-list ref] wrote:

quoted

On 5/28/26 7:38 AM, Stefano Brivio wrote:

quoted

Hi Fernando, Jakub,

On Wed, 27 May 2026 23:59:47 +0200
Fernando Fernandez Mancera [off-list ref] wrote:
   

quoted

On 5/27/26 11:51 PM, Chris Adams wrote:

quoted

Once upon a time, Jakub Kicinski [off-list ref] said:

quoted

On Tue, 26 May 2026 20:06:41 -0500 Chris Adams wrote:

quoted

quoted

Hi! Adding more people to CC. Do you know if you upgraded from 6.18
or 6.19?

It was 6.19 to 7.0.
     

quoted

Would you be able to try testing with some commits reverted?
On a quick look the candidates would be:

cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")

It's this one.

Phew, the second one was mine :)
     

quoted

I figured out that it happens after stopping a VM (and I usually
start/stop a VM for a bit in the morning, which is why it happened more
than once).  So I set up a VM with a nested VM, running up-to-date
Fedora 44, and then was able to bisect pretty easily, and it landed on
this commit.

Fedora is using NetworkManager, and IIRC NM does some part of privacy
address management (right?).  NM didn't change, so maybe this commit is
confusing something in NM?

Sounds plausible, pretty sure we knew this commit was risky to begin
with, but we had no direct proof that it'd break real life users.

Revert is the right course of action here. Would you be willing/able
to send the revert with your problem description and a Fixes tag
pointing to the reverted commit?

I did want to add a little more test note:

It's definitely an interaction with NetworkManager.  If I stop NM and
run my VM start/stop test, nothing unexpected happens after.  If NM is
running and I do my VM test, when the next router advertisement is
received, NM replaces the privacy addresses.
      

As someone that is experienced in NetworkManager, I can confirm it is
related. NetworkManager is querying the IPv6 address and when the
connection is configured with ipv6.ip6-privacy=2 (prefer-temp-addr),
NetworkManager creates a route to make the system use the temporary
address for outgoing connections by default.

If the order is messed up, the address picked will likely be too. One
could argue that this is partially fault of NetworkManager and that it
should check the timestamps or preferred times rather than order.. but
well, the rule is "do not break userspace".

I hope this clarifies things.

Not entirely. I'm looking into this right now, but note that the purpose
of that commit is to *preserve* the order of addresses as they were
inserted, not to mess it up.

Before that, addresses were stored and returned via netlink *reversed*
compared to the insertion, which is rather surprising, also because
it's the opposite of what we do for IPv4 addresses, and caused issues
with pasta(1) as it copies addresses into containers in the same order
as reported via netlink.

Right, to make it clear. I agree with you here. The current order is the
*right* or rather intuitive order.

quoted

Do you happen to know exactly in which way the order happens to be wrong
now?

It is not wrong, but userspace applications were parsing addresses and
expecting them in the previous order whether it was correct or not. Now,
that parsing is broken because they are coming in an unexpected (maybe
more intuitive) order.

Ouch, I see.

Two questions:

- could there be other issues coming from the fact that NetworkManager
   relies on a particular order (whether right or wrong) of addresses? If
   yes, I guess NetworkManager would benefit from a fix in any case
   (that is, we broke it a bit harder, but maybe it was already broken?)

Yes, I believe the right way to pick a temporary address is to use the most recent one and the right way to see that is by looking to timestamps not looking at the order.

Ugh, that is a mess.

   Now, I suppose NetworkManager is much more universally used and it's
   definitely a more mature project so it's a bigger userspace breakage,
   but at the same time it's a ton of containers (Podman uses pasta by
   default, Docker/moby optionally via rootlesskit) we might introduce a
   regression for.

I think the main argument here is: while we know it is affecting NetworkManager, we cannot know if it is breaking something else. The kernel rule is usually "do not break userspace" and this is likely considered a regression from my understanding. Cc'ing Thorsten as he is an expert on regressions.

So I'm wondering if in practice it wouldn't be better to fix
NetworkManager instead.

We can fix future versions of NetworkManager but we cannot fix released versions. That is, if someone upgrade kernel but not NetworkManager, it would break. I guess the same applies to pasta.. so we must understand what has been the most prevalent behavior.

quoted

FWIW; Iñigo Huguet (CC'ed) pointed out that this is breaking a
NetworkManager test and they were going to report it.

quoted

Also note that userspace was broken and fixed a couple of times:

    https://bugs.passt.top/show_bug.cgi?id=175#c10

so I would look into fixing that properly if doable. If it takes too
long and this is causing issues for a lot of people meanwhile of course
it might make sense to revert, but I'd give it a quick try at least.

I think the solution shouldn't modify the order of the list by default.
Maybe we can introduce a flag for the dump operation?

I'm a bit struggling with the name and how complicated a description
would be. NLM_F_RIGHT_ORDER? I guess you're suggesting to get the
"wrong" / reversed order by default... even though it's the opposite
compared to IPv4? Or NLM_F_REVERSED, otherwise? Is it really worth it?

I don't have a clear idea on how to do it so it is acceptable, right now. FWIW; I think consistent behavior between IPv4 and IPv6 isn't a goal. They are designed in different ways and implemented in different ways too. There are many situations where they behave completely different e.g ECMP routing.

I have sent the revert patch but more likely for AI review + CI. I do not want it to be merged until we reach a consensus, we can discard it if we decide to do so.

Thanks,
Fernando.