From: Fernando Fernandez Mancera <hidden> Date: 2026-05-29 11:24:11
Chris Adams reported that preserving insertion order for same-scope
addresses is causing SSH connections to be dropped after stopping a VM
while running NetworkManager.
NetworkManager caches the IPv6 address configuration, when a RA arrives,
it determines the list of addresses to configure and checks if the
addresses are already in the right order in the kernel. If they aren't,
NetworkManager removes and re-adds them to achieve the desired order.
As the order changes, NetworkManager is confused and reconfigures the
addresses on every update. In addition, this would also affect to cloud
tooling that relies on IPv6 addresses order to identify primary and
secondaries addresses.
This reverts commit cb3de96eea66f5e4a580086c6a1be46e765f97f4.
Fixes: cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")
Reported-by: Chris Adams <redacted>
Closes: https://lore.kernel.org/netdev/20260521135310.GC977@cmadams.net/
Signed-off-by: Fernando Fernandez Mancera <redacted>
---
v2: updated commit description to make it more accurate
---
net/ipv6/addrconf.c | 2 +-
tools/testing/selftests/net/ioam6.sh | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
On Fri, 29 May 2026 13:23:57 +0200
Fernando Fernandez Mancera [off-list ref] wrote:
Chris Adams reported that preserving insertion order for same-scope
addresses is causing SSH connections to be dropped after stopping a VM
while running NetworkManager.
NetworkManager caches the IPv6 address configuration, when a RA arrives,
it determines the list of addresses to configure and checks if the
addresses are already in the right order in the kernel. If they aren't,
NetworkManager removes and re-adds them to achieve the desired order.
As the order changes, NetworkManager is confused and reconfigures the
addresses on every update. In addition, this would also affect to cloud
tooling that relies on IPv6 addresses order to identify primary and
secondaries addresses.
By the way, I'm still looking into this part, trying to find
"problematic" examples.
And I couldn't find any, yet, because it looks like there's always a
_single_ IPv6 address being used as a secondary for a primary IPv4
address.
quoted hunk
This reverts commit cb3de96eea66f5e4a580086c6a1be46e765f97f4.
Fixes: cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")
Reported-by: Chris Adams <redacted>
Closes: https://lore.kernel.org/netdev/20260521135310.GC977@cmadams.net/
Signed-off-by: Fernando Fernandez Mancera <redacted>
---
v2: updated commit description to make it more accurate
---
net/ipv6/addrconf.c | 2 +-
tools/testing/selftests/net/ioam6.sh | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
From: Fernando Fernandez Mancera <hidden> Date: 2026-05-29 11:45:43
On 5/29/26 1:41 PM, Stefano Brivio wrote:
On Fri, 29 May 2026 13:23:57 +0200
Fernando Fernandez Mancera [off-list ref] wrote:
quoted
Chris Adams reported that preserving insertion order for same-scope
addresses is causing SSH connections to be dropped after stopping a VM
while running NetworkManager.
NetworkManager caches the IPv6 address configuration, when a RA arrives,
it determines the list of addresses to configure and checks if the
addresses are already in the right order in the kernel. If they aren't,
NetworkManager removes and re-adds them to achieve the desired order.
As the order changes, NetworkManager is confused and reconfigures the
addresses on every update. In addition, this would also affect to cloud
tooling that relies on IPv6 addresses order to identify primary and
secondaries addresses.
By the way, I'm still looking into this part, trying to find
"problematic" examples.
And I couldn't find any, yet, because it looks like there's always a
_single_ IPv6 address being used as a secondary for a primary IPv4
address.
IIRC, Azure cloud should be one of them. I do not have an account there
to test it.. but some years ago I did some work supporting configuring
IPV6 primary and secondary addresses via IMDSv2.
If someone with an account could test it I would appreciate it.
quoted
This reverts commit cb3de96eea66f5e4a580086c6a1be46e765f97f4.
Fixes: cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")
Reported-by: Chris Adams <redacted>
Closes: https://lore.kernel.org/netdev/20260521135310.GC977@cmadams.net/
Signed-off-by: Fernando Fernandez Mancera <redacted>
---
v2: updated commit description to make it more accurate
---
net/ipv6/addrconf.c | 2 +-
tools/testing/selftests/net/ioam6.sh | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
From: Chris Adams <hidden> Date: 2026-05-29 12:06:50
Once upon a time, Stefano Brivio [off-list ref] said:
On Fri, 29 May 2026 13:23:57 +0200
Fernando Fernandez Mancera [off-list ref] wrote:
quoted
Chris Adams reported that preserving insertion order for same-scope
addresses is causing SSH connections to be dropped after stopping a VM
while running NetworkManager.
NetworkManager caches the IPv6 address configuration, when a RA arrives,
it determines the list of addresses to configure and checks if the
addresses are already in the right order in the kernel. If they aren't,
NetworkManager removes and re-adds them to achieve the desired order.
As the order changes, NetworkManager is confused and reconfigures the
addresses on every update. In addition, this would also affect to cloud
tooling that relies on IPv6 addresses order to identify primary and
secondaries addresses.
By the way, I'm still looking into this part, trying to find
"problematic" examples.
And I couldn't find any, yet, because it looks like there's always a
_single_ IPv6 address being used as a secondary for a primary IPv4
address.
I hit two cases that triggered the behavior:
- after an overnight suspend/resume cycle (not sure if there's a minimum
time suspended)
- with a bridge NIC, after stopping a VM with a NIC on the bridge (which
removed an fe80:: link-local address); this is the case I used to
bisect since it was easy to script
In both cases, the next IPv6 router-advertisement received was what
triggered NetworkManager to replace the current privacy addresses. This
could be 2-3 minutes after the trigger.
quoted
This reverts commit cb3de96eea66f5e4a580086c6a1be46e765f97f4.
Fixes: cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")
Reported-by: Chris Adams <redacted>
Closes: https://lore.kernel.org/netdev/20260521135310.GC977@cmadams.net/
Signed-off-by: Fernando Fernandez Mancera <redacted>
---
v2: updated commit description to make it more accurate
---
net/ipv6/addrconf.c | 2 +-
tools/testing/selftests/net/ioam6.sh | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
Hello:
This patch was applied to netdev/net.git (main)
by Jakub Kicinski [off-list ref]:
On Fri, 29 May 2026 13:23:57 +0200 you wrote:
Chris Adams reported that preserving insertion order for same-scope
addresses is causing SSH connections to be dropped after stopping a VM
while running NetworkManager.
NetworkManager caches the IPv6 address configuration, when a RA arrives,
it determines the list of addresses to configure and checks if the
addresses are already in the right order in the kernel. If they aren't,
NetworkManager removes and re-adds them to achieve the desired order.
[...]
Hi Stefano,
On 29/05/2026 21:41, Stefano Brivio wrote:
On Fri, 29 May 2026 13:23:57 +0200
Fernando Fernandez Mancera [off-list ref] wrote:
quoted
Chris Adams reported that preserving insertion order for same-scope
addresses is causing SSH connections to be dropped after stopping a VM
while running NetworkManager.
NetworkManager caches the IPv6 address configuration, when a RA arrives,
it determines the list of addresses to configure and checks if the
addresses are already in the right order in the kernel. If they aren't,
NetworkManager removes and re-adds them to achieve the desired order.
As the order changes, NetworkManager is confused and reconfigures the
addresses on every update. In addition, this would also affect to cloud
tooling that relies on IPv6 addresses order to identify primary and
secondaries addresses.
By the way, I'm still looking into this part, trying to find
"problematic" examples.
And I couldn't find any, yet, because it looks like there's always a
_single_ IPv6 address being used as a secondary for a primary IPv4
address.
FYI, the order change also affected some specific scripts, e.g. here
with MPTCP and packetdrill:
https://github.com/multipath-tcp/packetdrill/commit/1b7cd4482ce8
Because the order was not "natural" before, and different from IPv4, a
workaround was needed to keep the same order. I was happy to remove it,
but now it looks like I need to re-apply it :)
It would be nice to get the "natural" order back without breaking the
userspace (or with a way to choose the order).
Cheers,
Matt
--
Sponsored by the NGI0 Core fund.
On Mon, 1 Jun 2026 12:03:59 +1000
Matthieu Baerts [off-list ref] wrote:
Hi Stefano,
On 29/05/2026 21:41, Stefano Brivio wrote:
quoted
On Fri, 29 May 2026 13:23:57 +0200
Fernando Fernandez Mancera [off-list ref] wrote:
quoted
Chris Adams reported that preserving insertion order for same-scope
addresses is causing SSH connections to be dropped after stopping a VM
while running NetworkManager.
NetworkManager caches the IPv6 address configuration, when a RA arrives,
it determines the list of addresses to configure and checks if the
addresses are already in the right order in the kernel. If they aren't,
NetworkManager removes and re-adds them to achieve the desired order.
As the order changes, NetworkManager is confused and reconfigures the
addresses on every update. In addition, this would also affect to cloud
tooling that relies on IPv6 addresses order to identify primary and
secondaries addresses.
By the way, I'm still looking into this part, trying to find
"problematic" examples.
And I couldn't find any, yet, because it looks like there's always a
_single_ IPv6 address being used as a secondary for a primary IPv4
address.
FYI, the order change also affected some specific scripts, e.g. here
with MPTCP and packetdrill:
https://github.com/multipath-tcp/packetdrill/commit/1b7cd4482ce8
Because the order was not "natural" before, and different from IPv4, a
workaround was needed to keep the same order. I was happy to remove it,
but now it looks like I need to re-apply it :)
Ouch. :) We were are also pondering about some kind of workaround like
that (https://bugs.passt.top/show_bug.cgi?id=175#c9) but fixing the
kernel looked simpler and the right thing to do... until last week.
It would be nice to get the "natural" order back without breaking the
userspace (or with a way to choose the order).
I was thinking that if we implement a label like NLM_F_INSERT_LAST
(David's proposal for the name), we could patch iproute2 to set it on
'ip address restore' at least, other than using it in pasta(1).
But that wouldn't be enough for your case. At the same time always
adding it for RTM_NEWADDR requests in iproute2 could break somebody
else's scripts. I guess a reasonable solution could be to add an
additional parameter for ip-address... 'insert_last'? 'last'?
--
Stefano
On Mon, Jun 1, 2026 at 3:35 PM Stefano Brivio [off-list ref] wrote:
I was thinking that if we implement a label like NLM_F_INSERT_LAST
(David's proposal for the name), we could patch iproute2 to set it on
'ip address restore' at least, other than using it in pasta(1).
I'm not sure whether what I'm going to say is a good idea, or just a
plain stupid proposal:
By reverting the old odd behaviour, UAPI is restored, yes. On the
other side, userspace programs won't have any incentive to start using
NLM_F_INSERT_LAST, which is actually the most desirable behaviour. The
old odd behaviour is very error prone, as we can see with the `ip addr
restore` bug. That's why programs should adopt the "new" behaviour and
this will save them from introducing bugs like that one in the future.
And new programs should get the normal insertion by default, without
needing to read documentation or kernel headers to find
NLM_F_INSERT_LAST.
What if we reapply the patch and add a NLM_F_INSERT_FIRST /
NLM_F_PREPEND option instead? This will break UAPI, which is bad, but
probably not too bad per the conversations in this and the other
thread. Userspace programs affected by the change, like
NetworkManager, will need to be fixed, either by switching to natural
order insertion, or by passing the NLM_F_INSERT_FIRST flag to restore
the old behaviour.
Additionally, a NLM_F_INSERT_FIRST can be implemented for IPv4 too, at
least for consistency (I don't know whether it may be actually useful
for something, likely yes?).
As I said, this solution would break a few programs like
NetworkManager, but the fix is as easy as `if (ipv6) flags |=
NLM_F_INSERT_FIRST`. Would old kernels ignore this unknown flag?
Because, in that case, we don't even need a detection mechanism from
NetworkManager.
But that wouldn't be enough for your case. At the same time always
adding it for RTM_NEWADDR requests in iproute2 could break somebody
else's scripts. I guess a reasonable solution could be to add an
additional parameter for ip-address... 'insert_last'? 'last'?
--
Stefano
From: David Gibson <hidden> Date: 2026-06-02 06:44:24
I get the impression there's a rough consensus that the best we can do
now is revert this change (already done), and make a new patch which
changes the insertion order to the "correct" one conditional on a new
flag.
Stefano has enough other fires to fight, so I'm taking a look at
implementing that. Some initial thoughts, that I'm soliciting
feedback on:
1) I'm assuming the idea here is to add the new flag to nlmsg_flags in
nlmsghdr
ifa_flags in ifaddrmsg would be the other candidate, but it looks like
it's encoding properties of the address itself, not about the action
of inserting it. Plus all its bits are allocated, anyway.
2) Could we re-use NLM_F_APPEND?
The short description of this existing flag in linux/uapi/netlink.h is
"Add to end of list" which sounds like the right thing. Looking
closer, however, it seems like what is' used for so far is things
where the entity added with the NEW<whatever> operation is itself a
list, and NLM_F_APPEND causes it to be added to rather than replaced.
It's not used for addresses at present, AFAICT the list of addresses
is a semantic level above the address entity itself.
So maybe re-using it for the thing I tentatively called
NLM_F_INSERT_LAST would be confusing?
On the other hand, it's not used for addresses at the moment, so
AFAICT there's nothing actually preventing us reusing it for this
purpose. That would save a bit - we only have 2 general and 4 NEW
specific bits left, by the looks of it.
3) What other things might need changing to take advantage of the
kernel change
My innitial testing suggests that unknown nlmsg_flags bits are
currently ignored by the kernel. That means that tools for which the
new behviour is desirable, but not essential may be able to avoid
probing - they can set the bit and hope. I think that includes
passt/pasta and also iproute's save/restore functionality.
That said, things which might want updating:
- iproute2 to use this for save/restore
- netlink(7) man page to detail the new flag / new flag semantics
- passt/pasta
- network-manager, here we require the exact behaviour be maintained,
so it would have to attempt the new flag, then apply the existing
workaround (reversing the order itself) if that fails. Maybe more
trouble that it's worth?
Any others people can think of?
--
David Gibson (he or they) | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you, not the other way
| around.
http://www.ozlabs.org/~dgibson
From: Andrew Lunn <andrew@lunn.ch> Date: 2026-06-02 12:46:29
On Tue, Jun 02, 2026 at 04:44:19PM +1000, David Gibson wrote:
I get the impression there's a rough consensus that the best we can do
now is revert this change (already done), and make a new patch which
changes the insertion order to the "correct" one conditional on a new
flag.
Stefano has enough other fires to fight, so I'm taking a look at
implementing that. Some initial thoughts, that I'm soliciting
feedback on:
I've only been partially reading along...
Are we talking about RTM_NEWADDR?
I've never worked on the code dealing with addresses. But in general,
if you want to add new functionality to a netlink message, you add a
new attribute to the message.
https://elixir.bootlin.com/linux/v7.0.10/source/include/uapi/linux/if_addr.h#L26
Andrew
On Tue, Jun 02, 2026 at 04:44:19PM +1000, David Gibson wrote:
I get the impression there's a rough consensus that the best we can do
now is revert this change (already done), and make a new patch which
changes the insertion order to the "correct" one conditional on a new
flag.
Stefano has enough other fires to fight, so I'm taking a look at
implementing that. Some initial thoughts, that I'm soliciting
feedback on:
1) I'm assuming the idea here is to add the new flag to nlmsg_flags in
nlmsghdr
ifa_flags in ifaddrmsg would be the other candidate, but it looks like
it's encoding properties of the address itself, not about the action
of inserting it. Plus all its bits are allocated, anyway.
2) Could we re-use NLM_F_APPEND?
The short description of this existing flag in linux/uapi/netlink.h is
"Add to end of list" which sounds like the right thing. Looking
closer, however, it seems like what is' used for so far is things
where the entity added with the NEW<whatever> operation is itself a
list, and NLM_F_APPEND causes it to be added to rather than replaced.
It's not used for addresses at present, AFAICT the list of addresses
is a semantic level above the address entity itself.
So maybe re-using it for the thing I tentatively called
NLM_F_INSERT_LAST would be confusing?
On the other hand, it's not used for addresses at the moment, so
AFAICT there's nothing actually preventing us reusing it for this
purpose. That would save a bit - we only have 2 general and 4 NEW
specific bits left, by the looks of it.
3) What other things might need changing to take advantage of the
kernel change
My innitial testing suggests that unknown nlmsg_flags bits are
currently ignored by the kernel.
Which is a problem... From the looks of it, the same is true for
ifa_flags.
That means that tools for which the new behviour is desirable, but not
essential may be able to avoid probing - they can set the bit and
hope. I think that includes passt/pasta and also iproute's
save/restore functionality.
That said, things which might want updating:
- iproute2 to use this for save/restore
- netlink(7) man page to detail the new flag / new flag semantics
- passt/pasta
- network-manager, here we require the exact behaviour be maintained,
so it would have to attempt the new flag, then apply the existing
workaround (reversing the order itself) if that fails. Maybe more
trouble that it's worth?
Any others people can think of?
The safest route is probably a new attribute and a corresponding keyword
in iproute2. Note that new iproute2 versions need to be able to work
with old kernels, so iproute2 cannot use the new attribute by default
(it will be rejected by old kernels).
From: David Gibson <hidden> Date: 2026-06-03 02:37:46
On Tue, Jun 02, 2026 at 04:21:18PM +0300, Ido Schimmel wrote:
On Tue, Jun 02, 2026 at 04:44:19PM +1000, David Gibson wrote:
quoted
I get the impression there's a rough consensus that the best we can do
now is revert this change (already done), and make a new patch which
changes the insertion order to the "correct" one conditional on a new
flag.
Stefano has enough other fires to fight, so I'm taking a look at
implementing that. Some initial thoughts, that I'm soliciting
feedback on:
1) I'm assuming the idea here is to add the new flag to nlmsg_flags in
nlmsghdr
ifa_flags in ifaddrmsg would be the other candidate, but it looks like
it's encoding properties of the address itself, not about the action
of inserting it. Plus all its bits are allocated, anyway.
2) Could we re-use NLM_F_APPEND?
The short description of this existing flag in linux/uapi/netlink.h is
"Add to end of list" which sounds like the right thing. Looking
closer, however, it seems like what is' used for so far is things
where the entity added with the NEW<whatever> operation is itself a
list, and NLM_F_APPEND causes it to be added to rather than replaced.
It's not used for addresses at present, AFAICT the list of addresses
is a semantic level above the address entity itself.
So maybe re-using it for the thing I tentatively called
NLM_F_INSERT_LAST would be confusing?
On the other hand, it's not used for addresses at the moment, so
AFAICT there's nothing actually preventing us reusing it for this
purpose. That would save a bit - we only have 2 general and 4 NEW
specific bits left, by the looks of it.
Hmm. So, in this example case we have a known, widely deployed
userspace that was broken by the change. Similarly with the
original now-reverted "fix" for the ordering, we have a known, widely
deployed userspace that was broken.
That's a different case from a hypothetical userspace that incorrectly
used NLM_F_APPEND on RTM_NEWADDR. Moreover, to be broken it would
need to incorrectly use NLM_F_APPEND on RTM_NEWADDR *and also* rely on
the counterintuitive and inconsistent insertion order for IPv6
addresses. Absent a concrete example of something meeting both those
conditions, I'm inclined to breaking that hypothetical case when the
payoff is an easier route to get known cases working with the
preferred insertion semantics.
Fwiw, I did look at the most likely candidates: iproute2,
network-manager and libvirt, and I see no signs that they're misusing
NLM_F_APPEND in this way.
quoted
3) What other things might need changing to take advantage of the
kernel change
My innitial testing suggests that unknown nlmsg_flags bits are
currently ignored by the kernel.
Which is a problem... From the looks of it, the same is true for
ifa_flags.
ifa_flags is a no go anyway.
quoted
That means that tools for which the new behviour is desirable, but not
essential may be able to avoid probing - they can set the bit and
hope. I think that includes passt/pasta and also iproute's
save/restore functionality.
That said, things which might want updating:
- iproute2 to use this for save/restore
- netlink(7) man page to detail the new flag / new flag semantics
- passt/pasta
- network-manager, here we require the exact behaviour be maintained,
so it would have to attempt the new flag, then apply the existing
workaround (reversing the order itself) if that fails. Maybe more
trouble that it's worth?
Any others people can think of?
The safest route is probably a new attribute and a corresponding keyword
in iproute2.
I guess attribute+keyword does make it pretty straightforward to opt
in to the new behaviour in scripts, not just C code.
Note that new iproute2 versions need to be able to work
with old kernels, so iproute2 cannot use the new attribute by default
(it will be rejected by old kernels).
Yes, I'm aware.
--
David Gibson (he or they) | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you, not the other way
| around.
http://www.ozlabs.org/~dgibson
From: David Gibson <hidden> Date: 2026-06-03 02:37:46
On Tue, Jun 02, 2026 at 02:46:22PM +0200, Andrew Lunn wrote:
On Tue, Jun 02, 2026 at 04:44:19PM +1000, David Gibson wrote:
quoted
I get the impression there's a rough consensus that the best we can do
now is revert this change (already done), and make a new patch which
changes the insertion order to the "correct" one conditional on a new
flag.
Stefano has enough other fires to fight, so I'm taking a look at
implementing that. Some initial thoughts, that I'm soliciting
feedback on:
I've only been partially reading along...
Are we talking about RTM_NEWADDR?
Ah, good point, that's another option, and avoids using scarce flags
bits. It is a little bit odd, because generally the attributes are,
well, attributes, _of the new object_ being created (an address in
this case). Here we're adjusting where / how it is created, not
anything about the address itself.
Stil, definitely a better option that allocating a new flags bit.
Versus NLM_F_APPEND, I'll reply to Ido Schimmel.
--
David Gibson (he or they) | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you, not the other way
| around.
http://www.ozlabs.org/~dgibson
On Mon, Jun 1, 2026 at 3:35 PM Stefano Brivio [off-list ref] wrote:
quoted
I was thinking that if we implement a label like NLM_F_INSERT_LAST
(David's proposal for the name), we could patch iproute2 to set it on
'ip address restore' at least, other than using it in pasta(1).
(...)
What if we reapply the patch and add a NLM_F_INSERT_FIRST /
NLM_F_PREPEND option instead? This will break UAPI, which is bad, but
probably not too bad per the conversations in this and the other
thread.
I guess it is safer not to break UAPI and not reapplying the patch,
especially when there is no clear way to know which order is taken by
the kernel.
What about clearly stating that the default order is "random", except
when an additional flag is set to sort them chronologically or the opposite?
Cheers,
Matt
--
Sponsored by the NGI0 Core fund.
On Wed, Jun 3, 2026 at 7:46 AM Matthieu Baerts [off-list ref] wrote:
I guess it is safer not to break UAPI and not reapplying the patch,
especially when there is no clear way to know which order is taken by
the kernel.
What about clearly stating that the default order is "random", except
when an additional flag is set to sort them chronologically or the opposite?
Stating that the order is "random" implies that someone will change
the default insertion order in the future, as the documentation says
that it's not UAPI. So they'll cause the UAPI breakage that we are
trying to avoid now. Moreover, if we consider the default order
"random" and state that userspace programs must not rely on it, why
can't we change it now?
As mentioned in the other thread by David, we have programs broken by
the original "fix" (NetworkManager), and programs that will be broken
(again) by the revert (pasta). But more importantly, I think that,
with the revert, there may be programs that are broken but they don't
know it yet. Again, see how `ip addr restore` will be broken again.
So, both solutions are bad, but doing the revert is worse, IMO.
As I'm not much involved in NetworkManager any more, only Beniamino is
who can truly speak on behalf of the NetworkManager project, but as a
former NetworkManager maintainer, I think that my proposal earlier in
this thread is the way to go:
https://lore.kernel.org/netdev/CACT4oueDv=KRZmJw=1N58EN7eoFz5AqbBBMP8FE4=+rQ99oFEg@mail.gmail.com/.
It's slightly controversial because it may be considered an UAPI
break, but on the other side it may be considered as a fix to a wrong
implementation of the UAPI, instead, as Stefano pointed out:
https://lore.kernel.org/netdev/20260528165320.15b90ded@elisabeth/
--
Íñigo Huguet
Stating that the order is "random" implies that someone will change
the default insertion order in the future, as the documentation says
that it's not UAPI. So they'll cause the UAPI breakage that we are
trying to avoid now. Moreover, if we consider the default order
"random" and state that userspace programs must not rely on it, why
can't we change it now?
Because the UAPI is only indirectly important, as it's "something that
used to work doesn't anymore" is what matters; you in fact can break the
UAPI all you want, as long as everything continues to work. Or to quote
Linus from:
https://www.kernel.org/doc/html/latest/process/handling-regressions.html#on-what-qualifies-as-userspace-interface-abi-api-documented-interfaces-etc
"I just wanted to point out that the argument about whether it's an ABI
change or not is irrelevant. If it turns out that some program - not a
test script, but something with relevance to conscious user expectations
~ depended on the old broken behavior, then it needs to be done some
other way.""
Yes, he talks about ABIs here, but it's the same for APIs.
Ciao, Thorsten
From: Fernando Fernandez Mancera <hidden> Date: 2026-06-03 07:30:06
On 6/3/26 9:17 AM, Thorsten Leemhuis wrote:
On 6/3/26 08:53, Íñigo Huguet wrote:
quoted
Stating that the order is "random" implies that someone will change
the default insertion order in the future, as the documentation says
that it's not UAPI. So they'll cause the UAPI breakage that we are
trying to avoid now. Moreover, if we consider the default order
"random" and state that userspace programs must not rely on it, why
can't we change it now?
Because the UAPI is only indirectly important, as it's "something that
used to work doesn't anymore" is what matters; you in fact can break the
UAPI all you want, as long as everything continues to work. Or to quote
Linus from:
I agree with Thorsten here. We must not break the UAPI. Is the current
order kind of weird? Yes. But software that is running on real systems
depends on it.
AFAICS, it is not only NetworkManager but systemd-networkd, cloud
tooling and any other script that uses netlink. I think we must stick to
the rule of not breaking userspace and the argument of consistency with
IPv4 is not appealing to me because that isn't a goal of IPv6, plenty is
different when you compared both protocols.
We should document that the order matters, no that it is "random"
because it isn't.
On Wed, Jun 03, 2026 at 12:34:36PM +1000, David Gibson wrote:
On Tue, Jun 02, 2026 at 04:21:18PM +0300, Ido Schimmel wrote:
quoted
On Tue, Jun 02, 2026 at 04:44:19PM +1000, David Gibson wrote:
quoted
I get the impression there's a rough consensus that the best we can do
now is revert this change (already done), and make a new patch which
changes the insertion order to the "correct" one conditional on a new
flag.
Stefano has enough other fires to fight, so I'm taking a look at
implementing that. Some initial thoughts, that I'm soliciting
feedback on:
1) I'm assuming the idea here is to add the new flag to nlmsg_flags in
nlmsghdr
ifa_flags in ifaddrmsg would be the other candidate, but it looks like
it's encoding properties of the address itself, not about the action
of inserting it. Plus all its bits are allocated, anyway.
2) Could we re-use NLM_F_APPEND?
The short description of this existing flag in linux/uapi/netlink.h is
"Add to end of list" which sounds like the right thing. Looking
closer, however, it seems like what is' used for so far is things
where the entity added with the NEW<whatever> operation is itself a
list, and NLM_F_APPEND causes it to be added to rather than replaced.
It's not used for addresses at present, AFAICT the list of addresses
is a semantic level above the address entity itself.
So maybe re-using it for the thing I tentatively called
NLM_F_INSERT_LAST would be confusing?
On the other hand, it's not used for addresses at the moment, so
AFAICT there's nothing actually preventing us reusing it for this
purpose. That would save a bit - we only have 2 general and 4 NEW
specific bits left, by the looks of it.
Hmm. So, in this example case we have a known, widely deployed
userspace that was broken by the change. Similarly with the
original now-reverted "fix" for the ordering, we have a known, widely
deployed userspace that was broken.
It was also reported over three years after the kernel change went in.
Point is that we have no way of knowing how user space is using these
flags. Suddenly giving them meaning when we simply ignored them before
is risky.
That's a different case from a hypothetical userspace that incorrectly
used NLM_F_APPEND on RTM_NEWADDR. Moreover, to be broken it would
need to incorrectly use NLM_F_APPEND on RTM_NEWADDR *and also* rely on
the counterintuitive and inconsistent insertion order for IPv6
addresses. Absent a concrete example of something meeting both those
conditions, I'm inclined to breaking that hypothetical case when the
payoff is an easier route to get known cases working with the
preferred insertion semantics.
Fwiw, I did look at the most likely candidates: iproute2,
network-manager and libvirt, and I see no signs that they're misusing
NLM_F_APPEND in this way.
See above. I don't like this approach. IMO, it's not worth making it
slightly a bit easier for some user space programs to adopt when the
risk is breaking other programs and repeating this ordeal.
From: David Gibson <hidden> Date: 2026-06-03 08:02:09
On Wed, Jun 03, 2026 at 03:46:14PM +1000, Matthieu Baerts wrote:
On 02/06/2026 00:01, Íñigo Huguet wrote:
quoted
On Mon, Jun 1, 2026 at 3:35 PM Stefano Brivio [off-list ref] wrote:
quoted
I was thinking that if we implement a label like NLM_F_INSERT_LAST
(David's proposal for the name), we could patch iproute2 to set it on
'ip address restore' at least, other than using it in pasta(1).
(...)
quoted
What if we reapply the patch and add a NLM_F_INSERT_FIRST /
NLM_F_PREPEND option instead? This will break UAPI, which is bad, but
probably not too bad per the conversations in this and the other
thread.
I guess it is safer not to break UAPI and not reapplying the patch,
especially when there is no clear way to know which order is taken by
the kernel.
I think Íñigo wasn't suggesting reapplying the patch as is, but a
modified version that only makes the behavioural change when userspace
sets a new attribute/flag/whatever.
What about clearly stating that the default order is "random", except
when an additional flag is set to sort them chronologically or the
opposite?
We could, but since userspace relying on the existing behaviour is
already out there, I'm not sure what it buys us. The insertion order
also isn't random - it's sorted by scope at least. Theoretically
there could be other (existing) sorting criteria for other address
types (I haven't looked beyong IPv4 and IPv6). The problem arises
because IPv4 and IPv6 have different behaviours when inserting
something where the order isn't defined - IPv4 inserts after existing
entries of the same scope, IPv6 puts it before existing entries of the
same scope - it's literally a < versus a <=.
--
David Gibson (he or they) | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you, not the other way
| around.
http://www.ozlabs.org/~dgibson
Hi Íñigo,
On 03/06/2026 16:53, Íñigo Huguet wrote:
On Wed, Jun 3, 2026 at 7:46 AM Matthieu Baerts [off-list ref] wrote:
quoted
I guess it is safer not to break UAPI and not reapplying the patch,
especially when there is no clear way to know which order is taken by
the kernel.
What about clearly stating that the default order is "random", except
when an additional flag is set to sort them chronologically or the opposite?
Stating that the order is "random" implies that someone will change
the default insertion order in the future, as the documentation says
that it's not UAPI. So they'll cause the UAPI breakage that we are
trying to avoid now. Moreover, if we consider the default order
"random" and state that userspace programs must not rely on it, why
can't we change it now?
I was only suggesting "random", simply to force userspace programs to
ask for a specific order when the order matters. But also to avoid
having to describe the current order, which is different in v4 and v6
for entries of the same scope.
I know the order is not random, and even if it is confusing, I don't
think we can change it without breaking stuff.
Cheers,
Matt
--
Sponsored by the NGI0 Core fund.
On Wed, 3 Jun 2026 10:47:17 +0300
Ido Schimmel [off-list ref] wrote:
On Wed, Jun 03, 2026 at 12:34:36PM +1000, David Gibson wrote:
quoted
On Tue, Jun 02, 2026 at 04:21:18PM +0300, Ido Schimmel wrote:
quoted
On Tue, Jun 02, 2026 at 04:44:19PM +1000, David Gibson wrote:
quoted
I get the impression there's a rough consensus that the best we can do
now is revert this change (already done), and make a new patch which
changes the insertion order to the "correct" one conditional on a new
flag.
Stefano has enough other fires to fight, so I'm taking a look at
implementing that. Some initial thoughts, that I'm soliciting
feedback on:
1) I'm assuming the idea here is to add the new flag to nlmsg_flags in
nlmsghdr
ifa_flags in ifaddrmsg would be the other candidate, but it looks like
it's encoding properties of the address itself, not about the action
of inserting it. Plus all its bits are allocated, anyway.
2) Could we re-use NLM_F_APPEND?
The short description of this existing flag in linux/uapi/netlink.h is
"Add to end of list" which sounds like the right thing. Looking
closer, however, it seems like what is' used for so far is things
where the entity added with the NEW<whatever> operation is itself a
list, and NLM_F_APPEND causes it to be added to rather than replaced.
It's not used for addresses at present, AFAICT the list of addresses
is a semantic level above the address entity itself.
So maybe re-using it for the thing I tentatively called
NLM_F_INSERT_LAST would be confusing?
On the other hand, it's not used for addresses at the moment, so
AFAICT there's nothing actually preventing us reusing it for this
purpose. That would save a bit - we only have 2 general and 4 NEW
specific bits left, by the looks of it.
Hmm. So, in this example case we have a known, widely deployed
userspace that was broken by the change. Similarly with the
original now-reverted "fix" for the ordering, we have a known, widely
deployed userspace that was broken.
It was also reported over three years after the kernel change went in.
Point is that we have no way of knowing how user space is using these
flags. Suddenly giving them meaning when we simply ignored them before
is risky.
I think that's a very different type of issue because, there, *another*
existing flag (NLM_F_EXCL) was suddenly given a meaning, as it happened
to have the same value as NLM_F_BULK, and that's what broke libvirt.
Not support for NLM_F_BULK itself.
Here, NLM_F_APPEND doesn't share its value with any other flag, and it
really is documented as "Add to end of list", but we don't do that.
That's a bug.
I think it's actually more likely that some bits of userspace are
currently broken and causing subtle issues because the author expected
NLM_F_APPEND to actually do what it promises, but maybe they only
tested that with IPv4.
Allow me to draw a parallel that looks more fitting to me: in commit
1e47b4837f3b ("ipv6: Dump route exceptions if requested") I happened to
fix a two-year old issue that made 'ip -6 route list cache' show no
output and 'ip -6 route flush cache' have no effect.
You could take this to the extreme and say that it was risky to fix
that because some userspace application could meanwhile have started
relying on the fact that 'ip -6 route list cache' returned no output.
I guess we agree it was a good idea to fix that, though.
Of course there are several degrees of UAPI expectations in between,
but *not* allowing to use NLM_F_APPEND to append objects because
userspace might rely on NLM_F_APPEND to *not* append objects sounds
a bit like this extreme to me, or at least closer to it than the
NLM_F_BULK kind of breakage.
quoted
That's a different case from a hypothetical userspace that incorrectly
used NLM_F_APPEND on RTM_NEWADDR. Moreover, to be broken it would
need to incorrectly use NLM_F_APPEND on RTM_NEWADDR *and also* rely on
the counterintuitive and inconsistent insertion order for IPv6
addresses. Absent a concrete example of something meeting both those
conditions, I'm inclined to breaking that hypothetical case when the
payoff is an easier route to get known cases working with the
preferred insertion semantics.
Fwiw, I did look at the most likely candidates: iproute2,
network-manager and libvirt, and I see no signs that they're misusing
NLM_F_APPEND in this way.
See above. I don't like this approach. IMO, it's not worth making it
slightly a bit easier for some user space programs to adopt when the
risk is breaking other programs and repeating this ordeal.
Another fact we shouldn't ignore is that, compared to the NLM_F_BULK
incident, we're actively surveying userspace before touching this.
--
Stefano
From: Nicolas Dichtel <hidden> Date: 2026-06-03 15:47:29
Le 03/06/2026 à 09:47, Ido Schimmel a écrit :
On Wed, Jun 03, 2026 at 12:34:36PM +1000, David Gibson wrote:
quoted
On Tue, Jun 02, 2026 at 04:21:18PM +0300, Ido Schimmel wrote:
quoted
On Tue, Jun 02, 2026 at 04:44:19PM +1000, David Gibson wrote:
[snip]
quoted
quoted
quoted
2) Could we re-use NLM_F_APPEND?
The short description of this existing flag in linux/uapi/netlink.h is
"Add to end of list" which sounds like the right thing. Looking
closer, however, it seems like what is' used for so far is things
where the entity added with the NEW<whatever> operation is itself a
list, and NLM_F_APPEND causes it to be added to rather than replaced.
It's not used for addresses at present, AFAICT the list of addresses
is a semantic level above the address entity itself.
So maybe re-using it for the thing I tentatively called
NLM_F_INSERT_LAST would be confusing?
On the other hand, it's not used for addresses at the moment, so
AFAICT there's nothing actually preventing us reusing it for this
purpose. That would save a bit - we only have 2 general and 4 NEW
specific bits left, by the looks of it.
Hmm. So, in this example case we have a known, widely deployed
userspace that was broken by the change. Similarly with the
original now-reverted "fix" for the ordering, we have a known, widely
deployed userspace that was broken.
It was also reported over three years after the kernel change went in.
Point is that we have no way of knowing how user space is using these
flags. Suddenly giving them meaning when we simply ignored them before
is risky.
From: David Gibson <hidden> Date: 2026-06-04 01:28:07
On Wed, Jun 03, 2026 at 05:45:39PM +0200, Stefano Brivio wrote:
On Wed, 3 Jun 2026 10:47:17 +0300
Ido Schimmel [off-list ref] wrote:
quoted
On Wed, Jun 03, 2026 at 12:34:36PM +1000, David Gibson wrote:
quoted
On Tue, Jun 02, 2026 at 04:21:18PM +0300, Ido Schimmel wrote:
quoted
On Tue, Jun 02, 2026 at 04:44:19PM +1000, David Gibson wrote:
quoted
I get the impression there's a rough consensus that the best we can do
now is revert this change (already done), and make a new patch which
changes the insertion order to the "correct" one conditional on a new
flag.
Stefano has enough other fires to fight, so I'm taking a look at
implementing that. Some initial thoughts, that I'm soliciting
feedback on:
1) I'm assuming the idea here is to add the new flag to nlmsg_flags in
nlmsghdr
ifa_flags in ifaddrmsg would be the other candidate, but it looks like
it's encoding properties of the address itself, not about the action
of inserting it. Plus all its bits are allocated, anyway.
2) Could we re-use NLM_F_APPEND?
The short description of this existing flag in linux/uapi/netlink.h is
"Add to end of list" which sounds like the right thing. Looking
closer, however, it seems like what is' used for so far is things
where the entity added with the NEW<whatever> operation is itself a
list, and NLM_F_APPEND causes it to be added to rather than replaced.
It's not used for addresses at present, AFAICT the list of addresses
is a semantic level above the address entity itself.
So maybe re-using it for the thing I tentatively called
NLM_F_INSERT_LAST would be confusing?
On the other hand, it's not used for addresses at the moment, so
AFAICT there's nothing actually preventing us reusing it for this
purpose. That would save a bit - we only have 2 general and 4 NEW
specific bits left, by the looks of it.
Hmm. So, in this example case we have a known, widely deployed
userspace that was broken by the change. Similarly with the
original now-reverted "fix" for the ordering, we have a known, widely
deployed userspace that was broken.
It was also reported over three years after the kernel change went in.
Point is that we have no way of knowing how user space is using these
flags. Suddenly giving them meaning when we simply ignored them before
is risky.
I think that's a very different type of issue because, there, *another*
existing flag (NLM_F_EXCL) was suddenly given a meaning, as it happened
to have the same value as NLM_F_BULK, and that's what broke libvirt.
Not support for NLM_F_BULK itself.
Here, NLM_F_APPEND doesn't share its value with any other flag, and it
really is documented as "Add to end of list", but we don't do that.
That's a bug.
Eh.. that's the short description in the header. But looking at how
it's actually used it generally means "append as opposed to replace"
(which is not relevant for addresses) rather than "append as opposed
to prepend". So in that sense we would be assigning a new meaning.
I think it's actually more likely that some bits of userspace are
currently broken and causing subtle issues because the author expected
NLM_F_APPEND to actually do what it promises, but maybe they only
tested that with IPv4.
That's possible, although I'd guess far less likely that simply
expecting insert last behaviour without any extra flag.
Allow me to draw a parallel that looks more fitting to me: in commit
1e47b4837f3b ("ipv6: Dump route exceptions if requested") I happened to
fix a two-year old issue that made 'ip -6 route list cache' show no
output and 'ip -6 route flush cache' have no effect.
You could take this to the extreme and say that it was risky to fix
that because some userspace application could meanwhile have started
relying on the fact that 'ip -6 route list cache' returned no output.
I guess we agree it was a good idea to fix that, though.
Of course there are several degrees of UAPI expectations in between,
but *not* allowing to use NLM_F_APPEND to append objects because
userspace might rely on NLM_F_APPEND to *not* append objects sounds
a bit like this extreme to me, or at least closer to it than the
NLM_F_BULK kind of breakage.
quoted
quoted
That's a different case from a hypothetical userspace that incorrectly
used NLM_F_APPEND on RTM_NEWADDR. Moreover, to be broken it would
need to incorrectly use NLM_F_APPEND on RTM_NEWADDR *and also* rely on
the counterintuitive and inconsistent insertion order for IPv6
addresses. Absent a concrete example of something meeting both those
conditions, I'm inclined to breaking that hypothetical case when the
payoff is an easier route to get known cases working with the
preferred insertion semantics.
Fwiw, I did look at the most likely candidates: iproute2,
network-manager and libvirt, and I see no signs that they're misusing
NLM_F_APPEND in this way.
See above. I don't like this approach. IMO, it's not worth making it
slightly a bit easier for some user space programs to adopt when the
risk is breaking other programs and repeating this ordeal.
Another fact we shouldn't ignore is that, compared to the NLM_F_BULK
incident, we're actively surveying userspace before touching this.
So, I second Stefano's arguments for the most part, as well as
re-iterating that being broken by this change would require the
intersection of two unlikely conditions (misusing NLM_F_APPEND *and*
expecting the "wrong" order).
That said, Ido, if you're still not convinced I can do this as an
attribute. It's more hassle, but I can make it work.
--
David Gibson (he or they) | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you, not the other way
| around.
http://www.ozlabs.org/~dgibson
On Thu, Jun 04, 2026 at 11:26:08AM +1000, David Gibson wrote:
So, I second Stefano's arguments for the most part, as well as
re-iterating that being broken by this change would require the
intersection of two unlikely conditions (misusing NLM_F_APPEND *and*
expecting the "wrong" order).
That said, Ido, if you're still not convinced I can do this as an
attribute. It's more hassle, but I can make it work.
I appreciate the survey that Stefano and you conducted, but there is
still a non-zero chance of causing regressions by suddenly giving
NLM_F_APPEND a meaning in RTM_NEWADDR. We already tried the
"change-and-see-what-happens" methodology once with this feature and it
backfired, so it's going to be quite painful if we miss again.
As I see it, we have three options:
1. Use NLM_F_APPEND. Relatively easy change in both the kernel and user
space, but at the risk of reintroducing regressions.
2. Add a new attribute (e.g., IFA_INSERT_MODE with DEFAULT/APPEND
options). Less risky than #1, at the cost of a bit more code in both the
kernel and user space.
3. Do nothing. As I understand it, any production software (as opposed
to a test script) that cares about the in-scope order will have to
maintain a fallback anyway (e.g., iterating over IPv6 addresses in
reverse). Therefore, the changes in #1 and #2 are not strictly
necessary, yet they are uAPI that the kernel will have to maintain
forever.
Given the above, my preference would be #3 -> #2 -> #1. The first two
options expose the same capability to user space, so #1 doesn't buy us
anything over #2, except a bit less code, but we risk introducing a
regression.
Between #2 and #3, production software can't drop the fallback even if
we implement #2, yet #2 requires us to maintain uAPI forever. I think we
should accept that the divergence between IPv4 and IPv6 is not ideal,
but at least it's predictable and dependable (Fernando is working on a
ksft and documentation).
That being said, you can send an RFC for #1 and see what others think
since at this point it's unclear who is still following the thread.
Thanks
From: Jakub Kicinski <kuba@kernel.org> Date: 2026-06-04 22:55:37
On Thu, 4 Jun 2026 21:39:09 +0300 Ido Schimmel wrote:
Given the above, my preference would be #3 -> #2 -> #1. The first two
options expose the same capability to user space, so #1 doesn't buy us
anything over #2, except a bit less code, but we risk introducing a
regression.
Between #2 and #3, production software can't drop the fallback even if
we implement #2, yet #2 requires us to maintain uAPI forever. I think we
should accept that the divergence between IPv4 and IPv6 is not ideal,
but at least it's predictable and dependable (Fernando is working on a
ksft and documentation).
On Thu, 4 Jun 2026 21:39:09 +0300
Ido Schimmel [off-list ref] wrote:
On Thu, Jun 04, 2026 at 11:26:08AM +1000, David Gibson wrote:
quoted
So, I second Stefano's arguments for the most part, as well as
re-iterating that being broken by this change would require the
intersection of two unlikely conditions (misusing NLM_F_APPEND *and*
expecting the "wrong" order).
That said, Ido, if you're still not convinced I can do this as an
attribute. It's more hassle, but I can make it work.
I appreciate the survey that Stefano and you conducted, but there is
still a non-zero chance of causing regressions by suddenly giving
NLM_F_APPEND a meaning in RTM_NEWADDR. We already tried the
"change-and-see-what-happens" methodology once with this feature and it
backfired, so it's going to be quite painful if we miss again.
As I see it, we have three options:
Thanks for the helpful summary by the way.
1. Use NLM_F_APPEND. Relatively easy change in both the kernel and user
space, but at the risk of reintroducing regressions.
2. Add a new attribute (e.g., IFA_INSERT_MODE with DEFAULT/APPEND
options). Less risky than #1, at the cost of a bit more code in both the
kernel and user space.
3. Do nothing. As I understand it, any production software (as opposed
to a test script) that cares about the in-scope order will have to
maintain a fallback anyway (e.g., iterating over IPv6 addresses in
reverse).
It's not always enough, or needed, or practical, though:
a. 'ip address restore' could be "fixed" to load the addresses in a
reversed order for IPv6, but what should 'ip address showdump' do at
that point?
Also reverse the order? Or not, because that's the order addresses
were dumped in? I'm fairly sure 'ip address save' shouldn't reverse
it. It would be more convenient for the other operations, but
definitely wrong, because the kernel is picking addresses in the
opposite order.
All these are doable, some look questionable, but any of the possible
workarounds looks hard to document, or even remember.
b. as to pasta(1) and passt(1): the regression introduced by the revert
of the kernel change isn't a big one: after all, we had the right
behaviour for just a few months.
Similarly, if the kernel gives us a way to get it right, we would
just stick to it for the future and be done with it. Adding a
workaround for older kernels isn't a priority because there was no
regression.
Side note: the workaround would be rather impractical for us because
we don't use dynamic memory allocation, but we certainly can't blame
the kernel or anybody else for that.
c. regardless of how addresses are dumped to userspace, there would be
no way to get the kernel to do what one reasonably expects it to do
(also from established IPv4 practice): *use* addresses in order of
insertion
By the way of c., you're explicitly excluding test scripts as they are
certainly less important, but does it really make sense to force people
to to maintain the kind of workaround Matthieu mentioned? Reporting it
here for convenience:
https://github.com/multipath-tcp/packetdrill/commit/1b7cd4482ce8
Therefore, the changes in #1 and #2 are not strictly
necessary, yet they are uAPI that the kernel will have to maintain
forever.
Given the above, my preference would be #3 -> #2 -> #1. The first two
options expose the same capability to user space, so #1 doesn't buy us
anything over #2, except a bit less code, but we risk introducing a
regression.
Between #2 and #3, production software can't drop the fallback even if
we implement #2, yet #2 requires us to maintain uAPI forever. I think we
should accept that the divergence between IPv4 and IPv6 is not ideal,
but at least it's predictable and dependable (Fernando is working on a
ksft and documentation).
That being said, you can send an RFC for #1 and see what others think
since at this point it's unclear who is still following the thread.
I think there's general consensus against #1 ("abusing" NLM_F_APPEND),
but still, I think #2 is very much needed and fundamentally harmless.
--
Stefano