When the routing decision involves ECMP, the initial hash is calculated
with saddr being :: and the decision which interface to select from an
ECMP group is based on that hash. If the route lookup has to be
repeated, e.g. because a route was updated, the hash is calculated with
saddr set to the previously selected address. This can cause the
selected interface to change, breaking ongoing connections.

To ensure the initial interface selection is based on an actual saddr
the route lookup is repeated after the source address selection if a
hash was calculated.

Signed-off-by: Maximilian Moehl <redacted>
Link: https://lore.kernel.org/all/aOYLRyIlc7XU7-7n@shredder/ (local)
---
I've created a write-up of what I've done, including the steps taken
to test the patch: https://moehl.eu/blog/linux-ipv6-ecmp-instability.html

 net/ipv6/ip6_output.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c
index 8e2a6b28cea7..465fce51d017 100644
--- a/net/ipv6/ip6_output.c
+++ b/net/ipv6/ip6_output.c

@@ -1148,6 +1148,18 @@ static int ip6_dst_lookup_tail(struct net *net, const struct sock *sk,
 			*dst = NULL;
 		}
 
+		/* If ECMP was involved the initial hash was calculted
+		 * with saddr=:: which can result in instability
+		 * when it is later re-calculated with the selected
+		 * saddr. Lookup the route again with the chosen
+		 * saddr to get a stable result.
+		 */
+		if (fl6->mp_hash) {
+			fl6->mp_hash = 0;
+			dst_release(*dst);
+			*dst = NULL;
+		}
+
 		if (fl6->flowi6_oif)
 			flags |= RT6_LOOKUP_F_IFACE;
 	}