Re: [PATCH net-next 0/3] net: nexthop: per-nexthop UDP dst port for fdb (VXLAN) nexthops

On Wed, Jul 15, 2026 at 07:43:17AM +0000, Jack Ma wrote:

On Tue, Jul 14, 2026 at 08:42:01PM +0300, Ido Schimmel wrote:

quoted

It's unclear to me why you need the UDP port based demux. On a single
node, do you have multiple pods that belong to the same tenant and
therefore share the same VNI? From the above, my understanding is that
the answer is "no".

Good question, and I think my earlier framing sent you down the wrong
path, sorry about that. The honest answer is actually "yes". By design
a node can hold several VTEPs on the same VNI, as long as they belong
to different HA sets. Anti-affinity only keeps the two replicas within
one HA set off the same node. Two pods from different HA sets on the
same VNI are free to co-locate, and that co-location is the case that
needs a per-pod port.

quoted

Assuming the answer is "no", instead of having multiple VXLAN sockets
in the host netns (each bound to a different port), why not have a
single VXLAN socket bound to the standard port (4789) and let the
VXLAN driver perform the demux based on the VNI? [...] creating
multiple VXLAN devices in the host netns that only differ in their
VNI and then move each to the relevant pod netns.

For different-VNI pods that is a clean fit, and I want to try it out
regardless. The outer packet still carries the VNI, so a shared socket
can steer by VNI without decapsulating, and the host never touches the
tenant frame.

Where it stops working for me is two same-VNI pods on one node. They
share the underlay IP, the port (4789), and the VNI, so the outer
header is the same for both:

  outer: NodeIP:4789, VNI=X  ->  pod A (HA set 1)
  outer: NodeIP:4789, VNI=X  ->  pod B (HA set 2)

There is nothing left in the outer header to steer on, so the host
would have to look past it and terminate the tunnel itself, which is
the VTEP role I would rather keep inside the pod. A distinct port per
pod puts the distinguishing bit back in the outer header, so the host
can stay a stateless outer-UDP forwarder.

quoted

Given the anti-affinity constraint, you will only need to load balance
between pods on different nodes that use the standard port and that is
already supported by the FDB nexthop groups.

Agreed that the across-node load balancing already works. What the
VNI-demux version costs is packing density. With one shared underlay
IP per node, VNI demux allows at most one VTEP per (VNI, node), so N
same-VNI HA sets of two replicas need 2N nodes. With a per-pod port
the same sets fit on two nodes (replica 1 on node A, replica 2 on node
B, each pod on its own port), and anti-affinity still holds. For 5
sets on one VNI that is 10 nodes versus 2.

OK, I understand. Each tenant has its own VNI, but it can have multiple
HA sets on the same node and you want to distinguish between them using
the UDP destination port.

So the port is not strictly required, and your VNI-demux model is a
real option I will prototype. The tradeoff I am weighing is density
plus keeping the host out of the tenant datapath, against the uAPI
cost, which is small: NHA_FDB_PORT is the nexthop analog of the
per-entry NDA_PORT that already exists, it is control-plane only, and
a leg with no port behaves exactly as today.

I don't particularly like NHA_FDB_PORT because it's an FDB-specific
encapsulation attribute and these attributes naturally belong under
NHA_ENCAP. That being said, I fail to see a better alternative given
that FDB nexthops don't use NHA_ENCAP / LWT infra and probably
shouldn't. It's a bigger change in the datapath and I expect that we
will not need too many FDB-specific attributes in the future.

If you are going to post v2, please provide a more detailed motivation
in the cover letter.

Happy to keep digging if it is useful. Thanks again for thinking it
through with me.

In an earlier reply you wrote "shared inner MAC/IP". The "MAC" part was
intentional because you need to load balance pure L2 traffic? IOW, if
your traffic is only L3, can we do the load balancing using multipath
routes in the overlay instead of using FDB nexthop groups?

Related: What is the use case for NHA_FDB_VNI? In my experience, the VNI
is only translated when routing traffic between tenants. Why do you need
this in an FDB nexthop?