Re: VXLAN FDB nexthop groups: per-nexthop UDP dst port / VNI feasibility

Please use plain text instead of HTML. Otherwise your emails won't make
it to the list.

On Wed, Jul 08, 2026 at 05:52:05AM +0000, Jack Ma wrote:

   Hi David, Ido, Roopa,
   I am using VXLAN FDB nexthop groups (ip nexthop add ... fdb, then
   bridge fdb add ... nhid) to do L2 ECMP across a set of HA endpoints.
   In my design each leg is a distinct (VTEP IP, UDP destination port)
   pair: several endpoints live behind the same node IP and are
   disambiguated by the VXLAN UDP destination port.
   Reading the current code (v6.12), it looks like an FDB nexthop can
   only carry a gateway IP, so a per-leg port cannot be expressed:
     * vxlan_fdb_nh_path_select() copies only nhc_gw into the synthetic
       rdst; remote_port and remote_vni stay zero
       (include/net/vxlan.h).
     * vxlan_xmit_one() then falls back to the device-wide port:
       dst_port = rdst->remote_port ? rdst->remote_port
                                    : vxlan->cfg.dst_port
       (drivers/net/vxlan/vxlan_core.c).
     * rtm_to_nh_config() rejects NHA_ENCAP and NHA_OIF on fdb nexthops
       ("Fdb attribute can not be used with encap, oif or blackhole",
       net/ipv4/nexthop.c).
     * vxlan_fdb_parse() rejects NDA_PORT / NDA_VNI / NDA_IFINDEX when
       NDA_NH_ID is present ("DST, VNI, ifindex and port are mutually
       exclusive with NH_ID").
   The original commits (38428d68719c "nexthop: support for fdb ecmp
   nexthops" and 1274e1cc4226 "vxlan: ecmp support for mac fdb entries")
   say "these nexthops only have ip", which matches the EVPN multihoming
   use case where every VTEP in the segment shares one UDP destination
   port.
   My questions:
     1. Was IP-only a deliberate scoping decision for the EVPN-MH use
        case, or is there a deeper reason a per-nexthop UDP destination
        port (and VNI) could not be carried on an fdb nexthop, for
        example via NHA_ENCAP / lwtunnel, which route nexthops already
        accept?

IP-only was specific for EVPN-MH and it simply reuses the gateway IP
that is already present in the nexthop structure.

     2. If legs genuinely need distinct UDP destination ports, is the
        per-remote rdst path (bridge fdb ... dst ... port ...) the
        intended mechanism, with the understanding that it gives
        head-end replication semantics rather than nexthop-object ECMP?

Yes, but you will get packet replication instead of load balancing,
which I believe is not what you want.

     3. Would you be open in principle to a patch that lets an fdb
        nexthop carry a per-nexthop dst port (and VNI)? If so I am happy
        to prototype it as an RFC.

I have some questions for you before I answer yours.

Can you explain why different VTEPs need to share a single IP? With
different IPs you can simply reuse the existing FDB nexthops without any
changes.

In addition, why disambiguate by the VXLAN UDP destination port and not
VNI? Even if we allow attaching tunnel encapsulation to FDB nexthops via
NHA_ENCAP, there does not seem to be uAPI in place to encode the
destination port (see ip_tun_policy).

Assuming you can live with a fixed port, I think you can achieve what
you want with BPF and no kernel changes. You can call
bpf_skb_set_tunnel_key [1] from a TC bpf program attached to the egress
of the VXLAN device (configured in "external" mode) and have it encode
the relevant VTEP based on the calculated hash.

[1] https://docs.ebpf.io/linux/helper-function/bpf_skb_set_tunnel_key/