From: Vladimir Oltean <olteanv@gmail.com> Date: 2021-03-18 23:19:27
From: Vladimir Oltean <vladimir.oltean@nxp.com>
This series has two objectives:
- To make LAG uppers on top of DSA ports work regardless of which order
we link interfaces to their masters (first make the port join the LAG,
then the LAG join the bridge, or the other way around).
- To make DSA ports support non-offloaded LAG interfaces properly.
There was a design decision to be made in patches 2-4 on whether we
should adopt the "push" model, where the driver just calls:
switchdev_bridge_port_offloaded(brport_dev,
&atomic_notifier_block,
&blocking_notifier_block,
extack);
and the bridge just replays the entire collection of switchdev port
attributes and objects that it has, in some predefined order and with
some predefined error handling logic;
or the "pull" model, where the driver, apart from calling:
switchdev_bridge_port_offloaded(brport_dev, extack);
has the task of "dumpster diving" (as Tobias puts it) through the bridge
attributes and objects by itself, by calling:
- br_vlan_replay
- br_fdb_replay
- br_mdb_replay
- br_vlan_enabled
- br_port_flag_is_set
- br_port_get_stp_state
- br_multicast_router
- br_get_ageing_time
(not necessarily all of them, and not necessarily in this order, and
with driver-defined error handling).
Even though I'm not in love myself with the "pull" model, I chose it
because there is a fundamental trick with replaying switchdev events
like this:
ip link add br0 type bridge
ip link add bond0 type bond
ip link set bond0 master br0
ip link set swp0 master bond0 <- this will replay the objects once for
the bond0 bridge port, and the swp0
switchdev port will process them
ip link set swp1 master bond0 <- this will replay the objects again for
the bond0 bridge port, and the swp1
switchdev port will see them, but swp0
will see them for the second time now
Basically I believe that it is implementation defined whether the driver
wants to error out on switchdev objects seen twice on a port, and the
bridge should not enforce a certain model for that. For example, for FDB
entries added to a bonding interface, the underling switchdev driver
might have an abstraction for just that: an FDB entry pointing towards a
logical (as opposed to physical) port. So when the second port joins the
bridge, it doesn't realy need to replay FDB entries, since there is
already at least one hardware port which has been receiving those
events, and the FDB entries don't need to be added a second time to the
same logical port.
In the other corner, we have the drivers that handle switchdev port
attributes on a LAG as individual switchdev port attributes on physical
ports (example: VLAN filtering). In fact, the switchdev_handle_port_attr_set
helper facilitates this: it is a fan-out from a single orig_dev towards
multiple lowers that pass the check_cb().
But that's the point: switchdev_handle_port_attr_set is just a helper
which the driver _opts_ to use. The bridge can't enforce the "push"
model, because that would assume that all drivers handle port attributes
in the same way, which is probably false.
For this reason, I preferred to go with the "pull" mode for this patch
set. Just to see how bad it is for other switchdev drivers to copy-paste
this logic, I added the pull support to ocelot too, and I think it's
pretty manageable.
This patch set is RFC because it is minimally tested, and I would like
to get some feedback/agreement regarding the design decisions taken,
before I spend any more time on this.
There are also some things I probably broke, but I couldn't figure any
better. For example, I can't seem to figure out if mlxsw does the right
thing when joining a bonding interface that is already a bridge port.
I think it probably doesn't, so in that case, the placement I found for
the switchdev_bridge_port_offload() probably needs some adjustment when
there exists a LAG upper.
If possible, I would like the maintainers of the switchdev drivers to
tell me if this change introduces any regressions to how packets are
flooded (actually not flooded) in software by the bridge between two
ports belonging to the same ASIC ID.
I should mention that this patch series is written on top of Tobias'
series:
https://patchwork.kernel.org/project/netdevbpf/cover/20210318192540.895062-1-tobias@waldekranz.com/
which should get applied soon.
Vladimir Oltean (16):
net: dsa: call dsa_port_bridge_join when joining a LAG that is already
in a bridge
net: dsa: pass extack to dsa_port_{bridge,lag}_join
net: dsa: inherit the actual bridge port flags at join time
net: dsa: sync up with bridge port's STP state when joining
net: dsa: sync up VLAN filtering state when joining the bridge
net: dsa: sync multicast router state when joining the bridge
net: dsa: sync ageing time when joining the bridge
net: dsa: replay port and host-joined mdb entries when joining the
bridge
net: dsa: replay port and local fdb entries when joining the bridge
net: dsa: replay VLANs installed on port when joining the bridge
net: ocelot: support multiple bridges
net: ocelot: call ocelot_netdevice_bridge_join when joining a bridged
LAG
net: ocelot: replay switchdev events when joining bridge
net: dsa: don't set skb->offload_fwd_mark when not offloading the
bridge
net: dsa: return -EOPNOTSUPP when driver does not implement
.port_lag_join
net: bridge: switchdev: let drivers inform which bridge ports are
offloaded
drivers/net/dsa/ocelot/felix.c | 4 +-
.../ethernet/freescale/dpaa2/dpaa2-switch.c | 4 +-
.../marvell/prestera/prestera_switchdev.c | 7 +
.../mellanox/mlxsw/spectrum_switchdev.c | 4 +-
drivers/net/ethernet/mscc/ocelot.c | 90 ++++----
drivers/net/ethernet/mscc/ocelot_net.c | 210 +++++++++++++++---
drivers/net/ethernet/rocker/rocker_ofdpa.c | 8 +-
drivers/net/ethernet/ti/am65-cpsw-nuss.c | 7 +-
drivers/net/ethernet/ti/cpsw_new.c | 6 +-
include/linux/if_bridge.h | 56 +++++
include/net/switchdev.h | 1 +
include/soc/mscc/ocelot.h | 13 +-
net/bridge/br_fdb.c | 52 +++++
net/bridge/br_if.c | 11 +-
net/bridge/br_mdb.c | 84 +++++++
net/bridge/br_private.h | 8 +-
net/bridge/br_stp.c | 27 +++
net/bridge/br_switchdev.c | 94 +++++++-
net/bridge/br_vlan.c | 71 ++++++
net/dsa/dsa_priv.h | 23 +-
net/dsa/port.c | 201 +++++++++++++----
net/dsa/slave.c | 11 +-
net/dsa/switch.c | 4 +-
net/dsa/tag_brcm.c | 2 +-
net/dsa/tag_dsa.c | 15 +-
net/dsa/tag_hellcreek.c | 2 +-
net/dsa/tag_ksz.c | 2 +-
net/dsa/tag_lan9303.c | 3 +-
net/dsa/tag_mtk.c | 2 +-
net/dsa/tag_ocelot.c | 2 +-
net/dsa/tag_ocelot_8021q.c | 2 +-
net/dsa/tag_rtl4_a.c | 2 +-
net/dsa/tag_sja1105.c | 4 +-
net/dsa/tag_xrs700x.c | 2 +-
34 files changed, 845 insertions(+), 189 deletions(-)
--
2.25.1
From: Vladimir Oltean <olteanv@gmail.com> Date: 2021-03-18 23:19:28
From: Vladimir Oltean <vladimir.oltean@nxp.com>
This is a pretty noisy change that was broken out of the larger change
for replaying switchdev attributes and objects at bridge join time,
which is when these extack objects are actually used.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
net/dsa/dsa_priv.h | 6 ++++--
net/dsa/port.c | 8 +++++---
net/dsa/slave.c | 7 +++++--
3 files changed, 14 insertions(+), 7 deletions(-)
From: Vladimir Oltean <olteanv@gmail.com> Date: 2021-03-18 23:19:28
From: Vladimir Oltean <vladimir.oltean@nxp.com>
It may happen that we have the following topology:
ip link add br0 type bridge stp_state 1
ip link add bond0 type bond
ip link set bond0 master br0
ip link set swp0 master bond0
ip link set swp1 master bond0
STP decides that it should put bond0 into the BLOCKING state, and
that's that. The ports that are actively listening for the switchdev
port attributes emitted for the bond0 bridge port (because they are
offloading it) and have the honor of seeing that switchdev port
attribute can react to it, so we can program swp0 and swp1 into the
BLOCKING state.
But if then we do:
ip link set swp2 master bond0
then as far as the bridge is concerned, nothing has changed: it still
has one bridge port. But this new bridge port will not see any STP state
change notification and will remain FORWARDING, which is how the
standalone code leaves it in.
Add a function to the bridge which retrieves the current STP state, such
that drivers can synchronize to it when they may have missed switchdev
events.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
include/linux/if_bridge.h | 6 ++++++
net/bridge/br_stp.c | 14 ++++++++++++++
net/dsa/port.c | 7 +++++++
3 files changed, 27 insertions(+)
From: Vladimir Oltean <olteanv@gmail.com> Date: 2021-03-18 23:19:28
From: Vladimir Oltean <vladimir.oltean@nxp.com>
DSA can properly detect and offload this sequence of operations:
ip link add br0 type bridge
ip link add bond0 type bond
ip link set swp0 master bond0
ip link set bond0 master br0
But not this one:
ip link add br0 type bridge
ip link add bond0 type bond
ip link set bond0 master br0
ip link set swp0 master bond0
Actually the second one is more complicated, due to the elapsed time
between the enslavement of bond0 and the offloading of it via swp0, a
lot of things could have happened to the bond0 bridge port in terms of
switchdev objects (host MDBs, VLANs, altered STP state etc). So this is
a bit of a can of worms, and making sure that the DSA port's state is in
sync with this already existing bridge port is handled in the next
patches.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
net/dsa/port.c | 22 ++++++++++++++++++----
1 file changed, 18 insertions(+), 4 deletions(-)
From: Vladimir Oltean <olteanv@gmail.com> Date: 2021-03-18 23:19:28
From: Vladimir Oltean <vladimir.oltean@nxp.com>
DSA currently assumes that the bridge port starts off with this
constellation of bridge port flags:
- learning on
- unicast flooding on
- multicast flooding on
- broadcast flooding on
just by virtue of code copy-pasta from the bridge layer (new_nbp).
This was a simple enough strategy thus far, because the 'bridge join'
moment always coincided with the 'bridge port creation' moment.
But with sandwiched interfaces, such as:
br0
|
bond0
|
swp0
it may happen that the user has had time to change the bridge port flags
of bond0 before enslaving swp0 to it. In that case, swp0 will falsely
assume that the bridge port flags are those determined by new_nbp, when
in fact this can happen:
ip link add br0 type bridge
ip link add bond0 type bond
ip link set bond0 master br0
ip link set bond0 type bridge_slave learning off
ip link set swp0 master br0
Now swp0 has learning enabled, bond0 has learning disabled. Not nice.
Fix this by "dumpster diving" through the actual bridge port flags with
br_port_flag_is_set, at bridge join time.
We use this opportunity to split dsa_port_change_brport_flags into two
distinct functions called dsa_port_inherit_brport_flags and
dsa_port_clear_brport_flags, now that the implementation for the two
cases is no longer similar.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
net/dsa/port.c | 123 ++++++++++++++++++++++++++++++++-----------------
1 file changed, 82 insertions(+), 41 deletions(-)
@@ -122,26 +122,82 @@ void dsa_port_disable(struct dsa_port *dp)rtnl_unlock();}-staticvoiddsa_port_change_brport_flags(structdsa_port*dp,-boolbridge_offload)+staticvoiddsa_port_clear_brport_flags(structdsa_port*dp,+structnetlink_ext_ack*extack){structswitchdev_brport_flagsflags;-intflag;-flags.mask=BR_LEARNING|BR_FLOOD|BR_MCAST_FLOOD|BR_BCAST_FLOOD;-if(bridge_offload)-flags.val=flags.mask;-else-flags.val=flags.mask&~BR_LEARNING;+flags.mask=BR_LEARNING;+flags.val=0;+dsa_port_bridge_flags(dp,flags,extack);++flags.mask=BR_FLOOD;+flags.val=BR_FLOOD;+dsa_port_bridge_flags(dp,flags,extack);++flags.mask=BR_MCAST_FLOOD;+flags.val=BR_MCAST_FLOOD;+dsa_port_bridge_flags(dp,flags,extack);++flags.mask=BR_BCAST_FLOOD;+flags.val=BR_BCAST_FLOOD;+dsa_port_bridge_flags(dp,flags,extack);+}++staticintdsa_port_inherit_brport_flags(structdsa_port*dp,+structnetlink_ext_ack*extack)+{+constunsignedlongmask=BR_LEARNING|BR_FLOOD|BR_MCAST_FLOOD|+BR_BCAST_FLOOD;+structnet_device*brport_dev=dsa_port_to_bridge_port(dp);+intflag,err;++for_each_set_bit(flag,&mask,32){+structswitchdev_brport_flagsflags={0};-for_each_set_bit(flag,&flags.mask,32){-structswitchdev_brport_flagstmp;+flags.mask=BIT(flag);-tmp.val=flags.val&BIT(flag);-tmp.mask=BIT(flag);+if(br_port_flag_is_set(brport_dev,BIT(flag)))+flags.val=BIT(flag);-dsa_port_bridge_flags(dp,tmp,NULL);+err=dsa_port_bridge_flags(dp,flags,extack);+if(err&&err!=-EOPNOTSUPP)+returnerr;}++return0;+}++staticintdsa_port_switchdev_sync(structdsa_port*dp,+structnetlink_ext_ack*extack)+{+interr;++err=dsa_port_inherit_brport_flags(dp,extack);+if(err)+returnerr;++return0;+}++/* Configure the port for standalone mode (no address learning, flood+*everything,BR_STATE_FORWARDING,etc).+*ThebridgeonlyemitsSWITCHDEV_ATTR_ID_PORT_*eventswhentheuser+*requestsitthroughnetlinkorsysfs,butnotautomaticallyatport+*joinorleave,soweneedtohandleresettingthebrportflagsourselves.+*Butweevenpreferitthatway,becauseotherwise,somesetupsmightnever+*getthenotificationtheyneed,forexample,whenaportleavesaLAGthat+*offloadsthebridge,itbecomesstandalone,butasfarasthebridgeis+*concerned,noporteverleft.+*/+staticvoiddsa_port_switchdev_unsync(structdsa_port*dp)+{+dsa_port_clear_brport_flags(dp,NULL);++/* Port left the bridge, put in BR_STATE_DISABLED by the bridge layer,+*soallowittobeinBR_STATE_FORWARDINGtobekeptfunctional+*/+dsa_port_set_state_now(dp,BR_STATE_FORWARDING);}intdsa_port_bridge_join(structdsa_port*dp,structnet_device*br,
@@ -155,24 +211,25 @@ int dsa_port_bridge_join(struct dsa_port *dp, struct net_device *br,};interr;-/* Notify the port driver to set its configurable flags in a way that-*matchestheinitialsettingsofabridgeport.-*/-dsa_port_change_brport_flags(dp,true);-/* Here the interface is already bridged. Reflect the current*configurationsothatdriverscanprogramtheirchipsaccordingly.*/dp->bridge_dev=br;err=dsa_broadcast(DSA_NOTIFIER_BRIDGE_JOIN,&info);+if(err)+gotoout_rollback;-/* The bridging is rolled back on error */-if(err){-dsa_port_change_brport_flags(dp,false);-dp->bridge_dev=NULL;-}+err=dsa_port_switchdev_sync(dp,extack);+if(err)+gotoout_rollback_unbridge;+return0;++out_rollback_unbridge:+dsa_broadcast(DSA_NOTIFIER_BRIDGE_LEAVE,&info);+out_rollback:+dp->bridge_dev=NULL;returnerr;}
@@ -186,6 +243,8 @@ void dsa_port_bridge_leave(struct dsa_port *dp, struct net_device *br)};interr;+dsa_port_switchdev_unsync(dp);+/* Here the port is already unbridged. Reflect the current configuration*sothatdriverscanprogramtheirchipsaccordingly.*/
@@ -194,24 +253,6 @@ void dsa_port_bridge_leave(struct dsa_port *dp, struct net_device *br)err=dsa_broadcast(DSA_NOTIFIER_BRIDGE_LEAVE,&info);if(err)pr_err("DSA: failed to notify DSA_NOTIFIER_BRIDGE_LEAVE\n");--/* Configure the port for standalone mode (no address learning,-*floodeverything).-*ThebridgeonlyemitsSWITCHDEV_ATTR_ID_PORT_BRIDGE_FLAGSevents-*whentheuserrequestsitthroughnetlinkorsysfs,butnot-*automaticallyatportjoinorleave,soweneedtohandleresetting-*thebrportflagsourselves.Butweevenpreferitthatway,because-*otherwise,somesetupsmightnevergetthenotificationtheyneed,-*forexample,whenaportleavesaLAGthatoffloadsthebridge,-*itbecomesstandalone,butasfarasthebridgeisconcerned,no-*porteverleft.-*/-dsa_port_change_brport_flags(dp,false);--/* Port left the bridge, put in BR_STATE_DISABLED by the bridge layer,-*soallowittobeinBR_STATE_FORWARDINGtobekeptfunctional-*/-dsa_port_set_state_now(dp,BR_STATE_FORWARDING);}intdsa_port_lag_change(structdsa_port*dp,
From: Vladimir Oltean <olteanv@gmail.com> Date: 2021-03-18 23:19:58
From: Vladimir Oltean <vladimir.oltean@nxp.com>
This is the same situation as for other switchdev port attributes: if we
join an already-created bridge port, such as a bond master interface,
then we can miss the initial switchdev notification emitted by the
bridge for this port.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
net/dsa/port.c | 7 +++++++
1 file changed, 7 insertions(+)
From: Vladimir Oltean <olteanv@gmail.com> Date: 2021-03-18 23:19:58
From: Vladimir Oltean <vladimir.oltean@nxp.com>
Make sure that the multicast router setting of the bridge is picked up
correctly by DSA when joining, regardless of whether there are
sandwiched interfaces or not. The SWITCHDEV_ATTR_ID_BRIDGE_MROUTER port
attribute is only emitted from br_mc_router_state_change.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
net/dsa/port.c | 10 ++++++++++
1 file changed, 10 insertions(+)
@@ -189,6 +189,10 @@ static int dsa_port_switchdev_sync(struct dsa_port *dp,if(err&&err!=-EOPNOTSUPP)returnerr;+err=dsa_port_mrouter(dp->cpu_dp,br_multicast_router(br),extack);+if(err&&err!=-EOPNOTSUPP)+returnerr;+return0;}
@@ -212,6 +216,12 @@ static void dsa_port_switchdev_unsync(struct dsa_port *dp)dsa_port_set_state_now(dp,BR_STATE_FORWARDING);/* VLAN filtering is handled by dsa_switch_bridge_leave */++/* Some drivers treat the notification for having a local multicast+*routerbyallowingmulticasttobefloodedtotheCPU,soweshould+*allowthisinstandalonemodetoo.+*/+dsa_port_mrouter(dp->cpu_dp,true,NULL);}intdsa_port_bridge_join(structdsa_port*dp,structnet_device*br,
From: Vladimir Oltean <olteanv@gmail.com> Date: 2021-03-18 23:19:58
From: Vladimir Oltean <vladimir.oltean@nxp.com>
The SWITCHDEV_ATTR_ID_BRIDGE_AGEING_TIME attribute is only emitted from:
sysfs/ioctl/netlink
-> br_set_ageing_time
-> __set_ageing_time
therefore not at bridge port creation time, so:
(a) drivers had to hardcode the initial value for the address ageing time,
because they didn't get any notification
(b) that hardcoded value can be out of sync, if the user changes the
ageing time before enslaving the port to the bridge
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
include/linux/if_bridge.h | 6 ++++++
net/bridge/br_stp.c | 13 +++++++++++++
net/dsa/port.c | 10 ++++++++++
3 files changed, 29 insertions(+)
@@ -639,6 +639,19 @@ int br_set_ageing_time(struct net_bridge *br, clock_t ageing_time)return0;}+clock_tbr_get_ageing_time(structnet_device*br_dev)+{+structnet_bridge*br;++if(!netif_is_bridge_master(br_dev))+return0;++br=netdev_priv(br_dev);++returnjiffies_to_clock_t(br->ageing_time);+}+EXPORT_SYMBOL_GPL(br_get_ageing_time);+/* called under bridge lock */void__br_set_topology_change(structnet_bridge*br,unsignedcharval){
@@ -173,6 +173,7 @@ static int dsa_port_switchdev_sync(struct dsa_port *dp,{structnet_device*brport_dev=dsa_port_to_bridge_port(dp);structnet_device*br=dp->bridge_dev;+clock_tageing_time;u8stp_state;interr;
@@ -193,6 +194,11 @@ static int dsa_port_switchdev_sync(struct dsa_port *dp,if(err&&err!=-EOPNOTSUPP)returnerr;+ageing_time=br_get_ageing_time(br);+err=dsa_port_ageing_time(dp,ageing_time);+if(err&&err!=-EOPNOTSUPP)+returnerr;+return0;}
@@ -222,6 +228,10 @@ static void dsa_port_switchdev_unsync(struct dsa_port *dp)*allowthisinstandalonemodetoo.*/dsa_port_mrouter(dp->cpu_dp,true,NULL);++/* Ageing time may be global to the switch chip, so don't change it+*herebecausewehavenogoodreason(orvalue)tochangeitto.+*/}intdsa_port_bridge_join(structdsa_port*dp,structnet_device*br,
From: Vladimir Oltean <olteanv@gmail.com> Date: 2021-03-18 23:19:59
From: Vladimir Oltean <vladimir.oltean@nxp.com>
I have udhcpcd in my system and this is configured to bring interfaces
up as soon as they are created.
I create a bridge as follows:
ip link add br0 type bridge
As soon as I create the bridge and udhcpcd brings it up, I have some
other crap (avahi) that starts sending some random IPv6 packets to
advertise some local services, and from there, the br0 bridge joins the
following IPv6 groups:
33:33:ff:6d:c1:9c vid 0
33:33:00:00:00:6a vid 0
33:33:00:00:00:fb vid 0
br_dev_xmit
-> br_multicast_rcv
-> br_ip6_multicast_add_group
-> __br_multicast_add_group
-> br_multicast_host_join
-> br_mdb_notify
This is all fine, but inside br_mdb_notify we have br_mdb_switchdev_host
hooked up, and switchdev will attempt to offload the host joined groups
to an empty list of ports. Of course nobody offloads them.
Then when we add a port to br0:
ip link set swp0 master br0
the bridge doesn't replay the host-joined MDB entries from br_add_if,
and eventually the host joined addresses expire, and a switchdev
notification for deleting it is emitted, but surprise, the original
addition was already completely missed.
The strategy to address this problem is to replay the MDB entries (both
the port ones and the host joined ones) when the new port joins the
bridge, similar to what vxlan_fdb_replay does (in that case, its FDB can
be populated and only then attached to a bridge that you offload).
However there are 2 possibilities: the addresses can be 'pushed' by the
bridge into the port, or the port can 'pull' them from the bridge.
Considering that in the general case, the new port can be really late to
the party, and there may have been many other switchdev ports that
already received the initial notification, we would like to avoid
delivering duplicate events to them, since they might misbehave. And
currently, the bridge calls the entire switchdev notifier chain, whereas
for replaying it should just call the notifier block of the new guy.
But the bridge doesn't know what is the new guy's notifier block, it
just knows where the switchdev notifier chain is. So for simplification,
we make this a driver-initiated pull for now, and the notifier block is
passed as an argument.
To emulate the calling context for mdb objects (deferred and put on the
blocking notifier chain), we must iterate under RCU protection through
the bridge's mdb entries, queue them, and only call them once we're out
of the RCU read-side critical section.
Suggested-by: Ido Schimmel <redacted>
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
include/linux/if_bridge.h | 9 +++++
net/bridge/br_mdb.c | 84 +++++++++++++++++++++++++++++++++++++++
net/dsa/dsa_priv.h | 2 +
net/dsa/port.c | 6 +++
net/dsa/slave.c | 2 +-
5 files changed, 102 insertions(+), 1 deletion(-)
From: Vladimir Oltean <olteanv@gmail.com> Date: 2021-03-18 23:20:00
From: Vladimir Oltean <vladimir.oltean@nxp.com>
When a DSA port joins a LAG that already had an FDB entry pointing to it:
ip link set bond0 master br0
bridge fdb add dev bond0 00:01:02:03:04:05 master static
ip link set swp0 master bond0
the DSA port will have no idea that this FDB entry is there, because it
missed the switchdev event emitted at its creation.
Ido Schimmel pointed this out during a discussion about challenges with
switchdev offloading of stacked interfaces between the physical port and
the bridge, and recommended to just catch that condition and deny the
CHANGEUPPER event:
https://lore.kernel.org/netdev/20210210105949.GB287766@shredder.lan/
But in fact, we might need to deal with the hard thing anyway, which is
to replay all FDB addresses relevant to this port, because it isn't just
static FDB entries, but also local addresses (ones that are not
forwarded but terminated by the bridge). There, we can't just say 'oh
yeah, there was an upper already so I'm not joining that'.
So, similar to the logic for replaying MDB entries, add a function that
must be called by individual switchdev drivers and replays local FDB
entries as well as ones pointing towards a bridge port. This time, we
use the atomic switchdev notifier block, since that's what FDB entries
expect for some reason.
Reported-by: Ido Schimmel <redacted>
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
include/linux/if_bridge.h | 9 +++++++
include/net/switchdev.h | 1 +
net/bridge/br_fdb.c | 52 +++++++++++++++++++++++++++++++++++++++
net/dsa/dsa_priv.h | 1 +
net/dsa/port.c | 4 +++
net/dsa/slave.c | 2 +-
6 files changed, 68 insertions(+), 1 deletion(-)
@@ -205,6 +205,7 @@ struct switchdev_notifier_info {structswitchdev_notifier_fdb_info{structswitchdev_notifier_infoinfo;/* must be first */+structlist_headlist;constunsignedchar*addr;u16vid;u8added_by_user:1,
From: Vladimir Oltean <olteanv@gmail.com> Date: 2021-03-18 23:20:00
From: Vladimir Oltean <vladimir.oltean@nxp.com>
Currently this simple setup:
ip link add br0 type bridge vlan_filtering 1
ip link add bond0 type bond
ip link set bond0 master br0
ip link set swp0 master bond0
will not work because the bridge has created the PVID in br_add_if ->
nbp_vlan_init, and it has notified switchdev of the existence of VLAN 1,
but that was too early, since swp0 was not yet a lower of bond0, so it
had no reason to act upon that notification.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
include/linux/if_bridge.h | 10 ++++++
net/bridge/br_vlan.c | 71 +++++++++++++++++++++++++++++++++++++++
net/dsa/port.c | 6 ++++
3 files changed, 87 insertions(+)
@@ -1751,6 +1751,77 @@ void br_vlan_notify(const struct net_bridge *br,kfree_skb(skb);}+staticintbr_vlan_replay_one(structnotifier_block*nb,+structnet_device*dev,+structswitchdev_obj_port_vlan*vlan,+structnetlink_ext_ack*extack)+{+structswitchdev_notifier_port_obj_infoobj_info={+.info={+.dev=dev,+.extack=extack,+},+.obj=&vlan->obj,+};+interr;++err=nb->notifier_call(nb,SWITCHDEV_PORT_OBJ_ADD,&obj_info);+returnnotifier_to_errno(err);+}++intbr_vlan_replay(structnet_device*br_dev,structnet_device*dev,+structnotifier_block*nb,structnetlink_ext_ack*extack)+{+structnet_bridge_vlan_group*vg;+structnet_bridge_vlan*v;+structnet_bridge_port*p;+structnet_bridge*br;+interr=0;+u16pvid;++ASSERT_RTNL();++if(!netif_is_bridge_master(br_dev))+return-EINVAL;++if(!netif_is_bridge_master(dev)&&!netif_is_bridge_port(dev))+return-EINVAL;++if(netif_is_bridge_master(dev)){+br=netdev_priv(dev);+vg=br_vlan_group(br);+p=NULL;+}else{+p=br_port_get_rtnl(dev);+if(WARN_ON(!p))+return-EINVAL;+vg=nbp_vlan_group(p);+br=p->br;+}++if(!vg)+return0;++pvid=br_get_pvid(vg);++list_for_each_entry(v,&vg->vlan_list,vlist){+structswitchdev_obj_port_vlanvlan={+.obj.orig_dev=dev,+.obj.id=SWITCHDEV_OBJ_ID_PORT_VLAN,+.flags=br_vlan_flags(v,pvid),+.vid=v->vid,+};++if(!br_vlan_should_use(v))+continue;++br_vlan_replay_one(nb,dev,&vlan,extack);+if(err)+returnerr;+}++returnerr;+}/* check if v_curr can enter a range ending in range_end */boolbr_vlan_can_enter_range(conststructnet_bridge_vlan*v_curr,conststructnet_bridge_vlan*range_end)
From: Vladimir Oltean <olteanv@gmail.com> Date: 2021-03-18 23:20:00
From: Vladimir Oltean <vladimir.oltean@nxp.com>
The ocelot switches are a bit odd in that they do not have an STP state
to put the ports into. Instead, the forwarding configuration is delayed
from the typical port_bridge_join into stp_state_set, when the port enters
the BR_STATE_FORWARDING state.
I can only guess that the implementation of this quirk is the reason that
led to the simplification of the driver such that only one bridge could
be offloaded at a time.
We can simplify the data structures somewhat, and introduce a per-port
bridge device pointer and STP state, similar to how the LAG offload
works now (there we have a per-port bonding device pointer and TX
enabled state). This allows offloading multiple bridges with relative
ease, while still keeping in place the quirk to delay the programming of
the PGIDs.
We actually need this change now because we need to remove the bogus
restriction from ocelot_bridge_stp_state_set that ocelot->bridge_mask
needs to contain BIT(port), otherwise that function is a no-op.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
drivers/net/ethernet/mscc/ocelot.c | 72 +++++++++++++++---------------
include/soc/mscc/ocelot.h | 7 ++-
2 files changed, 39 insertions(+), 40 deletions(-)
@@ -766,7 +766,7 @@ int ocelot_xtr_poll_frame(struct ocelot *ocelot, int grp, struct sk_buff **nskb)/* Everything we see on an interface that is in the HW bridge*hasalreadybeenforwarded.*/-if(ocelot->bridge_mask&BIT(src_port))+if(ocelot->ports[src_port]->bridge)skb->offload_fwd_mark=1;skb->protocol=eth_type_trans(skb,dev);
@@ -1508,16 +1517,9 @@ EXPORT_SYMBOL(ocelot_port_mdb_del);intocelot_port_bridge_join(structocelot*ocelot,intport,structnet_device*bridge){-if(!ocelot->bridge_mask){-ocelot->hw_bridge_dev=bridge;-}else{-if(ocelot->hw_bridge_dev!=bridge)-/* This is adding the port to a second bridge, this is-*unsupported*/-return-ENODEV;-}+structocelot_port*ocelot_port=ocelot->ports[port];-ocelot->bridge_mask|=BIT(port);+ocelot_port->bridge=bridge;return0;}
From: Vladimir Oltean <olteanv@gmail.com> Date: 2021-03-18 23:20:00
From: Vladimir Oltean <vladimir.oltean@nxp.com>
DSA has gained the recent ability to deal gracefully with upper
interfaces it cannot offload, such as the bridge, bonding or team
drivers. When such uppers exist, the ports are still in standalone mode
as far as the hardware is concerned.
But when we deliver packets to the software bridge in order for that to
do the forwarding, there is an unpleasant surprise in that the bridge
will refuse to forward them. This is because we unconditionally set
skb->offload_fwd_mark = true, meaning that the bridge thinks the frames
were already forwarded in hardware by us.
Since dp->bridge_dev is populated only when there is hardware offload
for it, but not in the software fallback case, let's introduce a new
helper that can be called from the tagger data path which sets the
skb->offload_fwd_mark accordingly to zero when there is no hardware
offload for bridging. This lets the bridge forward packets back to other
interfaces of our switch, if needed.
Without this change, sending a packet to the CPU for an unoffloaded
interface triggers this WARN_ON:
void nbp_switchdev_frame_mark(const struct net_bridge_port *p,
struct sk_buff *skb)
{
if (skb->offload_fwd_mark && !WARN_ON_ONCE(!p->offload_fwd_mark))
BR_INPUT_SKB_CB(skb)->offload_fwd_mark = p->offload_fwd_mark;
}
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
Reviewed-by: Tobias Waldekranz <tobias@waldekranz.com>
---
net/dsa/dsa_priv.h | 14 ++++++++++++++
net/dsa/tag_brcm.c | 2 +-
net/dsa/tag_dsa.c | 15 +++++++++++----
net/dsa/tag_hellcreek.c | 2 +-
net/dsa/tag_ksz.c | 2 +-
net/dsa/tag_lan9303.c | 3 ++-
net/dsa/tag_mtk.c | 2 +-
net/dsa/tag_ocelot.c | 2 +-
net/dsa/tag_ocelot_8021q.c | 2 +-
net/dsa/tag_rtl4_a.c | 2 +-
net/dsa/tag_sja1105.c | 4 ++--
net/dsa/tag_xrs700x.c | 2 +-
12 files changed, 37 insertions(+), 15 deletions(-)
@@ -349,6 +349,20 @@ static inline struct sk_buff *dsa_untag_bridge_pvid(struct sk_buff *skb)returnskb;}+/* If the ingress port offloads the bridge, we mark the frame as autonomously+*forwardedbyhardware,sothesoftwarebridgedoesn'tforwardintwice,back+*tous,becausewealreadydid.However,ifwe'reinfallbackmodeandwedo+*softwarebridging,wearenotoffloadingit,thereforethedp->bridge_dev+*pointerisnotpopulated,andfloodingneedstobedonebysoftware(weare+*effectivelyoperatinginstandaloneportsmode).+*/+staticinlinevoiddsa_default_offload_fwd_mark(structsk_buff*skb)+{+structdsa_port*dp=dsa_slave_to_port(skb->dev);++skb->offload_fwd_mark=!!(dp->bridge_dev);+}+/* switch.c */intdsa_switch_register_notifier(structdsa_switch*ds);voiddsa_switch_unregister_notifier(structdsa_switch*ds);
@@ -202,6 +199,7 @@ static struct sk_buff *dsa_rcv_ll(struct sk_buff *skb, struct net_device *dev,/* Traps have, by definition, not been*forwardedbyhardware,sodon'tmarkthem.*/+trap=true;break;default:/* Reserved code, this could be anything. Drop
@@ -235,6 +233,15 @@ static struct sk_buff *dsa_rcv_ll(struct sk_buff *skb, struct net_device *dev,if(!skb->dev)returnNULL;+/* When using LAG offload, skb->dev is not a DSA slave interface,+*sowecannotcalldsa_default_offload_fwd_markandweneedto+*special-caseit.+*/+if(trunk)+skb->offload_fwd_mark=true;+elseif(!trap)+dsa_default_offload_fwd_mark(skb);+/* If the 'tagged' bit is set; convert the DSA tag to a 802.1Q*tag,anddeletetheethertype(extra)ifapplicable.Ifthe*'tagged'bitiscleared;deletetheDSAtag,andethertype
@@ -123,7 +123,7 @@ static struct sk_buff *ocelot_rcv(struct sk_buff *skb,*/returnNULL;-skb->offload_fwd_mark=1;+dsa_default_offload_fwd_mark(skb);skb->priority=qos_class;/* Ocelot switches copy frames unmodified to the CPU. However, it is
From: Vladimir Oltean <olteanv@gmail.com> Date: 2021-03-18 23:20:00
From: Vladimir Oltean <vladimir.oltean@nxp.com>
The DSA core has a layered structure, and even though we end up
returning 0 (success) to user space when setting a bonding/team upper
that can't be offloaded, some parts of the framework actually need to
know that we couldn't offload that.
For example, if dsa_switch_lag_join returns 0 as it currently does,
dsa_port_lag_join has no way to tell a successful offload from a
software fallback, and it will call dsa_port_bridge_join afterwards.
Then we'll think we're offloading the bridge master of the LAG, when in
fact we're not even offloading the LAG. In turn, this will make us set
skb->offload_fwd_mark = true, which is incorrect and the bridge doesn't
like it.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
net/dsa/switch.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
From: Vladimir Oltean <olteanv@gmail.com> Date: 2021-03-18 23:20:00
From: Vladimir Oltean <vladimir.oltean@nxp.com>
The premise of this change is that the switchdev port attributes and
objects offloaded by ocelot might have been missed when we are joining
an already existing bridge port, such as a bonding interface.
The patch pulls these switchdev attributes and objects from the bridge,
on behalf of the 'bridge port' net device which might be either the
ocelot switch interface, or the bonding upper interface.
The ocelot_net.c belongs strictly to the switchdev ocelot driver, while
ocelot.c is part of a library shared with the DSA felix driver.
The ocelot_port_bridge_leave function (part of the common library) used
to call ocelot_port_vlan_filtering(false), something which is not
necessary for DSA, since the framework deals with that already there.
So we move this function to ocelot_switchdev_unsync, which is specific
to the switchdev driver.
The code movement described above makes ocelot_port_bridge_leave no
longer return an error code, so we change its type from int to void.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
drivers/net/dsa/ocelot/felix.c | 4 +-
drivers/net/ethernet/mscc/ocelot.c | 18 ++--
drivers/net/ethernet/mscc/ocelot_net.c | 117 +++++++++++++++++++++----
include/soc/mscc/ocelot.h | 6 +-
4 files changed, 111 insertions(+), 34 deletions(-)
@@ -1514,34 +1514,28 @@ int ocelot_port_mdb_del(struct ocelot *ocelot, int port,}EXPORT_SYMBOL(ocelot_port_mdb_del);-intocelot_port_bridge_join(structocelot*ocelot,intport,-structnet_device*bridge)+voidocelot_port_bridge_join(structocelot*ocelot,intport,+structnet_device*bridge){structocelot_port*ocelot_port=ocelot->ports[port];ocelot_port->bridge=bridge;-return0;+ocelot_apply_bridge_fwd_mask(ocelot);}EXPORT_SYMBOL(ocelot_port_bridge_join);-intocelot_port_bridge_leave(structocelot*ocelot,intport,-structnet_device*bridge)+voidocelot_port_bridge_leave(structocelot*ocelot,intport,+structnet_device*bridge){structocelot_port*ocelot_port=ocelot->ports[port];structocelot_vlanpvid={0},native_vlan={0};-intret;ocelot_port->bridge=NULL;-ret=ocelot_port_vlan_filtering(ocelot,port,false);-if(ret)-returnret;-ocelot_port_set_pvid(ocelot,port,pvid);ocelot_port_set_native_vlan(ocelot,port,native_vlan);--return0;+ocelot_apply_bridge_fwd_mask(ocelot);}EXPORT_SYMBOL(ocelot_port_bridge_leave);
@@ -803,10 +803,10 @@ int ocelot_port_pre_bridge_flags(struct ocelot *ocelot, int port,structswitchdev_brport_flagsval);voidocelot_port_bridge_flags(structocelot*ocelot,intport,structswitchdev_brport_flagsval);-intocelot_port_bridge_join(structocelot*ocelot,intport,-structnet_device*bridge);-intocelot_port_bridge_leave(structocelot*ocelot,intport,+voidocelot_port_bridge_join(structocelot*ocelot,intport,structnet_device*bridge);+voidocelot_port_bridge_leave(structocelot*ocelot,intport,+structnet_device*bridge);intocelot_fdb_dump(structocelot*ocelot,intport,dsa_fdb_dump_cb_t*cb,void*data);intocelot_fdb_add(structocelot*ocelot,intport,
From: Vladimir Oltean <olteanv@gmail.com> Date: 2021-03-18 23:20:00
From: Vladimir Oltean <vladimir.oltean@nxp.com>
On reception of an skb, the bridge checks if it was marked as 'already
forwarded in hardware' (checks if skb->offload_fwd_mark == 1), and if it
is, it puts a mark of its own on that skb, with the switchdev mark of
the ingress port. Then during forwarding, it enforces that the egress
port must have a different switchdev mark than the ingress one (this is
done in nbp_switchdev_allowed_egress).
Non-switchdev drivers don't report any physical switch id (neither
through devlink nor .ndo_get_port_parent_id), therefore the bridge
assigns them a switchdev mark of 0, and packets coming from them will
always have skb->offload_fwd_mark = 0. So there aren't any restrictions.
Problems appear due to the fact that DSA would like to perform software
fallback for bonding and team interfaces that the physical switch cannot
offload.
+-- br0 -+
/ / | \
/ / | \
/ / | \
/ / | \
/ / | \
/ | | bond0
/ | | / \
swp0 swp1 swp2 swp3 swp4
There, it is desirable that the presence of swp3 and swp4 under a
non-offloaded LAG does not preclude us from doing hardware bridging
beteen swp0, swp1 and swp2. The bandwidth of the CPU is often times high
enough that software bridging between {swp0,swp1,swp2} and bond0 is not
impractical.
But this creates an impossible paradox given the current way in which
port switchdev marks are assigned. When the driver receives a packet
from swp0 (say, due to flooding), it must set skb->offload_fwd_mark to
something.
- If we set it to 0, then the bridge will forward it towards swp1, swp2
and bond0. But the switch has already forwarded it towards swp1 and
swp2 (not to bond0, remember, that isn't offloaded, so as far as the
switch is concerned, ports swp3 and swp4 are not looking up the FDB,
and the entire bond0 is a destination that is strictly behind the
CPU). But we don't want duplicated traffic towards swp1 and swp2, so
it's not ok to set skb->offload_fwd_mark = 0.
- If we set it to 1, then the bridge will not forward the skb towards
the ports with the same switchdev mark, i.e. not to swp1, swp2 and
bond0. Towards swp1 and swp2 that's ok, but towards bond0? It should
have forwarded the skb there.
So the real issue is that bond0 will be assigned the same switchdev mark
as {swp0,swp1,swp2}, because the function that assigns switchdev marks
to bridge ports, nbp_switchdev_mark_set, recurses through bond0's lower
interfaces until it finds something that implements devlink.
A solution is to give the bridge explicit hints as to what switchdev
mark it should use for each port.
Currently, the bridging offload is very 'silent': a driver registers a
netdevice notifier, which is put on the netns's notifier chain, and
which sniffs around for NETDEV_CHANGEUPPER events where the upper is a
bridge, and the lower is an interface it knows about (one registered by
this driver, normally). Then, from within that notifier, it does a bunch
of stuff behind the bridge's back, without the bridge necessarily
knowing that there's somebody offloading that port. It looks like this:
ip link set swp0 master br0
|
v
bridge calls netdev_master_upper_dev_link
|
v
call_netdevice_notifiers
|
v
dsa_slave_netdevice_event
|
v
oh, hey! it's for me!
|
v
.port_bridge_join
What we do to solve the conundrum is to be less silent, and emit a
notification back. Something like this:
ip link set swp0 master br0
|
v
bridge calls netdev_master_upper_dev_link
|
v bridge: Aye! I'll use this
call_netdevice_notifiers ^ ppid as the
| | switchdev mark for
v | this port, and zero
dsa_slave_netdevice_event | if I got nothing.
| |
v |
oh, hey! it's for me! |
| |
v |
.port_bridge_join |
| |
+------------------------+
switchdev_bridge_port_offload(swp0)
Then stacked interfaces (like bond0 on top of swp3/swp4) would be
treated differently in DSA, depending on whether we can or cannot
offload them.
The offload case:
ip link set bond0 master br0
|
v
bridge calls netdev_master_upper_dev_link
|
v bridge: Aye! I'll use this
call_netdevice_notifiers ^ ppid as the
| | switchdev mark for
v | bond0.
dsa_slave_netdevice_event | Coincidentally (or not),
| | bond0 and swp0, swp1, swp2
v | all have the same switchdev
hmm, it's not quite for me, | mark now, since the ASIC
but my driver has already | is able to forward towards
called .port_lag_join | all these ports in hw.
for it, because I have |
a port with dp->lag_dev == bond0. |
| |
v |
.port_bridge_join |
for swp3 and swp4 |
| |
+------------------------+
switchdev_bridge_port_offload(bond0)
And the non-offload case:
ip link set bond0 master br0
|
v
bridge calls netdev_master_upper_dev_link
|
v bridge waiting:
call_netdevice_notifiers ^ huh, switchdev_bridge_port_offload
| | wasn't called, okay, I'll use a
v | switchdev mark of zero for this one.
dsa_slave_netdevice_event : Then packets received on swp0 will
| : not be forwarded towards swp1, but
v : they will towards bond0.
it's not for me, but
bond0 is an upper of swp3
and swp4, but their dp->lag_dev
is NULL because they couldn't
offload it.
Basically we can draw the conclusion that the lowers of a bridge port
can come and go, so depending on the configuration of lowers for a
bridge port, it can dynamically toggle between offloaded and unoffloaded.
Therefore, we need an equivalent switchdev_bridge_port_unoffload too.
This patch changes the way any switchdev driver interacts with the
bridge. From now on, everybody needs to call switchdev_bridge_port_offload,
otherwise the bridge will treat the port as non-offloaded and allow
software flooding to other ports from the same ASIC.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
.../ethernet/freescale/dpaa2/dpaa2-switch.c | 4 +-
.../marvell/prestera/prestera_switchdev.c | 7 ++
.../mellanox/mlxsw/spectrum_switchdev.c | 4 +-
drivers/net/ethernet/mscc/ocelot_net.c | 4 +-
drivers/net/ethernet/rocker/rocker_ofdpa.c | 8 +-
drivers/net/ethernet/ti/am65-cpsw-nuss.c | 7 +-
drivers/net/ethernet/ti/cpsw_new.c | 6 +-
include/linux/if_bridge.h | 16 ++++
net/bridge/br_if.c | 11 +--
net/bridge/br_private.h | 8 +-
net/bridge/br_switchdev.c | 94 ++++++++++++++++---
11 files changed, 138 insertions(+), 31 deletions(-)
@@ -1518,7 +1518,7 @@ static int dpaa2_switch_port_bridge_join(struct net_device *netdev,if(err)gotoerr_egress_flood;-return0;+returnswitchdev_bridge_port_offload(netdev,NULL);err_egress_flood:dpaa2_switch_port_set_fdb(port_priv,NULL);
@@ -1552,6 +1552,8 @@ static int dpaa2_switch_port_bridge_leave(struct net_device *netdev)structethsw_core*ethsw=port_priv->ethsw_data;interr;+switchdev_bridge_port_unoffload(netdev);+/* First of all, fast age any learn FDB addresses on this switch port */dpaa2_switch_port_fast_age(port_priv);
@@ -671,13 +667,13 @@ int br_add_if(struct net_bridge *br, struct net_device *dev,*/err=dev_pre_changeaddr_notify(br->dev,dev->dev_addr,extack);if(err)-gotoerr7;+gotoerr6;}err=nbp_vlan_init(p,extack);if(err){netdev_err(dev,"failed to initialize vlan filtering on this port\n");-gotoerr7;+gotoerr6;}spin_lock_bh(&br->lock);
@@ -8,37 +8,109 @@#include"br_private.h"-staticintbr_switchdev_mark_get(structnet_bridge*br,structnet_device*dev)+staticintbr_switchdev_mark_get(structnet_bridge*br,+structnet_bridge_port*new_nbp){structnet_bridge_port*p;/* dev is yet to be added to the port list. */list_for_each_entry(p,&br->port_list,list){-if(netdev_port_same_parent_id(dev,p->dev))+if(!p->offload_count)+continue;++if(netdev_phys_item_id_same(&p->ppid,&new_nbp->ppid))returnp->offload_fwd_mark;}return++br->offload_fwd_mark;}-intnbp_switchdev_mark_set(structnet_bridge_port*p)+staticintnbp_switchdev_mark_set(structnet_bridge_port*p,+structnetdev_phys_item_idppid,+structnetlink_ext_ack*extack)+{+if(p->offload_count){+/* Prevent unsupported configurations such as a bridge port+*whichisabondinginterface,andthememberportsarefrom+*differenthardwareswitches.+*/+if(!netdev_phys_item_id_same(&p->ppid,&ppid)){+NL_SET_ERR_MSG_MOD(extack,+"Same bridge port cannot be offloaded by two physical switches");+return-EBUSY;+}+/* Be tolerant with drivers that call SWITCHDEV_BRPORT_OFFLOADED+*morethanonceforthesamebridgeport,suchaswhenthe+*bridgeportisanoffloadedbonding/teaminterface.+*/+p->offload_count++;+return0;+}++p->ppid=ppid;+p->offload_count=1;+p->offload_fwd_mark=br_switchdev_mark_get(p->br,p);++return0;+}++staticvoidnbp_switchdev_mark_clear(structnet_bridge_port*p,+structnetdev_phys_item_idppid)+{+if(WARN_ON(!netdev_phys_item_id_same(&p->ppid,&ppid)))+return;+if(WARN_ON(!p->offload_count))+return;++p->offload_count--;+if(p->offload_count)+return;++p->offload_fwd_mark=0;+}++/* Let the bridge know that this port is offloaded, so that it can use+*theportparentidobtainedbyrecursiontodeterminethebridge+*port'sswitchdevmark.+*/+intswitchdev_bridge_port_offload(structnet_device*dev,+structnetlink_ext_ack*extack){-structnetdev_phys_item_idppid={};+structnetdev_phys_item_idppid;+structnet_bridge_port*p;interr;-ASSERT_RTNL();+p=br_port_get_rtnl(dev);+if(!p)+return-ENODEV;-err=dev_get_port_parent_id(p->dev,&ppid,true);-if(err){-if(err==-EOPNOTSUPP)-return0;+err=dev_get_port_parent_id(dev,&ppid,true);+if(err)+returnerr;++returnnbp_switchdev_mark_set(p,ppid,extack);+}+EXPORT_SYMBOL_GPL(switchdev_bridge_port_offload);++intswitchdev_bridge_port_unoffload(structnet_device*dev)+{+structnetdev_phys_item_idppid;+structnet_bridge_port*p;+interr;++p=br_port_get_rtnl(dev);+if(!p)+return-ENODEV;++err=dev_get_port_parent_id(dev,&ppid,true);+if(err)returnerr;-}-p->offload_fwd_mark=br_switchdev_mark_get(p->br,p->dev);+nbp_switchdev_mark_clear(p,ppid);return0;}+EXPORT_SYMBOL_GPL(switchdev_bridge_port_unoffload);voidnbp_switchdev_frame_mark(conststructnet_bridge_port*p,structsk_buff*skb)
From: Vladimir Oltean <olteanv@gmail.com> Date: 2021-03-18 23:20:00
From: Vladimir Oltean <vladimir.oltean@nxp.com>
Similar to the DSA situation, ocelot supports LAG offload but treats
this scenario improperly:
ip link add br0 type bridge
ip link add bond0 type bond
ip link set bond0 master br0
ip link set swp0 master bond0
We do the same thing as we do there, which is to simulate a 'bridge join'
on 'lag join', if we detect that the bonding upper has a bridge upper.
Again, same as DSA, ocelot supports software fallback for LAG, and in
that case, we should avoid calling ocelot_netdevice_changeupper.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
drivers/net/ethernet/mscc/ocelot_net.c | 111 +++++++++++++++++++------
1 file changed, 86 insertions(+), 25 deletions(-)
@@ -1117,10 +1117,15 @@ static int ocelot_port_obj_del(struct net_device *dev,returnret;}-staticintocelot_netdevice_bridge_join(structocelot*ocelot,intport,-structnet_device*bridge)+staticintocelot_netdevice_bridge_join(structnet_device*dev,+structnet_device*bridge,+structnetlink_ext_ack*extack){+structocelot_port_private*priv=netdev_priv(dev);+structocelot_port*ocelot_port=&priv->port;+structocelot*ocelot=ocelot_port->ocelot;structswitchdev_brport_flagsflags;+intport=priv->chip_port;interr;flags.mask=BR_LEARNING|BR_FLOOD|BR_MCAST_FLOOD|BR_BCAST_FLOOD;
@@ -1135,10 +1140,14 @@ static int ocelot_netdevice_bridge_join(struct ocelot *ocelot, int port,return0;}-staticintocelot_netdevice_bridge_leave(structocelot*ocelot,intport,+staticintocelot_netdevice_bridge_leave(structnet_device*dev,structnet_device*bridge){+structocelot_port_private*priv=netdev_priv(dev);+structocelot_port*ocelot_port=&priv->port;+structocelot*ocelot=ocelot_port->ocelot;structswitchdev_brport_flagsflags;+intport=priv->chip_port;interr;flags.mask=BR_LEARNING|BR_FLOOD|BR_MCAST_FLOOD|BR_BCAST_FLOOD;
@@ -1151,43 +1160,89 @@ static int ocelot_netdevice_bridge_leave(struct ocelot *ocelot, int port,returnerr;}-staticintocelot_netdevice_changeupper(structnet_device*dev,-structnetdev_notifier_changeupper_info*info)+staticintocelot_netdevice_lag_join(structnet_device*dev,+structnet_device*bond,+structnetdev_lag_upper_info*info,+structnetlink_ext_ack*extack){structocelot_port_private*priv=netdev_priv(dev);structocelot_port*ocelot_port=&priv->port;structocelot*ocelot=ocelot_port->ocelot;+structnet_device*bridge_dev;intport=priv->chip_port;+interr;++err=ocelot_port_lag_join(ocelot,port,bond,info);+if(err==-EOPNOTSUPP){+NL_SET_ERR_MSG_MOD(extack,"Offloading not supported");+return0;+}++bridge_dev=netdev_master_upper_dev_get(bond);+if(!bridge_dev||!netif_is_bridge_master(bridge_dev))+return0;++err=ocelot_netdevice_bridge_join(dev,bridge_dev,extack);+if(err)+gotoerr_bridge_join;++return0;++err_bridge_join:+ocelot_port_lag_leave(ocelot,port,bond);+returnerr;+}++staticintocelot_netdevice_lag_leave(structnet_device*dev,+structnet_device*bond)+{+structocelot_port_private*priv=netdev_priv(dev);+structocelot_port*ocelot_port=&priv->port;+structocelot*ocelot=ocelot_port->ocelot;+structnet_device*bridge_dev;+intport=priv->chip_port;++ocelot_port_lag_leave(ocelot,port,bond);++bridge_dev=netdev_master_upper_dev_get(bond);+if(!bridge_dev||!netif_is_bridge_master(bridge_dev))+return0;++returnocelot_netdevice_bridge_leave(dev,bridge_dev);+}++staticintocelot_netdevice_changeupper(structnet_device*dev,+structnetdev_notifier_changeupper_info*info)+{+structnetlink_ext_ack*extack;interr=0;+extack=netdev_notifier_info_to_extack(&info->info);+if(netif_is_bridge_master(info->upper_dev)){-if(info->linking){-err=ocelot_netdevice_bridge_join(ocelot,port,-info->upper_dev);-}else{-err=ocelot_netdevice_bridge_leave(ocelot,port,-info->upper_dev);-}+if(info->linking)+err=ocelot_netdevice_bridge_join(dev,info->upper_dev,+extack);+else+err=ocelot_netdevice_bridge_leave(dev,info->upper_dev);}if(netif_is_lag_master(info->upper_dev)){-if(info->linking){-err=ocelot_port_lag_join(ocelot,port,-info->upper_dev,-info->upper_info);-if(err==-EOPNOTSUPP){-NL_SET_ERR_MSG_MOD(info->info.extack,-"Offloading not supported");-err=0;-}-}else{-ocelot_port_lag_leave(ocelot,port,-info->upper_dev);-}+if(info->linking)+err=ocelot_netdevice_lag_join(dev,info->upper_dev,+info->upper_info,extack);+else+ocelot_netdevice_lag_leave(dev,info->upper_dev);}returnnotifier_from_errno(err);}+/* Treat CHANGEUPPER events on an offloaded LAG as individual CHANGEUPPER+*eventsforthelowerphysicalportsoftheLAG.+*IftheLAGupperisn'toffloaded,ignoreitsCHANGEUPPERevents.+*IncasetheLAGjoinedabridge,notifythatweareoffloadingitandcando+*forwardinginhardwaretowardsit.+*/staticintocelot_netdevice_lag_changeupper(structnet_device*dev,structnetdev_notifier_changeupper_info*info)
From: DENG Qingfang <dqfext@gmail.com> Date: 2021-03-19 08:53:23
On Fri, Mar 19, 2021 at 01:18:27AM +0200, Vladimir Oltean wrote:
quoted hunk
From: Vladimir Oltean <vladimir.oltean@nxp.com>
DSA has gained the recent ability to deal gracefully with upper
interfaces it cannot offload, such as the bridge, bonding or team
drivers. When such uppers exist, the ports are still in standalone mode
as far as the hardware is concerned.
But when we deliver packets to the software bridge in order for that to
do the forwarding, there is an unpleasant surprise in that the bridge
will refuse to forward them. This is because we unconditionally set
skb->offload_fwd_mark = true, meaning that the bridge thinks the frames
were already forwarded in hardware by us.
Since dp->bridge_dev is populated only when there is hardware offload
for it, but not in the software fallback case, let's introduce a new
helper that can be called from the tagger data path which sets the
skb->offload_fwd_mark accordingly to zero when there is no hardware
offload for bridging. This lets the bridge forward packets back to other
interfaces of our switch, if needed.
Without this change, sending a packet to the CPU for an unoffloaded
interface triggers this WARN_ON:
void nbp_switchdev_frame_mark(const struct net_bridge_port *p,
struct sk_buff *skb)
{
if (skb->offload_fwd_mark && !WARN_ON_ONCE(!p->offload_fwd_mark))
BR_INPUT_SKB_CB(skb)->offload_fwd_mark = p->offload_fwd_mark;
}
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
Reviewed-by: Tobias Waldekranz <tobias@waldekranz.com>
---
net/dsa/dsa_priv.h | 14 ++++++++++++++
net/dsa/tag_brcm.c | 2 +-
net/dsa/tag_dsa.c | 15 +++++++++++----
net/dsa/tag_hellcreek.c | 2 +-
net/dsa/tag_ksz.c | 2 +-
net/dsa/tag_lan9303.c | 3 ++-
net/dsa/tag_mtk.c | 2 +-
net/dsa/tag_ocelot.c | 2 +-
net/dsa/tag_ocelot_8021q.c | 2 +-
net/dsa/tag_rtl4_a.c | 2 +-
net/dsa/tag_sja1105.c | 4 ++--
net/dsa/tag_xrs700x.c | 2 +-
12 files changed, 37 insertions(+), 15 deletions(-)
@@ -349,6 +349,20 @@ static inline struct sk_buff *dsa_untag_bridge_pvid(struct sk_buff *skb)returnskb;}+/* If the ingress port offloads the bridge, we mark the frame as autonomously+*forwardedbyhardware,sothesoftwarebridgedoesn'tforwardintwice,back+*tous,becausewealreadydid.However,ifwe'reinfallbackmodeandwedo+*softwarebridging,wearenotoffloadingit,thereforethedp->bridge_dev+*pointerisnotpopulated,andfloodingneedstobedonebysoftware(weare+*effectivelyoperatinginstandaloneportsmode).+*/+staticinlinevoiddsa_default_offload_fwd_mark(structsk_buff*skb)+{+structdsa_port*dp=dsa_slave_to_port(skb->dev);++skb->offload_fwd_mark=!!(dp->bridge_dev);+}
So offload_fwd_mark is set iff the ingress port offloads the bridge.
Consider this set up on a switch which does NOT support LAG offload:
+----- br0 -----+
| |
bond0 |
| | (Linux interfaces)
+---+---+ +---+---+
| | | |
+-------+-------+-------+-------+
| sw0p0 | sw0p1 | sw0p2 | sw0p3 |
+-------+-------+-------+-------+
| | | |
+---A---+ B C (LAN clients)
sw0p0 and sw0p1 should be in standalone mode (offload_fwd_mark = 0),
while sw0p2 and sw0p3 are offloaded (offload_fwd_mark = 1).
When a frame is sent into sw0p2 or sw0p3, can it be forwarded to sw0p0 or
sw0p1?
Setting offload_fwd_mark to 0 could also cause potential packet loss on
switches that perform learning on the CPU port:
When client C is talking to client A, frames from C will:
1. Enter sw0p3, where the switch will learn C is reachable via sw0p3.
2. Be sent to the CPU port and bounced back, where the switch will learn C is
reachable via the CPU port, overwriting the previous learned FDB entry.
3. Be sent out of either sw0p0 or sw0p1, and reach its destination - A.
During step 2, if client B sends a frame to C, the frame will be forwarded to
the CPU, which will think it is already forwarded by the switch, and refuse to
forward it back, resulting in packet loss.
Many switch TX tags (mtk, qca, rtl) have a bit to disable source address
learning on a per-frame basis. We should utilise that.
@@ -202,6 +199,7 @@ static struct sk_buff *dsa_rcv_ll(struct sk_buff *skb, struct net_device *dev,/* Traps have, by definition, not been*forwardedbyhardware,sodon'tmarkthem.*/+trap=true;break;default:/* Reserved code, this could be anything. Drop
@@ -235,6 +233,15 @@ static struct sk_buff *dsa_rcv_ll(struct sk_buff *skb, struct net_device *dev,if(!skb->dev)returnNULL;+/* When using LAG offload, skb->dev is not a DSA slave interface,+*sowecannotcalldsa_default_offload_fwd_markandweneedto+*special-caseit.+*/+if(trunk)+skb->offload_fwd_mark=true;+elseif(!trap)+dsa_default_offload_fwd_mark(skb);+/* If the 'tagged' bit is set; convert the DSA tag to a 802.1Q*tag,anddeletetheethertype(extra)ifapplicable.Ifthe*'tagged'bitiscleared;deletetheDSAtag,andethertype
@@ -123,7 +123,7 @@ static struct sk_buff *ocelot_rcv(struct sk_buff *skb,*/returnNULL;-skb->offload_fwd_mark=1;+dsa_default_offload_fwd_mark(skb);skb->priority=qos_class;/* Ocelot switches copy frames unmodified to the CPU. However, it is
From: Vladimir Oltean <olteanv@gmail.com> Date: 2021-03-19 09:07:48
On Fri, Mar 19, 2021 at 04:52:31PM +0800, DENG Qingfang wrote:
On Fri, Mar 19, 2021 at 01:18:27AM +0200, Vladimir Oltean wrote:
quoted
From: Vladimir Oltean <vladimir.oltean@nxp.com>
DSA has gained the recent ability to deal gracefully with upper
interfaces it cannot offload, such as the bridge, bonding or team
drivers. When such uppers exist, the ports are still in standalone mode
as far as the hardware is concerned.
But when we deliver packets to the software bridge in order for that to
do the forwarding, there is an unpleasant surprise in that the bridge
will refuse to forward them. This is because we unconditionally set
skb->offload_fwd_mark = true, meaning that the bridge thinks the frames
were already forwarded in hardware by us.
Since dp->bridge_dev is populated only when there is hardware offload
for it, but not in the software fallback case, let's introduce a new
helper that can be called from the tagger data path which sets the
skb->offload_fwd_mark accordingly to zero when there is no hardware
offload for bridging. This lets the bridge forward packets back to other
interfaces of our switch, if needed.
Without this change, sending a packet to the CPU for an unoffloaded
interface triggers this WARN_ON:
void nbp_switchdev_frame_mark(const struct net_bridge_port *p,
struct sk_buff *skb)
{
if (skb->offload_fwd_mark && !WARN_ON_ONCE(!p->offload_fwd_mark))
BR_INPUT_SKB_CB(skb)->offload_fwd_mark = p->offload_fwd_mark;
}
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
Reviewed-by: Tobias Waldekranz <tobias@waldekranz.com>
---
net/dsa/dsa_priv.h | 14 ++++++++++++++
net/dsa/tag_brcm.c | 2 +-
net/dsa/tag_dsa.c | 15 +++++++++++----
net/dsa/tag_hellcreek.c | 2 +-
net/dsa/tag_ksz.c | 2 +-
net/dsa/tag_lan9303.c | 3 ++-
net/dsa/tag_mtk.c | 2 +-
net/dsa/tag_ocelot.c | 2 +-
net/dsa/tag_ocelot_8021q.c | 2 +-
net/dsa/tag_rtl4_a.c | 2 +-
net/dsa/tag_sja1105.c | 4 ++--
net/dsa/tag_xrs700x.c | 2 +-
12 files changed, 37 insertions(+), 15 deletions(-)
@@ -349,6 +349,20 @@ static inline struct sk_buff *dsa_untag_bridge_pvid(struct sk_buff *skb)returnskb;}+/* If the ingress port offloads the bridge, we mark the frame as autonomously+*forwardedbyhardware,sothesoftwarebridgedoesn'tforwardintwice,back+*tous,becausewealreadydid.However,ifwe'reinfallbackmodeandwedo+*softwarebridging,wearenotoffloadingit,thereforethedp->bridge_dev+*pointerisnotpopulated,andfloodingneedstobedonebysoftware(weare+*effectivelyoperatinginstandaloneportsmode).+*/+staticinlinevoiddsa_default_offload_fwd_mark(structsk_buff*skb)+{+structdsa_port*dp=dsa_slave_to_port(skb->dev);++skb->offload_fwd_mark=!!(dp->bridge_dev);+}
So offload_fwd_mark is set iff the ingress port offloads the bridge.
Consider this set up on a switch which does NOT support LAG offload:
+----- br0 -----+
| |
bond0 |
| | (Linux interfaces)
+---+---+ +---+---+
| | | |
+-------+-------+-------+-------+
| sw0p0 | sw0p1 | sw0p2 | sw0p3 |
+-------+-------+-------+-------+
| | | |
+---A---+ B C (LAN clients)
sw0p0 and sw0p1 should be in standalone mode (offload_fwd_mark = 0),
while sw0p2 and sw0p3 are offloaded (offload_fwd_mark = 1).
When a frame is sent into sw0p2 or sw0p3, can it be forwarded to sw0p0 or
sw0p1?
bool nbp_switchdev_allowed_egress(const struct net_bridge_port *p,
const struct sk_buff *skb)
{
return !skb->offload_fwd_mark ||
BR_INPUT_SKB_CB(skb)->offload_fwd_mark != p->offload_fwd_mark;
}
where p->offload_fwd_mark is the mark of the egress port, and
BR_INPUT_SKB_CB(skb) is the mark of the ingress port, assigned here:
void nbp_switchdev_frame_mark(const struct net_bridge_port *p,
struct sk_buff *skb)
{
if (skb->offload_fwd_mark && !WARN_ON_ONCE(!p->offload_fwd_mark))
BR_INPUT_SKB_CB(skb)->offload_fwd_mark = p->offload_fwd_mark;
}
Basically, sw0p0 and sw0p1 have a switchdev mark of 0, and sw0p2 and
sw0p3 have a non-zero switchdev mark, so nbp_switchdev_allowed_egress
returns true in both directions, regardless of the value of
skb->offload_fwd_mark.
Setting offload_fwd_mark to 0 could also cause potential packet loss on
switches that perform learning on the CPU port:
When client C is talking to client A, frames from C will:
1. Enter sw0p3, where the switch will learn C is reachable via sw0p3.
2. Be sent to the CPU port and bounced back, where the switch will learn C is
reachable via the CPU port, overwriting the previous learned FDB entry.
3. Be sent out of either sw0p0 or sw0p1, and reach its destination - A.
During step 2, if client B sends a frame to C, the frame will be forwarded to
the CPU, which will think it is already forwarded by the switch, and refuse to
forward it back, resulting in packet loss.
Many switch TX tags (mtk, qca, rtl) have a bit to disable source address
learning on a per-frame basis. We should utilise that.
This is a good point actually, which I thought about, but did not give a
lot of importance to for the moment. Either we go full steam ahead with
assisted learning on the CPU port for everybody, and we selectively
learn the addresses relevant to the bridging funciton only, or we do
what you say, but then it will be a little bit more complicated IMO, and
have hardware dependencies, which isn't as nice.
From: DENG Qingfang <dqfext@gmail.com> Date: 2021-03-19 09:30:24
On Fri, Mar 19, 2021 at 5:06 PM Vladimir Oltean [off-list ref] wrote:
This is a good point actually, which I thought about, but did not give a
lot of importance to for the moment. Either we go full steam ahead with
assisted learning on the CPU port for everybody, and we selectively
learn the addresses relevant to the bridging funciton only, or we do
what you say, but then it will be a little bit more complicated IMO, and
have hardware dependencies, which isn't as nice.
Are skb->offload_fwd_mark and source DSA switch kept in dsa_slave_xmit?
I think SA learning should be bypassed iff skb->offload_fwd_mark == 1 and
source DSA switch == destination DSA switch.
From: Vladimir Oltean <olteanv@gmail.com> Date: 2021-03-19 10:50:05
On Fri, Mar 19, 2021 at 05:29:12PM +0800, DENG Qingfang wrote:
On Fri, Mar 19, 2021 at 5:06 PM Vladimir Oltean [off-list ref] wrote:
quoted
This is a good point actually, which I thought about, but did not give a
lot of importance to for the moment. Either we go full steam ahead with
assisted learning on the CPU port for everybody, and we selectively
learn the addresses relevant to the bridging funciton only, or we do
what you say, but then it will be a little bit more complicated IMO, and
have hardware dependencies, which isn't as nice.
Are skb->offload_fwd_mark and source DSA switch kept in dsa_slave_xmit?
I think SA learning should be bypassed iff skb->offload_fwd_mark == 1 and
source DSA switch == destination DSA switch.
Why would you even want to look at the source net device for forwarding?
I'd say that if dp->bridge_dev is NULL in the xmit function, you certainly
want to bypass address learning if you can. Maybe also for link-local traffic.
From: Vladimir Oltean <vladimir.oltean@nxp.com>
DSA can properly detect and offload this sequence of operations:
ip link add br0 type bridge
ip link add bond0 type bond
ip link set swp0 master bond0
ip link set bond0 master br0
But not this one:
ip link add br0 type bridge
ip link add bond0 type bond
ip link set bond0 master br0
ip link set swp0 master bond0
Actually the second one is more complicated, due to the elapsed time
between the enslavement of bond0 and the offloading of it via swp0, a
lot of things could have happened to the bond0 bridge port in terms of
switchdev objects (host MDBs, VLANs, altered STP state etc). So this is
a bit of a can of worms, and making sure that the DSA port's state is in
sync with this already existing bridge port is handled in the next
patches.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
From: Vladimir Oltean <vladimir.oltean@nxp.com>
This is a pretty noisy change that was broken out of the larger change
for replaying switchdev attributes and objects at bridge join time,
which is when these extack objects are actually used.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
From: Vladimir Oltean <vladimir.oltean@nxp.com>
DSA currently assumes that the bridge port starts off with this
constellation of bridge port flags:
- learning on
- unicast flooding on
- multicast flooding on
- broadcast flooding on
just by virtue of code copy-pasta from the bridge layer (new_nbp).
This was a simple enough strategy thus far, because the 'bridge join'
moment always coincided with the 'bridge port creation' moment.
But with sandwiched interfaces, such as:
br0
|
bond0
|
swp0
it may happen that the user has had time to change the bridge port flags
of bond0 before enslaving swp0 to it. In that case, swp0 will falsely
assume that the bridge port flags are those determined by new_nbp, when
in fact this can happen:
ip link add br0 type bridge
ip link add bond0 type bond
ip link set bond0 master br0
ip link set bond0 type bridge_slave learning off
ip link set swp0 master br0
Now swp0 has learning enabled, bond0 has learning disabled. Not nice.
Fix this by "dumpster diving" through the actual bridge port flags with
br_port_flag_is_set, at bridge join time.
We use this opportunity to split dsa_port_change_brport_flags into two
distinct functions called dsa_port_inherit_brport_flags and
dsa_port_clear_brport_flags, now that the implementation for the two
cases is no longer similar.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
net/dsa/port.c | 123 ++++++++++++++++++++++++++++++++-----------------
1 file changed, 82 insertions(+), 41 deletions(-)
From: Vladimir Oltean <vladimir.oltean@nxp.com>
It may happen that we have the following topology:
ip link add br0 type bridge stp_state 1
ip link add bond0 type bond
ip link set bond0 master br0
ip link set swp0 master bond0
ip link set swp1 master bond0
STP decides that it should put bond0 into the BLOCKING state, and
that's that. The ports that are actively listening for the switchdev
port attributes emitted for the bond0 bridge port (because they are
offloading it) and have the honor of seeing that switchdev port
attribute can react to it, so we can program swp0 and swp1 into the
BLOCKING state.
But if then we do:
ip link set swp2 master bond0
then as far as the bridge is concerned, nothing has changed: it still
has one bridge port. But this new bridge port will not see any STP state
change notification and will remain FORWARDING, which is how the
standalone code leaves it in.
Add a function to the bridge which retrieves the current STP state, such
that drivers can synchronize to it when they may have missed switchdev
events.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
From: Vladimir Oltean <vladimir.oltean@nxp.com>
Make sure that the multicast router setting of the bridge is picked up
correctly by DSA when joining, regardless of whether there are
sandwiched interfaces or not. The SWITCHDEV_ATTR_ID_BRIDGE_MROUTER port
attribute is only emitted from br_mc_router_state_change.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
From: Vladimir Oltean <vladimir.oltean@nxp.com>
This is the same situation as for other switchdev port attributes: if we
join an already-created bridge port, such as a bond master interface,
then we can miss the initial switchdev notification emitted by the
bridge for this port.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
From: Vladimir Oltean <vladimir.oltean@nxp.com>
The SWITCHDEV_ATTR_ID_BRIDGE_AGEING_TIME attribute is only emitted from:
sysfs/ioctl/netlink
-> br_set_ageing_time
-> __set_ageing_time
therefore not at bridge port creation time, so:
(a) drivers had to hardcode the initial value for the address ageing time,
because they didn't get any notification
(b) that hardcoded value can be out of sync, if the user changes the
ageing time before enslaving the port to the bridge
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
include/linux/if_bridge.h | 6 ++++++
net/bridge/br_stp.c | 13 +++++++++++++
net/dsa/port.c | 10 ++++++++++
3 files changed, 29 insertions(+)
From: Vladimir Oltean <vladimir.oltean@nxp.com>
I have udhcpcd in my system and this is configured to bring interfaces
up as soon as they are created.
I create a bridge as follows:
ip link add br0 type bridge
As soon as I create the bridge and udhcpcd brings it up, I have some
other crap (avahi)
How dare you ;)
that starts sending some random IPv6 packets to
quoted hunk
advertise some local services, and from there, the br0 bridge joins the
following IPv6 groups:
33:33:ff:6d:c1:9c vid 0
33:33:00:00:00:6a vid 0
33:33:00:00:00:fb vid 0
br_dev_xmit
-> br_multicast_rcv
-> br_ip6_multicast_add_group
-> __br_multicast_add_group
-> br_multicast_host_join
-> br_mdb_notify
This is all fine, but inside br_mdb_notify we have br_mdb_switchdev_host
hooked up, and switchdev will attempt to offload the host joined groups
to an empty list of ports. Of course nobody offloads them.
Then when we add a port to br0:
ip link set swp0 master br0
the bridge doesn't replay the host-joined MDB entries from br_add_if,
and eventually the host joined addresses expire, and a switchdev
notification for deleting it is emitted, but surprise, the original
addition was already completely missed.
The strategy to address this problem is to replay the MDB entries (both
the port ones and the host joined ones) when the new port joins the
bridge, similar to what vxlan_fdb_replay does (in that case, its FDB can
be populated and only then attached to a bridge that you offload).
However there are 2 possibilities: the addresses can be 'pushed' by the
bridge into the port, or the port can 'pull' them from the bridge.
Considering that in the general case, the new port can be really late to
the party, and there may have been many other switchdev ports that
already received the initial notification, we would like to avoid
delivering duplicate events to them, since they might misbehave. And
currently, the bridge calls the entire switchdev notifier chain, whereas
for replaying it should just call the notifier block of the new guy.
But the bridge doesn't know what is the new guy's notifier block, it
just knows where the switchdev notifier chain is. So for simplification,
we make this a driver-initiated pull for now, and the notifier block is
passed as an argument.
To emulate the calling context for mdb objects (deferred and put on the
blocking notifier chain), we must iterate under RCU protection through
the bridge's mdb entries, queue them, and only call them once we're out
of the RCU read-side critical section.
Suggested-by: Ido Schimmel <redacted>
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
include/linux/if_bridge.h | 9 +++++
net/bridge/br_mdb.c | 84 +++++++++++++++++++++++++++++++++++++++
net/dsa/dsa_priv.h | 2 +
net/dsa/port.c | 6 +++
net/dsa/slave.c | 2 +-
5 files changed, 102 insertions(+), 1 deletion(-)
How you would feel about re-using br_mdb_switchdev_host_port() here and
pass a 'type' value that is neither RTM_NEWDB nor RTM_DELDB just so you
don't have to duplicate that code here and we ensure it is in sync?
--
Florian
From: Vladimir Oltean <vladimir.oltean@nxp.com>
Currently this simple setup:
ip link add br0 type bridge vlan_filtering 1
ip link add bond0 type bond
ip link set bond0 master br0
ip link set swp0 master bond0
will not work because the bridge has created the PVID in br_add_if ->
nbp_vlan_init, and it has notified switchdev of the existence of VLAN 1,
but that was too early, since swp0 was not yet a lower of bond0, so it
had no reason to act upon that notification.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
include/linux/if_bridge.h | 10 ++++++
net/bridge/br_vlan.c | 71 +++++++++++++++++++++++++++++++++++++++
net/dsa/port.c | 6 ++++
3 files changed, 87 insertions(+)
Same comment as patch 8, CONFIG_BRIDGE_VLAN_FILTERING can be turned off
even if this does not really make practical sense with a hardware
switch. Should we return -EOPNOTSUPP instead?
--
Florian
@@ -639,6 +639,19 @@ int br_set_ageing_time(struct net_bridge *br, clock_t ageing_time)return0;}+clock_tbr_get_ageing_time(structnet_device*br_dev)+{+structnet_bridge*br;++if(!netif_is_bridge_master(br_dev))+return0;++br=netdev_priv(br_dev);++returnjiffies_to_clock_t(br->ageing_time);
Don't you want an ASSERT_RTNL() in this function as well?
Hmm, I'm not sure. I don't think I'm accessing anything that is under
the protection of the rtnl_mutex. If anything, the ageing time is
protected by the "bridge lock", but I don't think there's much of an
issue if I read an unsigned int while not holding it.
From: Vladimir Oltean <olteanv@gmail.com> Date: 2021-03-20 11:13:10
On Fri, Mar 19, 2021 at 03:08:46PM -0700, Florian Fainelli wrote:
On 3/18/2021 4:18 PM, Vladimir Oltean wrote:
quoted
From: Vladimir Oltean <vladimir.oltean@nxp.com>
DSA currently assumes that the bridge port starts off with this
constellation of bridge port flags:
- learning on
- unicast flooding on
- multicast flooding on
- broadcast flooding on
just by virtue of code copy-pasta from the bridge layer (new_nbp).
This was a simple enough strategy thus far, because the 'bridge join'
moment always coincided with the 'bridge port creation' moment.
But with sandwiched interfaces, such as:
br0
|
bond0
|
swp0
it may happen that the user has had time to change the bridge port flags
of bond0 before enslaving swp0 to it. In that case, swp0 will falsely
assume that the bridge port flags are those determined by new_nbp, when
in fact this can happen:
ip link add br0 type bridge
ip link add bond0 type bond
ip link set bond0 master br0
ip link set bond0 type bridge_slave learning off
ip link set swp0 master br0
Now swp0 has learning enabled, bond0 has learning disabled. Not nice.
Fix this by "dumpster diving" through the actual bridge port flags with
br_port_flag_is_set, at bridge join time.
We use this opportunity to split dsa_port_change_brport_flags into two
distinct functions called dsa_port_inherit_brport_flags and
dsa_port_clear_brport_flags, now that the implementation for the two
cases is no longer similar.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
net/dsa/port.c | 123 ++++++++++++++++++++++++++++++++-----------------
1 file changed, 82 insertions(+), 41 deletions(-)
Would not you want to use the same for_each_set_bit() loop that
dsa_port_change_br_flags() uses, that would be a tad more compact.
--
Florian
The reworded version has an equal number of lines, but at least it
catches errors now:
static void dsa_port_clear_brport_flags(struct dsa_port *dp,
struct netlink_ext_ack *extack)
{
const unsigned long val = BR_FLOOD | BR_MCAST_FLOOD | BR_BCAST_FLOOD;
const unsigned long mask = BR_LEARNING | BR_FLOOD | BR_MCAST_FLOOD |
BR_BCAST_FLOOD;
int flag, err;
for_each_set_bit(flag, &mask, 32) {
struct switchdev_brport_flags flags = {0};
flags.mask = BIT(flag);
flags.val = val & BIT(flag);
err = dsa_port_bridge_flags(dp, flags, extack);
if (err && err != -EOPNOTSUPP)
dev_err(dp->ds->dev,
"failed to clear bridge port flag %d: %d (%pe)\n",
flag, err, ERR_PTR(err));
}
}
From: Vladimir Oltean <olteanv@gmail.com> Date: 2021-03-20 11:20:30
On Fri, Mar 19, 2021 at 03:20:38PM -0700, Florian Fainelli wrote:
On 3/18/2021 4:18 PM, Vladimir Oltean wrote:
quoted
From: Vladimir Oltean <vladimir.oltean@nxp.com>
I have udhcpcd in my system and this is configured to bring interfaces
up as soon as they are created.
I create a bridge as follows:
ip link add br0 type bridge
As soon as I create the bridge and udhcpcd brings it up, I have some
other crap (avahi)
How dare you ;)
Well, it comes preinstalled on my system, I don't need it, and it has
caused me nothing but trouble. So I think it has earned its title :D
quoted
that starts sending some random IPv6 packets to
advertise some local services, and from there, the br0 bridge joins the
following IPv6 groups:
33:33:ff:6d:c1:9c vid 0
33:33:00:00:00:6a vid 0
33:33:00:00:00:fb vid 0
br_dev_xmit
-> br_multicast_rcv
-> br_ip6_multicast_add_group
-> __br_multicast_add_group
-> br_multicast_host_join
-> br_mdb_notify
This is all fine, but inside br_mdb_notify we have br_mdb_switchdev_host
hooked up, and switchdev will attempt to offload the host joined groups
to an empty list of ports. Of course nobody offloads them.
Then when we add a port to br0:
ip link set swp0 master br0
the bridge doesn't replay the host-joined MDB entries from br_add_if,
and eventually the host joined addresses expire, and a switchdev
notification for deleting it is emitted, but surprise, the original
addition was already completely missed.
The strategy to address this problem is to replay the MDB entries (both
the port ones and the host joined ones) when the new port joins the
bridge, similar to what vxlan_fdb_replay does (in that case, its FDB can
be populated and only then attached to a bridge that you offload).
However there are 2 possibilities: the addresses can be 'pushed' by the
bridge into the port, or the port can 'pull' them from the bridge.
Considering that in the general case, the new port can be really late to
the party, and there may have been many other switchdev ports that
already received the initial notification, we would like to avoid
delivering duplicate events to them, since they might misbehave. And
currently, the bridge calls the entire switchdev notifier chain, whereas
for replaying it should just call the notifier block of the new guy.
But the bridge doesn't know what is the new guy's notifier block, it
just knows where the switchdev notifier chain is. So for simplification,
we make this a driver-initiated pull for now, and the notifier block is
passed as an argument.
To emulate the calling context for mdb objects (deferred and put on the
blocking notifier chain), we must iterate under RCU protection through
the bridge's mdb entries, queue them, and only call them once we're out
of the RCU read-side critical section.
Suggested-by: Ido Schimmel <redacted>
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
include/linux/if_bridge.h | 9 +++++
net/bridge/br_mdb.c | 84 +++++++++++++++++++++++++++++++++++++++
net/dsa/dsa_priv.h | 2 +
net/dsa/port.c | 6 +++
net/dsa/slave.c | 2 +-
5 files changed, 102 insertions(+), 1 deletion(-)
How you would feel about re-using br_mdb_switchdev_host_port() here and
pass a 'type' value that is neither RTM_NEWDB nor RTM_DELDB just so you
don't have to duplicate that code here and we ensure it is in sync?
The trouble is that br_mdb_switchdev_host calls switchdev_port_obj_add,
and I think the agreement was that replayed events should be a silent,
one-to-one conversation via a direct call to the notifier block of the
interested driver, as opposed to a call to the entire notifier chain
which would make everybody else in the system see duplicates. This is
the reason why I duplicated mostly everything.
From: DENG Qingfang <dqfext@gmail.com> Date: 2021-03-22 08:04:52
On Fri, Mar 19, 2021 at 6:49 PM Vladimir Oltean [off-list ref] wrote:
Why would you even want to look at the source net device for forwarding?
I'd say that if dp->bridge_dev is NULL in the xmit function, you certainly
want to bypass address learning if you can. Maybe also for link-local traffic.
Also for trapped traffic (snooping, tc-flower trap action) if the CPU
sends them back.
On Fri, Mar 19, 2021 at 01:18, Vladimir Oltean [off-list ref] wrote:
From: Vladimir Oltean <vladimir.oltean@nxp.com>
DSA can properly detect and offload this sequence of operations:
ip link add br0 type bridge
ip link add bond0 type bond
ip link set swp0 master bond0
ip link set bond0 master br0
But not this one:
ip link add br0 type bridge
ip link add bond0 type bond
ip link set bond0 master br0
ip link set swp0 master bond0
Actually the second one is more complicated, due to the elapsed time
between the enslavement of bond0 and the offloading of it via swp0, a
lot of things could have happened to the bond0 bridge port in terms of
switchdev objects (host MDBs, VLANs, altered STP state etc). So this is
a bit of a can of worms, and making sure that the DSA port's state is in
sync with this already existing bridge port is handled in the next
patches.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
On Fri, Mar 19, 2021 at 01:18, Vladimir Oltean [off-list ref] wrote:
From: Vladimir Oltean <vladimir.oltean@nxp.com>
This is a pretty noisy change that was broken out of the larger change
for replaying switchdev attributes and objects at bridge join time,
which is when these extack objects are actually used.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
On Fri, Mar 19, 2021 at 01:18, Vladimir Oltean [off-list ref] wrote:
From: Vladimir Oltean <vladimir.oltean@nxp.com>
It may happen that we have the following topology:
ip link add br0 type bridge stp_state 1
ip link add bond0 type bond
ip link set bond0 master br0
ip link set swp0 master bond0
ip link set swp1 master bond0
STP decides that it should put bond0 into the BLOCKING state, and
that's that. The ports that are actively listening for the switchdev
port attributes emitted for the bond0 bridge port (because they are
offloading it) and have the honor of seeing that switchdev port
attribute can react to it, so we can program swp0 and swp1 into the
BLOCKING state.
But if then we do:
ip link set swp2 master bond0
then as far as the bridge is concerned, nothing has changed: it still
has one bridge port. But this new bridge port will not see any STP state
change notification and will remain FORWARDING, which is how the
standalone code leaves it in.
Add a function to the bridge which retrieves the current STP state, such
that drivers can synchronize to it when they may have missed switchdev
events.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
On Fri, Mar 19, 2021 at 01:18, Vladimir Oltean [off-list ref] wrote:
From: Vladimir Oltean <vladimir.oltean@nxp.com>
This is the same situation as for other switchdev port attributes: if we
join an already-created bridge port, such as a bond master interface,
then we can miss the initial switchdev notification emitted by the
bridge for this port.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
On Fri, Mar 19, 2021 at 01:18, Vladimir Oltean [off-list ref] wrote:
quoted hunk
From: Vladimir Oltean <vladimir.oltean@nxp.com>
Make sure that the multicast router setting of the bridge is picked up
correctly by DSA when joining, regardless of whether there are
sandwiched interfaces or not. The SWITCHDEV_ATTR_ID_BRIDGE_MROUTER port
attribute is only emitted from br_mc_router_state_change.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
net/dsa/port.c | 10 ++++++++++
1 file changed, 10 insertions(+)
@@ -189,6 +189,10 @@ static int dsa_port_switchdev_sync(struct dsa_port *dp,if(err&&err!=-EOPNOTSUPP)returnerr;+err=dsa_port_mrouter(dp->cpu_dp,br_multicast_router(br),extack);+if(err&&err!=-EOPNOTSUPP)+returnerr;+return0;}
@@ -212,6 +216,12 @@ static void dsa_port_switchdev_unsync(struct dsa_port *dp)dsa_port_set_state_now(dp,BR_STATE_FORWARDING);/* VLAN filtering is handled by dsa_switch_bridge_leave */++/* Some drivers treat the notification for having a local multicast+*routerbyallowingmulticasttobefloodedtotheCPU,soweshould+*allowthisinstandalonemodetoo.+*/+dsa_port_mrouter(dp->cpu_dp,true,NULL);
Is this really for the DSA layer to decide? The driver has already been
notified that at least one port is now in standalone mode. So if that
particular driver then requires all multicast to be flooded towards the
CPU, it can make that decision on its own.
E.g. say that you implement standalone mode using a matchall TCAM rule
that maps all frames coming in on a particular port to the CPU. You
could still leave flooding of unknown multicast off in that case. Now
that driver has to figure out if the notification about a multicast
router on the CPU is a real router, or the DSA layer telling it
something that it can safely ignore.
Today I think that most (all?) DSA drivers treats mrouter in the same
way as the multicast flooding bridge flag. But AFAIK, the semantic
meaning of the setting is "flood IP multicast to this port because there
is a router behind it somewhere". This means unknown _IP_ multicast, but
also all known (IGMP/MLD) groups. As most smaller devices cannot
separate IP multicast from the non-IP variety, we flood everything. But
we should also make sure that the port in question receives all known
groups for the _bridge_ in question. Because this is really a bridge
setting, though that information is not carried over to the driver
today. So reusing it in this way feels like it could be problematic down
the road.
}
int dsa_port_bridge_join(struct dsa_port *dp, struct net_device *br,
--
2.25.1
On Fri, Mar 19, 2021 at 01:18, Vladimir Oltean [off-list ref] wrote:
From: Vladimir Oltean <vladimir.oltean@nxp.com>
The SWITCHDEV_ATTR_ID_BRIDGE_AGEING_TIME attribute is only emitted from:
sysfs/ioctl/netlink
-> br_set_ageing_time
-> __set_ageing_time
therefore not at bridge port creation time, so:
(a) drivers had to hardcode the initial value for the address ageing time,
because they didn't get any notification
(b) that hardcoded value can be out of sync, if the user changes the
ageing time before enslaving the port to the bridge
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
From: Vladimir Oltean <olteanv@gmail.com> Date: 2021-03-22 11:44:04
On Mon, Mar 22, 2021 at 12:17:33PM +0100, Tobias Waldekranz wrote:
On Fri, Mar 19, 2021 at 01:18, Vladimir Oltean [off-list ref] wrote:
quoted
From: Vladimir Oltean <vladimir.oltean@nxp.com>
Make sure that the multicast router setting of the bridge is picked up
correctly by DSA when joining, regardless of whether there are
sandwiched interfaces or not. The SWITCHDEV_ATTR_ID_BRIDGE_MROUTER port
attribute is only emitted from br_mc_router_state_change.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
net/dsa/port.c | 10 ++++++++++
1 file changed, 10 insertions(+)
@@ -189,6 +189,10 @@ static int dsa_port_switchdev_sync(struct dsa_port *dp,if(err&&err!=-EOPNOTSUPP)returnerr;+err=dsa_port_mrouter(dp->cpu_dp,br_multicast_router(br),extack);+if(err&&err!=-EOPNOTSUPP)+returnerr;+return0;}
@@ -212,6 +216,12 @@ static void dsa_port_switchdev_unsync(struct dsa_port *dp)dsa_port_set_state_now(dp,BR_STATE_FORWARDING);/* VLAN filtering is handled by dsa_switch_bridge_leave */++/* Some drivers treat the notification for having a local multicast+*routerbyallowingmulticasttobefloodedtotheCPU,soweshould+*allowthisinstandalonemodetoo.+*/+dsa_port_mrouter(dp->cpu_dp,true,NULL);
Is this really for the DSA layer to decide? The driver has already been
notified that at least one port is now in standalone mode. So if that
particular driver then requires all multicast to be flooded towards the
CPU, it can make that decision on its own.
E.g. say that you implement standalone mode using a matchall TCAM rule
that maps all frames coming in on a particular port to the CPU. You
could still leave flooding of unknown multicast off in that case. Now
that driver has to figure out if the notification about a multicast
router on the CPU is a real router, or the DSA layer telling it
something that it can safely ignore.
Today I think that most (all?) DSA drivers treats mrouter in the same
way as the multicast flooding bridge flag. But AFAIK, the semantic
meaning of the setting is "flood IP multicast to this port because there
is a router behind it somewhere". This means unknown _IP_ multicast, but
also all known (IGMP/MLD) groups. As most smaller devices cannot
separate IP multicast from the non-IP variety, we flood everything. But
we should also make sure that the port in question receives all known
groups for the _bridge_ in question. Because this is really a bridge
setting, though that information is not carried over to the driver
today. So reusing it in this way feels like it could be problematic down
the road.
I agree with your objections in principle, but somehow I would like to
make progress with this patch series which is not really about how we
deal with IP multicast flooding to the CPU port in standalone ports
mode, so I would like to not get bogged down too much into this for now.
Don't forget that up until recent commit a8b659e7ff75 ("net: dsa: act as
passthrough for bridge port flags"), DSA drivers had no real idea
whether multicast flooding was meant for IP or not. And in standalone
mode, the way things work now is that the CPU port should see all
traffic, so it isn't wrong to do what this patch does.
Unless you see a breaking change introduced by this patch, we can
revisit this discussion for the "RX filtering on DSA" series, where it
is more relevant.
On Fri, Mar 19, 2021 at 01:18, Vladimir Oltean [off-list ref] wrote:
quoted hunk
From: Vladimir Oltean <vladimir.oltean@nxp.com>
When a DSA port joins a LAG that already had an FDB entry pointing to it:
ip link set bond0 master br0
bridge fdb add dev bond0 00:01:02:03:04:05 master static
ip link set swp0 master bond0
the DSA port will have no idea that this FDB entry is there, because it
missed the switchdev event emitted at its creation.
Ido Schimmel pointed this out during a discussion about challenges with
switchdev offloading of stacked interfaces between the physical port and
the bridge, and recommended to just catch that condition and deny the
CHANGEUPPER event:
https://lore.kernel.org/netdev/20210210105949.GB287766@shredder.lan/
But in fact, we might need to deal with the hard thing anyway, which is
to replay all FDB addresses relevant to this port, because it isn't just
static FDB entries, but also local addresses (ones that are not
forwarded but terminated by the bridge). There, we can't just say 'oh
yeah, there was an upper already so I'm not joining that'.
So, similar to the logic for replaying MDB entries, add a function that
must be called by individual switchdev drivers and replays local FDB
entries as well as ones pointing towards a bridge port. This time, we
use the atomic switchdev notifier block, since that's what FDB entries
expect for some reason.
Reported-by: Ido Schimmel <redacted>
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
include/linux/if_bridge.h | 9 +++++++
include/net/switchdev.h | 1 +
net/bridge/br_fdb.c | 52 +++++++++++++++++++++++++++++++++++++++
net/dsa/dsa_priv.h | 1 +
net/dsa/port.c | 4 +++
net/dsa/slave.c | 2 +-
6 files changed, 68 insertions(+), 1 deletion(-)
@@ -205,6 +205,7 @@ struct switchdev_notifier_info {structswitchdev_notifier_fdb_info{structswitchdev_notifier_infoinfo;/* must be first */+structlist_headlist;constunsignedchar*addr;u16vid;u8added_by_user:1,
I do not know if it is a problem or not, more of an observation: This is
not guaranteed to be an exact replay of the events that the bridge port
(i.e. bond0 or whatever) has received since, in fdb_insert, we exit
early when adding local entries if that address is already in the
database.
Do we have to guard against this somehow? Or maybe we should consider
the current behavior a bug and make sure to always send the event in the
first place?
From: Vladimir Oltean <vladimir.oltean@nxp.com>
The DSA core has a layered structure, and even though we end up
returning 0 (success) to user space when setting a bonding/team upper
that can't be offloaded, some parts of the framework actually need to
know that we couldn't offload that.
For example, if dsa_switch_lag_join returns 0 as it currently does,
dsa_port_lag_join has no way to tell a successful offload from a
software fallback, and it will call dsa_port_bridge_join afterwards.
Then we'll think we're offloading the bridge master of the LAG, when in
fact we're not even offloading the LAG. In turn, this will make us set
skb->offload_fwd_mark = true, which is incorrect and the bridge doesn't
like it.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
On Fri, Mar 19, 2021 at 03:20:38PM -0700, Florian Fainelli wrote:
quoted
On 3/18/2021 4:18 PM, Vladimir Oltean wrote:
quoted
From: Vladimir Oltean <vladimir.oltean@nxp.com>
I have udhcpcd in my system and this is configured to bring interfaces
up as soon as they are created.
I create a bridge as follows:
ip link add br0 type bridge
As soon as I create the bridge and udhcpcd brings it up, I have some
other crap (avahi)
How dare you ;)
Well, it comes preinstalled on my system, I don't need it, and it has
caused me nothing but trouble. So I think it has earned its title :D
quoted
quoted
that starts sending some random IPv6 packets to
advertise some local services, and from there, the br0 bridge joins the
following IPv6 groups:
33:33:ff:6d:c1:9c vid 0
33:33:00:00:00:6a vid 0
33:33:00:00:00:fb vid 0
br_dev_xmit
-> br_multicast_rcv
-> br_ip6_multicast_add_group
-> __br_multicast_add_group
-> br_multicast_host_join
-> br_mdb_notify
This is all fine, but inside br_mdb_notify we have br_mdb_switchdev_host
hooked up, and switchdev will attempt to offload the host joined groups
to an empty list of ports. Of course nobody offloads them.
Then when we add a port to br0:
ip link set swp0 master br0
the bridge doesn't replay the host-joined MDB entries from br_add_if,
and eventually the host joined addresses expire, and a switchdev
notification for deleting it is emitted, but surprise, the original
addition was already completely missed.
The strategy to address this problem is to replay the MDB entries (both
the port ones and the host joined ones) when the new port joins the
bridge, similar to what vxlan_fdb_replay does (in that case, its FDB can
be populated and only then attached to a bridge that you offload).
However there are 2 possibilities: the addresses can be 'pushed' by the
bridge into the port, or the port can 'pull' them from the bridge.
Considering that in the general case, the new port can be really late to
the party, and there may have been many other switchdev ports that
already received the initial notification, we would like to avoid
delivering duplicate events to them, since they might misbehave. And
currently, the bridge calls the entire switchdev notifier chain, whereas
for replaying it should just call the notifier block of the new guy.
But the bridge doesn't know what is the new guy's notifier block, it
just knows where the switchdev notifier chain is. So for simplification,
we make this a driver-initiated pull for now, and the notifier block is
passed as an argument.
To emulate the calling context for mdb objects (deferred and put on the
blocking notifier chain), we must iterate under RCU protection through
the bridge's mdb entries, queue them, and only call them once we're out
of the RCU read-side critical section.
Suggested-by: Ido Schimmel <redacted>
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
include/linux/if_bridge.h | 9 +++++
net/bridge/br_mdb.c | 84 +++++++++++++++++++++++++++++++++++++++
net/dsa/dsa_priv.h | 2 +
net/dsa/port.c | 6 +++
net/dsa/slave.c | 2 +-
5 files changed, 102 insertions(+), 1 deletion(-)
How you would feel about re-using br_mdb_switchdev_host_port() here and
pass a 'type' value that is neither RTM_NEWDB nor RTM_DELDB just so you
don't have to duplicate that code here and we ensure it is in sync?
The trouble is that br_mdb_switchdev_host calls switchdev_port_obj_add,
and I think the agreement was that replayed events should be a silent,
one-to-one conversation via a direct call to the notifier block of the
interested driver, as opposed to a call to the entire notifier chain
which would make everybody else in the system see duplicates. This is
the reason why I duplicated mostly everything.
It's not a whole lot of notification but if you passed a type argument
that is neither of the two supported value (say -1),
br_mdb_switchdev_host_port() would end its execution there, and that
would avoid the duplication altogether. I am not stuck on that idea and
can hardly think for now of why this function would change, or why the
switchdev_obj_port_mdb would change, too.
--
Florian
On Fri, Mar 19, 2021 at 01:18, Vladimir Oltean [off-list ref] wrote:
From: Vladimir Oltean <vladimir.oltean@nxp.com>
The DSA core has a layered structure, and even though we end up
returning 0 (success) to user space when setting a bonding/team upper
that can't be offloaded, some parts of the framework actually need to
know that we couldn't offload that.
For example, if dsa_switch_lag_join returns 0 as it currently does,
dsa_port_lag_join has no way to tell a successful offload from a
software fallback, and it will call dsa_port_bridge_join afterwards.
Then we'll think we're offloading the bridge master of the LAG, when in
fact we're not even offloading the LAG. In turn, this will make us set
skb->offload_fwd_mark = true, which is incorrect and the bridge doesn't
like it.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
From: Vladimir Oltean <vladimir.oltean@nxp.com>
DSA has gained the recent ability to deal gracefully with upper
interfaces it cannot offload, such as the bridge, bonding or team
drivers. When such uppers exist, the ports are still in standalone mode
as far as the hardware is concerned.
But when we deliver packets to the software bridge in order for that to
do the forwarding, there is an unpleasant surprise in that the bridge
will refuse to forward them. This is because we unconditionally set
skb->offload_fwd_mark = true, meaning that the bridge thinks the frames
were already forwarded in hardware by us.
Since dp->bridge_dev is populated only when there is hardware offload
for it, but not in the software fallback case, let's introduce a new
helper that can be called from the tagger data path which sets the
skb->offload_fwd_mark accordingly to zero when there is no hardware
offload for bridging. This lets the bridge forward packets back to other
interfaces of our switch, if needed.
Without this change, sending a packet to the CPU for an unoffloaded
interface triggers this WARN_ON:
void nbp_switchdev_frame_mark(const struct net_bridge_port *p,
struct sk_buff *skb)
{
if (skb->offload_fwd_mark && !WARN_ON_ONCE(!p->offload_fwd_mark))
BR_INPUT_SKB_CB(skb)->offload_fwd_mark = p->offload_fwd_mark;
}
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
Reviewed-by: Tobias Waldekranz <tobias@waldekranz.com>
From: Vladimir Oltean <olteanv@gmail.com> Date: 2021-03-22 16:20:56
On Mon, Mar 22, 2021 at 04:44:41PM +0100, Tobias Waldekranz wrote:
I do not know if it is a problem or not, more of an observation: This is
not guaranteed to be an exact replay of the events that the bridge port
(i.e. bond0 or whatever) has received since, in fdb_insert, we exit
early when adding local entries if that address is already in the
database.
Do we have to guard against this somehow? Or maybe we should consider
the current behavior a bug and make sure to always send the event in the
first place?
I don't really understand what you're saying.
fdb_insert has:
fdb = br_fdb_find(br, addr, vid);
if (fdb) {
/* it is okay to have multiple ports with same
* address, just use the first one.
*/
if (test_bit(BR_FDB_LOCAL, &fdb->flags))
return 0;
br_warn(br, "adding interface %s with same address as a received packet (addr:%pM, vlan:%u)\n",
source ? source->dev->name : br->dev->name, addr, vid);
fdb_delete(br, fdb, true);
}
fdb = fdb_create(br, source, addr, vid,
BIT(BR_FDB_LOCAL) | BIT(BR_FDB_STATIC));
Basically, if the {addr, vid} pair already exists in the fdb, and it
points to a local entry, fdb_create is bypassed.
Whereas my br_fdb_replay() function iterates over br->fdb_list, which is
exactly where fdb_create() also lays its eggs. That is to say, unless
I'm missing something, that duplicate local FDB entries that skipped the
fdb_create() call in fdb_insert() because they were for already-existing
local FDB entries will also be skipped by br_fdb_replay(), because it
iterates over a br->fdb_list which contains unique local addresses.
Where am I wrong?
On Fri, Mar 19, 2021 at 01:18, Vladimir Oltean [off-list ref] wrote:
From: Vladimir Oltean <vladimir.oltean@nxp.com>
On reception of an skb, the bridge checks if it was marked as 'already
forwarded in hardware' (checks if skb->offload_fwd_mark == 1), and if it
is, it puts a mark of its own on that skb, with the switchdev mark of
the ingress port. Then during forwarding, it enforces that the egress
port must have a different switchdev mark than the ingress one (this is
done in nbp_switchdev_allowed_egress).
Non-switchdev drivers don't report any physical switch id (neither
through devlink nor .ndo_get_port_parent_id), therefore the bridge
assigns them a switchdev mark of 0, and packets coming from them will
always have skb->offload_fwd_mark = 0. So there aren't any restrictions.
Problems appear due to the fact that DSA would like to perform software
fallback for bonding and team interfaces that the physical switch cannot
offload.
+-- br0 -+
/ / | \
/ / | \
/ / | \
/ / | \
/ / | \
/ | | bond0
/ | | / \
swp0 swp1 swp2 swp3 swp4
There, it is desirable that the presence of swp3 and swp4 under a
non-offloaded LAG does not preclude us from doing hardware bridging
beteen swp0, swp1 and swp2. The bandwidth of the CPU is often times high
enough that software bridging between {swp0,swp1,swp2} and bond0 is not
impractical.
But this creates an impossible paradox given the current way in which
port switchdev marks are assigned. When the driver receives a packet
from swp0 (say, due to flooding), it must set skb->offload_fwd_mark to
something.
- If we set it to 0, then the bridge will forward it towards swp1, swp2
and bond0. But the switch has already forwarded it towards swp1 and
swp2 (not to bond0, remember, that isn't offloaded, so as far as the
switch is concerned, ports swp3 and swp4 are not looking up the FDB,
and the entire bond0 is a destination that is strictly behind the
CPU). But we don't want duplicated traffic towards swp1 and swp2, so
it's not ok to set skb->offload_fwd_mark = 0.
- If we set it to 1, then the bridge will not forward the skb towards
the ports with the same switchdev mark, i.e. not to swp1, swp2 and
bond0. Towards swp1 and swp2 that's ok, but towards bond0? It should
have forwarded the skb there.
So the real issue is that bond0 will be assigned the same switchdev mark
as {swp0,swp1,swp2}, because the function that assigns switchdev marks
to bridge ports, nbp_switchdev_mark_set, recurses through bond0's lower
interfaces until it finds something that implements devlink.
A solution is to give the bridge explicit hints as to what switchdev
mark it should use for each port.
Currently, the bridging offload is very 'silent': a driver registers a
netdevice notifier, which is put on the netns's notifier chain, and
which sniffs around for NETDEV_CHANGEUPPER events where the upper is a
bridge, and the lower is an interface it knows about (one registered by
this driver, normally). Then, from within that notifier, it does a bunch
of stuff behind the bridge's back, without the bridge necessarily
knowing that there's somebody offloading that port. It looks like this:
ip link set swp0 master br0
|
v
bridge calls netdev_master_upper_dev_link
|
v
call_netdevice_notifiers
|
v
dsa_slave_netdevice_event
|
v
oh, hey! it's for me!
|
v
.port_bridge_join
What we do to solve the conundrum is to be less silent, and emit a
notification back. Something like this:
ip link set swp0 master br0
|
v
bridge calls netdev_master_upper_dev_link
|
v bridge: Aye! I'll use this
call_netdevice_notifiers ^ ppid as the
| | switchdev mark for
v | this port, and zero
dsa_slave_netdevice_event | if I got nothing.
| |
v |
oh, hey! it's for me! |
| |
v |
.port_bridge_join |
| |
+------------------------+
switchdev_bridge_port_offload(swp0)
Then stacked interfaces (like bond0 on top of swp3/swp4) would be
treated differently in DSA, depending on whether we can or cannot
offload them.
The offload case:
ip link set bond0 master br0
|
v
bridge calls netdev_master_upper_dev_link
|
v bridge: Aye! I'll use this
call_netdevice_notifiers ^ ppid as the
| | switchdev mark for
v | bond0.
dsa_slave_netdevice_event | Coincidentally (or not),
| | bond0 and swp0, swp1, swp2
v | all have the same switchdev
hmm, it's not quite for me, | mark now, since the ASIC
but my driver has already | is able to forward towards
called .port_lag_join | all these ports in hw.
for it, because I have |
a port with dp->lag_dev == bond0. |
| |
v |
.port_bridge_join |
for swp3 and swp4 |
| |
+------------------------+
switchdev_bridge_port_offload(bond0)
And the non-offload case:
ip link set bond0 master br0
|
v
bridge calls netdev_master_upper_dev_link
|
v bridge waiting:
call_netdevice_notifiers ^ huh, switchdev_bridge_port_offload
| | wasn't called, okay, I'll use a
v | switchdev mark of zero for this one.
dsa_slave_netdevice_event : Then packets received on swp0 will
| : not be forwarded towards swp1, but
v : they will towards bond0.
it's not for me, but
bond0 is an upper of swp3
and swp4, but their dp->lag_dev
is NULL because they couldn't
offload it.
Basically we can draw the conclusion that the lowers of a bridge port
can come and go, so depending on the configuration of lowers for a
bridge port, it can dynamically toggle between offloaded and unoffloaded.
Therefore, we need an equivalent switchdev_bridge_port_unoffload too.
This patch changes the way any switchdev driver interacts with the
bridge. From now on, everybody needs to call switchdev_bridge_port_offload,
otherwise the bridge will treat the port as non-offloaded and allow
software flooding to other ports from the same ASIC.
Signed-off-by: Vladimir Oltean <vladimir.oltean@nxp.com>
---
.../ethernet/freescale/dpaa2/dpaa2-switch.c | 4 +-
.../marvell/prestera/prestera_switchdev.c | 7 ++
.../mellanox/mlxsw/spectrum_switchdev.c | 4 +-
drivers/net/ethernet/mscc/ocelot_net.c | 4 +-
drivers/net/ethernet/rocker/rocker_ofdpa.c | 8 +-
drivers/net/ethernet/ti/am65-cpsw-nuss.c | 7 +-
drivers/net/ethernet/ti/cpsw_new.c | 6 +-
@@ -1518,7 +1518,7 @@ static int dpaa2_switch_port_bridge_join(struct net_device *netdev,if(err)gotoerr_egress_flood;-return0;+returnswitchdev_bridge_port_offload(netdev,NULL);err_egress_flood:dpaa2_switch_port_set_fdb(port_priv,NULL);
@@ -1552,6 +1552,8 @@ static int dpaa2_switch_port_bridge_leave(struct net_device *netdev)structethsw_core*ethsw=port_priv->ethsw_data;interr;+switchdev_bridge_port_unoffload(netdev);+/* First of all, fast age any learn FDB addresses on this switch port */dpaa2_switch_port_fast_age(port_priv);
@@ -671,13 +667,13 @@ int br_add_if(struct net_bridge *br, struct net_device *dev,*/err=dev_pre_changeaddr_notify(br->dev,dev->dev_addr,extack);if(err)-gotoerr7;+gotoerr6;}err=nbp_vlan_init(p,extack);if(err){netdev_err(dev,"failed to initialize vlan filtering on this port\n");-gotoerr7;+gotoerr6;}spin_lock_bh(&br->lock);
@@ -8,37 +8,109 @@#include"br_private.h"-staticintbr_switchdev_mark_get(structnet_bridge*br,structnet_device*dev)+staticintbr_switchdev_mark_get(structnet_bridge*br,+structnet_bridge_port*new_nbp){structnet_bridge_port*p;/* dev is yet to be added to the port list. */list_for_each_entry(p,&br->port_list,list){-if(netdev_port_same_parent_id(dev,p->dev))+if(!p->offload_count)+continue;++if(netdev_phys_item_id_same(&p->ppid,&new_nbp->ppid))returnp->offload_fwd_mark;}return++br->offload_fwd_mark;}-intnbp_switchdev_mark_set(structnet_bridge_port*p)+staticintnbp_switchdev_mark_set(structnet_bridge_port*p,+structnetdev_phys_item_idppid,+structnetlink_ext_ack*extack)+{+if(p->offload_count){+/* Prevent unsupported configurations such as a bridge port+*whichisabondinginterface,andthememberportsarefrom+*differenthardwareswitches.+*/+if(!netdev_phys_item_id_same(&p->ppid,&ppid)){+NL_SET_ERR_MSG_MOD(extack,+"Same bridge port cannot be offloaded by two physical switches");+return-EBUSY;+}+/* Be tolerant with drivers that call SWITCHDEV_BRPORT_OFFLOADED+*morethanonceforthesamebridgeport,suchaswhenthe+*bridgeportisanoffloadedbonding/teaminterface.+*/+p->offload_count++;+return0;+}++p->ppid=ppid;+p->offload_count=1;+p->offload_fwd_mark=br_switchdev_mark_get(p->br,p);++return0;+}++staticvoidnbp_switchdev_mark_clear(structnet_bridge_port*p,+structnetdev_phys_item_idppid)+{+if(WARN_ON(!netdev_phys_item_id_same(&p->ppid,&ppid)))+return;+if(WARN_ON(!p->offload_count))+return;++p->offload_count--;+if(p->offload_count)+return;++p->offload_fwd_mark=0;+}++/* Let the bridge know that this port is offloaded, so that it can use+*theportparentidobtainedbyrecursiontodeterminethebridge+*port'sswitchdevmark.+*/+intswitchdev_bridge_port_offload(structnet_device*dev,+structnetlink_ext_ack*extack){-structnetdev_phys_item_idppid={};+structnetdev_phys_item_idppid;+structnet_bridge_port*p;interr;-ASSERT_RTNL();+p=br_port_get_rtnl(dev);+if(!p)+return-ENODEV;-err=dev_get_port_parent_id(p->dev,&ppid,true);-if(err){-if(err==-EOPNOTSUPP)-return0;+err=dev_get_port_parent_id(dev,&ppid,true);+if(err)+returnerr;++returnnbp_switchdev_mark_set(p,ppid,extack);+}+EXPORT_SYMBOL_GPL(switchdev_bridge_port_offload);++intswitchdev_bridge_port_unoffload(structnet_device*dev)+{+structnetdev_phys_item_idppid;+structnet_bridge_port*p;+interr;+
On Mon, Mar 22, 2021 at 18:19, Vladimir Oltean [off-list ref] wrote:
On Mon, Mar 22, 2021 at 04:44:41PM +0100, Tobias Waldekranz wrote:
quoted
I do not know if it is a problem or not, more of an observation: This is
not guaranteed to be an exact replay of the events that the bridge port
(i.e. bond0 or whatever) has received since, in fdb_insert, we exit
early when adding local entries if that address is already in the
database.
Do we have to guard against this somehow? Or maybe we should consider
the current behavior a bug and make sure to always send the event in the
first place?
I don't really understand what you're saying.
fdb_insert has:
fdb = br_fdb_find(br, addr, vid);
if (fdb) {
/* it is okay to have multiple ports with same
* address, just use the first one.
*/
if (test_bit(BR_FDB_LOCAL, &fdb->flags))
return 0;
br_warn(br, "adding interface %s with same address as a received packet (addr:%pM, vlan:%u)\n",
source ? source->dev->name : br->dev->name, addr, vid);
fdb_delete(br, fdb, true);
}
fdb = fdb_create(br, source, addr, vid,
BIT(BR_FDB_LOCAL) | BIT(BR_FDB_STATIC));
Basically, if the {addr, vid} pair already exists in the fdb, and it
points to a local entry, fdb_create is bypassed.
Whereas my br_fdb_replay() function iterates over br->fdb_list, which is
exactly where fdb_create() also lays its eggs. That is to say, unless
I'm missing something, that duplicate local FDB entries that skipped the
fdb_create() call in fdb_insert() because they were for already-existing
local FDB entries will also be skipped by br_fdb_replay(), because it
iterates over a br->fdb_list which contains unique local addresses.
Where am I wrong?
No you are right. I was thinking back to my attempt of offloading local
addresses and I distinctly remembered that local addresses could be
added without a notification being sent.
But that is not what is happening. It is just already inserted on
another port. So the notification would reach DSA, or not, depending on
ordering the of events. But there will be no discrepancy between that
and the replay.
From: Vladimir Oltean <olteanv@gmail.com> Date: 2021-03-22 17:14:15
On Mon, Mar 22, 2021 at 06:07:51PM +0100, Tobias Waldekranz wrote:
On Mon, Mar 22, 2021 at 18:19, Vladimir Oltean [off-list ref] wrote:
quoted
On Mon, Mar 22, 2021 at 04:44:41PM +0100, Tobias Waldekranz wrote:
quoted
I do not know if it is a problem or not, more of an observation: This is
not guaranteed to be an exact replay of the events that the bridge port
(i.e. bond0 or whatever) has received since, in fdb_insert, we exit
early when adding local entries if that address is already in the
database.
Do we have to guard against this somehow? Or maybe we should consider
the current behavior a bug and make sure to always send the event in the
first place?
I don't really understand what you're saying.
fdb_insert has:
fdb = br_fdb_find(br, addr, vid);
if (fdb) {
/* it is okay to have multiple ports with same
* address, just use the first one.
*/
if (test_bit(BR_FDB_LOCAL, &fdb->flags))
return 0;
br_warn(br, "adding interface %s with same address as a received packet (addr:%pM, vlan:%u)\n",
source ? source->dev->name : br->dev->name, addr, vid);
fdb_delete(br, fdb, true);
}
fdb = fdb_create(br, source, addr, vid,
BIT(BR_FDB_LOCAL) | BIT(BR_FDB_STATIC));
Basically, if the {addr, vid} pair already exists in the fdb, and it
points to a local entry, fdb_create is bypassed.
Whereas my br_fdb_replay() function iterates over br->fdb_list, which is
exactly where fdb_create() also lays its eggs. That is to say, unless
I'm missing something, that duplicate local FDB entries that skipped the
fdb_create() call in fdb_insert() because they were for already-existing
local FDB entries will also be skipped by br_fdb_replay(), because it
iterates over a br->fdb_list which contains unique local addresses.
Where am I wrong?
No you are right. I was thinking back to my attempt of offloading local
addresses and I distinctly remembered that local addresses could be
added without a notification being sent.
But that is not what is happening. It is just already inserted on
another port. So the notification would reach DSA, or not, depending on
ordering the of events. But there will be no discrepancy between that
and the replay.
I'm not saying that the bridge isn't broken, because it is, but for
different reasons, as explained here:
https://patchwork.kernel.org/project/netdevbpf/patch/20210224114350.2791260-9-olteanv@gmail.com/
What I can do is I can make br_switchdev_fdb_notify() skip fdb entries
with the BR_FDB_LOCAL bit set, and target that patch against "net", with
a Fixes: tag of 6b26b51b1d13 ("net: bridge: Add support for notifying
devices about FDB add/del").
Then I can also skip the entries with BR_FDB_LOCAL from br_fdb_replay.
Then, when I return to the "RX filtering for DSA" series, I can add the
"is_local" bit to switchdev FDB objects, and make all drivers reject
"is_local" entries (which is what the linked patch does) unless more
specific treatment is applied to those (trap to CPU).
Nikolay?
From: Vladimir Oltean <olteanv@gmail.com> Date: 2021-03-22 22:24:09
On Mon, Mar 22, 2021 at 04:04:01PM +0800, DENG Qingfang wrote:
On Fri, Mar 19, 2021 at 6:49 PM Vladimir Oltean [off-list ref] wrote:
quoted
Why would you even want to look at the source net device for forwarding?
I'd say that if dp->bridge_dev is NULL in the xmit function, you certainly
want to bypass address learning if you can. Maybe also for link-local traffic.
Also for trapped traffic (snooping, tc-flower trap action) if the CPU
sends them back.
This sounds line an interesting use case, please tell me more about what
commands I could run to reinject trapped packets into the hardware data
path.