From: Jiri Pirko <redacted>
This patchset introduces support for modular switch systems.
NVIDIA Mellanox SN4800 is an example of such. It contains 8 slots
to accomodate line cards. Available line cards include:
16X 100GbE (QSFP28)
8X 200GbE (QSFP56)
4X 400GbE (QSFP-DD)
Similar to split cabels, it is essencial for the correctness of
configuration and funcionality to treat the line card entities
in the same way, no matter the line card is inserted or not.
Meaning, the netdevice of a line card port cannot just disappear
when line card is removed. Also, system admin needs to be able
to apply configuration on netdevices belonging to line card port
even before the linecard gets inserted.
To resolve this, a concept of "provisioning" is introduced.
The user may "provision" certain slot with a line card type.
Driver then creates all instances (devlink ports, netdevices, etc)
related to this line card type. The carrier of netdevices stays down.
Once the line card is inserted and activated, the carrier of the
related netdevices goes up.
Once user does not want to use the line card related instances
anymore, he can "unprovision" the slot. Driver then removes the
instances.
Patches 1-5 are extending devlink driver API and UAPI in order to
register, show, dump, provision and activate the line card.
Patches 6-9 are implementing the introduced API in netdevsim
Example:
# Create a new netdevsim device, with no ports and 2 line cards:
$ echo "10 0 2" >/sys/bus/netdevsim/new_device
$ devlink port # No ports are listed
$ devlink lc
netdevsim/netdevsim10:
lc 0 state unprovisioned
supported_types:
card1port card2ports card4ports
lc 1 state unprovisioned
supported_types:
card1port card2ports card4ports
# Note that driver advertizes supported line card types. In case of
# netdevsim, these are 3.
$ devlink lc provision netdevsim/netdevsim10 lc 0 type card4ports
$ devlink lc
netdevsim/netdevsim10:
lc 0 state provisioned type card4ports
supported_types:
card1port card2ports card4ports
lc 1 state unprovisioned
supported_types:
card1port card2ports card4ports
$ devlink port
netdevsim/netdevsim10/1000: type eth netdev eni10nl0p1 flavour physical lc 0 port 1 splittable false
netdevsim/netdevsim10/1001: type eth netdev eni10nl0p2 flavour physical lc 0 port 2 splittable false
netdevsim/netdevsim10/1002: type eth netdev eni10nl0p3 flavour physical lc 0 port 3 splittable false
netdevsim/netdevsim10/1003: type eth netdev eni10nl0p4 flavour physical lc 0 port 4 splittable false
# ^^ ^^^^
# netdev name adjusted index of a line card this port belongs to
$ ip link set eni10nl0p1 up
$ ip link show eni10nl0p1
165: eni10nl0p1: <NO-CARRIER,BROADCAST,NOARP,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
link/ether 7e:2d:05:93:d3:d1 brd ff:ff:ff:ff:ff:ff
# Now activate the line card using debugfs. That emulates plug-in event
# on real hardware:
$ echo "Y"> /sys/kernel/debug/netdevsim/netdevsim10/linecards/0/active
$ ip link show eni10nl0p1
165: eni10nl0p1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 7e:2d:05:93:d3:d1 brd ff:ff:ff:ff:ff:ff
# The carrier is UP now.
Jiri Pirko (10):
devlink: add support to create line card and expose to user
devlink: implement line card provisioning
devlink: implement line card active state
devlink: append split port number to the port name
devlink: add port to line card relationship set
netdevsim: introduce line card support
netdevsim: allow port objects to be linked with line cards
netdevsim: create devlink line card object and implement provisioning
netdevsim: implement line card activation
selftests: add netdevsim devlink lc test
drivers/net/netdevsim/bus.c | 21 +-
drivers/net/netdevsim/dev.c | 370 ++++++++++++++-
drivers/net/netdevsim/netdev.c | 2 +
drivers/net/netdevsim/netdevsim.h | 23 +
include/net/devlink.h | 44 ++
include/uapi/linux/devlink.h | 25 +
net/core/devlink.c | 443 +++++++++++++++++-
.../drivers/net/netdevsim/devlink.sh | 62 ++-
8 files changed, 964 insertions(+), 26 deletions(-)
--
2.26.2
From: Jiri Pirko <redacted>
Extend the devlink API so the driver is going to be able to create and
destroy linecard instances. There can be multiple line cards per devlink
device. Expose this new type of object over devlink netlink API to the
userspace, with notifications.
Signed-off-by: Jiri Pirko <redacted>
---
include/net/devlink.h | 10 ++
include/uapi/linux/devlink.h | 7 ++
net/core/devlink.c | 227 ++++++++++++++++++++++++++++++++++-
3 files changed, 243 insertions(+), 1 deletion(-)
@@ -405,16 +445,18 @@ devlink_region_snapshot_get_by_id(struct devlink_region *region, u32 id)#define DEVLINK_NL_FLAG_NEED_PORT BIT(0)#define DEVLINK_NL_FLAG_NEED_DEVLINK_OR_PORT BIT(1)+#define DEVLINK_NL_FLAG_NEED_LINECARD BIT(2)/* The per devlink instance lock is taken by default in the pre-doit*operation,yetseveralcommandsdonotrequirethis.Theglobal*devlinklockistakenandprotectsfromdisruptionbyuser-calls.*/-#define DEVLINK_NL_FLAG_NO_LOCK BIT(2)+#define DEVLINK_NL_FLAG_NO_LOCK BIT(3)staticintdevlink_nl_pre_doit(conststructgenl_ops*ops,structsk_buff*skb,structgenl_info*info){+structdevlink_linecard*linecard;structdevlink_port*devlink_port;structdevlink*devlink;interr;
@@ -439,6 +481,13 @@ static int devlink_nl_pre_doit(const struct genl_ops *ops,devlink_port=devlink_port_get_from_info(devlink,info);if(!IS_ERR(devlink_port))info->user_ptr[1]=devlink_port;+}elseif(ops->internal_flags&DEVLINK_NL_FLAG_NEED_LINECARD){+linecard=devlink_linecard_get_from_info(devlink,info);+if(IS_ERR(linecard)){+err=PTR_ERR(linecard);+gotounlock;+}+info->user_ptr[1]=linecard;}return0;
@@ -1136,6 +1185,121 @@ static int devlink_nl_cmd_port_unsplit_doit(struct sk_buff *skb,returndevlink_port_unsplit(devlink,port_index,info->extack);}+staticintdevlink_nl_linecard_fill(structsk_buff*msg,+structdevlink*devlink,+structdevlink_linecard*linecard,+enumdevlink_commandcmd,u32portid,+u32seq,intflags,+structnetlink_ext_ack*extack)+{+void*hdr;++hdr=genlmsg_put(msg,portid,seq,&devlink_nl_family,flags,cmd);+if(!hdr)+return-EMSGSIZE;++if(devlink_nl_put_handle(msg,devlink))+gotonla_put_failure;+if(nla_put_u32(msg,DEVLINK_ATTR_LINECARD_INDEX,linecard->index))+gotonla_put_failure;++genlmsg_end(msg,hdr);+return0;++nla_put_failure:+genlmsg_cancel(msg,hdr);+return-EMSGSIZE;+}++staticvoiddevlink_linecard_notify(structdevlink_linecard*linecard,+enumdevlink_commandcmd)+{+structdevlink*devlink=linecard->devlink;+structsk_buff*msg;+interr;++WARN_ON(cmd!=DEVLINK_CMD_LINECARD_NEW&&+cmd!=DEVLINK_CMD_LINECARD_DEL);++msg=nlmsg_new(NLMSG_DEFAULT_SIZE,GFP_KERNEL);+if(!msg)+return;++err=devlink_nl_linecard_fill(msg,devlink,linecard,cmd,0,0,0,+NULL);+if(err){+nlmsg_free(msg);+return;+}++genlmsg_multicast_netns(&devlink_nl_family,devlink_net(devlink),+msg,0,DEVLINK_MCGRP_CONFIG,GFP_KERNEL);+}++staticintdevlink_nl_cmd_linecard_get_doit(structsk_buff*skb,+structgenl_info*info)+{+structdevlink_linecard*linecard=info->user_ptr[1];+structdevlink*devlink=linecard->devlink;+structsk_buff*msg;+interr;++msg=nlmsg_new(NLMSG_DEFAULT_SIZE,GFP_KERNEL);+if(!msg)+return-ENOMEM;++err=devlink_nl_linecard_fill(msg,devlink,linecard,+DEVLINK_CMD_LINECARD_NEW,+info->snd_portid,info->snd_seq,0,+info->extack);+if(err){+nlmsg_free(msg);+returnerr;+}++returngenlmsg_reply(msg,info);+}++staticintdevlink_nl_cmd_linecard_get_dumpit(structsk_buff*msg,+structnetlink_callback*cb)+{+structdevlink_linecard*linecard;+structdevlink*devlink;+intstart=cb->args[0];+intidx=0;+interr;++mutex_lock(&devlink_mutex);+list_for_each_entry(devlink,&devlink_list,list){+if(!net_eq(devlink_net(devlink),sock_net(msg->sk)))+continue;+mutex_lock(&devlink->lock);+list_for_each_entry(linecard,&devlink->linecard_list,list){+if(idx<start){+idx++;+continue;+}+err=devlink_nl_linecard_fill(msg,devlink,linecard,+DEVLINK_CMD_LINECARD_NEW,+NETLINK_CB(cb->skb).portid,+cb->nlh->nlmsg_seq,+NLM_F_MULTI,+cb->extack);+if(err){+mutex_unlock(&devlink->lock);+gotoout;+}+idx++;+}+mutex_unlock(&devlink->lock);+}+out:+mutex_unlock(&devlink_mutex);++cb->args[0]=idx;+returnmsg->len;+}+staticintdevlink_nl_sb_fill(structsk_buff*msg,structdevlink*devlink,structdevlink_sb*devlink_sb,enumdevlink_commandcmd,u32portid,
From: Jiri Pirko <redacted>
In order to be able to configure all needed stuff on a port/netdevice
of a line card without the line card being present, introduce line card
provisioning. Basically provisioning will create a placeholder for
instances (ports/netdevices) for a line card type.
Allow the user to query the supported line card types over line card
get command. Then implement two netlink commands to allow user to
provision/unprovision the line card with selected line card type.
On the driver API side, add provision/unprovision ops and supported
types array to be advertised. Upon provision op call, the driver should
take care of creating the instances for the particular line card type.
Introduce provision_set/clear() functions to be called by the driver
once the provisioning/unprovisioning is done on its side.
Signed-off-by: Jiri Pirko <redacted>
---
include/net/devlink.h | 31 +++++++-
include/uapi/linux/devlink.h | 17 +++++
net/core/devlink.c | 141 ++++++++++++++++++++++++++++++++++-
3 files changed, 185 insertions(+), 4 deletions(-)
@@ -131,6 +131,9 @@ enum devlink_command {DEVLINK_CMD_LINECARD_NEW,DEVLINK_CMD_LINECARD_DEL,+DEVLINK_CMD_LINECARD_PROVISION,+DEVLINK_CMD_LINECARD_UNPROVISION,+/* add new commands above here */__DEVLINK_CMD_MAX,DEVLINK_CMD_MAX=__DEVLINK_CMD_MAX-1
@@ -329,6 +332,17 @@ enum devlink_reload_limit {#define DEVLINK_RELOAD_LIMITS_VALID_MASK (_BITUL(__DEVLINK_RELOAD_LIMIT_MAX) - 1)+enumdevlink_linecard_state{+DEVLINK_LINECARD_STATE_UNSPEC,+DEVLINK_LINECARD_STATE_UNPROVISIONED,+DEVLINK_LINECARD_STATE_UNPROVISIONING,+DEVLINK_LINECARD_STATE_PROVISIONING,+DEVLINK_LINECARD_STATE_PROVISIONED,++__DEVLINK_LINECARD_STATE_MAX,+DEVLINK_LINECARD_STATE_MAX=__DEVLINK_LINECARD_STATE_MAX-1+};+enumdevlink_attr{/* don't change the order or add anything between, this is ABI! */DEVLINK_ATTR_UNSPEC,
@@ -535,6 +549,9 @@ enum devlink_attr {DEVLINK_ATTR_RELOAD_ACTION_STATS,/* nested */DEVLINK_ATTR_LINECARD_INDEX,/* u32 */+DEVLINK_ATTR_LINECARD_STATE,/* u8 */+DEVLINK_ATTR_LINECARD_TYPE,/* string */+DEVLINK_ATTR_LINECARD_SUPPORTED_TYPES,/* nested *//* add new attributes above here, update the policy in devlink.c */
@@ -1192,7 +1192,9 @@ static int devlink_nl_linecard_fill(struct sk_buff *msg,u32seq,intflags,structnetlink_ext_ack*extack){+structnlattr*attr;void*hdr;+inti;hdr=genlmsg_put(msg,portid,seq,&devlink_nl_family,flags,cmd);if(!hdr)
@@ -1202,6 +1204,22 @@ static int devlink_nl_linecard_fill(struct sk_buff *msg,gotonla_put_failure;if(nla_put_u32(msg,DEVLINK_ATTR_LINECARD_INDEX,linecard->index))gotonla_put_failure;+if(nla_put_u8(msg,DEVLINK_ATTR_LINECARD_STATE,linecard->state))+gotonla_put_failure;+if(linecard->state>=DEVLINK_LINECARD_STATE_PROVISIONED&&+nla_put_string(msg,DEVLINK_ATTR_LINECARD_TYPE,+linecard->provisioned_type))+gotonla_put_failure;++attr=nla_nest_start(msg,DEVLINK_ATTR_LINECARD_SUPPORTED_TYPES);+if(!attr)+return-EMSGSIZE;+for(i=0;i<linecard->ops->supported_types_count;i++){+if(nla_put_string(msg,DEVLINK_ATTR_LINECARD_TYPE,+linecard->ops->supported_types[i]))+gotonla_put_failure;+}+nla_nest_end(msg,attr);genlmsg_end(msg,hdr);return0;
@@ -1300,6 +1318,68 @@ static int devlink_nl_cmd_linecard_get_dumpit(struct sk_buff *msg,returnmsg->len;}+staticintdevlink_nl_cmd_linecard_provision_doit(structsk_buff*skb,+structgenl_info*info)+{+structdevlink_linecard*linecard=info->user_ptr[1];+constchar*type;+inti;++if(linecard->state==DEVLINK_LINECARD_STATE_PROVISIONING){+NL_SET_ERR_MSG_MOD(info->extack,"Linecard is currently being provisioned");+return-EBUSY;+}+if(linecard->state==DEVLINK_LINECARD_STATE_UNPROVISIONING){+NL_SET_ERR_MSG_MOD(info->extack,"Linecard is currently being unprovisioned");+return-EBUSY;+}+if(linecard->state!=DEVLINK_LINECARD_STATE_UNPROVISIONED){+NL_SET_ERR_MSG_MOD(info->extack,"Linecard already provisioned");+return-EBUSY;+}++if(!info->attrs[DEVLINK_ATTR_LINECARD_TYPE]){+NL_SET_ERR_MSG_MOD(info->extack,"Provision type not provided");+return-EINVAL;+}++type=nla_data(info->attrs[DEVLINK_ATTR_LINECARD_TYPE]);+for(i=0;i<linecard->ops->supported_types_count;i++){+if(!strcmp(linecard->ops->supported_types[i],type)){+linecard->state=DEVLINK_LINECARD_STATE_PROVISIONING;+devlink_linecard_notify(linecard,DEVLINK_CMD_LINECARD_NEW);+returnlinecard->ops->provision(linecard,+linecard->priv,i,+info->extack);+}+}+NL_SET_ERR_MSG_MOD(info->extack,"Unsupported provision type provided");+return-EINVAL;+}++staticintdevlink_nl_cmd_linecard_unprovision_doit(structsk_buff*skb,+structgenl_info*info)+{+structdevlink_linecard*linecard=info->user_ptr[1];++if(linecard->state==DEVLINK_LINECARD_STATE_PROVISIONING){+NL_SET_ERR_MSG_MOD(info->extack,"Linecard is currently being provisioned");+return-EBUSY;+}+if(linecard->state==DEVLINK_LINECARD_STATE_UNPROVISIONING){+NL_SET_ERR_MSG_MOD(info->extack,"Linecard is currently being unprovisioned");+return-EBUSY;+}+if(linecard->state==DEVLINK_LINECARD_STATE_UNPROVISIONED){+NL_SET_ERR_MSG_MOD(info->extack,"Linecard is not provisioned");+return-EOPNOTSUPP;+}+linecard->state=DEVLINK_LINECARD_STATE_UNPROVISIONING;+devlink_linecard_notify(linecard,DEVLINK_CMD_LINECARD_NEW);+returnlinecard->ops->unprovision(linecard,linecard->priv,+info->extack);+}+staticintdevlink_nl_sb_fill(structsk_buff*msg,structdevlink*devlink,structdevlink_sb*devlink_sb,enumdevlink_commandcmd,u32portid,
From: Jiri Pirko <redacted>
Allow driver to mark a lin ecards as active. Expose this state to the
userspace over devlink netlink interface with proper notifications.
Signed-off-by: Jiri Pirko <redacted>
---
include/net/devlink.h | 4 ++++
include/uapi/linux/devlink.h | 1 +
net/core/devlink.c | 46 ++++++++++++++++++++++++++++++++++++
3 files changed, 51 insertions(+)
From: Jiri Pirko <redacted>
Instead of doing sprintf twice in case the port is split or not, append
the split port suffix in case the port is split.
Signed-off-by: Jiri Pirko <redacted>
---
net/core/devlink.c | 10 ++++------
1 file changed, 4 insertions(+), 6 deletions(-)
From: Jiri Pirko <redacted>
Add support for line card objects. Expose them over debugfs and allow
user to specify number of line cards to be created for a new device.
Similar to ports, the number of line cards is fixed.
Extend "new_device" sysfs file write by third number to allow to specify
number line cards like this:
$ echo "10 4 2" >/sys/bus/netdevsim/new_device
This command asks to create two line cards. By default, if this number
is not preset, no line card is created.
Signed-off-by: Jiri Pirko <redacted>
---
drivers/net/netdevsim/bus.c | 17 +++--
drivers/net/netdevsim/dev.c | 108 +++++++++++++++++++++++++++++-
drivers/net/netdevsim/netdevsim.h | 15 +++++
3 files changed, 133 insertions(+), 7 deletions(-)
@@ -179,29 +179,34 @@ static struct device_type nsim_bus_dev_type = {};staticstructnsim_bus_dev*-nsim_bus_dev_new(unsignedintid,unsignedintport_count);+nsim_bus_dev_new(unsignedintid,unsignedintport_count,+unsignedintlinecard_count);staticssize_tnew_device_store(structbus_type*bus,constchar*buf,size_tcount){structnsim_bus_dev*nsim_bus_dev;+unsignedintlinecard_count;unsignedintport_count;unsignedintid;interr;-err=sscanf(buf,"%u %u",&id,&port_count);+err=sscanf(buf,"%u %u %u",&id,&port_count,&linecard_count);switch(err){case1:port_count=1;fallthrough;case2:+linecard_count=0;+fallthrough;+case3:if(id>INT_MAX){pr_err("Value of \"id\" is too big.\n");return-EINVAL;}break;default:-pr_err("Format for adding new device is \"id port_count\" (uint uint).\n");+pr_err("Format for adding new device is \"id port_count linecard_count\" (uint uint uint).\n");return-EINVAL;}
@@ -328,6 +334,7 @@ nsim_bus_dev_new(unsigned int id, unsigned int port_count)nsim_bus_dev->dev.bus=&nsim_bus;nsim_bus_dev->dev.type=&nsim_bus_dev_type;nsim_bus_dev->port_count=port_count;+nsim_bus_dev->linecard_count=linecard_count;nsim_bus_dev->initial_net=current->nsproxy->net_ns;mutex_init(&nsim_bus_dev->nsim_bus_reload_lock);/* Disallow using nsim_bus_dev */
@@ -206,6 +219,7 @@ struct nsim_dev {structnetdev_phys_item_idswitch_id;structlist_headport_list;structmutexport_list_lock;/* protects port list */+structlist_headlinecard_list;boolfw_update_status;u32fw_update_overwrite_mask;u32max_macs;
@@ -287,6 +301,7 @@ struct nsim_bus_dev {structdevicedev;structlist_headlist;unsignedintport_count;+unsignedintlinecard_count;structnet*initial_net;/* Purpose of this is to carry net pointer*duringtheprobetimeonly.*/
From: Jiri Pirko <redacted>
Line cards contain ports. Allow ports to be places on the line cards.
Track the ports that belong under certain line card. Make sure that
the line card port carrier is down, as it will be taken up later on
during "activation".
Signed-off-by: Jiri Pirko <redacted>
---
drivers/net/netdevsim/bus.c | 4 +--
drivers/net/netdevsim/dev.c | 48 +++++++++++++++++++++++++------
drivers/net/netdevsim/netdev.c | 2 ++
drivers/net/netdevsim/netdevsim.h | 4 +++
4 files changed, 48 insertions(+), 10 deletions(-)
From: Jiri Pirko <redacted>
Use devlink_linecard_create/destroy() to register the line card with
devlink core. Implement provisioning ops with a list of supported
line cards. To avoid deadlock and to mimic actual HW flow, use workqueue
to add/del ports during provisioning as the port add/del calls
devlink_port_register/unregister() which take devlink mutex.
Signed-off-by: Jiri Pirko <redacted>
---
drivers/net/netdevsim/dev.c | 135 +++++++++++++++++++++++++++++-
drivers/net/netdevsim/netdevsim.h | 4 +
2 files changed, 138 insertions(+), 1 deletion(-)
From: Jiri Pirko <redacted>
On real HW, the activation typically happens upon line card insertion.
Emulate such event using write to debugfs file "active".
Signed-off-by: Jiri Pirko <redacted>
---
drivers/net/netdevsim/dev.c | 81 +++++++++++++++++++++++++++++++++++++
1 file changed, 81 insertions(+)
@@ -507,10 +508,67 @@ dummy_reporter_test()log_test"dummy reporter test"}+check_linecards_state()+{+localexpected_state_0=$1+localexpected_state_1=$2++localstate=$(devlinklcshow$DL_HANDLElc0-j|jq-e-r".[][][].state")+check_err$?"Failed to get linecard 0 state"++["$state"=="$expected_state_0"]+check_err$?"Unexpected linecard 0 state (got $state, expected $expected_state_0)"++localstate=$(devlinklcshow$DL_HANDLElc1-j|jq-e-r".[][][].state")+check_err$?"Failed to get linecard 1 state"++["$state"=="$expected_state_1"]+check_err$?"Unexpected linecard 1 state (got $state, expected $expected_state_1)"+}++linecard_test()+{+RET=0++check_linecards_state"unprovisioned""unprovisioned"++devlinklcprovision$DL_HANDLElc0typecard2ports+check_err$?"Failed to provision linecard 0 with card2ports"++check_linecards_state"provisioned""unprovisioned"++devlinklcprovision$DL_HANDLElc1typecard4ports+check_err$?"Failed to provision linecard 0 with card4ports"++check_linecards_state"provisioned""provisioned"++echo"Y">$DEBUGFS_DIR/linecards/0/active+check_err$?"Failed to set lincard 0 active"++check_linecards_state"active""provisioned"++echo"Y">$DEBUGFS_DIR/linecards/1/active+check_err$?"Failed to set lincard 1 active"++check_linecards_state"active""active"++devlinklcunprovision$DL_HANDLElc0+check_err$?"Failed to unprovision linecard 0"++check_linecards_state"unprovisioned""active"++devlinklcunprovision$DL_HANDLElc1+check_err$?"Failed to unprovision linecard 1"++check_linecards_state"unprovisioned""unprovisioned"++log_test"linecard test"+}+ setup_prepare(){modprobenetdevsim-echo"$BUS_ADDR$PORT_COUNT">/sys/bus/netdevsim/new_device+echo"$BUS_ADDR$PORT_COUNT$LINECARD_COUNT">/sys/bus/netdevsim/new_devicewhile[!-d$SYSFS_NET_DIR];do:;done}
From: Jiri Pirko <redacted>
In order to properly inform user about relationship between port and
line card, introduce a driver API to set line card for a port. Use this
information to extend port devlink netlink message by line card index
and also include the line card index into phys_port_name and by that
into a netdevice name.
Signed-off-by: Jiri Pirko <redacted>
---
include/net/devlink.h | 3 +++
net/core/devlink.c | 25 ++++++++++++++++++++++++-
2 files changed, 27 insertions(+), 1 deletion(-)
@@ -1414,6 +1418,8 @@ static const struct dl_args_metadata dl_args_required[] = {{DL_OPT_TRAP_NAME,"Trap's name is expected."},{DL_OPT_TRAP_GROUP_NAME,"Trap group's name is expected."},{DL_OPT_PORT_FUNCTION_HW_ADDR,"Port function's hardware address is expected."},+{DL_OPT_LINECARD,"Linecard index expected."},+{DL_OPT_LINECARD_TYPE,"Linecard type expected."},};staticintdl_args_finding_required_validate(uint64_to_required,
@@ -4005,6 +4040,156 @@ static int cmd_port(struct dl *dl)return-ENOENT;}+staticvoidcmd_linecard_help(void)+{+pr_err("Usage: devlink lc show [ DEV [ lc LC_INDEX ] ]\n");+pr_err(" devlink lc provision DEV lc LC_INDEX type LC_TYPE\n");+pr_err(" devlink lc unprovision DEV lc LC_INDEX\n");+}++staticconstchar*linecard_state_name(uint16_tflavour)+{+switch(flavour){+caseDEVLINK_LINECARD_STATE_UNPROVISIONED:+return"unprovisioned";+caseDEVLINK_LINECARD_STATE_UNPROVISIONING:+return"unprovisioning";+caseDEVLINK_LINECARD_STATE_PROVISIONING:+return"provisioning";+caseDEVLINK_LINECARD_STATE_PROVISIONED:+return"provisioned";+caseDEVLINK_LINECARD_STATE_ACTIVE:+return"active";+default:+return"<unknown state>";+}+}++staticvoidpr_out_linecard_supported_types(structdl*dl,structnlattr**tb)+{+structnlattr*nla_types=tb[DEVLINK_ATTR_LINECARD_SUPPORTED_TYPES];+structnlattr*nla_type;++if(!nla_types)+return;++pr_out_array_start(dl,"supported_types");+check_indent_newline(dl);+mnl_attr_for_each_nested(nla_type,nla_types){+print_string(PRINT_ANY,NULL," %s",+mnl_attr_get_str(nla_type));+}+pr_out_array_end(dl);+}++staticvoidpr_out_linecard(structdl*dl,structnlattr**tb)+{+uint8_tstate;++pr_out_handle_start_arr(dl,tb);+check_indent_newline(dl);+print_uint(PRINT_ANY,"lc","lc %u",+mnl_attr_get_u32(tb[DEVLINK_ATTR_LINECARD_INDEX]));+state=mnl_attr_get_u8(tb[DEVLINK_ATTR_LINECARD_STATE]);+print_string(PRINT_ANY,"state"," state %s",+linecard_state_name(state));+if(tb[DEVLINK_ATTR_LINECARD_TYPE])+print_string(PRINT_ANY,"type"," type %s",+mnl_attr_get_str(tb[DEVLINK_ATTR_LINECARD_TYPE]));+pr_out_linecard_supported_types(dl,tb);+pr_out_handle_end(dl);+}++staticintcmd_linecard_show_cb(conststructnlmsghdr*nlh,void*data)+{+structdl*dl=data;+structnlattr*tb[DEVLINK_ATTR_MAX+1]={};+structgenlmsghdr*genl=mnl_nlmsg_get_payload(nlh);++mnl_attr_parse(nlh,sizeof(*genl),attr_cb,tb);+if(!tb[DEVLINK_ATTR_BUS_NAME]||!tb[DEVLINK_ATTR_DEV_NAME]||+!tb[DEVLINK_ATTR_LINECARD_INDEX]||+!tb[DEVLINK_ATTR_LINECARD_STATE])+returnMNL_CB_ERROR;+pr_out_linecard(dl,tb);+returnMNL_CB_OK;+}++staticintcmd_linecard_show(structdl*dl)+{+structnlmsghdr*nlh;+uint16_tflags=NLM_F_REQUEST|NLM_F_ACK;+interr;++if(dl_argc(dl)==0)+flags|=NLM_F_DUMP;++nlh=mnlg_msg_prepare(dl->nlg,DEVLINK_CMD_LINECARD_GET,flags);++if(dl_argc(dl)>0){+err=dl_argv_parse_put(nlh,dl,DL_OPT_HANDLE,+DL_OPT_LINECARD);+if(err)+returnerr;+}++pr_out_section_start(dl,"lc");+err=_mnlg_socket_sndrcv(dl->nlg,nlh,cmd_linecard_show_cb,dl);+pr_out_section_end(dl);+returnerr;+}++staticintcmd_linecard_provision(structdl*dl)+{+structnlmsghdr*nlh;+interr;++nlh=mnlg_msg_prepare(dl->nlg,DEVLINK_CMD_LINECARD_PROVISION,+NLM_F_REQUEST|NLM_F_ACK);++err=dl_argv_parse_put(nlh,dl,DL_OPT_HANDLE|DL_OPT_LINECARD|+DL_OPT_LINECARD_TYPE,0);+if(err)+returnerr;++return_mnlg_socket_sndrcv(dl->nlg,nlh,NULL,NULL);+}++staticintcmd_linecard_unprovision(structdl*dl)+{+structnlmsghdr*nlh;+interr;++nlh=mnlg_msg_prepare(dl->nlg,DEVLINK_CMD_LINECARD_UNPROVISION,+NLM_F_REQUEST|NLM_F_ACK);++err=dl_argv_parse_put(nlh,dl,DL_OPT_HANDLE|DL_OPT_LINECARD,0);+if(err)+returnerr;++return_mnlg_socket_sndrcv(dl->nlg,nlh,NULL,NULL);+}++staticintcmd_linecard(structdl*dl)+{+if(dl_argv_match(dl,"help")){+cmd_linecard_help();+return0;+}elseif(dl_argv_match(dl,"show")||+dl_argv_match(dl,"list")||dl_no_arg(dl)){+dl_arg_inc(dl);+returncmd_linecard_show(dl);+}elseif(dl_argv_match(dl,"provision")){+dl_arg_inc(dl);+returncmd_linecard_provision(dl);+}elseif(dl_argv_match(dl,"unprovision")){+dl_arg_inc(dl);+returncmd_linecard_unprovision(dl);+}+pr_err("Command \"%s\" not found\n",dl_argv(dl));+return-ENOENT;+}+staticvoidcmd_sb_help(void){pr_err("Usage: devlink sb show [ DEV [ sb SB_INDEX ] ]\n");
@@ -5059,6 +5253,18 @@ static int cmd_mon_show_cb(const struct nlmsghdr *nlh, void *data)pr_out_mon_header(genl->cmd);pr_out_trap_policer(dl,tb,false);break;+caseDEVLINK_CMD_LINECARD_GET:/* fall through */+caseDEVLINK_CMD_LINECARD_SET:/* fall through */+caseDEVLINK_CMD_LINECARD_NEW:/* fall through */+caseDEVLINK_CMD_LINECARD_DEL:+mnl_attr_parse(nlh,sizeof(*genl),attr_cb,tb);+if(!tb[DEVLINK_ATTR_BUS_NAME]||!tb[DEVLINK_ATTR_DEV_NAME]||+!tb[DEVLINK_ATTR_LINECARD_INDEX])+returnMNL_CB_ERROR;+pr_out_mon_header(genl->cmd);+pr_out_linecard(dl,tb);+pr_out_mon_footer();+break;}fflush(stdout);returnMNL_CB_OK;
From: Andrew Lunn <andrew@lunn.ch> Date: 2021-01-14 02:08:35
$ devlink lc provision netdevsim/netdevsim10 lc 0 type card4ports
$ devlink lc
netdevsim/netdevsim10:
lc 0 state provisioned type card4ports
supported_types:
card1port card2ports card4ports
lc 1 state unprovisioned
supported_types:
card1port card2ports card4ports
Hi Jiri
# Now activate the line card using debugfs. That emulates plug-in event
# on real hardware:
$ echo "Y"> /sys/kernel/debug/netdevsim/netdevsim10/linecards/0/active
$ ip link show eni10nl0p1
165: eni10nl0p1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 7e:2d:05:93:d3:d1 brd ff:ff:ff:ff:ff:ff
# The carrier is UP now.
What is missing from the devlink lc view is what line card is actually
in the slot. Say if i provision for a card4port, but actually insert a
card2port. It would be nice to have something like:
$ devlink lc
netdevsim/netdevsim10:
lc 0 state provisioned type card4ports
supported_types:
card1port card2ports card4ports
inserted_type:
card2ports;
lc 1 state unprovisioned
supported_types:
card1port card2ports card4ports
inserted_type:
None
I assume if i prevision for card4ports but actually install a
card2ports, all the interfaces stay down?
Maybe
should actually be
echo "card2ports" > /sys/kernel/debug/netdevsim/netdevsim10/linecards/0/active
so you can emulate somebody putting the wrong card in the slot?
Andrew
From: Jakub Kicinski <kuba@kernel.org> Date: 2021-01-14 02:27:59
On Wed, 13 Jan 2021 13:12:12 +0100 Jiri Pirko wrote:
This patchset introduces support for modular switch systems.
NVIDIA Mellanox SN4800 is an example of such. It contains 8 slots
to accomodate line cards. Available line cards include:
16X 100GbE (QSFP28)
8X 200GbE (QSFP56)
4X 400GbE (QSFP-DD)
Similar to split cabels, it is essencial for the correctness of
configuration and funcionality to treat the line card entities
in the same way, no matter the line card is inserted or not.
Meaning, the netdevice of a line card port cannot just disappear
when line card is removed. Also, system admin needs to be able
to apply configuration on netdevices belonging to line card port
even before the linecard gets inserted.
I don't understand why that would be. Please provide reasoning,
e.g. what the FW/HW limitation is.
To resolve this, a concept of "provisioning" is introduced.
The user may "provision" certain slot with a line card type.
Driver then creates all instances (devlink ports, netdevices, etc)
related to this line card type. The carrier of netdevices stays down.
Once the line card is inserted and activated, the carrier of the
related netdevices goes up.
Dunno what "line card" means for Mellovidia but I don't think
the analogy of port splitting works. To my knowledge traditional
line cards often carry processors w/ full MACs etc. so I'd say
plugging in a line card is much more like plugging in a new NIC.
There is no way to tell a breakout cable from normal one, so the
system has no chance to magically configure itself. Besides SFP
is just plugging a cable, not a module of the system..
Thu, Jan 14, 2021 at 03:07:18AM CET, andrew@lunn.ch wrote:
quoted
$ devlink lc provision netdevsim/netdevsim10 lc 0 type card4ports
$ devlink lc
netdevsim/netdevsim10:
lc 0 state provisioned type card4ports
supported_types:
card1port card2ports card4ports
lc 1 state unprovisioned
supported_types:
card1port card2ports card4ports
Hi Jiri
quoted
# Now activate the line card using debugfs. That emulates plug-in event
# on real hardware:
$ echo "Y"> /sys/kernel/debug/netdevsim/netdevsim10/linecards/0/active
$ ip link show eni10nl0p1
165: eni10nl0p1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 7e:2d:05:93:d3:d1 brd ff:ff:ff:ff:ff:ff
# The carrier is UP now.
What is missing from the devlink lc view is what line card is actually
in the slot. Say if i provision for a card4port, but actually insert a
card2port. It would be nice to have something like:
$ devlink lc
netdevsim/netdevsim10:
lc 0 state provisioned type card4ports
supported_types:
card1port card2ports card4ports
inserted_type:
card2ports;
lc 1 state unprovisioned
supported_types:
card1port card2ports card4ports
inserted_type:
None
I see. Yes, that might be doable. I'm noting this down.
I assume if i prevision for card4ports but actually install a
card2ports, all the interfaces stay down?
Yes, the card won't get activated in case or provision mismatch.
should actually be
echo "card2ports" > /sys/kernel/debug/netdevsim/netdevsim10/linecards/0/active
so you can emulate somebody putting the wrong card in the slot?
Thu, Jan 14, 2021 at 03:27:16AM CET, kuba@kernel.org wrote:
On Wed, 13 Jan 2021 13:12:12 +0100 Jiri Pirko wrote:
quoted
This patchset introduces support for modular switch systems.
NVIDIA Mellanox SN4800 is an example of such. It contains 8 slots
to accomodate line cards. Available line cards include:
16X 100GbE (QSFP28)
8X 200GbE (QSFP56)
4X 400GbE (QSFP-DD)
Similar to split cabels, it is essencial for the correctness of
configuration and funcionality to treat the line card entities
in the same way, no matter the line card is inserted or not.
Meaning, the netdevice of a line card port cannot just disappear
when line card is removed. Also, system admin needs to be able
to apply configuration on netdevices belonging to line card port
even before the linecard gets inserted.
I don't understand why that would be. Please provide reasoning,
e.g. what the FW/HW limitation is.
Well, for split cable, you need to be able to say:
port 2, split into 4. And you will have 4 netdevices. These netdevices
you can use to put into bridge, configure mtu, speeds, routes, etc.
These will exist no matter if the splitter cable is actually inserted or
not.
With linecards, this is very similar. By provisioning, you also create
certain number of ports, according to the linecard that you plan to
insert. And similarly to the splitter, the netdevices are created.
You may combine the linecard/splitter config when splitter cable is
connected to a linecard port. Then you provision a linecard,
port is going to appear and you will split this port.
quoted
To resolve this, a concept of "provisioning" is introduced.
The user may "provision" certain slot with a line card type.
Driver then creates all instances (devlink ports, netdevices, etc)
related to this line card type. The carrier of netdevices stays down.
Once the line card is inserted and activated, the carrier of the
related netdevices goes up.
Dunno what "line card" means for Mellovidia but I don't think
the analogy of port splitting works. To my knowledge traditional
line cards often carry processors w/ full MACs etc. so I'd say
plugging in a line card is much more like plugging in a new NIC.
No. It is basically a phy gearbox. The mac is not there. The interface
between asic and linecard are lanes. The linecards is basically an
attachable phy.
There is no way to tell a breakout cable from normal one, so the
system has no chance to magically configure itself. Besides SFP
is just plugging a cable, not a module of the system..
From: Jacob Keller <jacob.e.keller@intel.com> Date: 2021-01-14 22:57:06
On 1/13/2021 11:39 PM, Jiri Pirko wrote:
Thu, Jan 14, 2021 at 03:07:18AM CET, andrew@lunn.ch wrote:
quoted
I assume if i prevision for card4ports but actually install a
card2ports, all the interfaces stay down?
Yes, the card won't get activated in case or provision mismatch.
If you're able to detect the line card type when plugging it in, I don't
understand why you need system administrator to pre-provision it using
such an interface? Wouldn't it make more sense to simply detect the
case? Or is it that you expect these things to be moved around and want
to make sure that you can configure the associated netdevices before the
card is plugged in?
From: Jacob Keller <jacob.e.keller@intel.com> Date: 2021-01-14 22:59:20
On 1/13/2021 6:27 PM, Jakub Kicinski wrote:
On Wed, 13 Jan 2021 13:12:12 +0100 Jiri Pirko wrote:
quoted
This patchset introduces support for modular switch systems.
NVIDIA Mellanox SN4800 is an example of such. It contains 8 slots
to accomodate line cards. Available line cards include:
16X 100GbE (QSFP28)
8X 200GbE (QSFP56)
4X 400GbE (QSFP-DD)
Similar to split cabels, it is essencial for the correctness of
configuration and funcionality to treat the line card entities
in the same way, no matter the line card is inserted or not.
Meaning, the netdevice of a line card port cannot just disappear
when line card is removed. Also, system admin needs to be able
to apply configuration on netdevices belonging to line card port
even before the linecard gets inserted.
I don't understand why that would be. Please provide reasoning,
e.g. what the FW/HW limitation is.
I agree, I wouldn't imagine that plugging or unplugging line cards is
expected to be done on a regular basis?
quoted
To resolve this, a concept of "provisioning" is introduced.
The user may "provision" certain slot with a line card type.
Driver then creates all instances (devlink ports, netdevices, etc)
related to this line card type. The carrier of netdevices stays down.
Once the line card is inserted and activated, the carrier of the
related netdevices goes up.
Dunno what "line card" means for Mellovidia but I don't think
the analogy of port splitting works. To my knowledge traditional
line cards often carry processors w/ full MACs etc. so I'd say
plugging in a line card is much more like plugging in a new NIC.
Even if they didn't...
There is no way to tell a breakout cable from normal one, so the
system has no chance to magically configure itself. Besides SFP
is just plugging a cable, not a module of the system..
If you're able to tell what is plugged in, why would we want to force
user to provision ahead of time? Wouldn't it make more sense to just
instantiate them as the card is plugged in? I guess it might be useful
to allow programming the netdevices before the cable is actually
inserted... I guess I don't see why that is valuable.
It would be sort of like if you provision a PCI slot before a device is
plugged into it..
From: Jakub Kicinski <kuba@kernel.org> Date: 2021-01-14 23:21:56
On Thu, 14 Jan 2021 14:58:33 -0800 Jacob Keller wrote:
quoted
There is no way to tell a breakout cable from normal one, so the
system has no chance to magically configure itself. Besides SFP
is just plugging a cable, not a module of the system..
If you're able to tell what is plugged in, why would we want to force
user to provision ahead of time? Wouldn't it make more sense to just
instantiate them as the card is plugged in? I guess it might be useful
to allow programming the netdevices before the cable is actually
inserted... I guess I don't see why that is valuable.
It would be sort of like if you provision a PCI slot before a device is
plugged into it..
From: Jakub Kicinski <kuba@kernel.org> Date: 2021-01-14 23:30:57
On Thu, 14 Jan 2021 08:48:04 +0100 Jiri Pirko wrote:
Thu, Jan 14, 2021 at 03:27:16AM CET, kuba@kernel.org wrote:
quoted
On Wed, 13 Jan 2021 13:12:12 +0100 Jiri Pirko wrote:
quoted
This patchset introduces support for modular switch systems.
NVIDIA Mellanox SN4800 is an example of such. It contains 8 slots
to accomodate line cards. Available line cards include:
16X 100GbE (QSFP28)
8X 200GbE (QSFP56)
4X 400GbE (QSFP-DD)
Similar to split cabels, it is essencial for the correctness of
configuration and funcionality to treat the line card entities
in the same way, no matter the line card is inserted or not.
Meaning, the netdevice of a line card port cannot just disappear
when line card is removed. Also, system admin needs to be able
to apply configuration on netdevices belonging to line card port
even before the linecard gets inserted.
I don't understand why that would be. Please provide reasoning,
e.g. what the FW/HW limitation is.
Well, for split cable, you need to be able to say:
port 2, split into 4. And you will have 4 netdevices. These netdevices
you can use to put into bridge, configure mtu, speeds, routes, etc.
These will exist no matter if the splitter cable is actually inserted or
not.
The difference is that the line card is more detectable (I hope).
I'm not a SFP experts so maybe someone will correct me but AFAIU
the QSFP (for optics) is the same regardless of breakout. It's the
passive optical strands that are either bundled or not. So there is
no way for the system to detect the cable type (AFAIK).
Or to put it differently IMO the netdev should be provisioned if the
system has a port into which user can plug in a cable. When there is
a line card-sized hole in the chassis, I'd be surprised to see ports.
That said I never worked with real world routers so maybe that's what
they do. Maybe some with a Cisco router in the basement can tell us? :)
With linecards, this is very similar. By provisioning, you also create
certain number of ports, according to the linecard that you plan to
insert. And similarly to the splitter, the netdevices are created.
You may combine the linecard/splitter config when splitter cable is
connected to a linecard port. Then you provision a linecard,
port is going to appear and you will split this port.
quoted
quoted
To resolve this, a concept of "provisioning" is introduced.
The user may "provision" certain slot with a line card type.
Driver then creates all instances (devlink ports, netdevices, etc)
related to this line card type. The carrier of netdevices stays down.
Once the line card is inserted and activated, the carrier of the
related netdevices goes up.
Dunno what "line card" means for Mellovidia but I don't think
the analogy of port splitting works. To my knowledge traditional
line cards often carry processors w/ full MACs etc. so I'd say
plugging in a line card is much more like plugging in a new NIC.
No. It is basically a phy gearbox. The mac is not there. The interface
between asic and linecard are lanes. The linecards is basically an
attachable phy.
If the device really needs this configuration / can't detect things
automatically, then we gotta do something like what you have.
The only question is do we still want to call it a line card.
Sounds more like a front panel module. At Netronome we called
those phymods.
Thu, Jan 14, 2021 at 11:56:15PM CET, jacob.e.keller@intel.com wrote:
On 1/13/2021 11:39 PM, Jiri Pirko wrote:
quoted
Thu, Jan 14, 2021 at 03:07:18AM CET, andrew@lunn.ch wrote:
quoted
I assume if i prevision for card4ports but actually install a
card2ports, all the interfaces stay down?
Yes, the card won't get activated in case or provision mismatch.
If you're able to detect the line card type when plugging it in, I don't
understand why you need system administrator to pre-provision it using
such an interface? Wouldn't it make more sense to simply detect the
case? Or is it that you expect these things to be moved around and want
to make sure that you can configure the associated netdevices before the
card is plugged in?
Fri, Jan 15, 2021 at 12:30:13AM CET, kuba@kernel.org wrote:
On Thu, 14 Jan 2021 08:48:04 +0100 Jiri Pirko wrote:
quoted
Thu, Jan 14, 2021 at 03:27:16AM CET, kuba@kernel.org wrote:
quoted
On Wed, 13 Jan 2021 13:12:12 +0100 Jiri Pirko wrote:
quoted
This patchset introduces support for modular switch systems.
NVIDIA Mellanox SN4800 is an example of such. It contains 8 slots
to accomodate line cards. Available line cards include:
16X 100GbE (QSFP28)
8X 200GbE (QSFP56)
4X 400GbE (QSFP-DD)
Similar to split cabels, it is essencial for the correctness of
configuration and funcionality to treat the line card entities
in the same way, no matter the line card is inserted or not.
Meaning, the netdevice of a line card port cannot just disappear
when line card is removed. Also, system admin needs to be able
to apply configuration on netdevices belonging to line card port
even before the linecard gets inserted.
I don't understand why that would be. Please provide reasoning,
e.g. what the FW/HW limitation is.
Well, for split cable, you need to be able to say:
port 2, split into 4. And you will have 4 netdevices. These netdevices
you can use to put into bridge, configure mtu, speeds, routes, etc.
These will exist no matter if the splitter cable is actually inserted or
not.
The difference is that the line card is more detectable (I hope).
I'm not a SFP experts so maybe someone will correct me but AFAIU
the QSFP (for optics) is the same regardless of breakout. It's the
passive optical strands that are either bundled or not. So there is
no way for the system to detect the cable type (AFAIK).
For SFP module, you are able to detect those.
Or to put it differently IMO the netdev should be provisioned if the
system has a port into which user can plug in a cable. When there is
Not really. For slit cables, the ports are provisioned not matter which
cable is connected, slitter 1->2/1->4 or 1->1 cable.
a line card-sized hole in the chassis, I'd be surprised to see ports.
That said I never worked with real world routers so maybe that's what
they do. Maybe some with a Cisco router in the basement can tell us? :)
The need for provision/pre-configure splitter/linecard is that the
ports/netdevices do not disapper/reappear when you replace
splitter/linecard. Consider a faulty linecard with one port burned. You
just want to replace it with new one. And in that case, you really don't
want kernel to remove netdevices and possibly mess up routing for
example.
quoted
With linecards, this is very similar. By provisioning, you also create
certain number of ports, according to the linecard that you plan to
insert. And similarly to the splitter, the netdevices are created.
You may combine the linecard/splitter config when splitter cable is
connected to a linecard port. Then you provision a linecard,
port is going to appear and you will split this port.
quoted
quoted
To resolve this, a concept of "provisioning" is introduced.
The user may "provision" certain slot with a line card type.
Driver then creates all instances (devlink ports, netdevices, etc)
related to this line card type. The carrier of netdevices stays down.
Once the line card is inserted and activated, the carrier of the
related netdevices goes up.
Dunno what "line card" means for Mellovidia but I don't think
the analogy of port splitting works. To my knowledge traditional
line cards often carry processors w/ full MACs etc. so I'd say
plugging in a line card is much more like plugging in a new NIC.
No. It is basically a phy gearbox. The mac is not there. The interface
between asic and linecard are lanes. The linecards is basically an
attachable phy.
If the device really needs this configuration / can't detect things
automatically, then we gotta do something like what you have.
The only question is do we still want to call it a line card.
Sounds more like a front panel module. At Netronome we called
those phymods.
Sure, the name is up to the discussion. We call it "linecard"
internally. I don't care about the name.
Fri, Jan 15, 2021 at 12:20:58AM CET, kuba@kernel.org wrote:
On Thu, 14 Jan 2021 14:58:33 -0800 Jacob Keller wrote:
quoted
quoted
There is no way to tell a breakout cable from normal one, so the
system has no chance to magically configure itself. Besides SFP
is just plugging a cable, not a module of the system..
If you're able to tell what is plugged in, why would we want to force
user to provision ahead of time? Wouldn't it make more sense to just
instantiate them as the card is plugged in? I guess it might be useful
to allow programming the netdevices before the cable is actually
inserted... I guess I don't see why that is valuable.
It would be sort of like if you provision a PCI slot before a device is
plugged into it..
Yup, that's pretty much my thinking as well.
Please see my reply in the other sub-tread of this thread. Thanks!
On Wed, Jan 13, 2021 at 01:12:12PM +0100, Jiri Pirko wrote:
# Create a new netdevsim device, with no ports and 2 line cards:
$ echo "10 0 2" >/sys/bus/netdevsim/new_device
$ devlink port # No ports are listed
$ devlink lc
netdevsim/netdevsim10:
lc 0 state unprovisioned
supported_types:
card1port card2ports card4ports
lc 1 state unprovisioned
supported_types:
card1port card2ports card4ports
# Note that driver advertizes supported line card types. In case of
# netdevsim, these are 3.
$ devlink lc provision netdevsim/netdevsim10 lc 0 type card4ports
Why do we need a separate command for that? You actually introduced
'DEVLINK_CMD_LINECARD_SET' in patch #1, but it's never used.
I prefer:
devlink lc set netdevsim/netdevsim10 index 0 state provision type card4ports
devlink lc set netdevsim/netdevsim10 index 0 state unprovision
It is consistent with the GET/SET/NEW/DEL pattern used by other
commands.
$ devlink lc
netdevsim/netdevsim10:
lc 0 state provisioned type card4ports
supported_types:
card1port card2ports card4ports
lc 1 state unprovisioned
supported_types:
card1port card2ports card4ports
$ devlink port
netdevsim/netdevsim10/1000: type eth netdev eni10nl0p1 flavour physical lc 0 port 1 splittable false
netdevsim/netdevsim10/1001: type eth netdev eni10nl0p2 flavour physical lc 0 port 2 splittable false
netdevsim/netdevsim10/1002: type eth netdev eni10nl0p3 flavour physical lc 0 port 3 splittable false
netdevsim/netdevsim10/1003: type eth netdev eni10nl0p4 flavour physical lc 0 port 4 splittable false
# ^^ ^^^^
# netdev name adjusted index of a line card this port belongs to
$ ip link set eni10nl0p1 up
$ ip link show eni10nl0p1
165: eni10nl0p1: <NO-CARRIER,BROADCAST,NOARP,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
link/ether 7e:2d:05:93:d3:d1 brd ff:ff:ff:ff:ff:ff
# Now activate the line card using debugfs. That emulates plug-in event
# on real hardware:
$ echo "Y"> /sys/kernel/debug/netdevsim/netdevsim10/linecards/0/active
$ ip link show eni10nl0p1
165: eni10nl0p1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 7e:2d:05:93:d3:d1 brd ff:ff:ff:ff:ff:ff
# The carrier is UP now.
Jiri Pirko (10):
devlink: add support to create line card and expose to user
devlink: implement line card provisioning
devlink: implement line card active state
devlink: append split port number to the port name
devlink: add port to line card relationship set
netdevsim: introduce line card support
netdevsim: allow port objects to be linked with line cards
netdevsim: create devlink line card object and implement provisioning
netdevsim: implement line card activation
selftests: add netdevsim devlink lc test
drivers/net/netdevsim/bus.c | 21 +-
drivers/net/netdevsim/dev.c | 370 ++++++++++++++-
drivers/net/netdevsim/netdev.c | 2 +
drivers/net/netdevsim/netdevsim.h | 23 +
include/net/devlink.h | 44 ++
include/uapi/linux/devlink.h | 25 +
net/core/devlink.c | 443 +++++++++++++++++-
.../drivers/net/netdevsim/devlink.sh | 62 ++-
8 files changed, 964 insertions(+), 26 deletions(-)
--
2.26.2
On Wed, Jan 13, 2021 at 01:12:14PM +0100, Jiri Pirko wrote:
quoted hunk
From: Jiri Pirko <redacted>
In order to be able to configure all needed stuff on a port/netdevice
of a line card without the line card being present, introduce line card
provisioning. Basically provisioning will create a placeholder for
instances (ports/netdevices) for a line card type.
Allow the user to query the supported line card types over line card
get command. Then implement two netlink commands to allow user to
provision/unprovision the line card with selected line card type.
On the driver API side, add provision/unprovision ops and supported
types array to be advertised. Upon provision op call, the driver should
take care of creating the instances for the particular line card type.
Introduce provision_set/clear() functions to be called by the driver
once the provisioning/unprovisioning is done on its side.
Signed-off-by: Jiri Pirko <redacted>
---
include/net/devlink.h | 31 +++++++-
include/uapi/linux/devlink.h | 17 +++++
net/core/devlink.c | 141 ++++++++++++++++++++++++++++++++++-
3 files changed, 185 insertions(+), 4 deletions(-)
Can you explain why these two states are necessary? Any reason the
provision operation can't be synchronous? This is somewhat explained in
patch #8, but it should really be explained here. Changelog says:
"To avoid deadlock and to mimic actual HW flow, use workqueue
to add/del ports during provisioning as the port add/del calls
devlink_port_register/unregister() which take devlink mutex."
The deadlock is not really a reason to have these states.
'DEVLINK_CMD_PORT_SPLIT' also calls devlink_port_register() /
devlink_port_unregister() and the deadlock is solved by:
'internal_flags = DEVLINK_NL_FLAG_NO_LOCK'
A hardware flow the requires it is something else...
quoted hunk
+ DEVLINK_LINECARD_STATE_PROVISIONED,
+
+ __DEVLINK_LINECARD_STATE_MAX,
+ DEVLINK_LINECARD_STATE_MAX = __DEVLINK_LINECARD_STATE_MAX - 1
+};
+
enum devlink_attr {
/* don't change the order or add anything between, this is ABI! */
DEVLINK_ATTR_UNSPEC,
@@ -1192,7 +1192,9 @@ static int devlink_nl_linecard_fill(struct sk_buff *msg,u32seq,intflags,structnetlink_ext_ack*extack){+structnlattr*attr;void*hdr;+inti;hdr=genlmsg_put(msg,portid,seq,&devlink_nl_family,flags,cmd);if(!hdr)
@@ -1202,6 +1204,22 @@ static int devlink_nl_linecard_fill(struct sk_buff *msg,gotonla_put_failure;if(nla_put_u32(msg,DEVLINK_ATTR_LINECARD_INDEX,linecard->index))gotonla_put_failure;+if(nla_put_u8(msg,DEVLINK_ATTR_LINECARD_STATE,linecard->state))+gotonla_put_failure;+if(linecard->state>=DEVLINK_LINECARD_STATE_PROVISIONED&&
This assumes that every state added after provisioned should report the
type. Better to check for the specific states
quoted hunk
+ nla_put_string(msg, DEVLINK_ATTR_LINECARD_TYPE,
+ linecard->provisioned_type))
+ goto nla_put_failure;
+
+ attr = nla_nest_start(msg, DEVLINK_ATTR_LINECARD_SUPPORTED_TYPES);
+ if (!attr)
+ return -EMSGSIZE;
+ for (i = 0; i < linecard->ops->supported_types_count; i++) {
+ if (nla_put_string(msg, DEVLINK_ATTR_LINECARD_TYPE,
+ linecard->ops->supported_types[i]))
+ goto nla_put_failure;
+ }
+ nla_nest_end(msg, attr);
genlmsg_end(msg, hdr);
return 0;
@@ -1300,6 +1318,68 @@ static int devlink_nl_cmd_linecard_get_dumpit(struct sk_buff *msg, return msg->len; }+static int devlink_nl_cmd_linecard_provision_doit(struct sk_buff *skb,+ struct genl_info *info)+{+ struct devlink_linecard *linecard = info->user_ptr[1];+ const char *type;+ int i;++ if (linecard->state == DEVLINK_LINECARD_STATE_PROVISIONING) {+ NL_SET_ERR_MSG_MOD(info->extack, "Linecard is currently being provisioned");+ return -EBUSY;+ }+ if (linecard->state == DEVLINK_LINECARD_STATE_UNPROVISIONING) {+ NL_SET_ERR_MSG_MOD(info->extack, "Linecard is currently being unprovisioned");+ return -EBUSY;+ }+ if (linecard->state != DEVLINK_LINECARD_STATE_UNPROVISIONED) {+ NL_SET_ERR_MSG_MOD(info->extack, "Linecard already provisioned");+ return -EBUSY;+ }++ if (!info->attrs[DEVLINK_ATTR_LINECARD_TYPE]) {+ NL_SET_ERR_MSG_MOD(info->extack, "Provision type not provided");+ return -EINVAL;+ }++ type = nla_data(info->attrs[DEVLINK_ATTR_LINECARD_TYPE]);+ for (i = 0; i < linecard->ops->supported_types_count; i++) {+ if (!strcmp(linecard->ops->supported_types[i], type)) {+ linecard->state = DEVLINK_LINECARD_STATE_PROVISIONING;+ devlink_linecard_notify(linecard, DEVLINK_CMD_LINECARD_NEW);+ return linecard->ops->provision(linecard,+ linecard->priv, i,+ info->extack);
So if this fails user space will see 'provisioning' although nothing is
being provisioned... Better to set the state and notify if this call did
not fail
quoted hunk
+ }
+ }
+ NL_SET_ERR_MSG_MOD(info->extack, "Unsupported provision type provided");
+ return -EINVAL;
+}
+
+static int devlink_nl_cmd_linecard_unprovision_doit(struct sk_buff *skb,
+ struct genl_info *info)
+{
+ struct devlink_linecard *linecard = info->user_ptr[1];
+
+ if (linecard->state == DEVLINK_LINECARD_STATE_PROVISIONING) {
+ NL_SET_ERR_MSG_MOD(info->extack, "Linecard is currently being provisioned");
+ return -EBUSY;
+ }
+ if (linecard->state == DEVLINK_LINECARD_STATE_UNPROVISIONING) {
+ NL_SET_ERR_MSG_MOD(info->extack, "Linecard is currently being unprovisioned");
+ return -EBUSY;
+ }
+ if (linecard->state == DEVLINK_LINECARD_STATE_UNPROVISIONED) {
+ NL_SET_ERR_MSG_MOD(info->extack, "Linecard is not provisioned");
+ return -EOPNOTSUPP;
+ }
+ linecard->state = DEVLINK_LINECARD_STATE_UNPROVISIONING;
+ devlink_linecard_notify(linecard, DEVLINK_CMD_LINECARD_NEW);
+ return linecard->ops->unprovision(linecard, linecard->priv,
+ info->extack);
+}
+
static int devlink_nl_sb_fill(struct sk_buff *msg, struct devlink *devlink,
struct devlink_sb *devlink_sb,
enum devlink_command cmd, u32 portid,
@@ -8653,6 +8757,39 @@ void devlink_linecard_destroy(struct devlink_linecard *linecard) } EXPORT_SYMBOL_GPL(devlink_linecard_create);+/**+ * devlink_linecard_provision_set - Set provisioning on linecard
'Set linecard as provisioned' maybe?
+ *
+ * @devlink_linecard: devlink linecard
+ * @type_index: index of the linecard type (in array of types in ops)
+ */
+void devlink_linecard_provision_set(struct devlink_linecard *linecard,
+ u32 type_index)
+{
+ WARN_ON(type_index >= linecard->ops->supported_types_count);
Wouldn't this explode below when you use the index to access the array?
Maybe better to just warn and return
@@ -8654,7 +8673,11 @@ static int __devlink_port_phys_port_name_get(struct devlink_port *devlink_port, switch (attrs->flavour) { case DEVLINK_PORT_FLAVOUR_PHYSICAL: case DEVLINK_PORT_FLAVOUR_VIRTUAL:- n = snprintf(name, len, "p%u", attrs->phys.port_number);+ if (devlink_port->linecard)+ n = snprintf(name, len, "l%u",+ devlink_port->linecard->index);+ n += snprintf(name + n, len - n, "p%u",+ attrs->phys.port_number); if (attrs->split) n += snprintf(name + n, len - n, "s%u", attrs->phys.split_subport_number);
Fri, Jan 15, 2021 at 05:03:19PM CET, idosch@idosch.org wrote:
On Wed, Jan 13, 2021 at 01:12:14PM +0100, Jiri Pirko wrote:
quoted
From: Jiri Pirko <redacted>
In order to be able to configure all needed stuff on a port/netdevice
of a line card without the line card being present, introduce line card
provisioning. Basically provisioning will create a placeholder for
instances (ports/netdevices) for a line card type.
Allow the user to query the supported line card types over line card
get command. Then implement two netlink commands to allow user to
provision/unprovision the line card with selected line card type.
On the driver API side, add provision/unprovision ops and supported
types array to be advertised. Upon provision op call, the driver should
take care of creating the instances for the particular line card type.
Introduce provision_set/clear() functions to be called by the driver
once the provisioning/unprovisioning is done on its side.
Signed-off-by: Jiri Pirko <redacted>
---
include/net/devlink.h | 31 +++++++-
include/uapi/linux/devlink.h | 17 +++++
net/core/devlink.c | 141 ++++++++++++++++++++++++++++++++++-
3 files changed, 185 insertions(+), 4 deletions(-)
I do not really see the point in these two commands. Better extend
DEVLINK_CMD_LINECARD_SET to carry these attributes.
Yeah, I was thinking about that. Not sure it is correct though. This is
single purpose command. It really does not change "an attribute" as the
"_SET" commands are usually doing. Consider extension of "_SET" by other
attributes. Then it looks wrong.
quoted
+
/* add new commands above here */
__DEVLINK_CMD_MAX,
DEVLINK_CMD_MAX = __DEVLINK_CMD_MAX - 1
Can you explain why these two states are necessary? Any reason the
provision operation can't be synchronous? This is somewhat explained in
patch #8, but it should really be explained here. Changelog says:
"To avoid deadlock and to mimic actual HW flow, use workqueue
to add/del ports during provisioning as the port add/del calls
devlink_port_register/unregister() which take devlink mutex."
The deadlock is not really a reason to have these states.
It is, need to avoid recursice locking
'DEVLINK_CMD_PORT_SPLIT' also calls devlink_port_register() /
devlink_port_unregister() and the deadlock is solved by:
'internal_flags = DEVLINK_NL_FLAG_NO_LOCK'
Yeah, however, there, the port_index is passed down to the driver, not
the actual object pointer. That's why it can be done like that.
A hardware flow the requires it is something else...
Hardware flow in case of Spectrum is async too.
quoted
+ DEVLINK_LINECARD_STATE_PROVISIONED,
+
+ __DEVLINK_LINECARD_STATE_MAX,
+ DEVLINK_LINECARD_STATE_MAX = __DEVLINK_LINECARD_STATE_MAX - 1
+};
+
enum devlink_attr {
/* don't change the order or add anything between, this is ABI! */
DEVLINK_ATTR_UNSPEC,
@@ -1192,7 +1192,9 @@ static int devlink_nl_linecard_fill(struct sk_buff *msg,u32seq,intflags,structnetlink_ext_ack*extack){+structnlattr*attr;void*hdr;+inti;hdr=genlmsg_put(msg,portid,seq,&devlink_nl_family,flags,cmd);if(!hdr)
@@ -1202,6 +1204,22 @@ static int devlink_nl_linecard_fill(struct sk_buff *msg,gotonla_put_failure;if(nla_put_u32(msg,DEVLINK_ATTR_LINECARD_INDEX,linecard->index))gotonla_put_failure;+if(nla_put_u8(msg,DEVLINK_ATTR_LINECARD_STATE,linecard->state))+gotonla_put_failure;+if(linecard->state>=DEVLINK_LINECARD_STATE_PROVISIONED&&
This assumes that every state added after provisioned should report the
type. Better to check for the specific states
Yes, that is correct assumption.
quoted
+ nla_put_string(msg, DEVLINK_ATTR_LINECARD_TYPE,
+ linecard->provisioned_type))
+ goto nla_put_failure;
+
+ attr = nla_nest_start(msg, DEVLINK_ATTR_LINECARD_SUPPORTED_TYPES);
+ if (!attr)
+ return -EMSGSIZE;
+ for (i = 0; i < linecard->ops->supported_types_count; i++) {
+ if (nla_put_string(msg, DEVLINK_ATTR_LINECARD_TYPE,
+ linecard->ops->supported_types[i]))
+ goto nla_put_failure;
+ }
+ nla_nest_end(msg, attr);
genlmsg_end(msg, hdr);
return 0;
@@ -1300,6 +1318,68 @@ static int devlink_nl_cmd_linecard_get_dumpit(struct sk_buff *msg, return msg->len; }+static int devlink_nl_cmd_linecard_provision_doit(struct sk_buff *skb,+ struct genl_info *info)+{+ struct devlink_linecard *linecard = info->user_ptr[1];+ const char *type;+ int i;++ if (linecard->state == DEVLINK_LINECARD_STATE_PROVISIONING) {+ NL_SET_ERR_MSG_MOD(info->extack, "Linecard is currently being provisioned");+ return -EBUSY;+ }+ if (linecard->state == DEVLINK_LINECARD_STATE_UNPROVISIONING) {+ NL_SET_ERR_MSG_MOD(info->extack, "Linecard is currently being unprovisioned");+ return -EBUSY;+ }+ if (linecard->state != DEVLINK_LINECARD_STATE_UNPROVISIONED) {+ NL_SET_ERR_MSG_MOD(info->extack, "Linecard already provisioned");+ return -EBUSY;+ }++ if (!info->attrs[DEVLINK_ATTR_LINECARD_TYPE]) {+ NL_SET_ERR_MSG_MOD(info->extack, "Provision type not provided");+ return -EINVAL;+ }++ type = nla_data(info->attrs[DEVLINK_ATTR_LINECARD_TYPE]);+ for (i = 0; i < linecard->ops->supported_types_count; i++) {+ if (!strcmp(linecard->ops->supported_types[i], type)) {+ linecard->state = DEVLINK_LINECARD_STATE_PROVISIONING;+ devlink_linecard_notify(linecard, DEVLINK_CMD_LINECARD_NEW);+ return linecard->ops->provision(linecard,+ linecard->priv, i,+ info->extack);
So if this fails user space will see 'provisioning' although nothing is
being provisioned... Better to set the state and notify if this call did
not fail
The driver is responsible to either call provision_set/provision_clear
helper. Note the async nature of this op.
quoted
+ }
+ }
+ NL_SET_ERR_MSG_MOD(info->extack, "Unsupported provision type provided");
+ return -EINVAL;
+}
+
+static int devlink_nl_cmd_linecard_unprovision_doit(struct sk_buff *skb,
+ struct genl_info *info)
+{
+ struct devlink_linecard *linecard = info->user_ptr[1];
+
+ if (linecard->state == DEVLINK_LINECARD_STATE_PROVISIONING) {
+ NL_SET_ERR_MSG_MOD(info->extack, "Linecard is currently being provisioned");
+ return -EBUSY;
+ }
+ if (linecard->state == DEVLINK_LINECARD_STATE_UNPROVISIONING) {
+ NL_SET_ERR_MSG_MOD(info->extack, "Linecard is currently being unprovisioned");
+ return -EBUSY;
+ }
+ if (linecard->state == DEVLINK_LINECARD_STATE_UNPROVISIONED) {
+ NL_SET_ERR_MSG_MOD(info->extack, "Linecard is not provisioned");
+ return -EOPNOTSUPP;
+ }
+ linecard->state = DEVLINK_LINECARD_STATE_UNPROVISIONING;
+ devlink_linecard_notify(linecard, DEVLINK_CMD_LINECARD_NEW);
+ return linecard->ops->unprovision(linecard, linecard->priv,
+ info->extack);
+}
+
static int devlink_nl_sb_fill(struct sk_buff *msg, struct devlink *devlink,
struct devlink_sb *devlink_sb,
enum devlink_command cmd, u32 portid,
@@ -8653,6 +8757,39 @@ void devlink_linecard_destroy(struct devlink_linecard *linecard) } EXPORT_SYMBOL_GPL(devlink_linecard_create);+/**+ * devlink_linecard_provision_set - Set provisioning on linecard
'Set linecard as provisioned' maybe?
Sure, why not.
quoted
+ *
+ * @devlink_linecard: devlink linecard
+ * @type_index: index of the linecard type (in array of types in ops)
+ */
+void devlink_linecard_provision_set(struct devlink_linecard *linecard,
+ u32 type_index)
+{
+ WARN_ON(type_index >= linecard->ops->supported_types_count);
Wouldn't this explode below when you use the index to access the array?
Maybe better to just warn and return
@@ -8654,7 +8673,11 @@ static int __devlink_port_phys_port_name_get(struct devlink_port *devlink_port, switch (attrs->flavour) { case DEVLINK_PORT_FLAVOUR_PHYSICAL: case DEVLINK_PORT_FLAVOUR_VIRTUAL:- n = snprintf(name, len, "p%u", attrs->phys.port_number);+ if (devlink_port->linecard)+ n = snprintf(name, len, "l%u",+ devlink_port->linecard->index);+ n += snprintf(name + n, len - n, "p%u",+ attrs->phys.port_number); if (attrs->split) n += snprintf(name + n, len - n, "s%u", attrs->phys.split_subport_number);
Fri, Jan 15, 2021 at 04:43:57PM CET, idosch@idosch.org wrote:
On Wed, Jan 13, 2021 at 01:12:12PM +0100, Jiri Pirko wrote:
quoted
# Create a new netdevsim device, with no ports and 2 line cards:
$ echo "10 0 2" >/sys/bus/netdevsim/new_device
$ devlink port # No ports are listed
$ devlink lc
netdevsim/netdevsim10:
lc 0 state unprovisioned
supported_types:
card1port card2ports card4ports
lc 1 state unprovisioned
supported_types:
card1port card2ports card4ports
# Note that driver advertizes supported line card types. In case of
# netdevsim, these are 3.
$ devlink lc provision netdevsim/netdevsim10 lc 0 type card4ports
Why do we need a separate command for that? You actually introduced
'DEVLINK_CMD_LINECARD_SET' in patch #1, but it's never used.
I prefer:
devlink lc set netdevsim/netdevsim10 index 0 state provision type card4ports
This is misleading. This is actually not setting state. The state gets
changed upon successful provisioning process. Also, one may think that
he can set other states, but he can't. I don't like this at all :/
devlink lc set netdevsim/netdevsim10 index 0 state unprovision
It is consistent with the GET/SET/NEW/DEL pattern used by other
commands.
Not really, see split port for example. This is similar to that.
quoted
$ devlink lc
netdevsim/netdevsim10:
lc 0 state provisioned type card4ports
supported_types:
card1port card2ports card4ports
lc 1 state unprovisioned
supported_types:
card1port card2ports card4ports
$ devlink port
netdevsim/netdevsim10/1000: type eth netdev eni10nl0p1 flavour physical lc 0 port 1 splittable false
netdevsim/netdevsim10/1001: type eth netdev eni10nl0p2 flavour physical lc 0 port 2 splittable false
netdevsim/netdevsim10/1002: type eth netdev eni10nl0p3 flavour physical lc 0 port 3 splittable false
netdevsim/netdevsim10/1003: type eth netdev eni10nl0p4 flavour physical lc 0 port 4 splittable false
# ^^ ^^^^
# netdev name adjusted index of a line card this port belongs to
$ ip link set eni10nl0p1 up
$ ip link show eni10nl0p1
165: eni10nl0p1: <NO-CARRIER,BROADCAST,NOARP,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
link/ether 7e:2d:05:93:d3:d1 brd ff:ff:ff:ff:ff:ff
# Now activate the line card using debugfs. That emulates plug-in event
# on real hardware:
$ echo "Y"> /sys/kernel/debug/netdevsim/netdevsim10/linecards/0/active
$ ip link show eni10nl0p1
165: eni10nl0p1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 7e:2d:05:93:d3:d1 brd ff:ff:ff:ff:ff:ff
# The carrier is UP now.
Jiri Pirko (10):
devlink: add support to create line card and expose to user
devlink: implement line card provisioning
devlink: implement line card active state
devlink: append split port number to the port name
devlink: add port to line card relationship set
netdevsim: introduce line card support
netdevsim: allow port objects to be linked with line cards
netdevsim: create devlink line card object and implement provisioning
netdevsim: implement line card activation
selftests: add netdevsim devlink lc test
drivers/net/netdevsim/bus.c | 21 +-
drivers/net/netdevsim/dev.c | 370 ++++++++++++++-
drivers/net/netdevsim/netdev.c | 2 +
drivers/net/netdevsim/netdevsim.h | 23 +
include/net/devlink.h | 44 ++
include/uapi/linux/devlink.h | 25 +
net/core/devlink.c | 443 +++++++++++++++++-
.../drivers/net/netdevsim/devlink.sh | 62 ++-
8 files changed, 964 insertions(+), 26 deletions(-)
--
2.26.2
On Fri, Jan 15, 2021 at 05:55:59PM +0100, Jiri Pirko wrote:
Fri, Jan 15, 2021 at 04:43:57PM CET, idosch@idosch.org wrote:
quoted
On Wed, Jan 13, 2021 at 01:12:12PM +0100, Jiri Pirko wrote:
quoted
# Create a new netdevsim device, with no ports and 2 line cards:
$ echo "10 0 2" >/sys/bus/netdevsim/new_device
$ devlink port # No ports are listed
$ devlink lc
netdevsim/netdevsim10:
lc 0 state unprovisioned
supported_types:
card1port card2ports card4ports
lc 1 state unprovisioned
supported_types:
card1port card2ports card4ports
# Note that driver advertizes supported line card types. In case of
# netdevsim, these are 3.
$ devlink lc provision netdevsim/netdevsim10 lc 0 type card4ports
Why do we need a separate command for that? You actually introduced
'DEVLINK_CMD_LINECARD_SET' in patch #1, but it's never used.
I prefer:
devlink lc set netdevsim/netdevsim10 index 0 state provision type card4ports
This is misleading. This is actually not setting state. The state gets
changed upon successful provisioning process. Also, one may think that
he can set other states, but he can't. I don't like this at all :/
So make state a read-only attribute. You really only care about setting
the type.
To provision:
# devlink lc set netdevsim/netdevsim10 index 0 type card4ports
To unprovsion:
# devlink lc set netdevsim/netdevsim10 index 0 type none
Or:
# devlink lc set netdevsim/netdevsim10 index 0 notype
quoted
devlink lc set netdevsim/netdevsim10 index 0 state unprovision
It is consistent with the GET/SET/NEW/DEL pattern used by other
commands.
Not really, see split port for example. This is similar to that.
It's not. The split command creates new objects whereas this command
modifies an existing object.
quoted
quoted
$ devlink lc
netdevsim/netdevsim10:
lc 0 state provisioned type card4ports
supported_types:
card1port card2ports card4ports
lc 1 state unprovisioned
supported_types:
card1port card2ports card4ports
$ devlink port
netdevsim/netdevsim10/1000: type eth netdev eni10nl0p1 flavour physical lc 0 port 1 splittable false
netdevsim/netdevsim10/1001: type eth netdev eni10nl0p2 flavour physical lc 0 port 2 splittable false
netdevsim/netdevsim10/1002: type eth netdev eni10nl0p3 flavour physical lc 0 port 3 splittable false
netdevsim/netdevsim10/1003: type eth netdev eni10nl0p4 flavour physical lc 0 port 4 splittable false
# ^^ ^^^^
# netdev name adjusted index of a line card this port belongs to
$ ip link set eni10nl0p1 up
$ ip link show eni10nl0p1
165: eni10nl0p1: <NO-CARRIER,BROADCAST,NOARP,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
link/ether 7e:2d:05:93:d3:d1 brd ff:ff:ff:ff:ff:ff
# Now activate the line card using debugfs. That emulates plug-in event
# on real hardware:
$ echo "Y"> /sys/kernel/debug/netdevsim/netdevsim10/linecards/0/active
$ ip link show eni10nl0p1
165: eni10nl0p1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 7e:2d:05:93:d3:d1 brd ff:ff:ff:ff:ff:ff
# The carrier is UP now.
Jiri Pirko (10):
devlink: add support to create line card and expose to user
devlink: implement line card provisioning
devlink: implement line card active state
devlink: append split port number to the port name
devlink: add port to line card relationship set
netdevsim: introduce line card support
netdevsim: allow port objects to be linked with line cards
netdevsim: create devlink line card object and implement provisioning
netdevsim: implement line card activation
selftests: add netdevsim devlink lc test
drivers/net/netdevsim/bus.c | 21 +-
drivers/net/netdevsim/dev.c | 370 ++++++++++++++-
drivers/net/netdevsim/netdev.c | 2 +
drivers/net/netdevsim/netdevsim.h | 23 +
include/net/devlink.h | 44 ++
include/uapi/linux/devlink.h | 25 +
net/core/devlink.c | 443 +++++++++++++++++-
.../drivers/net/netdevsim/devlink.sh | 62 ++-
8 files changed, 964 insertions(+), 26 deletions(-)
--
2.26.2
On Fri, Jan 15, 2021 at 05:51:57PM +0100, Jiri Pirko wrote:
Fri, Jan 15, 2021 at 05:03:19PM CET, idosch@idosch.org wrote:
quoted
On Wed, Jan 13, 2021 at 01:12:14PM +0100, Jiri Pirko wrote:
quoted
From: Jiri Pirko <redacted>
In order to be able to configure all needed stuff on a port/netdevice
of a line card without the line card being present, introduce line card
provisioning. Basically provisioning will create a placeholder for
instances (ports/netdevices) for a line card type.
Allow the user to query the supported line card types over line card
get command. Then implement two netlink commands to allow user to
provision/unprovision the line card with selected line card type.
On the driver API side, add provision/unprovision ops and supported
types array to be advertised. Upon provision op call, the driver should
take care of creating the instances for the particular line card type.
Introduce provision_set/clear() functions to be called by the driver
once the provisioning/unprovisioning is done on its side.
Signed-off-by: Jiri Pirko <redacted>
---
include/net/devlink.h | 31 +++++++-
include/uapi/linux/devlink.h | 17 +++++
net/core/devlink.c | 141 ++++++++++++++++++++++++++++++++++-
3 files changed, 185 insertions(+), 4 deletions(-)
I do not really see the point in these two commands. Better extend
DEVLINK_CMD_LINECARD_SET to carry these attributes.
Yeah, I was thinking about that. Not sure it is correct though. This is
single purpose command. It really does not change "an attribute" as the
"_SET" commands are usually doing. Consider extension of "_SET" by other
attributes. Then it looks wrong.
It is setting the type of the linecard, which is an attribute of the
linecard.
quoted
quoted
+
/* add new commands above here */
__DEVLINK_CMD_MAX,
DEVLINK_CMD_MAX = __DEVLINK_CMD_MAX - 1
Can you explain why these two states are necessary? Any reason the
provision operation can't be synchronous? This is somewhat explained in
patch #8, but it should really be explained here. Changelog says:
"To avoid deadlock and to mimic actual HW flow, use workqueue
to add/del ports during provisioning as the port add/del calls
devlink_port_register/unregister() which take devlink mutex."
The deadlock is not really a reason to have these states.
It is, need to avoid recursice locking
quoted
'DEVLINK_CMD_PORT_SPLIT' also calls devlink_port_register() /
devlink_port_unregister() and the deadlock is solved by:
'internal_flags = DEVLINK_NL_FLAG_NO_LOCK'
Yeah, however, there, the port_index is passed down to the driver, not
the actual object pointer. That's why it can be done like that.
quoted
A hardware flow the requires it is something else...
Hardware flow in case of Spectrum is async too.
OK, so the changelog needs to state that these states are necessary
because the nature of linecard provisioning is asynchronous.
quoted
quoted
+ DEVLINK_LINECARD_STATE_PROVISIONED,
+
+ __DEVLINK_LINECARD_STATE_MAX,
+ DEVLINK_LINECARD_STATE_MAX = __DEVLINK_LINECARD_STATE_MAX - 1
+};
+
enum devlink_attr {
/* don't change the order or add anything between, this is ABI! */
DEVLINK_ATTR_UNSPEC,
From: Jakub Kicinski <kuba@kernel.org> Date: 2021-01-15 19:26:59
On Fri, 15 Jan 2021 15:39:06 +0100 Jiri Pirko wrote:
quoted
I'm not a SFP experts so maybe someone will correct me but AFAIU
the QSFP (for optics) is the same regardless of breakout. It's the
passive optical strands that are either bundled or not. So there is
no way for the system to detect the cable type (AFAIK).
For SFP module, you are able to detect those.
Not sure you understand what I'm saying. Maybe you're thinking about
DACs? This is a optical cable for breakout:
https://www.fs.com/products/68048.html
There is no electronics in it to "detect" things AFAIU. Same QSFP can
be used with this cable or a non-breakout.
quoted
Or to put it differently IMO the netdev should be provisioned if the
system has a port into which user can plug in a cable. When there is
Not really. For slit cables, the ports are provisioned not matter which
cable is connected, slitter 1->2/1->4 or 1->1 cable.
quoted
a line card-sized hole in the chassis, I'd be surprised to see ports.
That said I never worked with real world routers so maybe that's what
they do. Maybe some with a Cisco router in the basement can tell us? :)
The need for provision/pre-configure splitter/linecard is that the
ports/netdevices do not disapper/reappear when you replace
splitter/linecard. Consider a faulty linecard with one port burned. You
just want to replace it with new one. And in that case, you really don't
want kernel to remove netdevices and possibly mess up routing for
example.
Having a single burned port sounds like a relatively rare scenario.
Reconfiguring routing is not the end of the world.
quoted
If the device really needs this configuration / can't detect things
automatically, then we gotta do something like what you have.
The only question is do we still want to call it a line card.
Sounds more like a front panel module. At Netronome we called
those phymods.
Sure, the name is up to the discussion. We call it "linecard"
internally. I don't care about the name.
Yeah, let's call it something more appropriate to indicate its
breakout/retimer/gearbox nature, and we'll be good :)
Fri, Jan 15, 2021 at 07:09:44PM CET, idosch@idosch.org wrote:
On Fri, Jan 15, 2021 at 05:51:57PM +0100, Jiri Pirko wrote:
quoted
Fri, Jan 15, 2021 at 05:03:19PM CET, idosch@idosch.org wrote:
quoted
On Wed, Jan 13, 2021 at 01:12:14PM +0100, Jiri Pirko wrote:
quoted
From: Jiri Pirko <redacted>
In order to be able to configure all needed stuff on a port/netdevice
of a line card without the line card being present, introduce line card
provisioning. Basically provisioning will create a placeholder for
instances (ports/netdevices) for a line card type.
Allow the user to query the supported line card types over line card
get command. Then implement two netlink commands to allow user to
provision/unprovision the line card with selected line card type.
On the driver API side, add provision/unprovision ops and supported
types array to be advertised. Upon provision op call, the driver should
take care of creating the instances for the particular line card type.
Introduce provision_set/clear() functions to be called by the driver
once the provisioning/unprovisioning is done on its side.
Signed-off-by: Jiri Pirko <redacted>
---
include/net/devlink.h | 31 +++++++-
include/uapi/linux/devlink.h | 17 +++++
net/core/devlink.c | 141 ++++++++++++++++++++++++++++++++++-
3 files changed, 185 insertions(+), 4 deletions(-)
I do not really see the point in these two commands. Better extend
DEVLINK_CMD_LINECARD_SET to carry these attributes.
Yeah, I was thinking about that. Not sure it is correct though. This is
single purpose command. It really does not change "an attribute" as the
"_SET" commands are usually doing. Consider extension of "_SET" by other
attributes. Then it looks wrong.
It is setting the type of the linecard, which is an attribute of the
linecard.
Hmm. Still, consider the async nature. Do you have any example of attr
set with async nature? I expect the attr to be set when cmd returns 0.
IDK. Does not feel correct...
quoted
quoted
quoted
+
/* add new commands above here */
__DEVLINK_CMD_MAX,
DEVLINK_CMD_MAX = __DEVLINK_CMD_MAX - 1
Can you explain why these two states are necessary? Any reason the
provision operation can't be synchronous? This is somewhat explained in
patch #8, but it should really be explained here. Changelog says:
"To avoid deadlock and to mimic actual HW flow, use workqueue
to add/del ports during provisioning as the port add/del calls
devlink_port_register/unregister() which take devlink mutex."
The deadlock is not really a reason to have these states.
It is, need to avoid recursice locking
quoted
'DEVLINK_CMD_PORT_SPLIT' also calls devlink_port_register() /
devlink_port_unregister() and the deadlock is solved by:
'internal_flags = DEVLINK_NL_FLAG_NO_LOCK'
Yeah, however, there, the port_index is passed down to the driver, not
the actual object pointer. That's why it can be done like that.
quoted
A hardware flow the requires it is something else...
Hardware flow in case of Spectrum is async too.
OK, so the changelog needs to state that these states are necessary
because the nature of linecard provisioning is asynchronous.
Ok.
quoted
quoted
quoted
+ DEVLINK_LINECARD_STATE_PROVISIONED,
+
+ __DEVLINK_LINECARD_STATE_MAX,
+ DEVLINK_LINECARD_STATE_MAX = __DEVLINK_LINECARD_STATE_MAX - 1
+};
+
enum devlink_attr {
/* don't change the order or add anything between, this is ABI! */
DEVLINK_ATTR_UNSPEC,
Fri, Jan 15, 2021 at 07:01:45PM CET, idosch@idosch.org wrote:
On Fri, Jan 15, 2021 at 05:55:59PM +0100, Jiri Pirko wrote:
quoted
Fri, Jan 15, 2021 at 04:43:57PM CET, idosch@idosch.org wrote:
quoted
On Wed, Jan 13, 2021 at 01:12:12PM +0100, Jiri Pirko wrote:
quoted
# Create a new netdevsim device, with no ports and 2 line cards:
$ echo "10 0 2" >/sys/bus/netdevsim/new_device
$ devlink port # No ports are listed
$ devlink lc
netdevsim/netdevsim10:
lc 0 state unprovisioned
supported_types:
card1port card2ports card4ports
lc 1 state unprovisioned
supported_types:
card1port card2ports card4ports
# Note that driver advertizes supported line card types. In case of
# netdevsim, these are 3.
$ devlink lc provision netdevsim/netdevsim10 lc 0 type card4ports
Why do we need a separate command for that? You actually introduced
'DEVLINK_CMD_LINECARD_SET' in patch #1, but it's never used.
I prefer:
devlink lc set netdevsim/netdevsim10 index 0 state provision type card4ports
This is misleading. This is actually not setting state. The state gets
changed upon successful provisioning process. Also, one may think that
he can set other states, but he can't. I don't like this at all :/
So make state a read-only attribute. You really only care about setting
the type.
To provision:
# devlink lc set netdevsim/netdevsim10 index 0 type card4ports
To unprovsion:
# devlink lc set netdevsim/netdevsim10 index 0 type none
Or:
# devlink lc set netdevsim/netdevsim10 index 0 notype
Hmm, okay, that might work. And I can add state "FAILED_PROVISION" what
would indicate that after the type was set by the user, driver was not
able to successfully provision. The the user has to set "notype" & "type
x" again. Sounds good?
quoted
quoted
devlink lc set netdevsim/netdevsim10 index 0 state unprovision
It is consistent with the GET/SET/NEW/DEL pattern used by other
commands.
Not really, see split port for example. This is similar to that.
It's not. The split command creates new objects whereas this command
modifies an existing object.
You are right.
quoted
quoted
quoted
$ devlink lc
netdevsim/netdevsim10:
lc 0 state provisioned type card4ports
supported_types:
card1port card2ports card4ports
lc 1 state unprovisioned
supported_types:
card1port card2ports card4ports
$ devlink port
netdevsim/netdevsim10/1000: type eth netdev eni10nl0p1 flavour physical lc 0 port 1 splittable false
netdevsim/netdevsim10/1001: type eth netdev eni10nl0p2 flavour physical lc 0 port 2 splittable false
netdevsim/netdevsim10/1002: type eth netdev eni10nl0p3 flavour physical lc 0 port 3 splittable false
netdevsim/netdevsim10/1003: type eth netdev eni10nl0p4 flavour physical lc 0 port 4 splittable false
# ^^ ^^^^
# netdev name adjusted index of a line card this port belongs to
$ ip link set eni10nl0p1 up
$ ip link show eni10nl0p1
165: eni10nl0p1: <NO-CARRIER,BROADCAST,NOARP,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
link/ether 7e:2d:05:93:d3:d1 brd ff:ff:ff:ff:ff:ff
# Now activate the line card using debugfs. That emulates plug-in event
# on real hardware:
$ echo "Y"> /sys/kernel/debug/netdevsim/netdevsim10/linecards/0/active
$ ip link show eni10nl0p1
165: eni10nl0p1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 7e:2d:05:93:d3:d1 brd ff:ff:ff:ff:ff:ff
# The carrier is UP now.
Jiri Pirko (10):
devlink: add support to create line card and expose to user
devlink: implement line card provisioning
devlink: implement line card active state
devlink: append split port number to the port name
devlink: add port to line card relationship set
netdevsim: introduce line card support
netdevsim: allow port objects to be linked with line cards
netdevsim: create devlink line card object and implement provisioning
netdevsim: implement line card activation
selftests: add netdevsim devlink lc test
drivers/net/netdevsim/bus.c | 21 +-
drivers/net/netdevsim/dev.c | 370 ++++++++++++++-
drivers/net/netdevsim/netdev.c | 2 +
drivers/net/netdevsim/netdevsim.h | 23 +
include/net/devlink.h | 44 ++
include/uapi/linux/devlink.h | 25 +
net/core/devlink.c | 443 +++++++++++++++++-
.../drivers/net/netdevsim/devlink.sh | 62 ++-
8 files changed, 964 insertions(+), 26 deletions(-)
--
2.26.2
Fri, Jan 15, 2021 at 08:26:17PM CET, kuba@kernel.org wrote:
On Fri, 15 Jan 2021 15:39:06 +0100 Jiri Pirko wrote:
quoted
quoted
I'm not a SFP experts so maybe someone will correct me but AFAIU
the QSFP (for optics) is the same regardless of breakout. It's the
passive optical strands that are either bundled or not. So there is
no way for the system to detect the cable type (AFAIK).
For SFP module, you are able to detect those.
Not sure you understand what I'm saying. Maybe you're thinking about
DACs? This is a optical cable for breakout:
https://www.fs.com/products/68048.html
There is no electronics in it to "detect" things AFAIU. Same QSFP can
be used with this cable or a non-breakout.
Ah, got you.
quoted
quoted
Or to put it differently IMO the netdev should be provisioned if the
system has a port into which user can plug in a cable. When there is
Not really. For slit cables, the ports are provisioned not matter which
cable is connected, slitter 1->2/1->4 or 1->1 cable.
quoted
a line card-sized hole in the chassis, I'd be surprised to see ports.
That said I never worked with real world routers so maybe that's what
they do. Maybe some with a Cisco router in the basement can tell us? :)
The need for provision/pre-configure splitter/linecard is that the
ports/netdevices do not disapper/reappear when you replace
splitter/linecard. Consider a faulty linecard with one port burned. You
just want to replace it with new one. And in that case, you really don't
want kernel to remove netdevices and possibly mess up routing for
example.
Having a single burned port sounds like a relatively rare scenario.
Hmm, rare in scale is common...
Reconfiguring routing is not the end of the world.
Well, yes, but you don't really want netdevices to come and go then you
plug in/out cables/modules. That's why we have split implemented as we
do. I don't understand why do you think linecards are different.
Plus, I'm not really sure that our hw can report the type, will check.
One way or another, I think that both configuration flows have valid
usecase. Some user may want pre-configuration, some user may want auto.
Btw, it is possible to implement splitter cable in auto mode as well.
quoted
quoted
If the device really needs this configuration / can't detect things
automatically, then we gotta do something like what you have.
The only question is do we still want to call it a line card.
Sounds more like a front panel module. At Netronome we called
those phymods.
Sure, the name is up to the discussion. We call it "linecard"
internally. I don't care about the name.
Yeah, let's call it something more appropriate to indicate its
breakout/retimer/gearbox nature, and we'll be good :)
Well, it can contain much more. It can contain a smartnic/fpga/whatever
for example. Not sure we can find something that fits to all cases.
I was thinking about it in the past, I think that the linecard is quite
appropriate. It connects with lines/lanes, and it does something,
either phy/gearbox, or just interconnects the lanes using smartnic/fpga
for example.
From: Jakub Kicinski <kuba@kernel.org> Date: 2021-01-18 18:00:45
On Mon, 18 Jan 2021 14:00:09 +0100 Jiri Pirko wrote:
quoted
quoted
quoted
Or to put it differently IMO the netdev should be provisioned if the
system has a port into which user can plug in a cable. When there is
Not really. For slit cables, the ports are provisioned not matter which
cable is connected, slitter 1->2/1->4 or 1->1 cable.
quoted
a line card-sized hole in the chassis, I'd be surprised to see ports.
That said I never worked with real world routers so maybe that's what
they do. Maybe some with a Cisco router in the basement can tell us? :)
The need for provision/pre-configure splitter/linecard is that the
ports/netdevices do not disapper/reappear when you replace
splitter/linecard. Consider a faulty linecard with one port burned. You
just want to replace it with new one. And in that case, you really don't
want kernel to remove netdevices and possibly mess up routing for
example.
Having a single burned port sounds like a relatively rare scenario.
Hmm, rare in scale is common...
Sure but at a scale of million switches it doesn't matter if a couple
are re-configuring their routing.
quoted
Reconfiguring routing is not the end of the world.
Well, yes, but you don't really want netdevices to come and go then you
plug in/out cables/modules. That's why we have split implemented as we
do. I don't understand why do you think linecards are different.
If I have an unused port it will still show up as a netdev.
If I have an unused phymod slot w/ a slot cover in it, why would there
be a netdev? Our definition of a physical port is something like "a
socket for a networking cable on the outside of the device". With your
code I can "provision" a phymod and there is no whole to plug in a
cable. If we follow the same logic, if I have a server with PCIe
hotplug, why can't I "provision" some netdevs for a NIC that I will
plug in later?
Plus, I'm not really sure that our hw can report the type, will check.
I think that's key.
One way or another, I think that both configuration flows have valid
usecase. Some user may want pre-configuration, some user may want auto.
Btw, it is possible to implement splitter cable in auto mode as well.
Auto as in iterate over possible configs until link up? That's nasty.
quoted
quoted
quoted
If the device really needs this configuration / can't detect things
automatically, then we gotta do something like what you have.
The only question is do we still want to call it a line card.
Sounds more like a front panel module. At Netronome we called
those phymods.
Sure, the name is up to the discussion. We call it "linecard"
internally. I don't care about the name.
Yeah, let's call it something more appropriate to indicate its
breakout/retimer/gearbox nature, and we'll be good :)
Well, it can contain much more. It can contain a smartnic/fpga/whatever
for example. Not sure we can find something that fits to all cases.
I was thinking about it in the past, I think that the linecard is quite
appropriate. It connects with lines/lanes, and it does something,
either phy/gearbox, or just interconnects the lanes using smartnic/fpga
for example.
If it has a FPGA / NPU in it, it's definitely auto-discoverable.
I don't understand why you think that it's okay to "provision" NICs
which aren't there but only for this particular use case.
On Wed, Jan 13, 2021 at 4:14 AM Jiri Pirko [off-list ref] wrote:
To resolve this, a concept of "provisioning" is introduced.
The user may "provision" certain slot with a line card type.
Driver then creates all instances (devlink ports, netdevices, etc)
related to this line card type. The carrier of netdevices stays down.
Once the line card is inserted and activated, the carrier of the
related netdevices goes up.
Do we need to start distinguishing different reasons for carrier down,
or have some kind of device not ready state instead?
I'm facing a similar issue with NIC firmware that isn't yet ready by
device open time, but have been resisting the urge to lie to the stack
about the state of the device and use link state as the next gate.
Sure, most things will just work most of the time, but the problems
with this approach are manifold. Firstly, at least in the NIC case,
the user may confuse this state for some kind of cable issue and go
looking in the wrong place for a solution. But, there are also several
ways the initialization can fail after this point and now the device
is administratively UP, but can never be UP, with no sanctioned way to
communicate the failure. Aren't the issues here similar?
Regards,
Edwin Peer
From: David Ahern <hidden> Date: 2021-01-18 22:57:05
On 1/18/21 6:00 AM, Jiri Pirko wrote:
quoted
Reconfiguring routing is not the end of the world.
Well, yes, but you don't really want netdevices to come and go then you
plug in/out cables/modules. That's why we have split implemented as we
And you don't want a routing daemon to use netdevices which are not
valid due to non-existence. Best case with what you want is carrier down
on the LC's netdevices and that destroys routing.
do. I don't understand why do you think linecards are different.
I still don't get why you expect linecards to be different than any
other hotplug device.
From: David Ahern <hidden> Date: 2021-01-18 22:59:08
On 1/18/21 11:01 AM, Edwin Peer wrote:
I'm facing a similar issue with NIC firmware that isn't yet ready by
device open time, but have been resisting the urge to lie to the stack
why not have the ndo_open return -EBUSY or -EAGAIN to tell S/W to try
again 'later'?
about the state of the device and use link state as the next gate.
Sure, most things will just work most of the time, but the problems
with this approach are manifold. Firstly, at least in the NIC case,
the user may confuse this state for some kind of cable issue and go
looking in the wrong place for a solution. But, there are also several
ways the initialization can fail after this point and now the device
is administratively UP, but can never be UP, with no sanctioned way to
communicate the failure. Aren't the issues here similar?
On Mon, Jan 18, 2021 at 2:57 PM David Ahern [off-list ref] wrote:
On 1/18/21 11:01 AM, Edwin Peer wrote:
quoted
I'm facing a similar issue with NIC firmware that isn't yet ready by
device open time, but have been resisting the urge to lie to the stack
why not have the ndo_open return -EBUSY or -EAGAIN to tell S/W to try
again 'later'?
Indeed, this is what we ended up doing, although we still need to
confirm Network Manager, systemd and whatever else our customers might
use do the necessary to satisfy the user requirement to handle the
delayed init.
Only reason I piped up is that this line card thing seems to introduce
a similar issue.
Regards,
Edwin Peer
From: David Ahern <hidden> Date: 2021-01-19 02:41:26
On 1/18/21 4:40 PM, Edwin Peer wrote:
On Mon, Jan 18, 2021 at 2:57 PM David Ahern [off-list ref] wrote:
quoted
On 1/18/21 11:01 AM, Edwin Peer wrote:
quoted
I'm facing a similar issue with NIC firmware that isn't yet ready by
device open time, but have been resisting the urge to lie to the stack
why not have the ndo_open return -EBUSY or -EAGAIN to tell S/W to try
again 'later'?
Indeed, this is what we ended up doing, although we still need to
confirm Network Manager, systemd and whatever else our customers might
use do the necessary to satisfy the user requirement to handle the
delayed init.
I am not surprised about the issue - boot times have been improved and
devices have gotten more complicated. And I was wondering how network
managers (add ifupdown{2} to that list) would handle an EAGAIN. You
could have an event sent -- e.g., IFLA_EVENT_FW_READY -- to allow
managers to avoid polling. Redundant for multiple netdev's per device,
but makes it event driven.
Only reason I piped up is that this line card thing seems to introduce
a similar issue.
On Mon, Jan 18, 2021 at 6:39 PM David Ahern [off-list ref] wrote:
quoted
Indeed, this is what we ended up doing, although we still need to
confirm Network Manager, systemd and whatever else our customers might
use do the necessary to satisfy the user requirement to handle the
delayed init.
I am not surprised about the issue - boot times have been improved and
devices have gotten more complicated. And I was wondering how network
managers (add ifupdown{2} to that list) would handle an EAGAIN. You
could have an event sent -- e.g., IFLA_EVENT_FW_READY -- to allow
managers to avoid polling. Redundant for multiple netdev's per device,
but makes it event driven.
This is what I was hinting at by talking about another device state.
For that, there would necessarily need to be an event to inform user
space about the transition out of said state into normal open/up. Of
course, the tools would need to be updated to know about such a new
event too. EAGAIN does seem simpler. In our case, we don't expect to
be polling too long, or frequently for that matter. It is
exceptionally rare that the firmware wouldn't be ready, but it can
happen.
Regards,
Edwin Peer
Mon, Jan 18, 2021 at 06:59:28PM CET, kuba@kernel.org wrote:
On Mon, 18 Jan 2021 14:00:09 +0100 Jiri Pirko wrote:
quoted
quoted
quoted
quoted
Or to put it differently IMO the netdev should be provisioned if the
system has a port into which user can plug in a cable. When there is
Not really. For slit cables, the ports are provisioned not matter which
cable is connected, slitter 1->2/1->4 or 1->1 cable.
quoted
a line card-sized hole in the chassis, I'd be surprised to see ports.
That said I never worked with real world routers so maybe that's what
they do. Maybe some with a Cisco router in the basement can tell us? :)
The need for provision/pre-configure splitter/linecard is that the
ports/netdevices do not disapper/reappear when you replace
splitter/linecard. Consider a faulty linecard with one port burned. You
just want to replace it with new one. And in that case, you really don't
want kernel to remove netdevices and possibly mess up routing for
example.
Having a single burned port sounds like a relatively rare scenario.
Hmm, rare in scale is common...
Sure but at a scale of million switches it doesn't matter if a couple
are re-configuring their routing.
quoted
quoted
Reconfiguring routing is not the end of the world.
Well, yes, but you don't really want netdevices to come and go then you
plug in/out cables/modules. That's why we have split implemented as we
do. I don't understand why do you think linecards are different.
If I have an unused port it will still show up as a netdev.
If I have an unused phymod slot w/ a slot cover in it, why would there
be a netdev? Our definition of a physical port is something like "a
socket for a networking cable on the outside of the device". With your
code I can "provision" a phymod and there is no whole to plug in a
cable. If we follow the same logic, if I have a server with PCIe
hotplug, why can't I "provision" some netdevs for a NIC that I will
plug in later?
quoted
Plus, I'm not really sure that our hw can report the type, will check.
I think that's key.
So, it can't. The driver is only aware of "activation" of the linecard
being successful or not.
quoted
One way or another, I think that both configuration flows have valid
usecase. Some user may want pre-configuration, some user may want auto.
Btw, it is possible to implement splitter cable in auto mode as well.
Auto as in iterate over possible configs until link up? That's nasty.
quoted
quoted
quoted
quoted
If the device really needs this configuration / can't detect things
automatically, then we gotta do something like what you have.
The only question is do we still want to call it a line card.
Sounds more like a front panel module. At Netronome we called
those phymods.
Sure, the name is up to the discussion. We call it "linecard"
internally. I don't care about the name.
Yeah, let's call it something more appropriate to indicate its
breakout/retimer/gearbox nature, and we'll be good :)
Well, it can contain much more. It can contain a smartnic/fpga/whatever
for example. Not sure we can find something that fits to all cases.
I was thinking about it in the past, I think that the linecard is quite
appropriate. It connects with lines/lanes, and it does something,
either phy/gearbox, or just interconnects the lanes using smartnic/fpga
for example.
If it has a FPGA / NPU in it, it's definitely auto-discoverable.
I don't understand why you think that it's okay to "provision" NICs
which aren't there but only for this particular use case.
Thu, Jan 14, 2021 at 03:07:18AM CET, andrew@lunn.ch wrote:
quoted
$ devlink lc provision netdevsim/netdevsim10 lc 0 type card4ports
$ devlink lc
netdevsim/netdevsim10:
lc 0 state provisioned type card4ports
supported_types:
card1port card2ports card4ports
lc 1 state unprovisioned
supported_types:
card1port card2ports card4ports
Hi Jiri
quoted
# Now activate the line card using debugfs. That emulates plug-in event
# on real hardware:
$ echo "Y"> /sys/kernel/debug/netdevsim/netdevsim10/linecards/0/active
$ ip link show eni10nl0p1
165: eni10nl0p1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 7e:2d:05:93:d3:d1 brd ff:ff:ff:ff:ff:ff
# The carrier is UP now.
What is missing from the devlink lc view is what line card is actually
in the slot. Say if i provision for a card4port, but actually insert a
card2port. It would be nice to have something like:
I checked, our hw does not support that. Only provides info that
linecard activation was/wasn't successful.
$ devlink lc
netdevsim/netdevsim10:
lc 0 state provisioned type card4ports
supported_types:
card1port card2ports card4ports
inserted_type:
card2ports;
lc 1 state unprovisioned
supported_types:
card1port card2ports card4ports
inserted_type:
None
I assume if i prevision for card4ports but actually install a
card2ports, all the interfaces stay down?
Maybe
should actually be
echo "card2ports" > /sys/kernel/debug/netdevsim/netdevsim10/linecards/0/active
so you can emulate somebody putting the wrong card in the slot?
Andrew
From: David Ahern <hidden> Date: 2021-01-19 18:27:51
On 1/19/21 4:56 AM, Jiri Pirko wrote:
Thu, Jan 14, 2021 at 03:07:18AM CET, andrew@lunn.ch wrote:
quoted
quoted
$ devlink lc provision netdevsim/netdevsim10 lc 0 type card4ports
$ devlink lc
netdevsim/netdevsim10:
lc 0 state provisioned type card4ports
supported_types:
card1port card2ports card4ports
lc 1 state unprovisioned
supported_types:
card1port card2ports card4ports
Hi Jiri
quoted
# Now activate the line card using debugfs. That emulates plug-in event
# on real hardware:
$ echo "Y"> /sys/kernel/debug/netdevsim/netdevsim10/linecards/0/active
$ ip link show eni10nl0p1
165: eni10nl0p1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 7e:2d:05:93:d3:d1 brd ff:ff:ff:ff:ff:ff
# The carrier is UP now.
What is missing from the devlink lc view is what line card is actually
in the slot. Say if i provision for a card4port, but actually insert a
card2port. It would be nice to have something like:
I checked, our hw does not support that. Only provides info that
linecard activation was/wasn't successful.
There is no way for the supervisor / management card to probe and see
what card is actually inserted in a given slot? That seems like a
serious design deficiency. What about some agent running on the line
card talking to an agent on the supervisor to provide that information?
From: Andrew Lunn <andrew@lunn.ch> Date: 2021-01-19 18:29:33
On Tue, Jan 19, 2021 at 12:56:10PM +0100, Jiri Pirko wrote:
Thu, Jan 14, 2021 at 03:07:18AM CET, andrew@lunn.ch wrote:
quoted
quoted
$ devlink lc provision netdevsim/netdevsim10 lc 0 type card4ports
$ devlink lc
netdevsim/netdevsim10:
lc 0 state provisioned type card4ports
supported_types:
card1port card2ports card4ports
lc 1 state unprovisioned
supported_types:
card1port card2ports card4ports
Hi Jiri
quoted
# Now activate the line card using debugfs. That emulates plug-in event
# on real hardware:
$ echo "Y"> /sys/kernel/debug/netdevsim/netdevsim10/linecards/0/active
$ ip link show eni10nl0p1
165: eni10nl0p1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 7e:2d:05:93:d3:d1 brd ff:ff:ff:ff:ff:ff
# The carrier is UP now.
What is missing from the devlink lc view is what line card is actually
in the slot. Say if i provision for a card4port, but actually insert a
card2port. It would be nice to have something like:
I checked, our hw does not support that. Only provides info that
linecard activation was/wasn't successful.
Hi Jiri
Is this a firmware limitation? There is no API to extract the
information from the firmware to the host? The firmware itself knows
there is a mismatch and refuses to configure the line card, and
prevents the MAC going up?
Even if you cannot do this now, it seems likely in future firmware
versions you will be able to, so maybe at least define the netlink
attributes now? As well as attributes indicating activation was
successful.
Andrew
Tue, Jan 19, 2021 at 03:51:49PM CET, andrew@lunn.ch wrote:
On Tue, Jan 19, 2021 at 12:56:10PM +0100, Jiri Pirko wrote:
quoted
Thu, Jan 14, 2021 at 03:07:18AM CET, andrew@lunn.ch wrote:
quoted
quoted
$ devlink lc provision netdevsim/netdevsim10 lc 0 type card4ports
$ devlink lc
netdevsim/netdevsim10:
lc 0 state provisioned type card4ports
supported_types:
card1port card2ports card4ports
lc 1 state unprovisioned
supported_types:
card1port card2ports card4ports
Hi Jiri
quoted
# Now activate the line card using debugfs. That emulates plug-in event
# on real hardware:
$ echo "Y"> /sys/kernel/debug/netdevsim/netdevsim10/linecards/0/active
$ ip link show eni10nl0p1
165: eni10nl0p1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 7e:2d:05:93:d3:d1 brd ff:ff:ff:ff:ff:ff
# The carrier is UP now.
What is missing from the devlink lc view is what line card is actually
in the slot. Say if i provision for a card4port, but actually insert a
card2port. It would be nice to have something like:
I checked, our hw does not support that. Only provides info that
linecard activation was/wasn't successful.
Hi Jiri
Is this a firmware limitation? There is no API to extract the
information from the firmware to the host? The firmware itself knows
there is a mismatch and refuses to configure the line card, and
prevents the MAC going up?
No, the FW does not know. The ASIC is not physically able to get the
linecard type. Yes, it is odd, I agree. The linecard type is known to
the driver which operates on i2c. This driver takes care of power
management of the linecard, among other tasks.
Even if you cannot do this now, it seems likely in future firmware
versions you will be able to, so maybe at least define the netlink
Sure, for netdevsim that is not problem. Our current hw does not support
it, the future may.
attributes now? As well as attributes indicating activation was
successful.
State "ACTIVATED" is that indication. It is in this RFC.
Tue, Jan 19, 2021 at 05:23:19PM CET, dsahern@gmail.com wrote:
On 1/19/21 4:56 AM, Jiri Pirko wrote:
quoted
Thu, Jan 14, 2021 at 03:07:18AM CET, andrew@lunn.ch wrote:
quoted
quoted
$ devlink lc provision netdevsim/netdevsim10 lc 0 type card4ports
$ devlink lc
netdevsim/netdevsim10:
lc 0 state provisioned type card4ports
supported_types:
card1port card2ports card4ports
lc 1 state unprovisioned
supported_types:
card1port card2ports card4ports
Hi Jiri
quoted
# Now activate the line card using debugfs. That emulates plug-in event
# on real hardware:
$ echo "Y"> /sys/kernel/debug/netdevsim/netdevsim10/linecards/0/active
$ ip link show eni10nl0p1
165: eni10nl0p1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 7e:2d:05:93:d3:d1 brd ff:ff:ff:ff:ff:ff
# The carrier is UP now.
What is missing from the devlink lc view is what line card is actually
in the slot. Say if i provision for a card4port, but actually insert a
card2port. It would be nice to have something like:
I checked, our hw does not support that. Only provides info that
linecard activation was/wasn't successful.
There is no way for the supervisor / management card to probe and see
what card is actually inserted in a given slot? That seems like a
serious design deficiency. What about some agent running on the line
card talking to an agent on the supervisor to provide that information?
The ASIC does not have this info. The linecard type is exposed over i2c
interface, different driver sits on top of it.
I agree it is odd, but that is how it is for our hw, unfortunatelly.
From: Andrew Lunn <andrew@lunn.ch> Date: 2021-01-20 20:36:42
No, the FW does not know. The ASIC is not physically able to get the
linecard type. Yes, it is odd, I agree. The linecard type is known to
the driver which operates on i2c. This driver takes care of power
management of the linecard, among other tasks.
So what does activated actually mean for your hardware? It seems to
mean something like: Some random card has been plugged in, we have no
idea what, but it has power, and we have enabled the MACs as
provisioned, which if you are lucky might match the hardware?
The foundations of this feature seems dubious.
Andrew
From: Andrew Lunn <andrew@lunn.ch> Date: 2021-01-21 00:57:27
On Wed, Jan 20, 2021 at 03:41:58PM -0800, Jakub Kicinski wrote:
On Wed, 20 Jan 2021 14:56:46 +0100 Andrew Lunn wrote:
quoted
quoted
No, the FW does not know. The ASIC is not physically able to get the
linecard type. Yes, it is odd, I agree. The linecard type is known to
the driver which operates on i2c. This driver takes care of power
management of the linecard, among other tasks.
So what does activated actually mean for your hardware? It seems to
mean something like: Some random card has been plugged in, we have no
idea what, but it has power, and we have enabled the MACs as
provisioned, which if you are lucky might match the hardware?
The foundations of this feature seems dubious.
But Jiri also says "The linecard type is known to the driver which
operates on i2c." which sounds like there is some i2c driver (in user
space?) which talks to the card and _does_ have the info? Maybe I'm
misreading it. What's the i2c driver?
Hi Jakub
A complete guess, but i think it will be the BMC, not the ASIC. There
have been patches from Mellanox in the past for a BMC, i think sent to
arm-soc, for the ASPEED devices often used as BMCs. And the BMC is
often the device doing power management. So what might be missing is
an interface between the driver and the BMC. But that then makes the
driver system specific. A OEM who buys ASICs and makes their own board
could have their own BMC running there own BMC firmware.
All speculation...
Andrew
From: Jakub Kicinski <kuba@kernel.org> Date: 2021-01-21 00:57:43
On Thu, 21 Jan 2021 01:01:21 +0100 Andrew Lunn wrote:
On Wed, Jan 20, 2021 at 03:41:58PM -0800, Jakub Kicinski wrote:
quoted
On Wed, 20 Jan 2021 14:56:46 +0100 Andrew Lunn wrote:
quoted
quoted
No, the FW does not know. The ASIC is not physically able to get the
linecard type. Yes, it is odd, I agree. The linecard type is known to
the driver which operates on i2c. This driver takes care of power
management of the linecard, among other tasks.
So what does activated actually mean for your hardware? It seems to
mean something like: Some random card has been plugged in, we have no
idea what, but it has power, and we have enabled the MACs as
provisioned, which if you are lucky might match the hardware?
The foundations of this feature seems dubious.
But Jiri also says "The linecard type is known to the driver which
operates on i2c." which sounds like there is some i2c driver (in user
space?) which talks to the card and _does_ have the info? Maybe I'm
misreading it. What's the i2c driver?
Hi Jakub
A complete guess, but i think it will be the BMC, not the ASIC. There
have been patches from Mellanox in the past for a BMC, i think sent to
arm-soc, for the ASPEED devices often used as BMCs. And the BMC is
often the device doing power management. So what might be missing is
an interface between the driver and the BMC. But that then makes the
driver system specific. A OEM who buys ASICs and makes their own board
could have their own BMC running there own BMC firmware.
All speculation...
From: Jakub Kicinski <kuba@kernel.org> Date: 2021-01-21 06:42:36
On Wed, 20 Jan 2021 14:56:46 +0100 Andrew Lunn wrote:
quoted
No, the FW does not know. The ASIC is not physically able to get the
linecard type. Yes, it is odd, I agree. The linecard type is known to
the driver which operates on i2c. This driver takes care of power
management of the linecard, among other tasks.
So what does activated actually mean for your hardware? It seems to
mean something like: Some random card has been plugged in, we have no
idea what, but it has power, and we have enabled the MACs as
provisioned, which if you are lucky might match the hardware?
The foundations of this feature seems dubious.
But Jiri also says "The linecard type is known to the driver which
operates on i2c." which sounds like there is some i2c driver (in user
space?) which talks to the card and _does_ have the info? Maybe I'm
misreading it. What's the i2c driver?
Thu, Jan 21, 2021 at 01:01:21AM CET, andrew@lunn.ch wrote:
On Wed, Jan 20, 2021 at 03:41:58PM -0800, Jakub Kicinski wrote:
quoted
On Wed, 20 Jan 2021 14:56:46 +0100 Andrew Lunn wrote:
quoted
quoted
No, the FW does not know. The ASIC is not physically able to get the
linecard type. Yes, it is odd, I agree. The linecard type is known to
the driver which operates on i2c. This driver takes care of power
management of the linecard, among other tasks.
So what does activated actually mean for your hardware? It seems to
mean something like: Some random card has been plugged in, we have no
idea what, but it has power, and we have enabled the MACs as
provisioned, which if you are lucky might match the hardware?
The foundations of this feature seems dubious.
But Jiri also says "The linecard type is known to the driver which
operates on i2c." which sounds like there is some i2c driver (in user
space?) which talks to the card and _does_ have the info? Maybe I'm
misreading it. What's the i2c driver?
Hi Jakub
A complete guess, but i think it will be the BMC, not the ASIC. There
have been patches from Mellanox in the past for a BMC, i think sent to
arm-soc, for the ASPEED devices often used as BMCs. And the BMC is
often the device doing power management. So what might be missing is
an interface between the driver and the BMC. But that then makes the
driver system specific. A OEM who buys ASICs and makes their own board
could have their own BMC running there own BMC firmware.
All speculation...
Basically all correct.
The thing is mlxsw and the i2c driver cannot talk to each other:
1) It would be ugly
2) They may likely be on a different host
Thu, Jan 21, 2021 at 12:41:58AM CET, kuba@kernel.org wrote:
On Wed, 20 Jan 2021 14:56:46 +0100 Andrew Lunn wrote:
quoted
quoted
No, the FW does not know. The ASIC is not physically able to get the
linecard type. Yes, it is odd, I agree. The linecard type is known to
the driver which operates on i2c. This driver takes care of power
management of the linecard, among other tasks.
So what does activated actually mean for your hardware? It seems to
mean something like: Some random card has been plugged in, we have no
idea what, but it has power, and we have enabled the MACs as
provisioned, which if you are lucky might match the hardware?
The foundations of this feature seems dubious.
But Jiri also says "The linecard type is known to the driver which
operates on i2c." which sounds like there is some i2c driver (in user
space?) which talks to the card and _does_ have the info? Maybe I'm
misreading it. What's the i2c driver?
That is Vadim's i2c kernel driver, this is going to upstream.
From: David Ahern <hidden> Date: 2021-01-21 16:45:02
On 1/21/21 8:32 AM, Jiri Pirko wrote:
Thu, Jan 21, 2021 at 12:41:58AM CET, kuba@kernel.org wrote:
quoted
On Wed, 20 Jan 2021 14:56:46 +0100 Andrew Lunn wrote:
quoted
quoted
No, the FW does not know. The ASIC is not physically able to get the
linecard type. Yes, it is odd, I agree. The linecard type is known to
the driver which operates on i2c. This driver takes care of power
management of the linecard, among other tasks.
So what does activated actually mean for your hardware? It seems to
mean something like: Some random card has been plugged in, we have no
idea what, but it has power, and we have enabled the MACs as
provisioned, which if you are lucky might match the hardware?
The foundations of this feature seems dubious.
But Jiri also says "The linecard type is known to the driver which
operates on i2c." which sounds like there is some i2c driver (in user
space?) which talks to the card and _does_ have the info? Maybe I'm
misreading it. What's the i2c driver?
That is Vadim's i2c kernel driver, this is going to upstream.
This pre-provisioning concept makes a fragile design to work around h/w
shortcomings. You really need a way for the management card to know
exactly what was plugged in to a slot so the control plane S/W can
respond accordingly. Surely there is a way for processes on the LC to
communicate with a process on the management card - even if it is inband
packets with special headers.
Thu, Jan 21, 2021 at 05:38:40PM CET, dsahern@gmail.com wrote:
On 1/21/21 8:32 AM, Jiri Pirko wrote:
quoted
Thu, Jan 21, 2021 at 12:41:58AM CET, kuba@kernel.org wrote:
quoted
On Wed, 20 Jan 2021 14:56:46 +0100 Andrew Lunn wrote:
quoted
quoted
No, the FW does not know. The ASIC is not physically able to get the
linecard type. Yes, it is odd, I agree. The linecard type is known to
the driver which operates on i2c. This driver takes care of power
management of the linecard, among other tasks.
So what does activated actually mean for your hardware? It seems to
mean something like: Some random card has been plugged in, we have no
idea what, but it has power, and we have enabled the MACs as
provisioned, which if you are lucky might match the hardware?
The foundations of this feature seems dubious.
But Jiri also says "The linecard type is known to the driver which
operates on i2c." which sounds like there is some i2c driver (in user
space?) which talks to the card and _does_ have the info? Maybe I'm
misreading it. What's the i2c driver?
That is Vadim's i2c kernel driver, this is going to upstream.
This pre-provisioning concept makes a fragile design to work around h/w
shortcomings. You really need a way for the management card to know
exactly what was plugged in to a slot so the control plane S/W can
respond accordingly. Surely there is a way for processes on the LC to
communicate with a process on the management card - even if it is inband
packets with special headers.
I don't see any way. The userspace is the one who can get the info, from
the i2c driver. The mlxsw driver has no means to get that info itself.
Mon, Jan 18, 2021 at 11:55:45PM CET, dsahern@gmail.com wrote:
On 1/18/21 6:00 AM, Jiri Pirko wrote:
quoted
quoted
Reconfiguring routing is not the end of the world.
Well, yes, but you don't really want netdevices to come and go then you
plug in/out cables/modules. That's why we have split implemented as we
And you don't want a routing daemon to use netdevices which are not
valid due to non-existence. Best case with what you want is carrier down
on the LC's netdevices and that destroys routing.
There are other things. The user may configure the netdev parameters in
advance, like mtu, put it in a bridge, setup TC filters on it etc.
The linecard unplug/plug does not destroy the settings. This is the same
thing with split ports and that is why we have implemented split ports
in "provision" mode as well.
quoted
do. I don't understand why do you think linecards are different.
I still don't get why you expect linecards to be different than any
other hotplug device.
It it not a device, does not have "struct device" related to it.
It is just a phy part of another device.
Thu, Jan 21, 2021 at 05:38:40PM CET, dsahern@gmail.com wrote:
On 1/21/21 8:32 AM, Jiri Pirko wrote:
quoted
Thu, Jan 21, 2021 at 12:41:58AM CET, kuba@kernel.org wrote:
quoted
On Wed, 20 Jan 2021 14:56:46 +0100 Andrew Lunn wrote:
quoted
quoted
No, the FW does not know. The ASIC is not physically able to get the
linecard type. Yes, it is odd, I agree. The linecard type is known to
the driver which operates on i2c. This driver takes care of power
management of the linecard, among other tasks.
So what does activated actually mean for your hardware? It seems to
mean something like: Some random card has been plugged in, we have no
idea what, but it has power, and we have enabled the MACs as
provisioned, which if you are lucky might match the hardware?
The foundations of this feature seems dubious.
But Jiri also says "The linecard type is known to the driver which
operates on i2c." which sounds like there is some i2c driver (in user
space?) which talks to the card and _does_ have the info? Maybe I'm
misreading it. What's the i2c driver?
That is Vadim's i2c kernel driver, this is going to upstream.
This pre-provisioning concept makes a fragile design to work around h/w
shortcomings. You really need a way for the management card to know
Not really. As I replied to you in the other part of this thread, the
linecard is basically very similar to a splitter cable. In a way, it is
a splitter cable. And should be threated in a similar way. As a phy. Not
as a device. Cables are replaceble without netdevice reappearing. This
linecards are the same. Therefore, the concept of provisioning makes
sense for them, as it does for splitter cable.
exactly what was plugged in to a slot so the control plane S/W can
respond accordingly. Surely there is a way for processes on the LC to
communicate with a process on the management card - even if it is inband
packets with special headers.
If a device is capable of splitter cable/linecard hotplug, sure, that
may be implemented. But the user has to configure it as such, to be
aware that "cable change" may move netdevices around.
From: Andrew Lunn <andrew@lunn.ch> Date: 2021-01-22 19:54:52
I don't see any way. The userspace is the one who can get the info, from
the i2c driver. The mlxsw driver has no means to get that info itself.
Hi Jiri
Please can you tell us more about this i2c driver. Do you have any
architecture pictures?
It is not unknown for one driver to embed another driver inside it. So
the i2c driver could be inside the mlxsw. It is also possible to link
drivers together, the mlxsw could go find the i2c driver and make use
of its services.
Andrew
Fri, Jan 22, 2021 at 03:13:12PM CET, andrew@lunn.ch wrote:
quoted
I don't see any way. The userspace is the one who can get the info, from
the i2c driver. The mlxsw driver has no means to get that info itself.
Hi Jiri
Please can you tell us more about this i2c driver. Do you have any
architecture pictures?
Quoting Vadim Pasternak:
"
Not upstreamed yet.
It will be mlxreg-lc driver for line card in drivers/platfrom/mellanox and
additional mlxreg-pm for line card powering on/off, setting enable/disable
and handling power off upon thermal shutdown event.
"
It is not unknown for one driver to embed another driver inside it. So
the i2c driver could be inside the mlxsw. It is also possible to link
drivers together, the mlxsw could go find the i2c driver and make use
of its services.
Okay. Do you have examples? How could the kernel figure out the relation
of the instances?
From: Andrew Lunn <andrew@lunn.ch> Date: 2021-01-26 13:58:47
On Tue, Jan 26, 2021 at 12:33:26PM +0100, Jiri Pirko wrote:
Fri, Jan 22, 2021 at 03:13:12PM CET, andrew@lunn.ch wrote:
quoted
quoted
I don't see any way. The userspace is the one who can get the info, from
the i2c driver. The mlxsw driver has no means to get that info itself.
Hi Jiri
Please can you tell us more about this i2c driver. Do you have any
architecture pictures?
Quoting Vadim Pasternak:
"
Not upstreamed yet.
It will be mlxreg-lc driver for line card in drivers/platfrom/mellanox and
additional mlxreg-pm for line card powering on/off, setting enable/disable
and handling power off upon thermal shutdown event.
"
quoted
It is not unknown for one driver to embed another driver inside it. So
the i2c driver could be inside the mlxsw. It is also possible to link
drivers together, the mlxsw could go find the i2c driver and make use
of its services.
Okay. Do you have examples? How could the kernel figure out the relation
of the instances?
Hi Jiri
One driver, embedded into another? You actually submitted an example:
commit 6882b0aee180f2797b8803bdf699aa45c2e5f2d6
Author: Vadim Pasternak [off-list ref]
Date: Wed Nov 16 15:20:44 2016 +0100
mlxsw: Introduce support for I2C bus
Add I2C bus implementation for Mellanox Technologies Switch ASICs.
This includes command interface implementation using input / out mailboxes,
whose location is retrieved from the firmware during probe time.
Signed-off-by: Vadim Pasternak [off-list ref]
Reviewed-by: Ido Schimmel [off-list ref]
Signed-off-by: Jiri Pirko [off-list ref]
Signed-off-by: David S. Miller [off-list ref]
There are Linux standard APIs for controlling the power to devices,
the regulator API. So i assume mlxreg-pm will make use of that. There
are also standard APIs for thermal management, which again, mlxreg-pm
should be using. The regulator API allows you to find regulators by
name. So just define a sensible naming convention, and the switch
driver can lookup the regulator, and turn it on/off as needed.
I'm guessing there are no standard Linux API which mlxreg-lc fits. I'm
also not sure it offers anything useful standalone. So i would
actually embed it inside the switchdev driver, and have internal APIs
to get information about the line card.
But i'm missing big picture architecture knowledge here, there could
be reasons why these suggestions don't work.
Andrew
Tue, Jan 26, 2021 at 02:56:08PM CET, andrew@lunn.ch wrote:
On Tue, Jan 26, 2021 at 12:33:26PM +0100, Jiri Pirko wrote:
quoted
Fri, Jan 22, 2021 at 03:13:12PM CET, andrew@lunn.ch wrote:
quoted
quoted
I don't see any way. The userspace is the one who can get the info, from
the i2c driver. The mlxsw driver has no means to get that info itself.
Hi Jiri
Please can you tell us more about this i2c driver. Do you have any
architecture pictures?
Quoting Vadim Pasternak:
"
Not upstreamed yet.
It will be mlxreg-lc driver for line card in drivers/platfrom/mellanox and
additional mlxreg-pm for line card powering on/off, setting enable/disable
and handling power off upon thermal shutdown event.
"
quoted
It is not unknown for one driver to embed another driver inside it. So
the i2c driver could be inside the mlxsw. It is also possible to link
drivers together, the mlxsw could go find the i2c driver and make use
of its services.
Okay. Do you have examples? How could the kernel figure out the relation
of the instances?
Hi Jiri
One driver, embedded into another? You actually submitted an example:
commit 6882b0aee180f2797b8803bdf699aa45c2e5f2d6
Author: Vadim Pasternak [off-list ref]
Date: Wed Nov 16 15:20:44 2016 +0100
mlxsw: Introduce support for I2C bus
Add I2C bus implementation for Mellanox Technologies Switch ASICs.
This includes command interface implementation using input / out mailboxes,
whose location is retrieved from the firmware during probe time.
Signed-off-by: Vadim Pasternak [off-list ref]
Reviewed-by: Ido Schimmel [off-list ref]
Signed-off-by: Jiri Pirko [off-list ref]
Signed-off-by: David S. Miller [off-list ref]
There are Linux standard APIs for controlling the power to devices,
the regulator API. So i assume mlxreg-pm will make use of that. There
are also standard APIs for thermal management, which again, mlxreg-pm
should be using. The regulator API allows you to find regulators by
name. So just define a sensible naming convention, and the switch
driver can lookup the regulator, and turn it on/off as needed.
I don't think it would apply. The thing is, i2c driver has a channel to
the linecard eeprom, from where it can read info about the linecard. The
i2c driver also knows when the linecard is plugged in, unlike mlxsw.
It acts as a standalone driver. Mlxsw has no way to directly find if the
card was plugged in (unpowered) and which type it is.
Not sure how to "embed" it. I don't think any existing API could help.
Basicall mlxsw would have to register a callback to the i2c driver
called every time card is inserted to do auto-provision.
Now consider a case when there are multiple instances of the ASIC on the
system. How to assemble a relationship between mlxsw instance and i2c
driver instance?
But again, auto-provision is only one usecase. Manual provisioning is
needed anyway. And that is exactly what my patchset is aiming to
introduce. Auto-provision can be added when/if needed later on.
I'm guessing there are no standard Linux API which mlxreg-lc fits. I'm
also not sure it offers anything useful standalone. So i would
actually embed it inside the switchdev driver, and have internal APIs
to get information about the line card.
But i'm missing big picture architecture knowledge here, there could
be reasons why these suggestions don't work.
Andrew
From: Andrew Lunn <andrew@lunn.ch> Date: 2021-01-27 14:16:39
quoted
There are Linux standard APIs for controlling the power to devices,
the regulator API. So i assume mlxreg-pm will make use of that. There
are also standard APIs for thermal management, which again, mlxreg-pm
should be using. The regulator API allows you to find regulators by
name. So just define a sensible naming convention, and the switch
driver can lookup the regulator, and turn it on/off as needed.
I don't think it would apply. The thing is, i2c driver has a channel to
the linecard eeprom, from where it can read info about the linecard. The
i2c driver also knows when the linecard is plugged in, unlike mlxsw.
It acts as a standalone driver. Mlxsw has no way to directly find if the
card was plugged in (unpowered) and which type it is.
Not sure how to "embed" it. I don't think any existing API could help.
Basicall mlxsw would have to register a callback to the i2c driver
called every time card is inserted to do auto-provision.
Now consider a case when there are multiple instances of the ASIC on the
system. How to assemble a relationship between mlxsw instance and i2c
driver instance?
You have that knowledge already, otherwise you cannot solve this
problem at all. The switch is an PCIe device right? So when the bus is
enumerated, the driver loads. How do you bind the i2c driver to the
i2c bus? You cannot enumerate i2c, so you must have some hard coded
knowledge somewhere? You just need to get that knowledge into the
mlxsw driver so it can bind its internal i2c client driver to the i2c
bus. That way you avoid user space, i guess maybe udev rules, or some
daemon monitoring propriety /sys files?
But again, auto-provision is only one usecase. Manual provisioning is
needed anyway. And that is exactly what my patchset is aiming to
introduce. Auto-provision can be added when/if needed later on.
I still don't actually get this use case. Why would i want to manually
provision?
Andrew
From: David Ahern <hidden> Date: 2021-01-27 15:06:32
On 1/27/21 7:14 AM, Andrew Lunn wrote:
quoted
I don't think it would apply. The thing is, i2c driver has a channel to
the linecard eeprom, from where it can read info about the linecard. The
i2c driver also knows when the linecard is plugged in, unlike mlxsw.
It acts as a standalone driver. Mlxsw has no way to directly find if the
card was plugged in (unpowered) and which type it is.
Not sure how to "embed" it. I don't think any existing API could help.
Basicall mlxsw would have to register a callback to the i2c driver
called every time card is inserted to do auto-provision.
Now consider a case when there are multiple instances of the ASIC on the
system. How to assemble a relationship between mlxsw instance and i2c
driver instance?
You have that knowledge already, otherwise you cannot solve this
problem at all. The switch is an PCIe device right? So when the bus is
enumerated, the driver loads. How do you bind the i2c driver to the
i2c bus? You cannot enumerate i2c, so you must have some hard coded
knowledge somewhere? You just need to get that knowledge into the
mlxsw driver so it can bind its internal i2c client driver to the i2c
bus. That way you avoid user space, i guess maybe udev rules, or some
daemon monitoring propriety /sys files?
quoted
But again, auto-provision is only one usecase. Manual provisioning is
needed anyway. And that is exactly what my patchset is aiming to
introduce. Auto-provision can be added when/if needed later on.
I still don't actually get this use case. Why would i want to manually
provision?
Wed, Jan 27, 2021 at 03:14:34PM CET, andrew@lunn.ch wrote:
quoted
quoted
There are Linux standard APIs for controlling the power to devices,
the regulator API. So i assume mlxreg-pm will make use of that. There
are also standard APIs for thermal management, which again, mlxreg-pm
should be using. The regulator API allows you to find regulators by
name. So just define a sensible naming convention, and the switch
driver can lookup the regulator, and turn it on/off as needed.
I don't think it would apply. The thing is, i2c driver has a channel to
the linecard eeprom, from where it can read info about the linecard. The
i2c driver also knows when the linecard is plugged in, unlike mlxsw.
It acts as a standalone driver. Mlxsw has no way to directly find if the
card was plugged in (unpowered) and which type it is.
Not sure how to "embed" it. I don't think any existing API could help.
Basicall mlxsw would have to register a callback to the i2c driver
called every time card is inserted to do auto-provision.
Now consider a case when there are multiple instances of the ASIC on the
system. How to assemble a relationship between mlxsw instance and i2c
driver instance?
You have that knowledge already, otherwise you cannot solve this
No I don't have it. I'm not sure why do you say so. The mlxsw and i2c
driver act independently.
problem at all. The switch is an PCIe device right? So when the bus is
enumerated, the driver loads. How do you bind the i2c driver to the
i2c bus? You cannot enumerate i2c, so you must have some hard coded
knowledge somewhere? You just need to get that knowledge into the
mlxsw driver so it can bind its internal i2c client driver to the i2c
There is no internal i2c client driver for this.
bus. That way you avoid user space, i guess maybe udev rules, or some
daemon monitoring propriety /sys files?
quoted
But again, auto-provision is only one usecase. Manual provisioning is
needed anyway. And that is exactly what my patchset is aiming to
introduce. Auto-provision can be added when/if needed later on.
I still don't actually get this use case. Why would i want to manually
provision?
Because user might want to see the system with all netdevices, configure
them, change the linecard if they got broken and all config, like
bridge, tc, etc will stay on the netdevices. Again, this is the same we
do for split port. This is important requirement, user don't want to see
netdevices come and go when he is plugging/unplugging cables. Linecards
are the same in this matter. Basically is is a "splitter module",
replacing the "splitter cable"
From: Andrew Lunn <andrew@lunn.ch> Date: 2021-01-28 14:18:06
On Thu, Jan 28, 2021 at 09:14:34AM +0100, Jiri Pirko wrote:
Wed, Jan 27, 2021 at 03:14:34PM CET, andrew@lunn.ch wrote:
quoted
quoted
quoted
There are Linux standard APIs for controlling the power to devices,
the regulator API. So i assume mlxreg-pm will make use of that. There
are also standard APIs for thermal management, which again, mlxreg-pm
should be using. The regulator API allows you to find regulators by
name. So just define a sensible naming convention, and the switch
driver can lookup the regulator, and turn it on/off as needed.
I don't think it would apply. The thing is, i2c driver has a channel to
the linecard eeprom, from where it can read info about the linecard. The
i2c driver also knows when the linecard is plugged in, unlike mlxsw.
It acts as a standalone driver. Mlxsw has no way to directly find if the
card was plugged in (unpowered) and which type it is.
Not sure how to "embed" it. I don't think any existing API could help.
Basicall mlxsw would have to register a callback to the i2c driver
called every time card is inserted to do auto-provision.
Now consider a case when there are multiple instances of the ASIC on the
system. How to assemble a relationship between mlxsw instance and i2c
driver instance?
You have that knowledge already, otherwise you cannot solve this
No I don't have it. I'm not sure why do you say so. The mlxsw and i2c
driver act independently.
Ah, so you just export some information in /sys from the i2c driver?
And you expect the poor user to look at the values, and copy paste
them to the correct mlxsw instance? 50/50 guess if you have two
switches, and hope they don't make a typO?
quoted
I still don't actually get this use case. Why would i want to manually
provision?
Because user might want to see the system with all netdevices, configure
them, change the linecard if they got broken and all config, like
bridge, tc, etc will stay on the netdevices. Again, this is the same we
do for split port. This is important requirement, user don't want to see
netdevices come and go when he is plugging/unplugging cables. Linecards
are the same in this matter. Basically is is a "splitter module",
replacing the "splitter cable"
So, what is the real use case here? Why might the user want to do
this?
Is it: The magic smoke has escaped. The user takes a spare switch, and
wants to put it on her desk to configure it where she has a comfy chair
and piece and quiet, unlike in the data centre, which is very noise,
only has hard plastic chair, no coffee allowed. She makes her best
guess at the configuration, up/downs the interfaces, reboots, to make
sure it is permanent, and only then moves to the data centre to swap
the dead router for the new one, and fix up whatever configuration
errors there are, while sat on the hard chair?
So this feature is about comfy chair vs hard chair?
I'm also wondering about the splitter port use case. At what point do
you tell the user that it is physically impossible to split the port
because the SFP simply does not support it? You say the netdevs don't
come/go. I assume the link never goes up, but how does the user know
the configuration is FUBAR, not the SFP? To me, it seems a lot more
intuitive that when i remove an SFP which has been split into 4, and
pop in an SFP which only supports a single stream, the 3 extra netdevs
would just vanish.
Andrew
Thu, Jan 28, 2021 at 03:17:13PM CET, andrew@lunn.ch wrote:
On Thu, Jan 28, 2021 at 09:14:34AM +0100, Jiri Pirko wrote:
quoted
Wed, Jan 27, 2021 at 03:14:34PM CET, andrew@lunn.ch wrote:
quoted
quoted
quoted
There are Linux standard APIs for controlling the power to devices,
the regulator API. So i assume mlxreg-pm will make use of that. There
are also standard APIs for thermal management, which again, mlxreg-pm
should be using. The regulator API allows you to find regulators by
name. So just define a sensible naming convention, and the switch
driver can lookup the regulator, and turn it on/off as needed.
I don't think it would apply. The thing is, i2c driver has a channel to
the linecard eeprom, from where it can read info about the linecard. The
i2c driver also knows when the linecard is plugged in, unlike mlxsw.
It acts as a standalone driver. Mlxsw has no way to directly find if the
card was plugged in (unpowered) and which type it is.
Not sure how to "embed" it. I don't think any existing API could help.
Basicall mlxsw would have to register a callback to the i2c driver
called every time card is inserted to do auto-provision.
Now consider a case when there are multiple instances of the ASIC on the
system. How to assemble a relationship between mlxsw instance and i2c
driver instance?
You have that knowledge already, otherwise you cannot solve this
No I don't have it. I'm not sure why do you say so. The mlxsw and i2c
driver act independently.
Ah, so you just export some information in /sys from the i2c driver?
And you expect the poor user to look at the values, and copy paste
them to the correct mlxsw instance? 50/50 guess if you have two
switches, and hope they don't make a typO?
Which values are you talking about here exactly?
quoted
quoted
I still don't actually get this use case. Why would i want to manually
provision?
Because user might want to see the system with all netdevices, configure
them, change the linecard if they got broken and all config, like
bridge, tc, etc will stay on the netdevices. Again, this is the same we
do for split port. This is important requirement, user don't want to see
netdevices come and go when he is plugging/unplugging cables. Linecards
are the same in this matter. Basically is is a "splitter module",
replacing the "splitter cable"
So, what is the real use case here? Why might the user want to do
this?
Is it: The magic smoke has escaped. The user takes a spare switch, and
wants to put it on her desk to configure it where she has a comfy chair
and piece and quiet, unlike in the data centre, which is very noise,
only has hard plastic chair, no coffee allowed. She makes her best
guess at the configuration, up/downs the interfaces, reboots, to make
sure it is permanent, and only then moves to the data centre to swap
the dead router for the new one, and fix up whatever configuration
errors there are, while sat on the hard chair?
So this feature is about comfy chair vs hard chair?
I don't really get the question, but configuring switch w/o any linecard
and plug the linecards in later on is definitelly a usecase.
I'm also wondering about the splitter port use case. At what point do
you tell the user that it is physically impossible to split the port
because the SFP simply does not support it? You say the netdevs don't
come/go. I assume the link never goes up, but how does the user know
the configuration is FUBAR, not the SFP? To me, it seems a lot more
intuitive that when i remove an SFP which has been split into 4, and
pop in an SFP which only supports a single stream, the 3 extra netdevs
would just vanish.
As I wrote easlier in this thread, for hw that supports it, there should
be possibility to turn on "autosplit" mode that would do exactly what
you describe. But depends on a usecase. User should be in power to
configure "autosplit" for split cables and "autodetect" for linecards.
Both should be treated in the same way I believe.