From: Jakub Kicinski <hidden> Date: 2017-06-06 00:02:15
Hi!
This series adds the ability to use one vNIC as a control channel
for passing messages to and from the application firmware. The
implementation restructures the existing netdev vNIC code to be able
to deal with nfp_nets with netdev pointer set to NULL. Control vNICs
are not visible to userspace (other than for dumping ring state), and
since they don't have netdevs we use a tasklet for RX and simple skb
list for TX queuing.
Due to special status of the control vNIC we have to reshuffle the
init code a bit to make sure control vNIC will be fully brought up
(and therefore communication with app FW can happen) before any netdev
or port is visible to user space.
FW will designate which vNIC is supposed to be used as control one
by setting _pf%u_net_ctrl_bar symbol. Some FWs depend on metadata
being prepended to control message, some prefer to look at queue ID
to decide that something is a control message. Our implementation
can cater to both.
First two users of this code will be eBPF maps and flower offloads.
Jakub Kicinski (16):
nfp: reorder open and close functions
nfp: split out the allocation part of open
nfp: reuse ring free code on close
nfp: move nfp_net_vecs_init()
nfp: prepare print macros for use without netdev
nfp: make sure debug accesses don't depend on netdevs
nfp: allow allocation and initialization of netdev-less vNICs
nfp: prepare config and enable for working without netdevs
nfp: add control vNIC datapath
nfp: make vNIC ctrl memory mapping function reusable
nfp: map all queue controllers at once
nfp: don't clutter init code passing fw_ver around
nfp: slice the netdev spawning function
nfp: allow non-equal distribution of IRQs
nfp: create control vNICs and wire up rx/tx
nfp: advertise support for NFD ABI 0.5
drivers/net/ethernet/netronome/nfp/nfp_app.c | 18 +
drivers/net/ethernet/netronome/nfp/nfp_app.h | 55 ++
drivers/net/ethernet/netronome/nfp/nfp_main.h | 13 +-
drivers/net/ethernet/netronome/nfp/nfp_net.h | 64 +-
.../net/ethernet/netronome/nfp/nfp_net_common.c | 709 +++++++++++++++------
drivers/net/ethernet/netronome/nfp/nfp_net_ctrl.h | 3 +
.../net/ethernet/netronome/nfp/nfp_net_debugfs.c | 9 +-
drivers/net/ethernet/netronome/nfp/nfp_net_main.c | 365 ++++++-----
.../net/ethernet/netronome/nfp/nfp_netvf_main.c | 4 +-
9 files changed, 872 insertions(+), 368 deletions(-)
--
2.11.0
From: Jakub Kicinski <hidden> Date: 2017-06-06 00:02:15
We will soon reuse parts of .ndo_stop() for clean up after errors
in .ndo_open(). Reorder the associated functions to make that possible.
No functional changes.
Signed-off-by: Jakub Kicinski <redacted>
---
.../net/ethernet/netronome/nfp/nfp_net_common.c | 140 ++++++++++-----------
1 file changed, 70 insertions(+), 70 deletions(-)
From: Jakub Kicinski <hidden> Date: 2017-06-06 00:02:16
Our open/close implementations have 3 stages:
- allocation/freeing of ring resources, irqs etc.,
- device config,
- device/stack enable (can't fail).
Right now all of those stages are placed in separate functions,
apart from allocation during open. Fix that. It will make it
easier for us to allocate resources for netdev-less vNICs.
Because we want to reuse allocation code in netdev-less vNICs
leave the netif_set_real_num_[rt]x_queues() calls inside open.
Signed-off-by: Jakub Kicinski <redacted>
---
.../net/ethernet/netronome/nfp/nfp_net_common.c | 57 +++++++++++++---------
1 file changed, 35 insertions(+), 22 deletions(-)
@@ -2364,16 +2364,10 @@ static void nfp_net_open_stack(struct nfp_net *nn)nfp_net_read_link_status(nn);}-staticintnfp_net_netdev_open(structnet_device*netdev)+staticintnfp_net_open_alloc_all(structnfp_net*nn){-structnfp_net*nn=netdev_priv(netdev);interr,r;-/* Step 1: Allocate resources for rings and the like-*-Requestinterrupts-*-AllocateRXandTXringresources-*-SetupinitialRSStable-*/err=nfp_net_aux_irq_request(nn,NFP_NET_CFG_EXN,"%s-exn",nn->exn_name,sizeof(nn->exn_name),NFP_NET_IRQ_EXN_IDX,nn->exn_handler);
@@ -2403,13 +2397,42 @@ static int nfp_net_netdev_open(struct net_device *netdev)for(r=0;r<nn->max_r_vecs;r++)nfp_net_vector_assign_rings(&nn->dp,&nn->r_vecs[r],r);+return0;++err_free_rx_rings:+nfp_net_rx_rings_free(&nn->dp);+err_cleanup_vec:+r=nn->dp.num_r_vecs;+err_cleanup_vec_p:+while(r--)+nfp_net_cleanup_vector(nn,&nn->r_vecs[r]);+nfp_net_aux_irq_free(nn,NFP_NET_CFG_LSC,NFP_NET_IRQ_LSC_IDX);+err_free_exn:+nfp_net_aux_irq_free(nn,NFP_NET_CFG_EXN,NFP_NET_IRQ_EXN_IDX);+returnerr;+}++staticintnfp_net_netdev_open(structnet_device*netdev)+{+structnfp_net*nn=netdev_priv(netdev);+interr;++/* Step 1: Allocate resources for rings and the like+*-Requestinterrupts+*-AllocateRXandTXringresources+*-SetupinitialRSStable+*/+err=nfp_net_open_alloc_all(nn);+if(err)+returnerr;+err=netif_set_real_num_tx_queues(netdev,nn->dp.num_stack_tx_rings);if(err)-gotoerr_free_rings;+gotoerr_free_all;err=netif_set_real_num_rx_queues(netdev,nn->dp.num_rx_rings);if(err)-gotoerr_free_rings;+gotoerr_free_all;/* Step 2: Configure the NFP*-Enableringsfrom0totx_rings/rx_rings-1.
@@ -2420,7 +2443,7 @@ static int nfp_net_netdev_open(struct net_device *netdev)*/err=nfp_net_set_config_and_enable(nn);if(err)-gotoerr_free_rings;+gotoerr_free_all;/* Step 3: Enable for kernel*-putsomefreelistdescriptorsoneachRXring
@@ -2432,18 +2455,8 @@ static int nfp_net_netdev_open(struct net_device *netdev)return0;-err_free_rings:-nfp_net_tx_rings_free(&nn->dp);-err_free_rx_rings:-nfp_net_rx_rings_free(&nn->dp);-err_cleanup_vec:-r=nn->dp.num_r_vecs;-err_cleanup_vec_p:-while(r--)-nfp_net_cleanup_vector(nn,&nn->r_vecs[r]);-nfp_net_aux_irq_free(nn,NFP_NET_CFG_LSC,NFP_NET_IRQ_LSC_IDX);-err_free_exn:-nfp_net_aux_irq_free(nn,NFP_NET_CFG_EXN,NFP_NET_IRQ_EXN_IDX);+err_free_all:+nfp_net_close_free_all(nn);returnerr;}
From: Jakub Kicinski <hidden> Date: 2017-06-06 00:02:17
On the close path reuse the ring free helpers introduced for runtime
reconfiguration.
Signed-off-by: Jakub Kicinski <redacted>
---
drivers/net/ethernet/netronome/nfp/nfp_net_common.c | 14 +++-----------
1 file changed, 3 insertions(+), 11 deletions(-)
From: Jakub Kicinski <hidden> Date: 2017-06-06 00:02:18
Move nfp_net_vecs_init() after all datapath functions. We will need
to init poll() callbacks from this function soon.
No functional changes.
Signed-off-by: Jakub Kicinski <redacted>
---
.../net/ethernet/netronome/nfp/nfp_net_common.c | 54 +++++++++++-----------
1 file changed, 27 insertions(+), 27 deletions(-)
From: Jakub Kicinski <hidden> Date: 2017-06-06 00:02:20
We want to be able to inspect the state of descriptor rings of
the control vNIC, so it will use the same interface as data vNICs.
Make sure the code doesn't use netdevs to determine state
of the rings and names things appropriately.
Signed-off-by: Jakub Kicinski <redacted>
---
drivers/net/ethernet/netronome/nfp/nfp_net.h | 11 +++++++++++
drivers/net/ethernet/netronome/nfp/nfp_net_debugfs.c | 9 ++++++---
2 files changed, 17 insertions(+), 3 deletions(-)
From: Jakub Kicinski <hidden> Date: 2017-06-06 00:02:21
Out of the three stages of ifup/ifdown (allocate, configure, start)
- this commit prepares the configuration stage for working with
control vNICs.
Signed-off-by: Jakub Kicinski <redacted>
---
drivers/net/ethernet/netronome/nfp/nfp_net_common.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
From: Jakub Kicinski <hidden> Date: 2017-06-06 00:02:21
vNICs used for sending and receiving control messages shouldn't
really have a netdev. Add the ability to initialize vNICs for
netdev-less operation.
Signed-off-by: Jakub Kicinski <redacted>
---
drivers/net/ethernet/netronome/nfp/nfp_net.h | 2 +-
.../net/ethernet/netronome/nfp/nfp_net_common.c | 170 ++++++++++++---------
drivers/net/ethernet/netronome/nfp/nfp_net_main.c | 7 +-
.../net/ethernet/netronome/nfp/nfp_netvf_main.c | 2 +-
4 files changed, 105 insertions(+), 76 deletions(-)
@@ -3162,52 +3174,13 @@ static void nfp_net_irqmod_init(struct nfp_net *nn)nn->tx_coalesce_max_frames=64;}-/**-*nfp_net_init()-Initialise/finalisethenfp_netstructure-*@nn:NFPNetdevicestructure-*-*Return:0onsuccessornegativeerrnoonerror.-*/-intnfp_net_init(structnfp_net*nn)+staticvoidnfp_net_netdev_init(structnfp_net*nn){structnet_device*netdev=nn->dp.netdev;-interr;--nn->dp.rx_dma_dir=DMA_FROM_DEVICE;--/* Get some of the read-only fields from the BAR */-nn->cap=nn_readl(nn,NFP_NET_CFG_CAP);-nn->max_mtu=nn_readl(nn,NFP_NET_CFG_MAX_MTU);--/* Chained metadata is signalled by capabilities except in version 4 */-nn->dp.chained_metadata_format=nn->fw_ver.major==4||-nn->cap&NFP_NET_CFG_CTRL_CHAIN_META;-if(nn->dp.chained_metadata_format&&nn->fw_ver.major!=4)-nn->cap&=~NFP_NET_CFG_CTRL_RSS;nfp_net_write_mac_addr(nn,nn->dp.netdev->dev_addr);-/* Determine RX packet/metadata boundary offset */-if(nn->fw_ver.major>=2){-u32reg;--reg=nn_readl(nn,NFP_NET_CFG_RX_OFFSET);-if(reg>NFP_NET_MAX_PREPEND){-nn_err(nn,"Invalid rx offset: %d\n",reg);-return-EINVAL;-}-nn->dp.rx_offset=reg;-}else{-nn->dp.rx_offset=NFP_NET_RX_OFFSET;-}--/* Set default MTU and Freelist buffer size */-if(nn->max_mtu<NFP_NET_DEFAULT_MTU)-netdev->mtu=nn->max_mtu;-else-netdev->mtu=NFP_NET_DEFAULT_MTU;-nn->dp.mtu=netdev->mtu;-nn->dp.fl_bufsz=nfp_net_calc_fl_bufsz(&nn->dp);+netdev->mtu=nn->dp.mtu;/* Advertise/enable offloads based on capabilities*
@@ -3237,12 +3210,8 @@ int nfp_net_init(struct nfp_net *nn)nn->dp.ctrl|=nn->cap&NFP_NET_CFG_CTRL_LSO2?:NFP_NET_CFG_CTRL_LSO;}-if(nn->cap&NFP_NET_CFG_CTRL_RSS_ANY){+if(nn->cap&NFP_NET_CFG_CTRL_RSS_ANY)netdev->hw_features|=NETIF_F_RXHASH;-nfp_net_rss_init(nn);-nn->dp.ctrl|=nn->cap&NFP_NET_CFG_CTRL_RSS2?:-NFP_NET_CFG_CTRL_RSS;-}if(nn->cap&NFP_NET_CFG_CTRL_VXLAN&&nn->cap&NFP_NET_CFG_CTRL_NVGRE){if(nn->cap&NFP_NET_CFG_CTRL_LSO)
@@ -3277,6 +3246,68 @@ int nfp_net_init(struct nfp_net *nn)netdev->features&=~(NETIF_F_TSO|NETIF_F_TSO6);nn->dp.ctrl&=~NFP_NET_CFG_CTRL_LSO_ANY;+/* Finalise the netdev setup */+netdev->netdev_ops=&nfp_net_netdev_ops;+netdev->watchdog_timeo=msecs_to_jiffies(5*1000);++/* MTU range: 68 - hw-specific max */+netdev->min_mtu=ETH_MIN_MTU;+netdev->max_mtu=nn->max_mtu;++netif_carrier_off(netdev);++nfp_net_set_ethtool_ops(netdev);+}++/**+*nfp_net_init()-Initialise/finalisethenfp_netstructure+*@nn:NFPNetdevicestructure+*+*Return:0onsuccessornegativeerrnoonerror.+*/+intnfp_net_init(structnfp_net*nn)+{+interr;++nn->dp.rx_dma_dir=DMA_FROM_DEVICE;++/* Get some of the read-only fields from the BAR */+nn->cap=nn_readl(nn,NFP_NET_CFG_CAP);+nn->max_mtu=nn_readl(nn,NFP_NET_CFG_MAX_MTU);++/* Chained metadata is signalled by capabilities except in version 4 */+nn->dp.chained_metadata_format=nn->fw_ver.major==4||+nn->cap&NFP_NET_CFG_CTRL_CHAIN_META;+if(nn->dp.chained_metadata_format&&nn->fw_ver.major!=4)+nn->cap&=~NFP_NET_CFG_CTRL_RSS;++/* Determine RX packet/metadata boundary offset */+if(nn->fw_ver.major>=2){+u32reg;++reg=nn_readl(nn,NFP_NET_CFG_RX_OFFSET);+if(reg>NFP_NET_MAX_PREPEND){+nn_err(nn,"Invalid rx offset: %d\n",reg);+return-EINVAL;+}+nn->dp.rx_offset=reg;+}else{+nn->dp.rx_offset=NFP_NET_RX_OFFSET;+}++/* Set default MTU and Freelist buffer size */+if(nn->max_mtu<NFP_NET_DEFAULT_MTU)+nn->dp.mtu=nn->max_mtu;+else+nn->dp.mtu=NFP_NET_DEFAULT_MTU;+nn->dp.fl_bufsz=nfp_net_calc_fl_bufsz(&nn->dp);++if(nn->cap&NFP_NET_CFG_CTRL_RSS_ANY){+nfp_net_rss_init(nn);+nn->dp.ctrl|=nn->cap&NFP_NET_CFG_CTRL_RSS2?:+NFP_NET_CFG_CTRL_RSS;+}+/* Allow L2 Broadcast and Multicast through by default, if supported */if(nn->cap&NFP_NET_CFG_CTRL_L2BC)nn->dp.ctrl|=NFP_NET_CFG_CTRL_L2BC;
@@ -3289,6 +3320,9 @@ int nfp_net_init(struct nfp_net *nn)nn->dp.ctrl|=NFP_NET_CFG_CTRL_IRQMOD;}+if(nn->dp.netdev)+nfp_net_netdev_init(nn);+/* Stash the re-configuration queue away. First odd queue in TX Bar */nn->qcp_cfg=nn->tx_bar+NFP_QCP_QUEUE_ADDR_SZ;
@@ -3301,20 +3335,11 @@ int nfp_net_init(struct nfp_net *nn)if(err)returnerr;-/* Finalise the netdev setup */-netdev->netdev_ops=&nfp_net_netdev_ops;-netdev->watchdog_timeo=msecs_to_jiffies(5*1000);--/* MTU range: 68 - hw-specific max */-netdev->min_mtu=ETH_MIN_MTU;-netdev->max_mtu=nn->max_mtu;--netif_carrier_off(netdev);--nfp_net_set_ethtool_ops(netdev);nfp_net_vecs_init(nn);-returnregister_netdev(netdev);+if(!nn->dp.netdev)+return0;+returnregister_netdev(nn->dp.netdev);}/**
@@ -3323,6 +3348,9 @@ int nfp_net_init(struct nfp_net *nn)*/voidnfp_net_clean(structnfp_net*nn){+if(!nn->dp.netdev)+return;+unregister_netdev(nn->dp.netdev);if(nn->dp.xdp_prog)
@@ -202,7 +202,7 @@ static int nfp_netvf_pci_probe(struct pci_dev *pdev,rx_bar_off=NFP_PCIE_QUEUE(startq);/* Allocate and initialise the netdev */-nn=nfp_net_alloc(pdev,max_tx_rings,max_rx_rings);+nn=nfp_net_alloc(pdev,true,max_tx_rings,max_rx_rings);if(IS_ERR(nn)){err=PTR_ERR(nn);gotoerr_ctrl_unmap;
From: Jakub Kicinski <hidden> Date: 2017-06-06 00:02:24
To be able to reuse print macros easily with control vNICs make the
macros check if netdev pointer is populated and use dev_* print
functions otherwise.
Signed-off-by: Jakub Kicinski <redacted>
---
drivers/net/ethernet/netronome/nfp/nfp_net.h | 33 +++++++++++++++++++++-------
1 file changed, 25 insertions(+), 8 deletions(-)
From: Jakub Kicinski <hidden> Date: 2017-06-06 00:02:25
RX and TX queue controllers are interleaved. Instead of creating
two mappings which map the same area at slightly different offset,
create only one mapping. Always map all queue controllers to simplify
the code and allow reusing the mapping for non-data vNICs.
Signed-off-by: Jakub Kicinski <redacted>
---
drivers/net/ethernet/netronome/nfp/nfp_main.h | 6 +-
drivers/net/ethernet/netronome/nfp/nfp_net.h | 1 +
drivers/net/ethernet/netronome/nfp/nfp_net_main.c | 122 +++++-----------------
3 files changed, 28 insertions(+), 101 deletions(-)
@@ -223,31 +223,6 @@ static int nfp_net_pf_get_app_id(struct nfp_pf *pf)NFP_APP_CORE_NIC);}-staticunsignedint-nfp_net_pf_total_qcs(structnfp_pf*pf,void__iomem*ctrl_bar,-unsignedintstride,u32start_off,u32num_off)-{-unsignedinti,min_qc,max_qc;--min_qc=readl(ctrl_bar+start_off);-max_qc=min_qc;--for(i=0;i<pf->max_data_vnics;i++){-/* To make our lives simpler only accept configuration where-*queuesareallocatedtoPFsinorder(queuesofPFnallhave-*indexeslowerthanPFn+1).-*/-if(max_qc>readl(ctrl_bar+start_off))-return0;--max_qc=readl(ctrl_bar+start_off);-max_qc+=readl(ctrl_bar+num_off)*stride;-ctrl_bar+=NFP_PF_CSR_SLICE_SIZE;-}--returnmax_qc-min_qc;-}-staticu8__iomem*nfp_net_pf_map_rtsym(structnfp_pf*pf,constchar*name,constchar*sym_fmt,unsignedintmin_size,structnfp_cpp_area**area)
@@ -439,8 +403,7 @@ nfp_net_pf_spawn_vnics(struct nfp_pf *pf,interr;/* Allocate the vnics and do basic init */-err=nfp_net_pf_alloc_vnics(pf,ctrl_bar,tx_bar,rx_bar,-stride,fw_ver);+err=nfp_net_pf_alloc_vnics(pf,ctrl_bar,qc_bar,stride,fw_ver);if(err)returnerr;
From: Jakub Kicinski <hidden> Date: 2017-06-06 00:02:26
Reading fw version from the BAR is trivial. Don't pass it around
through layers of init functions, simply read it again where needed.
This commit has the side effect of each vNIC having the exact NFD
version from its own control memory, rather than all data vNICs
assuming the version of the first one. This should not result in
user-visible changes, though. Capabilities of data vNICs of trival
apps are identical.
Signed-off-by: Jakub Kicinski <redacted>
---
drivers/net/ethernet/netronome/nfp/nfp_net_main.c | 17 +++++++----------
1 file changed, 7 insertions(+), 10 deletions(-)
From: Jakub Kicinski <hidden> Date: 2017-06-06 00:02:27
We want to be able to create a special vNIC for control messages.
This vNIC should be created before any netdev is registered to allow
nfp_app logic to exchange messages with the FW app before any netdev
is visible to user space. Unfortunately we can't enable IRQs until
we know how many vNICs we will need to spawn.
Divide the function which spawns netdevs for vNICs into three parts:
- vNIC/memory allocation;
- IRQ allocation;
- netdev init and register.
This will help us insert the initialization of the control channel
after IRQ allocation but before netdev init and register.
Signed-off-by: Jakub Kicinski <redacted>
---
drivers/net/ethernet/netronome/nfp/nfp_net_main.c | 62 ++++++++++++++---------
1 file changed, 37 insertions(+), 25 deletions(-)
@@ -691,14 +690,27 @@ int nfp_net_pci_probe(struct nfp_pf *pf)pf->ddir=nfp_net_debugfs_device_add(pf->pdev);-err=nfp_net_pf_spawn_vnics(pf,ctrl_bar,qc_bar,stride);+/* Allocate the vnics and do basic init */+err=nfp_net_pf_alloc_vnics(pf,ctrl_bar,qc_bar,stride);if(err)gotoerr_clean_ddir;+err=nfp_net_pf_alloc_irqs(pf);+if(err)+gotoerr_free_vnics;++err=nfp_net_pf_init_vnics(pf);+if(err)+gotoerr_free_irqs;+mutex_unlock(&pf->lock);return0;+err_free_irqs:+nfp_net_pf_free_irqs(pf);+err_free_vnics:+nfp_net_pf_free_vnics(pf);err_clean_ddir:nfp_net_debugfs_dir_clean(&pf->ddir);nfp_net_pf_app_clean(pf);
From: Jakub Kicinski <hidden> Date: 2017-06-06 00:02:28
Thus far the code assumed all vNICs will request similar number of IRQs.
This will be no longer true with control vNICs (where 1 IRQ will suffice).
Signed-off-by: Jakub Kicinski <redacted>
---
drivers/net/ethernet/netronome/nfp/nfp_net_main.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
From: Jakub Kicinski <hidden> Date: 2017-06-06 00:02:28
Since control vNICs don't have a netdev, they can't use napi and
queuing stack provides. Add simple tasklet-based data receive
and send of control messages with queuing on a skb_list.
Signed-off-by: Jakub Kicinski <redacted>
---
drivers/net/ethernet/netronome/nfp/nfp_app.h | 11 +
drivers/net/ethernet/netronome/nfp/nfp_net.h | 17 +-
.../net/ethernet/netronome/nfp/nfp_net_common.c | 323 ++++++++++++++++++++-
drivers/net/ethernet/netronome/nfp/nfp_net_ctrl.h | 3 +
4 files changed, 345 insertions(+), 9 deletions(-)
@@ -1742,6 +1754,231 @@ static int nfp_net_poll(struct napi_struct *napi, int budget)returnpkts_polled;}+/* Control device data path+*/++staticbool+nfp_ctrl_tx_one(structnfp_net*nn,structnfp_net_r_vector*r_vec,+structsk_buff*skb,boolold)+{+unsignedintreal_len=skb->len,meta_len=0;+structnfp_net_tx_ring*tx_ring;+structnfp_net_tx_buf*txbuf;+structnfp_net_tx_desc*txd;+structnfp_net_dp*dp;+dma_addr_tdma_addr;+intwr_idx;++dp=&r_vec->nfp_net->dp;+tx_ring=r_vec->tx_ring;++if(WARN_ON_ONCE(skb_shinfo(skb)->nr_frags)){+nn_dp_warn(dp,"Driver's CTRL TX does not implement gather\n");+gotoerr_free;+}++if(unlikely(nfp_net_tx_full(tx_ring,1))){+u64_stats_update_begin(&r_vec->tx_sync);+r_vec->tx_busy++;+u64_stats_update_end(&r_vec->tx_sync);+if(!old)+__skb_queue_tail(&r_vec->queue,skb);+else+__skb_queue_head(&r_vec->queue,skb);+returntrue;+}++if(nfp_app_ctrl_has_meta(nn->app)){+if(unlikely(skb_headroom(skb)<8)){+nn_dp_warn(dp,"CTRL TX on skb without headroom\n");+gotoerr_free;+}+meta_len=8;+put_unaligned_be32(NFP_META_PORT_ID_CTRL,skb_push(skb,4));+put_unaligned_be32(NFP_NET_META_PORTID,skb_push(skb,4));+}++/* Start with the head skbuf */+dma_addr=dma_map_single(dp->dev,skb->data,skb_headlen(skb),+DMA_TO_DEVICE);+if(dma_mapping_error(dp->dev,dma_addr))+gotoerr_dma_warn;++wr_idx=D_IDX(tx_ring,tx_ring->wr_p);++/* Stash the soft descriptor of the head then initialize it */+txbuf=&tx_ring->txbufs[wr_idx];+txbuf->skb=skb;+txbuf->dma_addr=dma_addr;+txbuf->fidx=-1;+txbuf->pkt_cnt=1;+txbuf->real_len=real_len;++/* Build TX descriptor */+txd=&tx_ring->txds[wr_idx];+txd->offset_eop=meta_len|PCIE_DESC_TX_EOP;+txd->dma_len=cpu_to_le16(skb_headlen(skb));+nfp_desc_set_dma_addr(txd,dma_addr);+txd->data_len=cpu_to_le16(skb->len);++txd->flags=0;+txd->mss=0;+txd->lso_hdrlen=0;++tx_ring->wr_p++;+tx_ring->wr_ptr_add++;+nfp_net_tx_xmit_more_flush(tx_ring);++returnfalse;++err_dma_warn:+nn_dp_warn(dp,"Failed to DMA map TX CTRL buffer\n");+err_free:+u64_stats_update_begin(&r_vec->tx_sync);+r_vec->tx_errors++;+u64_stats_update_end(&r_vec->tx_sync);+dev_kfree_skb_any(skb);+returnfalse;+}++boolnfp_ctrl_tx(structnfp_net*nn,structsk_buff*skb)+{+structnfp_net_r_vector*r_vec=&nn->r_vecs[0];+boolret;++spin_lock_bh(&r_vec->lock);+ret=nfp_ctrl_tx_one(nn,r_vec,skb,false);+spin_unlock_bh(&r_vec->lock);++returnret;+}++staticvoid__nfp_ctrl_tx_queued(structnfp_net_r_vector*r_vec)+{+structsk_buff*skb;++while((skb=__skb_dequeue(&r_vec->queue)))+if(nfp_ctrl_tx_one(r_vec->nfp_net,r_vec,skb,true))+return;+}++staticbool+nfp_ctrl_meta_ok(structnfp_net*nn,void*data,unsignedintmeta_len)+{+u32meta_type,meta_tag;++if(!nfp_app_ctrl_has_meta(nn->app))+return!meta_len;++if(meta_len!=8)+returnfalse;++meta_type=get_unaligned_be32(data);+meta_tag=get_unaligned_be32(data+4);++return(meta_type==NFP_NET_META_PORTID&&+meta_tag==NFP_META_PORT_ID_CTRL);+}++staticbool+nfp_ctrl_rx_one(structnfp_net*nn,structnfp_net_dp*dp,+structnfp_net_r_vector*r_vec,structnfp_net_rx_ring*rx_ring)+{+unsignedintmeta_len,data_len,meta_off,pkt_len,pkt_off;+structnfp_net_rx_buf*rxbuf;+structnfp_net_rx_desc*rxd;+dma_addr_tnew_dma_addr;+structsk_buff*skb;+void*new_frag;+intidx;++idx=D_IDX(rx_ring,rx_ring->rd_p);++rxd=&rx_ring->rxds[idx];+if(!(rxd->rxd.meta_len_dd&PCIE_DESC_RX_DD))+returnfalse;++/* Memory barrier to ensure that we won't do other reads+*beforetheDDbit.+*/+dma_rmb();++rx_ring->rd_p++;++rxbuf=&rx_ring->rxbufs[idx];+meta_len=rxd->rxd.meta_len_dd&PCIE_DESC_RX_META_LEN_MASK;+data_len=le16_to_cpu(rxd->rxd.data_len);+pkt_len=data_len-meta_len;++pkt_off=NFP_NET_RX_BUF_HEADROOM+dp->rx_dma_off;+if(dp->rx_offset==NFP_NET_CFG_RX_OFFSET_DYNAMIC)+pkt_off+=meta_len;+else+pkt_off+=dp->rx_offset;+meta_off=pkt_off-meta_len;++/* Stats update */+u64_stats_update_begin(&r_vec->rx_sync);+r_vec->rx_pkts++;+r_vec->rx_bytes+=pkt_len;+u64_stats_update_end(&r_vec->rx_sync);++nfp_net_dma_sync_cpu_rx(dp,rxbuf->dma_addr+meta_off,data_len);++if(unlikely(!nfp_ctrl_meta_ok(nn,rxbuf->frag+meta_off,meta_len))){+nn_dp_warn(dp,"incorrect metadata for ctrl packet (%d)\n",+meta_len);+nfp_net_rx_drop(dp,r_vec,rx_ring,rxbuf,NULL);+returntrue;+}++skb=build_skb(rxbuf->frag,dp->fl_bufsz);+if(unlikely(!skb)){+nfp_net_rx_drop(dp,r_vec,rx_ring,rxbuf,NULL);+returntrue;+}+new_frag=nfp_net_napi_alloc_one(dp,&new_dma_addr);+if(unlikely(!new_frag)){+nfp_net_rx_drop(dp,r_vec,rx_ring,rxbuf,skb);+returntrue;+}++nfp_net_dma_unmap_rx(dp,rxbuf->dma_addr);++nfp_net_rx_give_one(dp,rx_ring,new_frag,new_dma_addr);++skb_reserve(skb,pkt_off);+skb_put(skb,pkt_len);++dev_kfree_skb_any(skb);++returntrue;+}++staticvoidnfp_ctrl_rx(structnfp_net_r_vector*r_vec)+{+structnfp_net_rx_ring*rx_ring=r_vec->rx_ring;+structnfp_net*nn=r_vec->nfp_net;+structnfp_net_dp*dp=&nn->dp;++while(nfp_ctrl_rx_one(nn,dp,r_vec,rx_ring))+continue;+}++staticvoidnfp_ctrl_poll(unsignedlongarg)+{+structnfp_net_r_vector*r_vec=(void*)arg;++spin_lock_bh(&r_vec->lock);+nfp_net_tx_complete(r_vec->tx_ring);+__nfp_ctrl_tx_queued(r_vec);+spin_unlock_bh(&r_vec->lock);++nfp_ctrl_rx(r_vec);++nfp_net_irq_unmask(r_vec->nfp_net,r_vec->irq_entry);+}+/* Setup and Configuration*/
@@ -2338,6 +2597,24 @@ static int nfp_net_netdev_close(struct net_device *netdev)return0;}+voidnfp_ctrl_close(structnfp_net*nn)+{+intr;++rtnl_lock();++for(r=0;r<nn->dp.num_r_vecs;r++){+disable_irq(nn->r_vecs[r].irq_vector);+tasklet_disable(&nn->r_vecs[r].tasklet);+}++nfp_net_clear_config_and_disable(nn);++nfp_net_close_free_all(nn);++rtnl_unlock();+}+/***nfp_net_open_stack()-Startthedevicefromstack'sperspective*@nn:NFPNetdevicetoreconfigure
@@ -2453,6 +2730,35 @@ static int nfp_net_netdev_open(struct net_device *netdev)returnerr;}+intnfp_ctrl_open(structnfp_net*nn)+{+interr,r;++/* ring dumping depends on vNICs being opened/closed under rtnl */+rtnl_lock();++err=nfp_net_open_alloc_all(nn);+if(err)+gotoerr_unlock;++err=nfp_net_set_config_and_enable(nn);+if(err)+gotoerr_free_all;++for(r=0;r<nn->dp.num_r_vecs;r++)+enable_irq(nn->r_vecs[r].irq_vector);++rtnl_unlock();++return0;++err_free_all:+nfp_net_close_free_all(nn);+err_unlock:+rtnl_unlock();+returnerr;+}+staticvoidnfp_net_set_rx_mode(structnet_device*netdev){structnfp_net*nn=netdev_priv(netdev);
@@ -3278,6 +3584,7 @@ int nfp_net_init(struct nfp_net *nn)/* Chained metadata is signalled by capabilities except in version 4 */nn->dp.chained_metadata_format=nn->fw_ver.major==4||+!nn->dp.netdev||nn->cap&NFP_NET_CFG_CTRL_CHAIN_META;if(nn->dp.chained_metadata_format&&nn->fw_ver.major!=4)nn->cap&=~NFP_NET_CFG_CTRL_RSS;
From: Jakub Kicinski <hidden> Date: 2017-06-06 00:02:29
When driver encounters an nfp_app which has a control message handler
defined, allocate a control vNIC. This control channel will be used
to exchange data with the application FW such as flow table programming,
statistics and global datapath control.
Signed-off-by: Jakub Kicinski <redacted>
---
drivers/net/ethernet/netronome/nfp/nfp_app.c | 18 +++
drivers/net/ethernet/netronome/nfp/nfp_app.h | 44 ++++++++
drivers/net/ethernet/netronome/nfp/nfp_main.h | 7 ++
.../net/ethernet/netronome/nfp/nfp_net_common.c | 2 +-
drivers/net/ethernet/netronome/nfp/nfp_net_main.c | 121 ++++++++++++++++++---
5 files changed, 177 insertions(+), 15 deletions(-)
From: Jakub Kicinski <hidden> Date: 2017-06-06 00:02:34
We will soon need to map control vNIC PCI memory as well as data vNIC
memory. Make the function for mapping areas pointed to by an RTsym
reusable.
Signed-off-by: Jakub Kicinski <redacted>
---
drivers/net/ethernet/netronome/nfp/nfp_net_main.c | 55 ++++++++++++-----------
1 file changed, 28 insertions(+), 27 deletions(-)
@@ -248,40 +248,37 @@ nfp_net_pf_total_qcs(struct nfp_pf *pf, void __iomem *ctrl_bar,returnmax_qc-min_qc;}-staticu8__iomem*nfp_net_pf_map_ctrl_bar(structnfp_pf*pf)+staticu8__iomem*+nfp_net_pf_map_rtsym(structnfp_pf*pf,constchar*name,constchar*sym_fmt,+unsignedintmin_size,structnfp_cpp_area**area){-conststructnfp_rtsym*ctrl_sym;-u8__iomem*ctrl_bar;+conststructnfp_rtsym*sym;charpf_symbol[256];+u8__iomem*mem;-snprintf(pf_symbol,sizeof(pf_symbol),"_pf%u_net_bar0",+snprintf(pf_symbol,sizeof(pf_symbol),sym_fmt,nfp_cppcore_pcie_unit(pf->cpp));-ctrl_sym=nfp_rtsym_lookup(pf->cpp,pf_symbol);-if(!ctrl_sym){-dev_err(&pf->pdev->dev,-"Failed to find PF BAR0 symbol %s\n",pf_symbol);-returnNULL;+sym=nfp_rtsym_lookup(pf->cpp,pf_symbol);+if(!sym){+nfp_err(pf->cpp,"Failed to find PF symbol %s\n",pf_symbol);+return(u8__iomem*)ERR_PTR(-ENOENT);}-if(ctrl_sym->size<pf->max_data_vnics*NFP_PF_CSR_SLICE_SIZE){-dev_err(&pf->pdev->dev,-"PF BAR0 too small to contain %d vNICs\n",-pf->max_data_vnics);-returnNULL;+if(sym->size<min_size){+nfp_err(pf->cpp,"PF symbol %s too small\n",pf_symbol);+return(u8__iomem*)ERR_PTR(-EINVAL);}-ctrl_bar=nfp_net_map_area(pf->cpp,"net.ctrl",-ctrl_sym->domain,ctrl_sym->target,-ctrl_sym->addr,ctrl_sym->size,-&pf->data_vnic_bar);-if(IS_ERR(ctrl_bar)){-dev_err(&pf->pdev->dev,"Failed to map PF BAR0: %ld\n",-PTR_ERR(ctrl_bar));-returnNULL;+mem=nfp_net_map_area(pf->cpp,name,sym->domain,sym->target,+sym->addr,sym->size,area);+if(IS_ERR(mem)){+nfp_err(pf->cpp,"Failed to map PF symbol %s: %ld\n",+pf_symbol,PTR_ERR(mem));+returnmem;}-returnctrl_bar;+returnmem;}staticvoidnfp_net_pf_free_vnic(structnfp_pf*pf,structnfp_net*nn)
@@ -662,10 +659,10 @@ int nfp_net_refresh_eth_port(struct nfp_port *port)*/intnfp_net_pci_probe(structnfp_pf*pf){+u32ctrl_bar_sz,tx_area_sz,rx_area_sz;u8__iomem*ctrl_bar,*tx_bar,*rx_bar;u32total_tx_qcs,total_rx_qcs;structnfp_net_fw_versionfw_ver;-u32tx_area_sz,rx_area_sz;u32start_q;intstride;interr;
@@ -685,9 +682,13 @@ int nfp_net_pci_probe(struct nfp_pf *pf)gotoerr_unlock;}-ctrl_bar=nfp_net_pf_map_ctrl_bar(pf);-if(!ctrl_bar){-err=pf->fw_loaded?-EINVAL:-EPROBE_DEFER;+ctrl_bar_sz=pf->max_data_vnics*NFP_PF_CSR_SLICE_SIZE;+ctrl_bar=nfp_net_pf_map_rtsym(pf,"net.ctrl","_pf%d_net_bar0",+ctrl_bar_sz,&pf->data_vnic_bar);+if(IS_ERR(ctrl_bar)){+err=PTR_ERR(ctrl_bar);+if(!pf->fw_loaded&&err==-ENOENT)+err=-EPROBE_DEFER;gotoerr_unlock;}
From: Jakub Kicinski <hidden> Date: 2017-06-06 00:02:39
NFD ABI 0.5 is equivalent to NFD ABI 3.0 but requires that the
driver checks the APP id symbol and makes sure it can support
given app. Most advanced apps will likely require control vNIC
(ability to exchange control messages between the driver and
app FW). Detailed app version checking and capability exchange
is left to app-specific code.
Signed-off-by: Jakub Kicinski <redacted>
---
drivers/net/ethernet/netronome/nfp/nfp_net_main.c | 2 +-
drivers/net/ethernet/netronome/nfp/nfp_netvf_main.c | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
Tue, Jun 06, 2017 at 02:01:41AM CEST, jakub.kicinski@netronome.com wrote:
Hi!
This series adds the ability to use one vNIC as a control channel
for passing messages to and from the application firmware. The
implementation restructures the existing netdev vNIC code to be able
to deal with nfp_nets with netdev pointer set to NULL. Control vNICs
are not visible to userspace (other than for dumping ring state), and
since they don't have netdevs we use a tasklet for RX and simple skb
list for TX queuing.
Due to special status of the control vNIC we have to reshuffle the
init code a bit to make sure control vNIC will be fully brought up
(and therefore communication with app FW can happen) before any netdev
or port is visible to user space.
FW will designate which vNIC is supposed to be used as control one
by setting _pf%u_net_ctrl_bar symbol. Some FWs depend on metadata
being prepended to control message, some prefer to look at queue ID
to decide that something is a control message. Our implementation
can cater to both.
First two users of this code will be eBPF maps and flower offloads.
How do you actually do the configuration from the userspace? I did not
find it in the patches.
I'm not really sure that doing it using one "control netdevice" is the
correct way to go. The configuration is asic-wide, should be done by a
devlink parent handle which was introduced for that exact purpose.
Am I missing something? We need to sync in this. In mlxsw we need to do
some pre-netdev configuraton as well.
Thanks.
From: Jakub Kicinski <hidden> Date: 2017-06-06 07:21:55
On Tue, 6 Jun 2017 08:16:10 +0200, Jiri Pirko wrote:
Tue, Jun 06, 2017 at 02:01:41AM CEST, jakub.kicinski@netronome.com wrote:
quoted
Hi!
This series adds the ability to use one vNIC as a control channel
for passing messages to and from the application firmware. The
implementation restructures the existing netdev vNIC code to be able
to deal with nfp_nets with netdev pointer set to NULL. Control vNICs
are not visible to userspace (other than for dumping ring state), and
since they don't have netdevs we use a tasklet for RX and simple skb
list for TX queuing.
Due to special status of the control vNIC we have to reshuffle the
init code a bit to make sure control vNIC will be fully brought up
(and therefore communication with app FW can happen) before any netdev
or port is visible to user space.
FW will designate which vNIC is supposed to be used as control one
by setting _pf%u_net_ctrl_bar symbol. Some FWs depend on metadata
being prepended to control message, some prefer to look at queue ID
to decide that something is a control message. Our implementation
can cater to both.
First two users of this code will be eBPF maps and flower offloads.
How do you actually do the configuration from the userspace? I did not
find it in the patches.
Yes, there is nothing interesting in those patches, really. It's all
internal to the driver. This set basically allows us to dedicate some
queue pairs to high-speed communication with the FW (sending commands,
populating/dumping eBPF and Flower tables).
The eBPF maps and Flower offload patches should follow in coming weeks.
I'm not really sure that doing it using one "control netdevice" is the
correct way to go.
The control queues are not exposed to user space at all. It's up to
the driver logic to generate and consume all messages passing on those
queues. I had to make this a separate series because of the amount
of code churn. I'm also not a fan of exposing control netdevices.
Reasons ranging from NetworkManager latching onto them (even if it's
configured not to) to someone inevitably trying to control the device
from user space with vendor commands and make the driver-kept state go
out of whack :/
The configuration is asic-wide, should be done by a devlink parent
handle which was introduced for that exact purpose.
Am I missing something? We need to sync in this. In mlxsw we need to do
some pre-netdev configuraton as well.
For programmable NICs we still need to come up with some API for setting
the target application/firmware name/adapter mode, but I don't think we
have any more fine-grained parameters to set per-device just yet.
I was thinking of either devlink, or perhaps, since I'm hearing people
want those to be persistent/written into flash, the recent MTD
discussion got me wondering if we should just expose the entire flash
and teach ethtool to modify the vendor-specific parameter table
directly in the flash. That seems like something that could quickly
get out of hand, though :S
What were your plans with pre-netdev config?
I hope this makes things slightly clearer, sorry for not doing a good
job on the cover letter :)
Tue, Jun 06, 2017 at 09:21:45AM CEST, kubakici@wp.pl wrote:
On Tue, 6 Jun 2017 08:16:10 +0200, Jiri Pirko wrote:
quoted
Tue, Jun 06, 2017 at 02:01:41AM CEST, jakub.kicinski@netronome.com wrote:
quoted
Hi!
This series adds the ability to use one vNIC as a control channel
for passing messages to and from the application firmware. The
implementation restructures the existing netdev vNIC code to be able
to deal with nfp_nets with netdev pointer set to NULL. Control vNICs
are not visible to userspace (other than for dumping ring state), and
since they don't have netdevs we use a tasklet for RX and simple skb
list for TX queuing.
Due to special status of the control vNIC we have to reshuffle the
init code a bit to make sure control vNIC will be fully brought up
(and therefore communication with app FW can happen) before any netdev
or port is visible to user space.
FW will designate which vNIC is supposed to be used as control one
by setting _pf%u_net_ctrl_bar symbol. Some FWs depend on metadata
being prepended to control message, some prefer to look at queue ID
to decide that something is a control message. Our implementation
can cater to both.
First two users of this code will be eBPF maps and flower offloads.
How do you actually do the configuration from the userspace? I did not
find it in the patches.
Yes, there is nothing interesting in those patches, really. It's all
internal to the driver. This set basically allows us to dedicate some
queue pairs to high-speed communication with the FW (sending commands,
populating/dumping eBPF and Flower tables).
The eBPF maps and Flower offload patches should follow in coming weeks.
Okay. So you don't let the user to dicide for which purpose (bpf/flowe)
he wants to use the nic, right?
quoted
I'm not really sure that doing it using one "control netdevice" is the
correct way to go.
The control queues are not exposed to user space at all. It's up to
the driver logic to generate and consume all messages passing on those
queues. I had to make this a separate series because of the amount
of code churn. I'm also not a fan of exposing control netdevices.
Reasons ranging from NetworkManager latching onto them (even if it's
configured not to) to someone inevitably trying to control the device
from user space with vendor commands and make the driver-kept state go
out of whack :/
Okay, that got me confused. I thought that you want the control
netdevice. Now that is clear we both don't want to see that, I'm fine :)
quoted
The configuration is asic-wide, should be done by a devlink parent
handle which was introduced for that exact purpose.
Am I missing something? We need to sync in this. In mlxsw we need to do
some pre-netdev configuraton as well.
For programmable NICs we still need to come up with some API for setting
the target application/firmware name/adapter mode, but I don't think we
have any more fine-grained parameters to set per-device just yet.
I was thinking of either devlink, or perhaps, since I'm hearing people
want those to be persistent/written into flash, the recent MTD
discussion got me wondering if we should just expose the entire flash
and teach ethtool to modify the vendor-specific parameter table
directly in the flash. That seems like something that could quickly
get out of hand, though :S
Yeah, I'm not fan of the mtd exposing some fw internals. I believe that
for the config like yours (switching the operation mode of nic), we
should have well defined user api. And since this is per-device, not
per-netdev, devlink should be the vahicle to carry this.
What were your plans with pre-netdev config?
We need to pass come initial resource division. Generally the consensus
is to have these options exposed through devlink, let the user configure
them all and then to have a trigger that would cause driver
re-orchestration according to the new values. The flow would look like
this:
-driver loads with defaults, inits hw and instantiates netdevs
-driver exposes config options via devlink
-user sets up the options
-user pushes the "go" trigger
-upon the trigger command, devlink calls the driver re-init callback
-driver shuts down the current instances, re-initializes hw,
re-instantiates the netdevs
Makes sense?
I hope this makes things slightly clearer, sorry for not doing a good
job on the cover letter :)
From: Jakub Kicinski <hidden> Date: 2017-06-06 09:09:53
On Tue, 6 Jun 2017 10:23:36 +0200, Jiri Pirko wrote:
Tue, Jun 06, 2017 at 09:21:45AM CEST, kubakici@wp.pl wrote:
quoted
On Tue, 6 Jun 2017 08:16:10 +0200, Jiri Pirko wrote:
quoted
Tue, Jun 06, 2017 at 02:01:41AM CEST, jakub.kicinski@netronome.com wrote:
quoted
Hi!
This series adds the ability to use one vNIC as a control channel
for passing messages to and from the application firmware. The
implementation restructures the existing netdev vNIC code to be able
to deal with nfp_nets with netdev pointer set to NULL. Control vNICs
are not visible to userspace (other than for dumping ring state), and
since they don't have netdevs we use a tasklet for RX and simple skb
list for TX queuing.
Due to special status of the control vNIC we have to reshuffle the
init code a bit to make sure control vNIC will be fully brought up
(and therefore communication with app FW can happen) before any netdev
or port is visible to user space.
FW will designate which vNIC is supposed to be used as control one
by setting _pf%u_net_ctrl_bar symbol. Some FWs depend on metadata
being prepended to control message, some prefer to look at queue ID
to decide that something is a control message. Our implementation
can cater to both.
First two users of this code will be eBPF maps and flower offloads.
How do you actually do the configuration from the userspace? I did not
find it in the patches.
Yes, there is nothing interesting in those patches, really. It's all
internal to the driver. This set basically allows us to dedicate some
queue pairs to high-speed communication with the FW (sending commands,
populating/dumping eBPF and Flower tables).
The eBPF maps and Flower offload patches should follow in coming weeks.
Okay. So you don't let the user to dicide for which purpose (bpf/flowe)
he wants to use the nic, right?
Yes, right now we just expect the user to drop the FW file he wants
to run into /lib/firmware/netronome. Driver will interrogate the
firmware once loaded to see which "app" it is.
quoted
quoted
The configuration is asic-wide, should be done by a devlink parent
handle which was introduced for that exact purpose.
Am I missing something? We need to sync in this. In mlxsw we need to do
some pre-netdev configuraton as well.
For programmable NICs we still need to come up with some API for setting
the target application/firmware name/adapter mode, but I don't think we
have any more fine-grained parameters to set per-device just yet.
I was thinking of either devlink, or perhaps, since I'm hearing people
want those to be persistent/written into flash, the recent MTD
discussion got me wondering if we should just expose the entire flash
and teach ethtool to modify the vendor-specific parameter table
directly in the flash. That seems like something that could quickly
get out of hand, though :S
Yeah, I'm not fan of the mtd exposing some fw internals. I believe that
for the config like yours (switching the operation mode of nic), we
should have well defined user api. And since this is per-device, not
per-netdev, devlink should be the vahicle to carry this.
Well defined API would be nice. Perhaps we will just start with
something very simple and then extend it if need be. E.g. I'm hearing
people would like to select not only the app but also the version of
it... Yes, starting simple is probably the way forward.
quoted
What were your plans with pre-netdev config?
We need to pass come initial resource division. Generally the consensus
is to have these options exposed through devlink, let the user configure
them all and then to have a trigger that would cause driver
re-orchestration according to the new values. The flow would look like
this:
-driver loads with defaults, inits hw and instantiates netdevs
-driver exposes config options via devlink
-user sets up the options
-user pushes the "go" trigger
-upon the trigger command, devlink calls the driver re-init callback
-driver shuts down the current instances, re-initializes hw,
re-instantiates the netdevs
Makes sense?
I like the idea of a "go"/apply/reload trigger and extending devlink.
Do you plan on adding a way to persist the settings? I'm concerned NIC
users may want to boot into the right mode once it's set, without
reloads and reconfigs upon boot. Also is there going to be a way to
query the pending/running config? Sounds like we may want to expose
three value sets - persistent/default, running and pending/to be
applied.
Tue, Jun 06, 2017 at 11:09:45AM CEST, kubakici@wp.pl wrote:
On Tue, 6 Jun 2017 10:23:36 +0200, Jiri Pirko wrote:
quoted
Tue, Jun 06, 2017 at 09:21:45AM CEST, kubakici@wp.pl wrote:
quoted
On Tue, 6 Jun 2017 08:16:10 +0200, Jiri Pirko wrote:
quoted
Tue, Jun 06, 2017 at 02:01:41AM CEST, jakub.kicinski@netronome.com wrote:
quoted
Hi!
This series adds the ability to use one vNIC as a control channel
for passing messages to and from the application firmware. The
implementation restructures the existing netdev vNIC code to be able
to deal with nfp_nets with netdev pointer set to NULL. Control vNICs
are not visible to userspace (other than for dumping ring state), and
since they don't have netdevs we use a tasklet for RX and simple skb
list for TX queuing.
Due to special status of the control vNIC we have to reshuffle the
init code a bit to make sure control vNIC will be fully brought up
(and therefore communication with app FW can happen) before any netdev
or port is visible to user space.
FW will designate which vNIC is supposed to be used as control one
by setting _pf%u_net_ctrl_bar symbol. Some FWs depend on metadata
being prepended to control message, some prefer to look at queue ID
to decide that something is a control message. Our implementation
can cater to both.
First two users of this code will be eBPF maps and flower offloads.
How do you actually do the configuration from the userspace? I did not
find it in the patches.
Yes, there is nothing interesting in those patches, really. It's all
internal to the driver. This set basically allows us to dedicate some
queue pairs to high-speed communication with the FW (sending commands,
populating/dumping eBPF and Flower tables).
The eBPF maps and Flower offload patches should follow in coming weeks.
Okay. So you don't let the user to dicide for which purpose (bpf/flowe)
he wants to use the nic, right?
Yes, right now we just expect the user to drop the FW file he wants
to run into /lib/firmware/netronome. Driver will interrogate the
firmware once loaded to see which "app" it is.
quoted
quoted
quoted
The configuration is asic-wide, should be done by a devlink parent
handle which was introduced for that exact purpose.
Am I missing something? We need to sync in this. In mlxsw we need to do
some pre-netdev configuraton as well.
For programmable NICs we still need to come up with some API for setting
the target application/firmware name/adapter mode, but I don't think we
have any more fine-grained parameters to set per-device just yet.
I was thinking of either devlink, or perhaps, since I'm hearing people
want those to be persistent/written into flash, the recent MTD
discussion got me wondering if we should just expose the entire flash
and teach ethtool to modify the vendor-specific parameter table
directly in the flash. That seems like something that could quickly
get out of hand, though :S
Yeah, I'm not fan of the mtd exposing some fw internals. I believe that
for the config like yours (switching the operation mode of nic), we
should have well defined user api. And since this is per-device, not
per-netdev, devlink should be the vahicle to carry this.
Well defined API would be nice. Perhaps we will just start with
something very simple and then extend it if need be. E.g. I'm hearing
people would like to select not only the app but also the version of
it... Yes, starting simple is probably the way forward.
quoted
quoted
What were your plans with pre-netdev config?
We need to pass come initial resource division. Generally the consensus
is to have these options exposed through devlink, let the user configure
them all and then to have a trigger that would cause driver
re-orchestration according to the new values. The flow would look like
this:
-driver loads with defaults, inits hw and instantiates netdevs
-driver exposes config options via devlink
-user sets up the options
-user pushes the "go" trigger
-upon the trigger command, devlink calls the driver re-init callback
-driver shuts down the current instances, re-initializes hw,
re-instantiates the netdevs
Makes sense?
I like the idea of a "go"/apply/reload trigger and extending devlink.
Do you plan on adding a way to persist the settings? I'm concerned NIC
users may want to boot into the right mode once it's set, without
reloads and reconfigs upon boot. Also is there going to be a way to
query the pending/running config? Sounds like we may want to expose
three value sets - persistent/default, running and pending/to be
applied.
I don't think it is a good idea to introduce any kind of configuration
persistency in HW. I believe that user is the master and he has all
needed info. He can store it persistently, but it is up to him.
So basicaly during boot, we need the devlink configuration to happen
early on, before the netdevices get configured. udev? Not sure how
exactly to do this. Have to ask around :)
We need to pass come initial resource division. Generally the
consensus is to have these options exposed through devlink, let the
user configure them all and then to have a trigger that would cause
driver re-orchestration according to the new values. The flow would
look like
this:
-driver loads with defaults, inits hw and instantiates netdevs
-driver exposes config options via devlink -user sets up the options
-user pushes the "go" trigger -upon the trigger command, devlink
calls the driver re-init callback -driver shuts down the current
instances, re-initializes hw, re-instantiates the netdevs
Makes sense?
I like the idea of a "go"/apply/reload trigger and extending devlink.
Do you plan on adding a way to persist the settings? I'm concerned NIC
users may want to boot into the right mode once it's set, without
reloads and reconfigs upon boot. Also is there going to be a way to
query the pending/running config? Sounds like we may want to expose
three value sets - persistent/default, running and pending/to be
applied.
I don't think it is a good idea to introduce any kind of configuration
persistency in HW. I believe that user is the master and he has all needed
info. He can store it persistently, but it is up to him.
So basicaly during boot, we need the devlink configuration to happen early
on, before the netdevices get configured. udev? Not sure how exactly to do
this. Have to ask around :)
Thinking about use cases where we'd want information available at probe time,
it might have been even better to have it separated from the netdevice,
e.g., providing clients with node to configure (generic?) information on top of
their PCI nodes.
Tue, Jun 06, 2017 at 11:35:08AM CEST, Yuval.Mintz@cavium.com wrote:
quoted
quoted
quoted
quoted
What were your plans with pre-netdev config?
We need to pass come initial resource division. Generally the
consensus is to have these options exposed through devlink, let the
user configure them all and then to have a trigger that would cause
driver re-orchestration according to the new values. The flow would
look like
this:
-driver loads with defaults, inits hw and instantiates netdevs
-driver exposes config options via devlink -user sets up the options
-user pushes the "go" trigger -upon the trigger command, devlink
calls the driver re-init callback -driver shuts down the current
instances, re-initializes hw, re-instantiates the netdevs
Makes sense?
I like the idea of a "go"/apply/reload trigger and extending devlink.
Do you plan on adding a way to persist the settings? I'm concerned NIC
users may want to boot into the right mode once it's set, without
reloads and reconfigs upon boot. Also is there going to be a way to
query the pending/running config? Sounds like we may want to expose
three value sets - persistent/default, running and pending/to be
applied.
quoted
I don't think it is a good idea to introduce any kind of configuration
persistency in HW. I believe that user is the master and he has all needed
info. He can store it persistently, but it is up to him.
So basicaly during boot, we need the devlink configuration to happen early
on, before the netdevices get configured. udev? Not sure how exactly to do
this. Have to ask around :)
Thinking about use cases where we'd want information available at probe time,
it might have been even better to have it separated from the netdevice,
e.g., providing clients with node to configure (generic?) information on top of
their PCI nodes.
Yuval, devlink is separated from the netdevices....
From: Jakub Kicinski <hidden> Date: 2017-06-06 19:55:03
On Tue, 6 Jun 2017 11:17:57 +0200, Jiri Pirko wrote:
quoted
quoted
quoted
What were your plans with pre-netdev config?
We need to pass come initial resource division. Generally the consensus
is to have these options exposed through devlink, let the user configure
them all and then to have a trigger that would cause driver
re-orchestration according to the new values. The flow would look like
this:
-driver loads with defaults, inits hw and instantiates netdevs
-driver exposes config options via devlink
-user sets up the options
-user pushes the "go" trigger
-upon the trigger command, devlink calls the driver re-init callback
-driver shuts down the current instances, re-initializes hw,
re-instantiates the netdevs
Makes sense?
I like the idea of a "go"/apply/reload trigger and extending devlink.
Do you plan on adding a way to persist the settings? I'm concerned NIC
users may want to boot into the right mode once it's set, without
reloads and reconfigs upon boot. Also is there going to be a way to
query the pending/running config? Sounds like we may want to expose
three value sets - persistent/default, running and pending/to be
applied.
I don't think it is a good idea to introduce any kind of configuration
persistency in HW. I believe that user is the master and he has all
needed info. He can store it persistently, but it is up to him.
So basicaly during boot, we need the devlink configuration to happen
early on, before the netdevices get configured. udev? Not sure how
exactly to do this. Have to ask around :)
Happy to hear that. Now there is two of us, I'll try again with the
marketing dept :)
From: David Miller <davem@davemloft.net> Date: 2017-06-07 16:52:31
From: Jakub Kicinski <redacted>
Date: Mon, 5 Jun 2017 17:01:41 -0700
This series adds the ability to use one vNIC as a control channel
for passing messages to and from the application firmware. The
implementation restructures the existing netdev vNIC code to be able
to deal with nfp_nets with netdev pointer set to NULL. Control vNICs
are not visible to userspace (other than for dumping ring state), and
since they don't have netdevs we use a tasklet for RX and simple skb
list for TX queuing.
Due to special status of the control vNIC we have to reshuffle the
init code a bit to make sure control vNIC will be fully brought up
(and therefore communication with app FW can happen) before any netdev
or port is visible to user space.
FW will designate which vNIC is supposed to be used as control one
by setting _pf%u_net_ctrl_bar symbol. Some FWs depend on metadata
being prepended to control message, some prefer to look at queue ID
to decide that something is a control message. Our implementation
can cater to both.
First two users of this code will be eBPF maps and flower offloads.
Ok, I read this over and also checked out your discussion with Jiri.
So far this looks OK to me, so series applied.
I look forward to seeing the eBPF maps and flower follow-on stuff.
We need to pass come initial resource division. Generally the
consensus is to have these options exposed through devlink, let
the user configure them all and then to have a trigger that would
cause driver re-orchestration according to the new values. The
flow would look like
this:
-driver loads with defaults, inits hw and instantiates netdevs
-driver exposes config options via devlink -user sets up the
options -user pushes the "go" trigger -upon the trigger command,
devlink calls the driver re-init callback -driver shuts down the
current instances, re-initializes hw, re-instantiates the netdevs
Makes sense?
I like the idea of a "go"/apply/reload trigger and extending devlink.
Do you plan on adding a way to persist the settings? I'm concerned
NIC users may want to boot into the right mode once it's set,
without reloads and reconfigs upon boot. Also is there going to be
a way to query the pending/running config? Sounds like we may want
to expose three value sets - persistent/default, running and
pending/to be applied.
quoted
I don't think it is a good idea to introduce any kind of
configuration persistency in HW. I believe that user is the master
and he has all needed info. He can store it persistently, but it is up to
him.
quoted
quoted
So basicaly during boot, we need the devlink configuration to happen
early on, before the netdevices get configured. udev? Not sure how
exactly to do this. Have to ask around :)
Thinking about use cases where we'd want information available at probe
time, it might have been even better to have it separated from the
netdevice, e.g., providing clients with node to configure (generic?)
information on top of their PCI nodes.
Yuval, devlink is separated from the netdevices....
Separate from the netdevices, yes. But I think it's still a networking facility.
I.e., would it make sense creating devlink nodes for PCI devices?
Anyway, I don't think there's any *strong* need for what I was asking for;
It's simply that I was thinking of qed where there's quite a bit going on
during the pci probe, and thought how re-doing it can be avoided.
Wed, Jun 07, 2017 at 07:48:08PM CEST, Yuval.Mintz@cavium.com wrote:
quoted
quoted
quoted
quoted
quoted
quoted
What were your plans with pre-netdev config?
We need to pass come initial resource division. Generally the
consensus is to have these options exposed through devlink, let
the user configure them all and then to have a trigger that would
cause driver re-orchestration according to the new values. The
flow would look like
this:
-driver loads with defaults, inits hw and instantiates netdevs
-driver exposes config options via devlink -user sets up the
options -user pushes the "go" trigger -upon the trigger command,
devlink calls the driver re-init callback -driver shuts down the
current instances, re-initializes hw, re-instantiates the netdevs
Makes sense?
I like the idea of a "go"/apply/reload trigger and extending devlink.
Do you plan on adding a way to persist the settings? I'm concerned
NIC users may want to boot into the right mode once it's set,
without reloads and reconfigs upon boot. Also is there going to be
a way to query the pending/running config? Sounds like we may want
to expose three value sets - persistent/default, running and
pending/to be applied.
quoted
I don't think it is a good idea to introduce any kind of
configuration persistency in HW. I believe that user is the master
and he has all needed info. He can store it persistently, but it is up to
him.
quoted
quoted
So basicaly during boot, we need the devlink configuration to happen
early on, before the netdevices get configured. udev? Not sure how
exactly to do this. Have to ask around :)
Thinking about use cases where we'd want information available at probe
time, it might have been even better to have it separated from the
netdevice, e.g., providing clients with node to configure (generic?)
information on top of their PCI nodes.
Yuval, devlink is separated from the netdevices....
Separate from the netdevices, yes. But I think it's still a networking facility.
I.e., would it make sense creating devlink nodes for PCI devices?
True that devlink is placed in net/core. But from the very beginning, it
was designed to be multipurpose. So not only for net devices, but for
any device where is is suitable.
Anyway, I don't think there's any *strong* need for what I was asking for;
It's simply that I was thinking of qed where there's quite a bit going on
during the pci probe, and thought how re-doing it can be avoided.
You can (should) register devlink early on way before netdevs.
We need to pass come initial resource division. Generally the
consensus is to have these options exposed through devlink, let
the user configure them all and then to have a trigger that
would cause driver re-orchestration according to the new
values. The flow would look like
this:
-driver loads with defaults, inits hw and instantiates netdevs
-driver exposes config options via devlink -user sets up the
options -user pushes the "go" trigger -upon the trigger
command, devlink calls the driver re-init callback -driver
shuts down the current instances, re-initializes hw,
re-instantiates the netdevs
Makes sense?
I like the idea of a "go"/apply/reload trigger and extending devlink.
Do you plan on adding a way to persist the settings? I'm
concerned NIC users may want to boot into the right mode once
it's set, without reloads and reconfigs upon boot. Also is there
going to be a way to query the pending/running config? Sounds
like we may want to expose three value sets - persistent/default,
running and pending/to be applied.
quoted
I don't think it is a good idea to introduce any kind of
configuration persistency in HW. I believe that user is the master
and he has all needed info. He can store it persistently, but it
is up to
him.
quoted
quoted
So basicaly during boot, we need the devlink configuration to
happen early on, before the netdevices get configured. udev? Not
sure how exactly to do this. Have to ask around :)
Thinking about use cases where we'd want information available at
probe time, it might have been even better to have it separated from
the netdevice, e.g., providing clients with node to configure
(generic?) information on top of their PCI nodes.
Yuval, devlink is separated from the netdevices....
Separate from the netdevices, yes. But I think it's still a networking facility.
I.e., would it make sense creating devlink nodes for PCI devices?
True that devlink is placed in net/core. But from the very beginning, it was
designed to be multipurpose. So not only for net devices, but for any device
where is is suitable.
quoted
Anyway, I don't think there's any *strong* need for what I was asking
for; It's simply that I was thinking of qed where there's quite a bit
going on during the pci probe, and thought how re-doing it can be avoided.
You can (should) register devlink early on way before netdevs.
Given that I can't really register it sooner than the PCI probe,
how would shortly-before or way-before differ?
Assume you have some userspace daemon monitoring devlink and applying
user-provided configuration.
How could it reliably apply the configuration between devlink registration
and netdevice registeration?
The merit of the udev suggestion was that the sync. point was in userspace,
so user could always make sure that devlink rules apply before netdev state changes.
Thu, Jun 08, 2017 at 08:33:11AM CEST, Yuval.Mintz@cavium.com wrote:
quoted
quoted
quoted
quoted
quoted
quoted
quoted
quoted
What were your plans with pre-netdev config?
We need to pass come initial resource division. Generally the
consensus is to have these options exposed through devlink, let
the user configure them all and then to have a trigger that
would cause driver re-orchestration according to the new
values. The flow would look like
this:
-driver loads with defaults, inits hw and instantiates netdevs
-driver exposes config options via devlink -user sets up the
options -user pushes the "go" trigger -upon the trigger
command, devlink calls the driver re-init callback -driver
shuts down the current instances, re-initializes hw,
re-instantiates the netdevs
Makes sense?
I like the idea of a "go"/apply/reload trigger and extending devlink.
Do you plan on adding a way to persist the settings? I'm
concerned NIC users may want to boot into the right mode once
it's set, without reloads and reconfigs upon boot. Also is there
going to be a way to query the pending/running config? Sounds
like we may want to expose three value sets - persistent/default,
running and pending/to be applied.
quoted
I don't think it is a good idea to introduce any kind of
configuration persistency in HW. I believe that user is the master
and he has all needed info. He can store it persistently, but it
is up to
him.
quoted
quoted
So basicaly during boot, we need the devlink configuration to
happen early on, before the netdevices get configured. udev? Not
sure how exactly to do this. Have to ask around :)
Thinking about use cases where we'd want information available at
probe time, it might have been even better to have it separated from
the netdevice, e.g., providing clients with node to configure
(generic?) information on top of their PCI nodes.
Yuval, devlink is separated from the netdevices....
Separate from the netdevices, yes. But I think it's still a networking facility.
I.e., would it make sense creating devlink nodes for PCI devices?
True that devlink is placed in net/core. But from the very beginning, it was
designed to be multipurpose. So not only for net devices, but for any device
where is is suitable.
quoted
Anyway, I don't think there's any *strong* need for what I was asking
for; It's simply that I was thinking of qed where there's quite a bit
going on during the pci probe, and thought how re-doing it can be avoided.
You can (should) register devlink early on way before netdevs.
Given that I can't really register it sooner than the PCI probe,
how would shortly-before or way-before differ?
Yeah. Pci probe is the limit :)
Assume you have some userspace daemon monitoring devlink and applying
user-provided configuration.
How could it reliably apply the configuration between devlink registration
and netdevice registeration?
You cannot assume that. That is why I suggested devlink config and then
a trigger to re-initialize the hw and the netdevices.
The merit of the udev suggestion was that the sync. point was in userspace,
so user could always make sure that devlink rules apply before netdev state changes.