Re: [RFC PATCH v3 0/6] Configuring traffic classes via new hardware offload mechanism in tc/mqprio

On 9/7/2017 11:34 AM, Florian Fainelli wrote:

On 09/07/2017 04:00 AM, Amritha Nambiar wrote:

quoted

The following series introduces a new hardware offload mode in 
tc/mqprio where the TCs, the queue configurations and bandwidth 
rate limits are offloaded to the hardware. The existing mqprio 
framework is extended to configure the queue counts and layout and
 also added support for rate limiting. This is achieved through new
 netlink attributes for the 'mode' option which takes values such
as 'dcb' (default) and 'channel' and a 'shaper' option for QoS 
attributes such as bandwidth rate limits in hw mode 1.

So "dcb" defines a default priorities to queue mapping?

In the default offload implementation, only the basic hw offload was
supported factoring only the 'number of TCs' values. The rest of the
queue configuration was being ignored. This is the legacy behavior with
hw mode set to 1.
Example:
# tc qdisc add dev eth0 root mqprio num_tc 4  map 0 0 0 0 1 1 1 1 queues
4@0 4@4 hw 1
I just named this default behavior to 'dcb' while introducing a new
offload mechanism.

quoted

Legacy devices can fall back to the existing setup supporting hw 
mode 1 without these additional options where only the TCs are 
offloaded and then the 'mode' and 'shaper' options defaults to DCB
 support.

That's the last part that confuses me, see below.

As I introduced new options for 'mode' and 'shaper', I set the defaults
for these options to 'dcb' so existing offloaders can continue to work
without supporting these new options. Patch 1 has a detailed description
on how this is done.

quoted

The i40e driver enables the new mqprio hardware offload mechanism 
factoring the TCs, queue configuration and bandwidth rates by 
creating HW channel VSIs.

I am really confused by what you call hw_mode 1, as I understand it 
there are really 3 different modes:

There are actually 2 modes now with 'hw' option set to 1, legacy/dcb and
channel.

- legacy: you don't define any traffic class mapping, but you can 
still chain this scheduler with a match + action (like what 
Documentation/networking/multiqueue.txt) you can optionally also add 
"shaper" arguments, but there should not be any default DCB queue 
mapping either?

- dcb: a default mapping for traffic classes to queues is defined, 
optional "shaper" arguments

The legacy mode now becomes the dcb mode. In this mode, although the TC
values, the queue configurations, prio-tc-mapping are all offloaded to
the device, the existing implementation in current drivers support only
a basic hw offload factoring only the TC values.

Examples:
# ... num_tc 2  map 0 0 0 0 1 1 1 1 queues 4@0 4@4 hw 1
# ... num_tc 2  map 0 0 0 0 1 1 1 1 queues 4@0 4@4 hw 1 mode dcb
# ... num_tc 2  map 0 0 0 0 1 1 1 1 queues 4@0 4@4 hw 1 mode dcb\
  shaper dcb

- channel: (maybe calling that "custom_tc_map" would be clearer?) 
where you express the exact traffic classes to queue mapping and 
optional "shaper" arguments

In the channel mode, a full hw offload is supported, the TC values, the
queue configurations and additionally QoS attributes (optional) are all
used in the new implementation.

Examples:
# ... num_tc 2  map 0 0 0 0 1 1 1 1 queues 4@0 4@4 hw 1 mode channel
# ... num_tc 2  map 0 0 0 0 1 1 1 1 queues 4@0 4@4 hw 1 mode channel\
shaper bw_rlimit max_rate 4Gbit 5Gbit

I think that's what you are doing, but I just got confused by the 
cover letter.

quoted

In this new mode, the priority to traffic class mapping and the 
user specified queue ranges are used to configure the traffic class
when the 'mode' option is set to 'channel'. This is achieved by
creating HW channels(VSI). A new channel is created for each of the
traffic class configuration offloaded via mqprio framework except
for the first TC (TC0) which is for the main VSI. TC0 for the main
VSI is also reconfigured as per user provided queue parameters.
Finally, bandwidth rate limits are set on these traffic classes
through the shaper attribute by sending these rates in addition to
the number of TCs and the queue configurations.

Example: # tc qdisc add dev eth0 root mqprio num_tc 2 map 0 0 0 0 1
1 1 1\ queues 4@0 4@4 hw 1 mode channel shaper bw_rlimit\

Do you see a case where you can declare a different number of
traffic classes say 4 and map them onto just 2 hardware queues? If
not, it seems a tiny bit redundant to have to specify both the map
and the queue mapping should be sufficient, right?

This will be subjected to validation of the user input and will be
treated as invalid configuration. The 'map' specifies the mapping
between user priorities and traffic classes while the queue mapping is
the queue layout specifying the queue count and offsets.

quoted

min_rate 1Gbit 2Gbit max_rate 4Gbit 5Gbit

To dump the bandwidth rates:

# tc qdisc show dev eth0

qdisc mqprio 804a: root  tc 2 map 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 
queues:(0:3) (4:7) mode:channel shaper:bw_rlimit   min_rate:1Gbit 
2Gbit   max_rate:4Gbit 5Gbit

I am not well versed into tc, but being able to specify "shaper" 
arguments has actually value outside of just the multiq scheduler and
it could probably be an action on its own?

The mqprio scheduler already supports configuring the traffic classes
and enabling support for HW shapers was just a matter to extending it
with new netlink based attributes.

quoted

---

Amritha Nambiar (6): mqprio: Introduce new hardware offload mode 
and shaper in mqprio i40e: Add macro for PF reset bit i40e: Add 
infrastructure for queue channel support i40e: Enable 'channel' 
mode in mqprio for TC configs i40e: Refactor VF BW rate limiting 
i40e: Add support setting TC max bandwidth rates


drivers/net/ethernet/intel/i40e/i40e.h             |   44 + 
drivers/net/ethernet/intel/i40e/i40e_debugfs.c     |    3 
drivers/net/ethernet/intel/i40e/i40e_ethtool.c     |    8 
drivers/net/ethernet/intel/i40e/i40e_main.c        | 1463 
+++++++++++++++++--- drivers/net/ethernet/intel/i40e/i40e_txrx.h | 
2 drivers/net/ethernet/intel/i40e/i40e_virtchnl_pf.c |   50 - 
include/net/pkt_cls.h                              |    9 
include/uapi/linux/pkt_sched.h                     |   32 
net/sched/sch_mqprio.c                             |  183 ++- 9 
files changed, 1551 insertions(+), 243 deletions(-)

--