Re: [RFI] ucmatose: No effect to set service type for QoS

On Fri, Aug 12, 2016 at 3:03 PM, Hal Rosenstock [off-list ref] wrote:

On 8/12/2016 7:55 AM, Hal Rosenstock wrote:

quoted

On 8/12/2016 4:15 AM, Jinpu Wang wrote:

quoted

On Thu, Aug 11, 2016 at 11:15 PM, Hal Rosenstock [off-list ref] wrote:

quoted

On 8/11/2016 8:29 AM, Jinpu Wang wrote:

quoted

On Wed, Aug 10, 2016 at 8:52 PM, Hal Rosenstock [off-list ref] wrote:

quoted

On 8/9/2016 12:26 PM, Jinpu Wang wrote:

quoted

Hi Sean,

I'm testing QoS support for IB. I notice ucmatose has equally
performance when set different service type, but set SL in ib_send_bw
works well (different SL show different performance base on opensm
settings)

I capature packats using ibdump, it shows in in LRH the service level
fields are all 0 when running traffic with ucmatose.

When running ib_send_bw, it carries the right service level I set.

Seems in rdma_set_service_type, it sets to tos to id_priv->tos, and
lter set to path_rec->qos_class or traffic_class but not to sl
directly, what's the consideration here?
code snip:
        switch (cma_family(id_priv)) {
        case AF_INET:
                path_rec->qos_class = cpu_to_be16((u16) id_priv->tos);
                comp_mask |= IB_SA_PATH_REC_QOS_CLASS;
                break;
        case AF_INET6:
                sin6 = (struct sockaddr_in6 *) cma_src_addr(id_priv);
                path_rec->traffic_class = (u8)
(be32_to_cpu(sin6->sin6_flowinfo) >> 20);
                comp_mask |= IB_SA_PATH_REC_TRAFFIC_CLASS;
                break;
        case AF_IB:
                sib = (struct sockaddr_ib *) cma_src_addr(id_priv);
                path_rec->traffic_class = (u8)
(be32_to_cpu(sib->sib_flowinfo) >> 20);


Does it make sense we also set sl here, or service type for ucmatose
is totally different with SL for ib_send_bw?

I think this is an OpenSM configuration issue. QoS policy needs to be
setup to return the proper SL to use for QoS class or TClass in the
PathRecord response.

-- Hal

Thanks Hal,

Configure extra QoS policy seems quite complex.

Configuration complexity varies depending on the requirements of the QoS
needs.

Which type of RDMA CM connections are being used (IPv4, IPv6, or native
IB) ?

quoted

Do you think patch attached make sense?

Attached patch doesn't appear to relate to upstream.

Indeed, it's based on MLNXOFED 3.2

quoted

It also looks incomplete to me. What invokes rdma_set_service_level ? Is
it some option in ucma.c:ucma_set_option ?

The main purpose is for our in house transport kernel module,

One more thing:

How does transport module know which SL to request ?

In general, SL is based on SM configuration.

Service ID and QoS Class or Traffic Class are the "higher level" IB
architected ways to obtain the SL.

quoted

quoted

it
supports all 3 connections
(IPv4, IPv6, and native IB, IB is the default).

quoted

quoted

Current patch doesn't appear to me to be backward compatible. If
rdma_set_service_level is not called in flow, then SL should not be set
in SA PR query which is what happens today.

Good point, I will add check only set SL if not 0,

0 is a valid SL so an extra bit somewhere is needed to indicate whether
a specific SL is being requested.

quoted

but if
rdma_set_service_level is not called,
SL should be 0 as before, shouldn't change SA PR query behavior, or I
missed something?

Component mask for SL in SA PR query is not on currently so that means
it's wildcarded rather than 0.

quoted

quoted

Also, if SL is set in query, you probably don't need some of the other
fields that are being set.

Do you mean SL shouldn't be set with other fields, what's the side effect there?

Never mind. It's probably best to leave those other fields as is.

Thanks, I update my patch to address your comments.

For configuration, we introduce a module parameter, sysadmin will set that.
similar for ib_send_bw kinds of perf tools.


-- 
Mit freundlichen Grüßen,
Best Regards,

Jack Wang

Linux Kernel Developer Storage
ProfitBricks GmbH  The IaaS-Company.

ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin
Tel: +49 30 5770083-42
Fax: +49 30 5770085-98
Email: jinpu.wang-EIkl63zCoXaH+58JC4qpiA@public.gmane.org
URL: http://www.profitbricks.de

Sitz der Gesellschaft: Berlin.
Registergericht: Amtsgericht Charlottenburg, HRB 125506 B.
Geschäftsführer: Andreas Gauger, Achim Weiss.