Re: Unable to Connect Using Active Speed 2.5 Gbps and Active Width of 1

On 3/28/2011 4:41 PM, Alan Cook wrote:

Jason Gunthorpe<jgunthorpe@...>  writes:

quoted

Are you using IPoIB to test your ping? You might be having multicast
group membership join problems due to a rate mismatch? Try one of the
low level verbs only diags.

I found the following issue when running ibdiagnet (which I had yet to run).

-I---------------------------------------------------
-I- IPoIB Subnets Check
-I---------------------------------------------------
-I- Subnet: IPv4 PKey:0x7fff QKey:0x00000b1b MTU:2048Byte rate:10Gbps SL:0x00
-W- Port raid1/P1 lid=0x0002 guid=0x0002c903000aa6d7 dev=26428 can not join due
     to rate:2.5Gbps<  group:10Gbps
-W- Port raid2/P1 lid=0x0001 guid=0x0002c903000aa6cf dev=26428 can not join due
     to rate:2.5Gbps<  group:10Gbps
-W- Suboptimal rate for group. Lowest member rate:120Gbps>  group-rate:10Gbps

I also changed the speed to 5 Gbps (which I thought had worked earlier) and I am
unable to transfer using a width 1X.  I receive the same warnings as above.

The default is rate 3 (10 Gbps) so I would think anything less than that wouldn't work without reconfiguration.

After doing some searching, I created a partitions.conf (which did not exist at
configured location in opensm.conf).  I added the following line:

Test,ipoib,rate=1,defmember=full:0x0002c903000aa6d7,0x0002c903000aa6cf;

How about if rather than Test, you change it to Default=0x7fff ?

After restarting opensm.conf, I find that the above warnings in ibdiagnet no
longer appear, but I am still unable to communicate between PCs.

Did I miss something or do something incorrectly?

The above is pertinent to IPoIB ping but couldn't explain ibping lack of connectivity.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hal Rosenstock <hal@...> writes:

How about if rather than Test, you change it to Default=0x7fff ?

I tried using the PKey since posting, but it alone did not solve the issue.

The above is pertinent to IPoIB ping but couldn't explain ibping lack of 
connectivity.

Turns out I was using the wrong port GUID (bad copy and paste) when testing with
ibping. It has been working all along.

I have just now solved the issue.

The partitions.conf file took me in the right direction.  The solution was to
use rate=2 instead of as rate=1 (which apparently is a reserved value).  I was
unable find documentation for the rate-value pairs, so I was assuming that
rate=1 was 2.5.  I finally found documentation for the rates in a post on the
old ofa-general mailing list:

#define IB_PATH_RECORD_RATE_2_5_GBS             2
#define IB_PATH_RECORD_RATE_10_GBS              3
#define IB_PATH_RECORD_RATE_30_GBS              4
#define IB_PATH_RECORD_RATE_5_GBS               5
#define IB_PATH_RECORD_RATE_20_GBS              6
#define IB_PATH_RECORD_RATE_40_GBS              7
#define IB_PATH_RECORD_RATE_60_GBS              8
#define IB_PATH_RECORD_RATE_80_GBS              9
#define IB_PATH_RECORD_RATE_120_GBS             10

#define IB_MIN_RATE    IB_PATH_RECORD_RATE_2_5_GBS
#define IB_MAX_RATE    IB_PATH_RECORD_RATE_120_GBS

Setting my partitions.conf as follows has solved the issue:

Default=0x7fff,ipoib,rate=2:ALL=full;

Thanks everyone for the help.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html