opensm + ceph RDMA = Down OSDs
From: westjoshuaalan@gmail.com <hidden>
Date: 2021-03-16 12:16:42
Hello, I understand that this mailing list may be able to help me with an issue I am experiencing specific to opensm/infiniband rdma! Forgive me if I am not in the right place though, if not, I would appreciate a pointer to the next step in my journey! I am having a recurring issue where after some period of working without issue, I come to find several OSDs in my ceph cluster offline, each reporting the same clock skew error in their systemd unit log (`systemctl status ceph-osd@##`):
Mar 12 00:39:28 rd240 ceph-osd[1655164]: 2021-03-12T00:39:28.006-0700 7f0d635b2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2021-03-11T23:39:28.012146-0700) Mar 12 00:39:29 rd240 ceph-osd[1655164]: 2021-03-12T00:39:29.006-0700 7f0d635b2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2021-03-11T23:39:29.012281-0700) Mar 12 00:39:30 rd240 ceph-osd[1655164]: 2021-03-12T00:39:30.007-0700 7f0d635b2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2021-03-11T23:39:30.012420-0700) (Ignore the fact it's right at midnight, which I beleive is simply from log rotation.)
When resetting the osds, they all come back online without issue, but at some point in the future are expected to go down again. See attached the two opensm logs from the node on which all osd failures occurred today. Here is the only message from the opensm master (I have three nodes in total, opensm running on each, and different opensm priorities for each port in an attempt to create redundancy)
Mar 11 19:32:26 717944 [EC430700] 0x01 -> log_send_error: ERR 5411: DR SMP Send completed with error (IB_TIMEOUT) -- dropping
Method 0x1, Attr 0x20, TID 0x11f51
Mar 11 19:32:26 718212 [EC430700] 0x01 -> Received SMP on a 2 hop path: Initial path = 0,1,3, Return path = 0,0,0
Mar 11 19:32:26 718223 [EC430700] 0x01 -> sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT): SubnGet(SMInfo), attr_mod 0x0, TID 0x11f51
Mar 11 19:32:26 718227 [EC430700] 0x01 -> sm_mad_ctrl_send_err_cb: ERR 3120: Timeout while getting attribute 0x20 (SMInfo); Possible mis-set mkey?
Mar 11 19:34:56 733934 [EC430700] 0x01 -> log_send_error: ERR 5411: DR SMP Send completed with error (IB_TIMEOUT) -- dropping
Method 0x1, Attr 0x20, TID 0x11f8d
Mar 11 19:34:56 733965 [EC430700] 0x01 -> Received SMP on a 2 hop path: Initial path = 0,1,3, Return path = 0,0,0
Mar 11 19:34:56 733972 [EC430700] 0x01 -> sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT): SubnGet(SMInfo), attr_mod 0x0, TID 0x11f8d
Mar 11 19:34:56 733977 [EC430700] 0x01 -> sm_mad_ctrl_send_err_cb: ERR 3120: Timeout while getting attribute 0x20 (SMInfo); Possible mis-set mkey?
Mar 11 19:34:56 733993 [EC430700] 0x01 -> log_send_error: ERR 5411: DR SMP Send completed with error (IB_TIMEOUT) -- dropping
Method 0x1, Attr 0x20, TID 0x11f8e
Mar 11 19:34:56 733999 [EC430700] 0x01 -> Received SMP on a 2 hop path: Initial path = 0,1,2, Return path = 0,0,0
Mar 11 19:34:56 734003 [EC430700] 0x01 -> sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT): SubnGet(SMInfo), attr_mod 0x0, TID 0x11f8e
Mar 11 19:34:56 734007 [EC430700] 0x01 -> sm_mad_ctrl_send_err_cb: ERR 3120: Timeout while getting attribute 0x20 (SMInfo); Possible mis-set mkey?
Mar 11 19:35:26 725924 [EC430700] 0x01 -> log_send_error: ERR 5411: DR SMP Send completed with error (IB_TIMEOUT) -- dropping
Method 0x1, Attr 0x20, TID 0x11f99--> That "Return path = 0,0,0" strikes me as possibly incorrect? --> This is just repeated until failed ceph osd's restarted Investigating the "Possible mis-set mkey?" line, I believe it is the default, as I haven't expressly set it to my memory, but haven't figured out how to check either. I suspect the error is in my opensm config.. opensm are each run via: /etc/init.d/opensm:
... start-stop-daemon --start --quiet --make-pidfile --pidfile /var/run/opensm-0x7cfe900300179b31 --background --exec \ /usr/sbin/opensm -- \ --daemon \ -g 0x7cfe900300179b31 \ -p /etc/opensm/partitions.conf \ -f /var/log/opensm.0x7cfe900300179b31.log \ --priority 11 \ #changed per port, no overlapping priorities --port-shifting \ --ucast_cache \ --do_mesh_analysis \ --lmc 0 \ -R ftree,updn,minhop \ --part_enforce both \ --allow_both_pkeys ...
Digging into this error, I checked and found iblink error counters as follows, mainly on (i believe) the IB switch itself?
# ibqueryerrors Errors for "dl380g7 mlx4_0" GUID 0x7cfe900300179b31 port 1: [VL15Dropped == 14] [PortXmitWait == 4294967295] GUID 0x7cfe900300179b32 port 2: [PortXmitWait == 2] Errors for "server mlx4_0" GUID 0x2c9030042dff1 port 1: [PortXmitWait == 4294967295] Errors for 0x2c903006e29b0 "MF0;ys23ib4:SX6036/U1" GUID 0x2c903006e29b0 port ALL: [LinkErrorRecoveryCounter == 15] [LinkDownedCounter == 48] [PortRcvSwitchRelayErrors == 2461] [PortXmitDiscards == 15854] [VL15Dropped == 5] [PortXmitWait == 4294967295] GUID 0x2c903006e29b0 port 0: [PortXmitWait == 76959324] GUID 0x2c903006e29b0 port 1: [LinkDownedCounter == 8] [PortRcvSwitchRelayErrors == 2] [PortXmitWait == 4294967295] GUID 0x2c903006e29b0 port 2: [LinkDownedCounter == 5] [PortRcvSwitchRelayErrors == 4] [VL15Dropped == 2] [PortXmitWait == 4294967295] GUID 0x2c903006e29b0 port 3: [LinkDownedCounter == 5] [PortRcvSwitchRelayErrors == 2] [VL15Dropped == 3] [PortXmitWait == 4294967295] GUID 0x2c903006e29b0 port 4: [LinkDownedCounter == 8] [PortRcvSwitchRelayErrors == 2] [PortXmitWait == 2] GUID 0x2c903006e29b0 port 5: [SymbolErrorCounter == 65535] [LinkErrorRecoveryCounter == 8] [LinkDownedCounter == 2] [PortRcvSwitchRelayErrors == 149] [PortXmitDiscards == 649] [PortXmitWait == 84704351] GUID 0x2c903006e29b0 port 6: [LinkDownedCounter == 9] [PortRcvSwitchRelayErrors == 2285] [PortXmitDiscards == 14176] [PortXmitWait == 4294967295] GUID 0x2c903006e29b0 port 7: [SymbolErrorCounter == 65535] [LinkErrorRecoveryCounter == 7] [LinkDownedCounter == 2] [PortRcvSwitchRelayErrors == 1] GUID 0x2c903006e29b0 port 8: [LinkDownedCounter == 9] [PortRcvSwitchRelayErrors == 16] [PortXmitDiscards == 1029] Errors for "rd240 mlx4_0" GUID 0xe41d2d0300e0bae1 port 1: [PortXmitWait == 4294967295] GUID 0xe41d2d0300e0bae2 port 2: [PortXmitWait == 1] ## Summary: 4 nodes checked, 4 bad nodes found ## 43 ports checked, 14 ports have errors beyond threshold ## Thresholds: ## Suppressed:
par I do not have an opensm.conf, instead setting the config via the above init.d script, but I do have partitions.conf as:
Default=0x7fff, rate=7, mtu=4, scope=2, defmember=full:
ALL=full, SELF=full, ALL_SWITCHES=full;
Default=0x7fff, ipoib, rate=7, mtu=4, scope=2:
mgid=ff12:401b:ffff::ffff:ffff # JW from error log
mgid=ff12:401b::ffff:ffff # IPv4 Broadcast address
mgid=ff12:401b::1 # IPv4 All Hosts group
mgid=ff12:401b::2 # IPv4 All Routers group
mgid=ff12:401b::16 # IPv4 IGMP group
mgid=ff12:401b::fb # IPv4 mDNS group
mgid=ff12:401b::fc # IPv4 Multicast Link Local Name Resolution group
mgid=ff12:401b::101 # IPv4 NTP group
mgid=ff12:401b::202 # IPv4 Sun RPC
mgid=ff12:601b::1 # IPv6 All Hosts group
mgid=ff12:601b::2 # IPv6 All Routers group
mgid=ff12:601b::16 # IPv6 MLDv2-capable Routers group
mgid=ff12:601b::fb # IPv6 mDNS group
mgid=ff12:601b::101 # IPv6 NTP group
mgid=ff12:601b::202 # IPv6 Sun RPC group
mgid=ff12:601b::1:3 # IPv6 Multicast Link Local Name Resolution group
ALL=full, SELF=full, ALL_SWITCHES=full;I am hopeful that this group may be able to point me in the right direction. I am 100% self taught, and have come a long way, but this issue has been haunting me for awhile now. It's not a production cluster, I am just learning and playing with stuff I find really interesting, but the double edged sword is that now I am feeling pretty lost... Any help would be greatly appreciated!
Attachments
- opensm.1.log.gz [application/gzip] 4537 bytes
- opensm.2.log.gz [application/gzip] 4507 bytes