Re: [IPoIB] Missing join mcast events causing full machine lockup

On 08/02/2016 11:29 PM, Doug Ledford wrote:

On Tue, 2016-08-02 at 23:18 +0300, Nikolay Borisov wrote:

quoted

On Tue, Aug 2, 2016 at 10:21 PM, Doug Ledford [off-list ref]
wrote:

quoted

On Thu, 2016-07-21 at 10:31 +0300, Nikolay Borisov wrote:

quoted

Hello,

With running the risk of sounding like a broken record, I came
across
another case where ipoib can cause the machine to go haywire due
to
missed join requests. This is on 4.4.14 kernel. Here is what I
believe
happens:

[ snip long traces ]

quoted

This makes me wonder if using timeouts is actually better than
blindly relying on completing the join.

Blindly relying on the join completions is not what we do.  We are
very
careful to make sure we always have the right locking so that we
never
leave a join request in the BUSY state without running the
completion
at some time.  If you are seeing us do that, then it means we have
a
bug in our locking or state processing.  The answer then is to find
that bug and not to paper over it with a timeout.  Can you find
some
way to reproduce this with a 4.7 kernel?

Unfortunately my environment is constrained to 4.4 kernel. I will,
however,
try and check if I can get a couple of IB-enabled nodes on 4.7 and
see
if something
shows up. And while I don't have a 100% reproducer for it I see those
symptoms rather regularly
on production nodes. I'm able and happy to extract any runtime state
that might be useful in debugging this i.e I can obtain crashdumps
and
reverse the state of the ipoib stacks. I've seen this issue on 3.12
and on 4.4.
Some of my previous emails also show this manifesting in hangs in
cm_destroy_id
as well. So clearly there is a problem there but it proves very
elusive.

Can you give any clues as to what's causing it?  Do you have link flap?
SM bounces?  Lots of multicast joins/leaves?

I spoke with the network admins and they said that the network is not flapping, 
we shouldn't have a lot of join/leaves since the network is not that big and is 
stable. E.g. once nodes joins they usually are not restarted. 

Here are some messages which result after the said hangs happen: 

Aug  1 04:53:51 node1 kernel: [29100.763267] ib0: Budget exhausted after napi rescheduled 
Jul 31 21:29:46 node1 kernel: [ 2457.666476] NETDEV WATCHDOG: ib0 (ib_qib): transmit queue 0 timed out
Jul 29 05:17:36 node1 kernel: [ 8797.968402] ib0: dev_queue_xmit failed to requeue packet
Jul 23 19:27:22 node1 kernel: ib0: packet len 2200 (> 2048) too long to send, dropping
Jul 25 01:01:52 node1 kernel: ib0: queue stopped 1, tx_head 124520708, tx_tail 124520580

Aug  2 10:05:26 node15 bird6: LocalIPv6: Socket error on ib0: No buffer space available

Also I'm being told that *sometimes* doing a remote port reset actually fixes the issue,
but only sometimes as otherwise the port is completely inactive. 

I realize this is not much information but this issue really just rears its ugly head out
of nowhere and usually there doesn't seem to be that much information ;(

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html