Re: [IPoIB] Missing join mcast events causing full machine lockup

On Tue, 2016-08-02 at 23:18 +0300, Nikolay Borisov wrote:

On Tue, Aug 2, 2016 at 10:21 PM, Doug Ledford [off-list ref]
wrote:

quoted

On Thu, 2016-07-21 at 10:31 +0300, Nikolay Borisov wrote:

quoted

Hello,

With running the risk of sounding like a broken record, I came
across
another case where ipoib can cause the machine to go haywire due
to
missed join requests. This is on 4.4.14 kernel. Here is what I
believe
happens:

[ snip long traces ]

quoted

This makes me wonder if using timeouts is actually better than
blindly relying on completing the join.

Blindly relying on the join completions is not what we do.  We are
very
careful to make sure we always have the right locking so that we
never
leave a join request in the BUSY state without running the
completion
at some time.  If you are seeing us do that, then it means we have
a
bug in our locking or state processing.  The answer then is to find
that bug and not to paper over it with a timeout.  Can you find
some
way to reproduce this with a 4.7 kernel?

Unfortunately my environment is constrained to 4.4 kernel. I will,
however,
try and check if I can get a couple of IB-enabled nodes on 4.7 and
see
if something
shows up. And while I don't have a 100% reproducer for it I see those
symptoms rather regularly
on production nodes. I'm able and happy to extract any runtime state
that might be useful in debugging this i.e I can obtain crashdumps
and
reverse the state of the ipoib stacks. I've seen this issue on 3.12
and on 4.4.
Some of my previous emails also show this manifesting in hangs in
cm_destroy_id
as well. So clearly there is a problem there but it proves very
elusive.

Can you give any clues as to what's causing it?  Do you have link flap?
SM bounces?  Lots of multicast joins/leaves?

-- 
Doug Ledford [off-list ref]
              GPG KeyID: 0E572FDD