Re: Soft lockup issue in Linux 4.1.9

On 10/01/15 13:29, Eric Dumazet wrote:

On Thu, Oct 1, 2015 at 3:59 AM, Holger Hoffstätte
[off-list ref] wrote:

quoted

On Thu, 01 Oct 2015 06:41:46 +0200, Andre Tomt wrote:

quoted

On 01. okt. 2015 00:37, Holger Hoffstätte wrote:

quoted

On Wed, 30 Sep 2015 23:59:43 +0200, Olivier Bonvalet wrote:

quoted

for information, I've just upgraded 6 servers from Linux 4.1.8 to Linux
4.1.9, and have some random soft lockup. If this can help :

Congratulations! You're not the first one to get hit by this, but
you are probably the first one to get a meaningful stacktrace! \o/

quoted

[  204.478380] Call Trace:
[  204.478381]  <IRQ>
[  204.478385]  [<ffffffff81076121>] ? try_to_del_timer_sync+0x43/0x4d
[  204.478386]  [<ffffffff810760de>] ? del_timer+0x4d/0x4d
[  204.478388]  [<ffffffff8107614b>] ? del_timer_sync+0x20/0x3d

Can you try to revert

    [PATCH 4.1 157/159] inet: fix races with reqsk timers

and see how that works for you? I'll do the same on my end. So far the
only thing I ever could gleam was an rcu stall after cpuidle_enter(),
but never anything regarding the timer - though it was definitely
related to NIC activity after idle.

I'm running with this patch reverted now as well. 2 hours no issues so
far, but I can't conclude anything yet as I've seen it take up to 6+
hours to explode here. As a result the bisect was going veeery slowly.

Now 12+ hours going without problems, never got this far with the patch
included, as it would usually freeze during idle periods.

As far as I'm concerned this is the culprit and should be reverted in
4.1.x, unless Eric can suggest how to fix this. (cc'ed).

Looks an old and known problem...

Following commit should be sent/added for 4.1 stable tree :

commit 83fccfc3940c4a2db90fd7e7079f5b465cd8c6af
Author: Eric Dumazet [off-list ref]
Date:   Thu Aug 13 15:44:51 2015 -0700

    inet: fix potential deadlock in reqsk_queue_unlink()

    When replacing del_timer() with del_timer_sync(), I introduced
    a deadlock condition :

    reqsk_queue_unlink() is called from inet_csk_reqsk_queue_drop()

    inet_csk_reqsk_queue_drop() can be called from many contexts,
    one being the timer handler itself (reqsk_timer_handler()).

    In this case, del_timer_sync() loops forever.

    Simple fix is to test if timer is pending.

    Fixes: 2235f2ac75fd ("inet: fix races with reqsk timers")
    Signed-off-by: Eric Dumazet [off-list ref]
    Signed-off-by: David S. Miller [off-list ref]

Whohoo! It applies/builds cleanly to 4.1.10-rc1 and is running as
we speak. Let's hope that this fixes the lockups.

Thanks for the quick reply!

Holger