Can you try to revert
[PATCH 4.1 157/159] inet: fix races with reqsk timers
and see how that works for you? I'll do the same on my end. So far the
only thing I ever could gleam was an rcu stall after cpuidle_enter(),
but never anything regarding the timer - though it was definitely
related to NIC activity after idle.
I'm running with this patch reverted now as well. 2 hours no issues so
far, but I can't conclude anything yet as I've seen it take up to 6+
hours to explode here. As a result the bisect was going veeery slowly.
Now 12+ hours going without problems, never got this far with the patch
included, as it would usually freeze during idle periods.
As far as I'm concerned this is the culprit and should be reverted in
4.1.x, unless Eric can suggest how to fix this. (cc'ed).
Looks an old and known problem...
Following commit should be sent/added for 4.1 stable tree :
commit 83fccfc3940c4a2db90fd7e7079f5b465cd8c6af
Author: Eric Dumazet [off-list ref]
Date: Thu Aug 13 15:44:51 2015 -0700
inet: fix potential deadlock in reqsk_queue_unlink()
When replacing del_timer() with del_timer_sync(), I introduced
a deadlock condition :
reqsk_queue_unlink() is called from inet_csk_reqsk_queue_drop()
inet_csk_reqsk_queue_drop() can be called from many contexts,
one being the timer handler itself (reqsk_timer_handler()).
In this case, del_timer_sync() loops forever.
Simple fix is to test if timer is pending.
Fixes: 2235f2ac75fd ("inet: fix races with reqsk timers")
Signed-off-by: Eric Dumazet [off-list ref]
Signed-off-by: David S. Miller [off-list ref]
Whohoo! It applies/builds cleanly to 4.1.10-rc1 and is running as
we speak. Let's hope that this fixes the lockups.
Thanks for the quick reply!
Holger
From: Eric Dumazet <edumazet@google.com> Date: 2015-10-01 11:52:23
On Thu, Oct 1, 2015 at 4:43 AM, Holger Hoffstätte
[off-list ref] wrote:
On 10/01/15 13:29, Eric Dumazet wrote:
quoted
commit 83fccfc3940c4a2db90fd7e7079f5b465cd8c6af
Author: Eric Dumazet [off-list ref]
Date: Thu Aug 13 15:44:51 2015 -0700
inet: fix potential deadlock in reqsk_queue_unlink()
When replacing del_timer() with del_timer_sync(), I introduced
a deadlock condition :
reqsk_queue_unlink() is called from inet_csk_reqsk_queue_drop()
inet_csk_reqsk_queue_drop() can be called from many contexts,
one being the timer handler itself (reqsk_timer_handler()).
In this case, del_timer_sync() loops forever.
Simple fix is to test if timer is pending.
Fixes: 2235f2ac75fd ("inet: fix races with reqsk timers")
Signed-off-by: Eric Dumazet [off-list ref]
Signed-off-by: David S. Miller [off-list ref]
Whohoo! It applies/builds cleanly to 4.1.10-rc1 and is running as
we speak. Let's hope that this fixes the lockups.
Seems to fix it for me as well. 3 systems have been running varying
types of production-like loads with it for 14+ hours without hanging.
Just got up, and yes - my systems survived the night as well, no issues.
Greg, any chance you can drop this into the pending 4.1.10? Otherwise people
will get another broken release.
cheers
Holger
Seems to fix it for me as well. 3 systems have been running varying
types of production-like loads with it for 14+ hours without hanging.
Just got up, and yes - my systems survived the night as well, no issues.
Greg, any chance you can drop this into the pending 4.1.10? Otherwise people
will get another broken release.
Fixes the problem here, too.
Regards,
--
Wolfgang Walter
Studentenwerk München
Anstalt des öffentlichen Rechts
From: Thomas Gleixner <hidden> Date: 2015-10-02 20:05:32
On Thu, 1 Oct 2015, Eric Dumazet wrote:
On Thu, Oct 1, 2015 at 4:43 AM, Holger Hoffstätte
[off-list ref] wrote:
quoted
On 10/01/15 13:29, Eric Dumazet wrote:
quoted
quoted
commit 83fccfc3940c4a2db90fd7e7079f5b465cd8c6af
Author: Eric Dumazet [off-list ref]
Date: Thu Aug 13 15:44:51 2015 -0700
inet: fix potential deadlock in reqsk_queue_unlink()
When replacing del_timer() with del_timer_sync(), I introduced
a deadlock condition :
reqsk_queue_unlink() is called from inet_csk_reqsk_queue_drop()
inet_csk_reqsk_queue_drop() can be called from many contexts,
one being the timer handler itself (reqsk_timer_handler()).
In this case, del_timer_sync() loops forever.
Simple fix is to test if timer is pending.
Fixes: 2235f2ac75fd ("inet: fix races with reqsk timers")
Signed-off-by: Eric Dumazet [off-list ref]
Signed-off-by: David S. Miller [off-list ref]
Whohoo! It applies/builds cleanly to 4.1.10-rc1 and is running as
we speak. Let's hope that this fixes the lockups.
It definitely should help !
What makes sure, that the timer cannot be readded while that timer
callback is running?
Thanks,
tglx
From: Thomas Gleixner <hidden> Date: 2015-10-02 21:05:16
On Fri, 2 Oct 2015, Eric Dumazet wrote:
On Fri, 2015-10-02 at 22:04 +0200, Thomas Gleixner wrote:
quoted
What makes sure, that the timer cannot be readded while that timer
callback is running?
What is exactly your question ?
CPU0 CPU1
timer expires
callback
add timer
timer_pending() == true
===> del_timer_sync()
I was just curious how this is prevented as I got lost in the
networking code as usual :)
Thanks,
tglx
From: Eric Dumazet <hidden> Date: 2015-10-02 21:32:44
On Fri, 2015-10-02 at 23:04 +0200, Thomas Gleixner wrote:
On Fri, 2 Oct 2015, Eric Dumazet wrote:
quoted
On Fri, 2015-10-02 at 22:04 +0200, Thomas Gleixner wrote:
quoted
What makes sure, that the timer cannot be readded while that timer
callback is running?
What is exactly your question ?
CPU0 CPU1
timer expires
callback
add timer
timer_pending() == true
===> del_timer_sync()
I was just curious how this is prevented as I got lost in the
networking code as usual :)
Sure ;)
I believe this can not happen for following reasons :
mod_timer_pinned() is used only when req is created, while timer cannot
possibly be running on the same req. The _pinned part is critical
because we set the req->refcnt _after_ starting the timer,
to avoid being visible and caught from rcu lookups in hash tables.
Then, timer might be modified only by mod_timer_pending() from
tcp_check_req() : This should not re-start timer if another cpu is in
the timer callback.
Thanks