hung task in mac80211

Hi,

I have an hung task on vanilla 4.13 kernel which I haven't on 4.12.
The problem is present both on my AP and on my notebook,
so it seems it affects AP and STA mode as well.
The generated messages are:

INFO: task kworker/u16:6:120 blocked for more than 120 seconds.
      Not tainted 4.13.0 #57
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kworker/u16:6   D    0   120      2 0x00000000
Workqueue: phy0 ieee80211_ba_session_work [mac80211]
Call Trace:
 ? __schedule+0x174/0x5b0
 ? schedule+0x31/0x80
 ? schedule_preempt_disabled+0x9/0x10
 ? __mutex_lock.isra.2+0x163/0x480
 ? select_task_rq_fair+0xb9f/0xc60
 ? __ieee80211_start_rx_ba_session+0x135/0x4d0 [mac80211]
 ? __ieee80211_start_rx_ba_session+0x135/0x4d0 [mac80211]
 ? try_to_wake_up+0x1f1/0x340
 ? update_curr+0x88/0xd0
 ? ieee80211_ba_session_work+0x148/0x230 [mac80211]
 ? process_one_work+0x1a5/0x330
 ? worker_thread+0x42/0x3c0
 ? create_worker+0x170/0x170
 ? kthread+0x10d/0x130
 ? kthread_create_on_node+0x40/0x40
 ? ret_from_fork+0x22/0x30

I did a bisect and the offending commit is:

commit 699cb58c8a52ff39bf659bff7971893ebe111bf2
Author: Johannes Berg [off-list ref]
Date:   Tue May 30 16:34:46 2017 +0200

    mac80211: manage RX BA session offload without SKB queue

    Instead of using the SKB queue with the fake pkt_type for the
    offloaded RX BA session management, also handle this with the
    normal aggregation state machine worker. This also makes the
    use of this more reliable since it gets rid of the allocation
    of the fake skb.

    Combined with the previous patch, this finally allows us to
    get rid of the pkt_type hack entirely, so do that as well.

    Signed-off-by: Johannes Berg [off-list ref]

Regards,
-- 
Matteo Croce
per aspera ad upstream

On Wednesday, September 6, 2017 1:57:47 PM CEST Matteo Croce wrote:

Hi,

I have an hung task on vanilla 4.13 kernel which I haven't on 4.12.
The problem is present both on my AP and on my notebook,
so it seems it affects AP and STA mode as well.
The generated messages are:

INFO: task kworker/u16:6:120 blocked for more than 120 seconds.
      Not tainted 4.13.0 #57
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kworker/u16:6   D    0   120      2 0x00000000
Workqueue: phy0 ieee80211_ba_session_work [mac80211]
Call Trace:
 ? __schedule+0x174/0x5b0
 ? schedule+0x31/0x80
 ? schedule_preempt_disabled+0x9/0x10
 ? __mutex_lock.isra.2+0x163/0x480
 ? select_task_rq_fair+0xb9f/0xc60
 ? __ieee80211_start_rx_ba_session+0x135/0x4d0 [mac80211]
 ? __ieee80211_start_rx_ba_session+0x135/0x4d0 [mac80211]
 ? try_to_wake_up+0x1f1/0x340
 ? update_curr+0x88/0xd0
 ? ieee80211_ba_session_work+0x148/0x230 [mac80211]

I did a bisect and the offending commit is:

commit 699cb58c8a52ff39bf659bff7971893ebe111bf2
Author: Johannes Berg [off-list ref]
Date:   Tue May 30 16:34:46 2017 +0200

    mac80211: manage RX BA session offload without SKB queue

I looked at this briefly:

ieee80211_ba_session_work acquires:
mutex_lock(&sta->ampdu_mlme.mtx) @
<http://elixir.free-electrons.com/linux/v4.13/source/net/mac80211/ht.c#L321>

But it now also calls
__ieee80211_start_rx_ba_session() @
http://elixir.free-electrons.com/linux/v4.13/source/net/mac80211/ht.c#L336

which also wants to hold mutex_lock(&sta->ampdu_mlme.mtx) in:
http://elixir.free-electrons.com/linux/v4.13/source/net/mac80211/agg-rx.c#L314

I guess this is where it deadlocks?

Regards,
Christian

On Wed, 6 Sep 2017 13:57:47 +0200
Matteo Croce [off-list ref] wrote:

Hi,

I have an hung task on vanilla 4.13 kernel which I haven't on 4.12.
The problem is present both on my AP and on my notebook,
so it seems it affects AP and STA mode as well.
The generated messages are:

INFO: task kworker/u16:6:120 blocked for more than 120 seconds.
      Not tainted 4.13.0 #57
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kworker/u16:6   D    0   120      2 0x00000000
Workqueue: phy0 ieee80211_ba_session_work [mac80211]
Call Trace:
 ? __schedule+0x174/0x5b0
 ? schedule+0x31/0x80
 ? schedule_preempt_disabled+0x9/0x10
 ? __mutex_lock.isra.2+0x163/0x480
 ? select_task_rq_fair+0xb9f/0xc60
 ? __ieee80211_start_rx_ba_session+0x135/0x4d0 [mac80211]
 ? __ieee80211_start_rx_ba_session+0x135/0x4d0 [mac80211]

This is ugly and maybe wrong, but you could check perhaps...:

diff --git a/net/mac80211/ht.c b/net/mac80211/ht.c
index c92df492e898..bd7512a656f2 100644
--- a/net/mac80211/ht.c
+++ b/net/mac80211/ht.c

@@ -320,28 +320,40 @@ void ieee80211_ba_session_work(struct work_struct *work)
 
 	mutex_lock(&sta->ampdu_mlme.mtx);
 	for (tid = 0; tid < IEEE80211_NUM_TIDS; tid++) {
-		if (test_and_clear_bit(tid, sta->ampdu_mlme.tid_rx_timer_expired))
+		if (test_and_clear_bit(tid, sta->ampdu_mlme.tid_rx_timer_expired)) {
+			mutex_unlock(&sta->ampdu_mlme.mtx);
 			___ieee80211_stop_rx_ba_session(
 				sta, tid, WLAN_BACK_RECIPIENT,
 				WLAN_REASON_QSTA_TIMEOUT, true);
+			mutex_lock(&sta->ampdu_mlme.mtx);
+		}
 
 		if (test_and_clear_bit(tid,
-				       sta->ampdu_mlme.tid_rx_stop_requested))
+				       sta->ampdu_mlme.tid_rx_stop_requested)) {
+			mutex_unlock(&sta->ampdu_mlme.mtx);
 			___ieee80211_stop_rx_ba_session(
 				sta, tid, WLAN_BACK_RECIPIENT,
 				WLAN_REASON_UNSPECIFIED, true);
+			mutex_lock(&sta->ampdu_mlme.mtx);
+		}
 
 		if (test_and_clear_bit(tid,
-				       sta->ampdu_mlme.tid_rx_manage_offl))
+				       sta->ampdu_mlme.tid_rx_manage_offl)) {
+			mutex_unlock(&sta->ampdu_mlme.mtx);
 			__ieee80211_start_rx_ba_session(sta, 0, 0, 0, 1, tid,
 							IEEE80211_MAX_AMPDU_BUF,
 							false, true);
+			mutex_lock(&sta->ampdu_mlme.mtx);
+		}
 
 		if (test_and_clear_bit(tid + IEEE80211_NUM_TIDS,
-				       sta->ampdu_mlme.tid_rx_manage_offl))
+				       sta->ampdu_mlme.tid_rx_manage_offl)) {
+			mutex_unlock(&sta->ampdu_mlme.mtx);
 			___ieee80211_stop_rx_ba_session(
 				sta, tid, WLAN_BACK_RECIPIENT,
 				0, false);
+			mutex_lock(&sta->ampdu_mlme.mtx);
+		}
 
 		spin_lock_bh(&sta->lock);
 

-- 
Stefano

On Wed, Sep 6, 2017 at 2:40 PM, Stefano Brivio [off-list ref] wrote:

ACK, I have it running since 12 minutes.
The hang usually appears shortly after boot as I set
kernel.hung_task_timeout_secs=10

-- 
Matteo Croce
per aspera ad upstream