Possible BT deadlock due to recursive read locking (Was: Re: kernel BUG at kernel/locking/rtmutex.c:1059)

On 2017-08-30 23:25:53 [+0200], Mart van de Wege wrote:

Hi,

Hi,

The issue persists up to v4.11.12-rt10

Does
	CONFIG_RWLOCK_RT_READER_BIASED=y
make it go away?

Stacktrace:

Aug 29 17:28:04 localhost kernel: [   46.483810] ------------[ cut here ]------------
Aug 29 17:28:04 localhost kernel: [   46.483812] kernel BUG at kernel/locking/rtmutex.c:1059!

this is a deadlock on RT however !RT has a hidden problem which not
yelled at by lockdep. We have the following call path:

| hci_send_monitor_ctrl_event()
|   read_lock(&hci_sk_list.lock);
|
|   hci_send_to_channel()
|      read_lock(&hci_sk_list.lock);

So both functions acquire the same read_lock. If a write comes along
between the first read_lock() and second read_lock() then we have a
deadlock because the write_lock() will lock down further readers from
acquiring the read-lock until the writer completed its task.

This recursive locking was introduced in 38ceaa00d02d ("Bluetooth: Add
support for sending MGMT commands and events to monitor").

Sebastian

On Thu, 31 Aug 2017, Sebastian Andrzej Siewior wrote:

On 2017-08-30 23:25:53 [+0200], Mart van de Wege wrote:

quoted

Hi,

Hi,

quoted

The issue persists up to v4.11.12-rt10

Does
	CONFIG_RWLOCK_RT_READER_BIASED=y
make it go away?

quoted

Stacktrace:

Aug 29 17:28:04 localhost kernel: [   46.483810] ------------[ cut here ]------------
Aug 29 17:28:04 localhost kernel: [   46.483812] kernel BUG at kernel/locking/rtmutex.c:1059!

this is a deadlock on RT however !RT has a hidden problem which not
yelled at by lockdep. We have the following call path:

| hci_send_monitor_ctrl_event()
|   read_lock(&hci_sk_list.lock);
|
|   hci_send_to_channel()
|      read_lock(&hci_sk_list.lock);

So both functions acquire the same read_lock. If a write comes along
between the first read_lock() and second read_lock() then we have a
deadlock because the write_lock() will lock down further readers from
acquiring the read-lock until the writer completed its task.

This recursive locking was introduced in 38ceaa00d02d ("Bluetooth: Add
support for sending MGMT commands and events to monitor").

Just for clarification: This is a mainline issue because qrlock based
rwlocks are writer fair, while the traditional rwlocks are reader
biased. Lockdep does not yell about it yet, because the wrapper function
use read-recursive mode.

Thanks,

	tglx

On Thu, 31 Aug 2017 16:52:12 +0200
Sebastian Andrzej Siewior [off-list ref] wrote:

On 2017-08-30 23:25:53 [+0200], Mart van de Wege wrote:

quoted

Hi,  

Hi,

quoted

The issue persists up to v4.11.12-rt10  

Does
	CONFIG_RWLOCK_RT_READER_BIASED=y
make it go away?

Yes, that does make it go away. -rt8 now boots cleanly, and works fine.
Building -rt10 now to be sure.

quoted

Stacktrace:

Aug 29 17:28:04 localhost kernel: [   46.483810] ------------[ cut
here ]------------ Aug 29 17:28:04 localhost kernel: [   46.483812]
kernel BUG at kernel/locking/rtmutex.c:1059!  

this is a deadlock on RT however !RT has a hidden problem which not
yelled at by lockdep. We have the following call path:

| hci_send_monitor_ctrl_event()
|   read_lock(&hci_sk_list.lock);
|
|   hci_send_to_channel()
|      read_lock(&hci_sk_list.lock);

So both functions acquire the same read_lock. If a write comes along
between the first read_lock() and second read_lock() then we have a
deadlock because the write_lock() will lock down further readers from
acquiring the read-lock until the writer completed its task.

This recursive locking was introduced in 38ceaa00d02d ("Bluetooth: Add
support for sending MGMT commands and events to monitor").

Sebastian

On Fri, 1 Sep 2017 10:51:07 +0200
Mart van de Wege [off-list ref] wrote:

On Thu, 31 Aug 2017 16:52:12 +0200
Sebastian Andrzej Siewior [off-list ref] wrote:

quoted

On 2017-08-30 23:25:53 [+0200], Mart van de Wege wrote:  

quoted

Hi,    

Hi,
  

quoted

The issue persists up to v4.11.12-rt10    

Does
	CONFIG_RWLOCK_RT_READER_BIASED=y
make it go away?
  

Yes, that does make it go away. -rt8 now boots cleanly, and works
fine. Building -rt10 now to be sure.

Confirm that as of 4.11.12-rt11 the deadlock still goes away when
enabling CONFIG_RWLOCK_RT_READER_BIASED

Mart

Mart reported a deadlock in -RT in the call path:
  hci_send_monitor_ctrl_event() -> hci_send_to_channel()

because both functions acquire the same read lock hci_sk_list.lock. This
is also a mainline issue because the qrwlock implementation is writer
fair (the traditional rwlock implementation is reader biased).

To avoid the deadlock there is now __hci_send_to_channel() which expects
the readlock to be held.

Cc: Marcel Holtmann <marcel@holtmann.org>
Cc: Johan Hedberg <redacted>
Fixes: 38ceaa00d02d ("Bluetooth: Add support for sending MGMT commands and events to monitor")
Reported-by: Mart van de Wege <redacted>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
 net/bluetooth/hci_sock.c | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)

diff --git a/net/bluetooth/hci_sock.c b/net/bluetooth/hci_sock.c
index 638bf0e1a2e3..f1ee820d871c 100644
--- a/net/bluetooth/hci_sock.c
+++ b/net/bluetooth/hci_sock.c

@@ -251,15 +251,13 @@ void hci_send_to_sock(struct hci_dev *hdev, struct sk_buff *skb)
 }
 
 /* Send frame to sockets with specific channel */
-void hci_send_to_channel(unsigned short channel, struct sk_buff *skb,
-			 int flag, struct sock *skip_sk)
+static void __hci_send_to_channel(unsigned short channel, struct sk_buff *skb,
+				  int flag, struct sock *skip_sk)
 {
 	struct sock *sk;
 
 	BT_DBG("channel %u len %d", channel, skb->len);
 
-	read_lock(&hci_sk_list.lock);
-
 	sk_for_each(sk, &hci_sk_list.head) {
 		struct sk_buff *nskb;
 

@@ -285,6 +283,13 @@ void hci_send_to_channel(unsigned short channel, struct sk_buff *skb,
 			kfree_skb(nskb);
 	}
 
+}
+
+void hci_send_to_channel(unsigned short channel, struct sk_buff *skb,
+			 int flag, struct sock *skip_sk)
+{
+	read_lock(&hci_sk_list.lock);
+	__hci_send_to_channel(channel, skb, flag, skip_sk);
 	read_unlock(&hci_sk_list.lock);
 }
 

@@ -388,8 +393,8 @@ void hci_send_monitor_ctrl_event(struct hci_dev *hdev, u16 event,
 		hdr->index = index;
 		hdr->len = cpu_to_le16(skb->len - HCI_MON_HDR_SIZE);
 
-		hci_send_to_channel(HCI_CHANNEL_MONITOR, skb,
-				    HCI_SOCK_TRUSTED, NULL);
+		__hci_send_to_channel(HCI_CHANNEL_MONITOR, skb,
+				      HCI_SOCK_TRUSTED, NULL);
 		kfree_skb(skb);
 	}
 

-- 
2.14.1

Confirm that 4.11.12-rt14 with this fix boots normally.

On Thu, 21 Sep 2017 15:51:23 +0200
Sebastian Andrzej Siewior [off-list ref] wrote:

Hi Sebastian,

Mart reported a deadlock in -RT in the call path:
 hci_send_monitor_ctrl_event() -> hci_send_to_channel()

because both functions acquire the same read lock hci_sk_list.lock. This
is also a mainline issue because the qrwlock implementation is writer
fair (the traditional rwlock implementation is reader biased).

To avoid the deadlock there is now __hci_send_to_channel() which expects
the readlock to be held.

Cc: Marcel Holtmann <marcel@holtmann.org>
Cc: Johan Hedberg <redacted>
Fixes: 38ceaa00d02d ("Bluetooth: Add support for sending MGMT commands and events to monitor")
Reported-by: Mart van de Wege <redacted>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
net/bluetooth/hci_sock.c | 17 +++++++++++------
1 file changed, 11 insertions(+), 6 deletions(-)

patch has been applied to bluetooth-next tree.

Regards

Marcel