[PATCH net] ppp: Fix a scheduling-while-atomic bug in del_chan

Subsystems: networking drivers, the rest

STALE3318d

7 messages, 3 authors, 2017-08-07 · open the first message on its own page

[PATCH net] ppp: Fix a scheduling-while-atomic bug in del_chan

From: <hidden>
Date: 2017-07-31 10:07:54

From: Gao Feng <redacted>

The PPTP set the pptp_sock_destruct as the sock's sk_destruct, it would
trigger this bug when __sk_free is invoked in atomic context, because of
the call path pptp_sock_destruct->del_chan->synchronize_rcu.

Now move the synchronize_rcu to pptp_release from del_chan. This is the
only one case which would free the sock and need the synchronize_rcu.

The following is the panic I met with kernel 3.3.8, but this issue should
exist in current kernel too according to the codes.

BUG: scheduling while atomic
__schedule_bug+0x5e/0x64
__schedule+0x55/0x580
? ppp_unregister_channel+0x1cd5/0x1de0 [ppp_generic]
? dev_hard_start_xmit+0x423/0x530
? sch_direct_xmit+0x73/0x170
__cond_resched+0x16/0x30
_cond_resched+0x22/0x30
wait_for_common+0x18/0x110
? call_rcu_bh+0x10/0x10
wait_for_completion+0x12/0x20
wait_rcu_gp+0x34/0x40
? wait_rcu_gp+0x40/0x40
synchronize_sched+0x1e/0x20
0xf8417298
0xf8417484
? sock_queue_rcv_skb+0x109/0x130
__sk_free+0x16/0x110
? udp_queue_rcv_skb+0x1f2/0x290
sk_free+0x16/0x20
__udp4_lib_rcv+0x3b8/0x650

Signed-off-by: Gao Feng <redacted>
---
 drivers/net/ppp/pptp.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/net/ppp/pptp.c b/drivers/net/ppp/pptp.c
index eac499c..6dde9a0 100644
--- a/drivers/net/ppp/pptp.c
+++ b/drivers/net/ppp/pptp.c
@@ -131,7 +131,6 @@ static void del_chan(struct pppox_sock *sock)
 	clear_bit(sock->proto.pptp.src_addr.call_id, callid_bitmap);
 	RCU_INIT_POINTER(callid_sock[sock->proto.pptp.src_addr.call_id], NULL);
 	spin_unlock(&chan_lock);
-	synchronize_rcu();
 }
 
 static int pptp_xmit(struct ppp_channel *chan, struct sk_buff *skb)
@@ -520,6 +519,7 @@ static int pptp_release(struct socket *sock)
 
 	po = pppox_sk(sk);
 	del_chan(po);
+	synchronize_rcu();
 
 	pppox_unbind_sock(sk);
 	sk->sk_state = PPPOX_DEAD;
-- 
1.9.1

Re: [PATCH net] ppp: Fix a scheduling-while-atomic bug in del_chan

From: David Miller <davem@davemloft.net>
Date: 2017-08-01 04:59:49

From: gfree.wind@vip.163.com
Date: Mon, 31 Jul 2017 18:07:38 +0800
From: Gao Feng <redacted>

The PPTP set the pptp_sock_destruct as the sock's sk_destruct, it would
trigger this bug when __sk_free is invoked in atomic context, because of
the call path pptp_sock_destruct->del_chan->synchronize_rcu.

Now move the synchronize_rcu to pptp_release from del_chan. This is the
only one case which would free the sock and need the synchronize_rcu.

The following is the panic I met with kernel 3.3.8, but this issue should
exist in current kernel too according to the codes.
 ...
Signed-off-by: Gao Feng <redacted>
Applied, thanks.

Re: [PATCH net] ppp: Fix a scheduling-while-atomic bug in del_chan

From: Cong Wang <hidden>
Date: 2017-08-01 20:39:43

On Mon, Jul 31, 2017 at 3:07 AM,  [off-list ref] wrote:
From: Gao Feng <redacted>

The PPTP set the pptp_sock_destruct as the sock's sk_destruct, it would
trigger this bug when __sk_free is invoked in atomic context, because of
the call path pptp_sock_destruct->del_chan->synchronize_rcu.

Now move the synchronize_rcu to pptp_release from del_chan. This is the
only one case which would free the sock and need the synchronize_rcu.
I don't understand the last part.
From my understanding, this RCU is supposed to protect the pppox_sock
pointers in 'callid_sock' which could be NULL'ed in del_chan(). And the
pppox_sock is freed when the last refcnt is gone, that is, when sock
dctor is called. pptp_release() is ONLY called when the fd in user-space
is gone, not necessarily the last refcnt.

Re: [PATCH net] ppp: Fix a scheduling-while-atomic bug in del_chan

From: Cong Wang <hidden>
Date: 2017-08-02 17:13:58

Hi, Gao

On Tue, Aug 1, 2017 at 1:39 PM, Cong Wang [off-list ref] wrote:
From my understanding, this RCU is supposed to protect the pppox_sock
pointers in 'callid_sock' which could be NULL'ed in del_chan(). And the
pppox_sock is freed when the last refcnt is gone, that is, when sock
dctor is called. pptp_release() is ONLY called when the fd in user-space
is gone, not necessarily the last refcnt.
Your commit is probably not the right fix. Can you try the following fix?
diff --git a/drivers/net/ppp/pptp.c b/drivers/net/ppp/pptp.c
index 6dde9a0cfe76..e75bb95c107f 100644
--- a/drivers/net/ppp/pptp.c
+++ b/drivers/net/ppp/pptp.c
@@ -519,7 +519,6 @@ static int pptp_release(struct socket *sock)

        po = pppox_sk(sk);
        del_chan(po);
-       synchronize_rcu();

        pppox_unbind_sock(sk);
        sk->sk_state = PPPOX_DEAD;
@@ -564,6 +563,7 @@ static int pptp_create(struct net *net, struct
socket *sock, int kern)
        sk->sk_family      = PF_PPPOX;
        sk->sk_protocol    = PX_PROTO_PPTP;
        sk->sk_destruct    = pptp_sock_destruct;
+       sock_set_flag(sk, SOCK_RCU_FREE);

        po = pppox_sk(sk);
        opt = &po->proto.pptp;

Re:Re: [PATCH net] ppp: Fix a scheduling-while-atomic bug in del_chan

From: Gao Feng <hidden>
Date: 2017-08-07 01:37:35

At 2017-08-03 01:13:36, "Cong Wang" [off-list ref] wrote:
Hi, Gao

On Tue, Aug 1, 2017 at 1:39 PM, Cong Wang [off-list ref] wrote:
quoted
From my understanding, this RCU is supposed to protect the pppox_sock
pointers in 'callid_sock' which could be NULL'ed in del_chan(). And the
pppox_sock is freed when the last refcnt is gone, that is, when sock
dctor is called. pptp_release() is ONLY called when the fd in user-space
is gone, not necessarily the last refcnt.
Hi Cong,

I am sorry to reply you so late, because I took a short trip recently, and didn't check my emails.

I think the RCU should be supposed to avoid the race between del_chan and lookup_chan.
The synchronize_rcu could make sure if there was one which calls lookup_chan in this period, it would be finished and the sock refcnt is increased if necessary.

So I think it is ok to invoke sock_put directly without SOCK_RCU_FREE, because the lookup_chan caller has already hold the sock refcnt, 

Best Regards
Feng
quoted hunk
Your commit is probably not the right fix. Can you try the following fix?
diff --git a/drivers/net/ppp/pptp.c b/drivers/net/ppp/pptp.c
index 6dde9a0cfe76..e75bb95c107f 100644
--- a/drivers/net/ppp/pptp.c
+++ b/drivers/net/ppp/pptp.c
@@ -519,7 +519,6 @@ static int pptp_release(struct socket *sock)

       po = pppox_sk(sk);
       del_chan(po);
-       synchronize_rcu();

       pppox_unbind_sock(sk);
       sk->sk_state = PPPOX_DEAD;
@@ -564,6 +563,7 @@ static int pptp_create(struct net *net, struct
socket *sock, int kern)
       sk->sk_family      = PF_PPPOX;
       sk->sk_protocol    = PX_PROTO_PPTP;
       sk->sk_destruct    = pptp_sock_destruct;
+       sock_set_flag(sk, SOCK_RCU_FREE);

       po = pppox_sk(sk);
       opt = &po->proto.pptp;

Re: Re: [PATCH net] ppp: Fix a scheduling-while-atomic bug in del_chan

From: Cong Wang <hidden>
Date: 2017-08-07 17:17:24

On Sun, Aug 6, 2017 at 6:32 PM, Gao Feng [off-list ref] wrote:
I think the RCU should be supposed to avoid the race between del_chan and lookup_chan.
More precisely, it is callid_sock which is protected by RCU.

Unless I miss any other code path, pptp_exit_module() is
problematic too, I don't think it can just vfree() the whole thing.

The synchronize_rcu could make sure if there was one which calls lookup_chan in this period, it would be finished and the sock refcnt is increased if necessary.

So I think it is ok to invoke sock_put directly without SOCK_RCU_FREE, because the lookup_chan caller has already hold the sock refcnt,
If you mean the sock_hold() inside lookup_chan(), no,
it doesn't help because we already dereference the sock
before it.

Also, lookup_chan_dst() does not have a refcnt, I don't
find any code preventing it deref'ing other sock in callid_sock
than the calling one.

Re: Re: [PATCH net] ppp: Fix a scheduling-while-atomic bug in del_chan

From: Cong Wang <hidden>
Date: 2017-08-07 17:34:55

On Mon, Aug 7, 2017 at 10:17 AM, Cong Wang [off-list ref] wrote:
Unless I miss any other code path, pptp_exit_module() is
problematic too, I don't think it can just vfree() the whole thing.
This path should be fine, because:

1. sock holds a refcnt to module via proto->owner
2. gre_del_protocol() already waits for readers
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help