[PATCH] net bridge: add null pointer check, fix panic

Subsystems: ethernet bridge, networking [general], the rest

STALE4647d

7 messages, 3 authors, 2013-11-11 · open the first message on its own page

[PATCH] net bridge: add null pointer check, fix panic

From: xiaoming gao <hidden>
Date: 2013-06-20 03:08:19

From: newtongao <redacted>
Date: Wed, 19 Jun 2013 14:58:33 +0800
Subject: [PATCH] net bridge: add null pointer check,fix panic

in kernel 3.0, br_port_get_rcu() may return NULL when network interface be deleting from bridge,
but in function br_handle_frame and br_handle_local_finish, the pointer didn't be checked before using,
so all br_port_get_rcu callers must do null check,or there occurs the null pointer panic.

kernel 3.4 also has this bug,i have verified.
mainline kernel still did not  check br_port_get_rcu()'s NULL pointer, but i have not tested it yet.

method to reproduce null pointer panic:
1. in function del_nbp,amplify the gap between "dev->priv_flags &= ~IFF_BRIDGE_PORT;" and "netdev_rx_handler_unregister(dev);" , such as add new line "while(1);".
2. create net bridge testbr and bind some network interface(e.g. testif) on it.
3. send packets to testif frequetly,such as ping testif.
4. brctl delif testbr testif.

backtrace:
[1002130.426411] FS:  00007f2235153700(0000) GS:ffff88007b020000(0000) knlGS:0000000000000000
[1002130.524656] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[1002130.594931] CR2: 0000000000000021 CR3: 0000000001dc7000 CR4: 0000000000002660
[1002130.681798] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[1002130.768654] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[1002130.855525] Process ksoftirqd/1 (pid: 9, threadinfo ffff880062642000, task ffff880062640800)
[1002130.957868] Stack:
[1002130.983758]  0000000080000000 ffff88007bc06a40 ffff88005e3c6c80 ffffffff81901d10
[1002131.073887]  ffff88005efc2000 0000000000000001 ffff880062643bf0 ffffffff818497d9
[1002131.163945]  ffff88006147e090 0000000000000020 ffff88007bc06a40 ffff88005e3c6c80
[1002131.254023] Call Trace:
[1002131.285086]  [<ffffffff81901d10>] ? br_handle_frame_finish+0x2b0/0x2b0
[1002131.364677]  [<ffffffff818497d9>] __netif_receive_skb+0x109/0x4a0
[1002131.439081]  [<ffffffff81849b12>] __netif_receive_skb+0x442/0x4a0
[1002131.513493]  [<ffffffff810e3a6e>] ? __kmalloc_node+0x3e/0x50
[1002131.582744]  [<ffffffff8184b7d8>] netif_receive_skb+0x78/0x80
[1002131.653053]  [<ffffffff8184b928>] napi_skb_finish+0x48/0x60
[1002131.721269]  [<ffffffff8184bf1d>] napi_gro_receive+0xfd/0x130
[1002131.791551]  [<ffffffff8192b2f6>] vlan_gro_receive+0x16/0x20
[1002131.860796]  [<ffffffff816a41d1>] igb_poll+0x6f1/0xdc0
[1002131.923871]  [<ffffffff8196c0d5>] ? __schedule+0x2e5/0x810
[1002131.991071]  [<ffffffff8196e38b>] ? _raw_spin_lock_irq+0xb/0x30
[1002132.063421]  [<ffffffff8184c0c1>] net_rx_action+0xa1/0x1d0
[1002132.130619]  [<ffffffff81057449>] __do_softirq+0x99/0x130
[1002132.196803]  [<ffffffff8105759a>] run_ksoftirqd+0xba/0x170
[1002132.264000]  [<ffffffff810574e0>] ? __do_softirq+0x130/0x130
[1002132.333258]  [<ffffffff8106d306>] kthread+0x96/0xa0
[1002132.393224]  [<ffffffff8196ffe4>] kernel_thread_helper+0x4/0x10
[1002132.465580]  [<ffffffff8196f0f6>] ? int_ret_from_sys_call+0x7/0x1b
[1002132.541016]  [<ffffffff8196e8a1>] ? retint_restore_args+0x5/0x6
[1002132.613368]  [<ffffffff8196ffe0>] ? gs_change+0x13/0x13
[1002132.677466] Code: 02 f6 81 a5 01 00 00 40 48 8b 81 e0 02 00 00 89 d6 4c 0f 45 f0 48 8b 05 50 76 20 00 66 33 70 04 66 f7 c6 ff f0 0f 84 d0 00 00 00
[1002132.835500]  0f b6 46 21 3c 02 74 50 3c 03 0f 85 60 ff ff ff 48 8b 05 e1
[1002132.920864] RIP  [<ffffffff81901e17>] br_handle_frame+0x107/0x260
[1002132.995296]  RSP <ffff880062643b50>
[1002133.038735] CR2: 0000000000000021

Signed-off-by: Xiaoming Gao <redacted>
---
 net/bridge/br_input.c |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)
diff --git a/net/bridge/br_input.c b/net/bridge/br_input.c
index f06ee39..c8b365f 100644
--- a/net/bridge/br_input.c
+++ b/net/bridge/br_input.c
@@ -122,7 +122,8 @@ static int br_handle_local_finish(struct sk_buff *skb)
 {
 	struct net_bridge_port *p = br_port_get_rcu(skb->dev);
 
-	br_fdb_update(p->br, p, eth_hdr(skb)->h_source);
+	if (p)
+		br_fdb_update(p->br, p, eth_hdr(skb)->h_source);
 	return 0;	 /* process further */
 }
 
@@ -160,6 +161,8 @@ rx_handler_result_t br_handle_frame(struct sk_buff **pskb)
 		return RX_HANDLER_CONSUMED;
 
 	p = br_port_get_rcu(skb->dev);
+	if (!p)
+		goto drop;
 
 	if (unlikely(is_link_local(dest))) {
 		/* Pause frames shouldn't be passed up by driver anyway */
-- 
1.7.1


Re: [PATCH] net bridge: add null pointer check, fix panic

From: Eric Dumazet <hidden>
Date: 2013-06-20 04:55:35

On Thu, 2013-06-20 at 11:08 +0800, xiaoming gao wrote:
From: newtongao <redacted>
Date: Wed, 19 Jun 2013 14:58:33 +0800
Subject: [PATCH] net bridge: add null pointer check,fix panic

in kernel 3.0, br_port_get_rcu() may return NULL when network interface be deleting from bridge,
but in function br_handle_frame and br_handle_local_finish, the pointer didn't be checked before using,
so all br_port_get_rcu callers must do null check,or there occurs the null pointer panic.

kernel 3.4 also has this bug,i have verified.
mainline kernel still did not  check br_port_get_rcu()'s NULL pointer, but i have not tested it yet.
Please check current version before sending a patch.

This was most probably fixed in commit 00cfec37484761a44
("net: add a synchronize_net() in netdev_rx_handler_unregister()")

Thanks

Re: [PATCH] net bridge: add null pointer check, fix panic

From: xiaoming gao <hidden>
Date: 2013-06-20 07:01:07

Eric Dumazet said, at 2013-6-20 12:55:
On Thu, 2013-06-20 at 11:08 +0800, xiaoming gao wrote:
quoted
From: newtongao <redacted>
Date: Wed, 19 Jun 2013 14:58:33 +0800
Subject: [PATCH] net bridge: add null pointer check,fix panic

in kernel 3.0, br_port_get_rcu() may return NULL when network interface be deleting from bridge,
but in function br_handle_frame and br_handle_local_finish, the pointer didn't be checked before using,
so all br_port_get_rcu callers must do null check,or there occurs the null pointer panic.

kernel 3.4 also has this bug,i have verified.
mainline kernel still did not  check br_port_get_rcu()'s NULL pointer, but i have not tested it yet.
Please check current version before sending a patch.

This was most probably fixed in commit 00cfec37484761a44
("net: add a synchronize_net() in netdev_rx_handler_unregister()")

Thanks
HI Eric
the problem is as follow:
br_del_if()-->del_nbp():

list_del_rcu(&p->list);
dev->priv_flags &= ~IFF_BRIDGE_PORT;

------>at this point, the nic be deleting still have rx_handler , so , may in br_handle_frame()
------>br_port_exists() will return false,so br_get_port_rcu() will return NULL
------>so in br_handle_frame , there will be a null panic.

netdev_rx_handler_unregister(dev);
synchronize_net();


i have checked commit 00cfec37484761a44, i think it didn't fix this bug..

thanks.

Re: [PATCH] net bridge: add null pointer check, fix panic

From: Eric Dumazet <hidden>
Date: 2013-06-20 07:29:10

On Thu, 2013-06-20 at 15:00 +0800, xiaoming gao wrote:
HI Eric
the problem is as follow:
br_del_if()-->del_nbp():

list_del_rcu(&p->list);
dev->priv_flags &= ~IFF_BRIDGE_PORT;

------>at this point, the nic be deleting still have rx_handler , so , may in br_handle_frame()
------>br_port_exists() will return false,so br_get_port_rcu() will return NULL
------>so in br_handle_frame , there will be a null panic.

netdev_rx_handler_unregister(dev);
synchronize_net();
This code is no longer like that in current tree.
Check commit 4a0b5ec12f0ffc3024616e6dc62cf8a04c54edcd
("bridge: remove a redundant synchronize_net()")

i have checked commit 00cfec37484761a44, i think it didn't fix this bug..
I claim adding NULL tests is not needed in the fast path, it was clearly
stated in the changelog.

I would change the dismantle path to make sure br_get_port_rcu() does
not return NULL in the fast path, and remove the test on FF_BRIDGE_PORT

Try something like that :
diff --git a/net/bridge/br_input.c b/net/bridge/br_input.c
index 1b8b8b8..2edfb80 100644
--- a/net/bridge/br_input.c
+++ b/net/bridge/br_input.c
@@ -60,7 +60,7 @@ static int br_pass_frame_up(struct sk_buff *skb)
 int br_handle_frame_finish(struct sk_buff *skb)
 {
 	const unsigned char *dest = eth_hdr(skb)->h_dest;
-	struct net_bridge_port *p = br_port_get_rcu(skb->dev);
+	struct net_bridge_port *p = __br_port_get_rcu(skb->dev);
 	struct net_bridge *br;
 	struct net_bridge_fdb_entry *dst;
 	struct net_bridge_mdb_entry *mdst;
@@ -68,7 +68,7 @@ int br_handle_frame_finish(struct sk_buff *skb)
 	bool unicast = true;
 	u16 vid = 0;
 
-	if (!p || p->state == BR_STATE_DISABLED)
+	if (p->state == BR_STATE_DISABLED)
 		goto drop;
 
 	if (!br_allowed_ingress(p->br, nbp_get_vlan_info(p), skb, &vid))
@@ -142,7 +142,7 @@ drop:
 /* note: already called with rcu_read_lock */
 static int br_handle_local_finish(struct sk_buff *skb)
 {
-	struct net_bridge_port *p = br_port_get_rcu(skb->dev);
+	struct net_bridge_port *p = __br_port_get_rcu(skb->dev);
 	u16 vid = 0;
 
 	br_vlan_get_tag(skb, &vid);
@@ -172,7 +172,7 @@ rx_handler_result_t br_handle_frame(struct sk_buff **pskb)
 	if (!skb)
 		return RX_HANDLER_CONSUMED;
 
-	p = br_port_get_rcu(skb->dev);
+	p = __br_port_get_rcu(skb->dev);
 
 	if (unlikely(is_link_local_ether_addr(dest))) {
 		/*
diff --git a/net/bridge/br_private.h b/net/bridge/br_private.h
index 3be89b3..9fdd467 100644
--- a/net/bridge/br_private.h
+++ b/net/bridge/br_private.h
@@ -184,6 +184,11 @@ struct net_bridge_port
 
 #define br_port_exists(dev) (dev->priv_flags & IFF_BRIDGE_PORT)
 
+static inline struct net_bridge_port *__br_port_get_rcu(const struct net_device *dev)
+{
+	return rcu_dereference(dev->rx_handler_data);
+}
+
 static inline struct net_bridge_port *br_port_get_rcu(const struct net_device *dev)
 {
 	struct net_bridge_port *port =

Re: [PATCH] net bridge: add null pointer check, fix panic

From: xiaoming gao <hidden>
Date: 2013-06-20 07:48:01

Eric Dumazet said, at 2013-6-20 15:29:
quoted hunk
On Thu, 2013-06-20 at 15:00 +0800, xiaoming gao wrote:
quoted
HI Eric
the problem is as follow:
br_del_if()-->del_nbp():

list_del_rcu(&p->list);
dev->priv_flags &= ~IFF_BRIDGE_PORT;

------>at this point, the nic be deleting still have rx_handler , so , may in br_handle_frame()
------>br_port_exists() will return false,so br_get_port_rcu() will return NULL
------>so in br_handle_frame , there will be a null panic.

netdev_rx_handler_unregister(dev);
synchronize_net();
This code is no longer like that in current tree.
Check commit 4a0b5ec12f0ffc3024616e6dc62cf8a04c54edcd
("bridge: remove a redundant synchronize_net()")
quoted

i have checked commit 00cfec37484761a44, i think it didn't fix this bug..
I claim adding NULL tests is not needed in the fast path, it was clearly
stated in the changelog.

I would change the dismantle path to make sure br_get_port_rcu() does
not return NULL in the fast path, and remove the test on FF_BRIDGE_PORT

Try something like that :
diff --git a/net/bridge/br_input.c b/net/bridge/br_input.c
index 1b8b8b8..2edfb80 100644
--- a/net/bridge/br_input.c
+++ b/net/bridge/br_input.c
@@ -60,7 +60,7 @@ static int br_pass_frame_up(struct sk_buff *skb)
 int br_handle_frame_finish(struct sk_buff *skb)
 {
 	const unsigned char *dest = eth_hdr(skb)->h_dest;
-	struct net_bridge_port *p = br_port_get_rcu(skb->dev);
+	struct net_bridge_port *p = __br_port_get_rcu(skb->dev);
 	struct net_bridge *br;
 	struct net_bridge_fdb_entry *dst;
 	struct net_bridge_mdb_entry *mdst;
@@ -68,7 +68,7 @@ int br_handle_frame_finish(struct sk_buff *skb)
 	bool unicast = true;
 	u16 vid = 0;
 
-	if (!p || p->state == BR_STATE_DISABLED)
+	if (p->state == BR_STATE_DISABLED)
 		goto drop;
 
 	if (!br_allowed_ingress(p->br, nbp_get_vlan_info(p), skb, &vid))
@@ -142,7 +142,7 @@ drop:
 /* note: already called with rcu_read_lock */
 static int br_handle_local_finish(struct sk_buff *skb)
 {
-	struct net_bridge_port *p = br_port_get_rcu(skb->dev);
+	struct net_bridge_port *p = __br_port_get_rcu(skb->dev);
 	u16 vid = 0;
 
 	br_vlan_get_tag(skb, &vid);
@@ -172,7 +172,7 @@ rx_handler_result_t br_handle_frame(struct sk_buff **pskb)
 	if (!skb)
 		return RX_HANDLER_CONSUMED;
 
-	p = br_port_get_rcu(skb->dev);
+	p = __br_port_get_rcu(skb->dev);
 
 	if (unlikely(is_link_local_ether_addr(dest))) {
 		/*
diff --git a/net/bridge/br_private.h b/net/bridge/br_private.h
index 3be89b3..9fdd467 100644
--- a/net/bridge/br_private.h
+++ b/net/bridge/br_private.h
@@ -184,6 +184,11 @@ struct net_bridge_port
 
 #define br_port_exists(dev) (dev->priv_flags & IFF_BRIDGE_PORT)
 
+static inline struct net_bridge_port *__br_port_get_rcu(const struct net_device *dev)
+{
+	return rcu_dereference(dev->rx_handler_data);
+}
+
 static inline struct net_bridge_port *br_port_get_rcu(const struct net_device *dev)
 {
 	struct net_bridge_port *port =
if you remove the test on FF_BRIDGE_PORT, and br_port_get_rcu never returns NULL, the problem is fixed.
but the codes in mainline is still bugged, am i right??
by the way, kernel-stable 3.0 and 3.4 tree also have this bug, and is very easily to reproduce.

Re: [PATCH] net bridge: add null pointer check, fix panic

From: Eric Dumazet <hidden>
Date: 2013-06-20 08:14:10

On Thu, 2013-06-20 at 15:47 +0800, xiaoming gao wrote:
if you remove the test on FF_BRIDGE_PORT, and br_port_get_rcu never returns NULL, the problem is fixed.
but the codes in mainline is still bugged, am i right??
by the way, kernel-stable 3.0 and 3.4 tree also have this bug, and is very easily to reproduce.
Commit 00cfec37484761a4 has to be backported anyway, if not already
backported.

I would rather fix things properly, instead of adding defensive code.

Re: [PATCH] net bridge: add null pointer check, fix panic

From: Alexander Y. Fomichev <hidden>
Date: 2013-11-11 10:27:32

On Thu, Jun 20, 2013 at 11:29 AM, Eric Dumazet [off-list ref] wrote:
quoted hunk
On Thu, 2013-06-20 at 15:00 +0800, xiaoming gao wrote:
quoted
HI Eric
the problem is as follow:
br_del_if()-->del_nbp():

list_del_rcu(&p->list);
dev->priv_flags &= ~IFF_BRIDGE_PORT;

------>at this point, the nic be deleting still have rx_handler , so , may in br_handle_frame()
------>br_port_exists() will return false,so br_get_port_rcu() will return NULL
------>so in br_handle_frame , there will be a null panic.

netdev_rx_handler_unregister(dev);
synchronize_net();
This code is no longer like that in current tree.
Check commit 4a0b5ec12f0ffc3024616e6dc62cf8a04c54edcd
("bridge: remove a redundant synchronize_net()")
quoted

i have checked commit 00cfec37484761a44, i think it didn't fix this bug..
I claim adding NULL tests is not needed in the fast path, it was clearly
stated in the changelog.

I would change the dismantle path to make sure br_get_port_rcu() does
not return NULL in the fast path, and remove the test on FF_BRIDGE_PORT
Try something like that :
diff --git a/net/bridge/br_input.c b/net/bridge/br_input.c
index 1b8b8b8..2edfb80 100644
--- a/net/bridge/br_input.c
+++ b/net/bridge/br_input.c
@@ -60,7 +60,7 @@ static int br_pass_frame_up(struct sk_buff *skb)
 int br_handle_frame_finish(struct sk_buff *skb)
 {
        const unsigned char *dest = eth_hdr(skb)->h_dest;
-       struct net_bridge_port *p = br_port_get_rcu(skb->dev);
+       struct net_bridge_port *p = __br_port_get_rcu(skb->dev);
        struct net_bridge *br;
        struct net_bridge_fdb_entry *dst;
        struct net_bridge_mdb_entry *mdst;
@@ -68,7 +68,7 @@ int br_handle_frame_finish(struct sk_buff *skb)
        bool unicast = true;
        u16 vid = 0;

-       if (!p || p->state == BR_STATE_DISABLED)
+       if (p->state == BR_STATE_DISABLED)
                goto drop;

        if (!br_allowed_ingress(p->br, nbp_get_vlan_info(p), skb, &vid))
@@ -142,7 +142,7 @@ drop:
 /* note: already called with rcu_read_lock */
 static int br_handle_local_finish(struct sk_buff *skb)
 {
-       struct net_bridge_port *p = br_port_get_rcu(skb->dev);
+       struct net_bridge_port *p = __br_port_get_rcu(skb->dev);
        u16 vid = 0;

        br_vlan_get_tag(skb, &vid);
@@ -172,7 +172,7 @@ rx_handler_result_t br_handle_frame(struct sk_buff **pskb)
        if (!skb)
                return RX_HANDLER_CONSUMED;

-       p = br_port_get_rcu(skb->dev);
+       p = __br_port_get_rcu(skb->dev);

        if (unlikely(is_link_local_ether_addr(dest))) {
                /*
diff --git a/net/bridge/br_private.h b/net/bridge/br_private.h
index 3be89b3..9fdd467 100644
--- a/net/bridge/br_private.h
+++ b/net/bridge/br_private.h
@@ -184,6 +184,11 @@ struct net_bridge_port

 #define br_port_exists(dev) (dev->priv_flags & IFF_BRIDGE_PORT)

+static inline struct net_bridge_port *__br_port_get_rcu(const struct net_device *dev)
+{
+       return rcu_dereference(dev->rx_handler_data);
+}
+
 static inline struct net_bridge_port *br_port_get_rcu(const struct net_device *dev)
 {
        struct net_bridge_port *port =


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
I claim adding NULL tests is not needed in the fast path, it was clearly
stated in the changelog.
Hello,

This commit makes trouble for current STP.
Two days ago i tried to switch to 3.10.18 and i've caught "bad magic"
on uninitialized br->lock:
./net/bridge/br_stp_bpdu.c +158 in br_stp_rcv (trace attached):

        p = br_port_get_rcu(dev);

        br = p->br;
        spin_lock(&br->lock); <- here
-----------------------------------------------

 Bisect pointed to this commit
(linux-stable: 960b8e5018a552f62cfbc0dfe94be7b6ba178f13)
(mainline 716ec052d2280d511e10e90ad54a86f5b5d4dcc2)

As far as i can see this happens when:

a) bridge module had been loaded but there was no bridge interface,
br->lock had not been initialized.
b) interface had been in promiscuous mod (tcpdump)
c) stp broadcasts 01:80:c2:00:00:00 coming to this iface
   (llc_rcv drops PACKET_OTHERHOST to protect us in promiscuous mode
but seems like not a broadcasts)
d) and finally rx_handler_data had been initialised for this interface
( by macvlan in my case)

It seems like STP needs its own IFF_BRIDGE_PORT check.
probably an easiest option to check it in br_stp_rcv as before (or
probably in llc_rcv)...

-- 
Best regards.
       Alexander Y. Fomichev [off-list ref]
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help