[PATCH net v1 0/3] add support for skb with sk ref cloning

Hello,

this series tries to fix a long standing problem in the CAN echo SKB
handling. The problem shows up if an echo SKB for a SKB that references
an already closed socket is created.

It looks like the mac80211/tx.c has the same problem, see RFC patch 3
for details.

regards,
Oleksij

Oleksij Rempel (3):
  skbuff: skb_clone_sk_optional(): add function to always clone a skb
    and increase refcount on sk if valid
  can: fix ref count warning if socket was closed before skb was cloned
  [RFC] mac80211: ieee80211_store_ack_skb(): make use of
    skb_clone_sk_optional()

 include/linux/can/skb.h   |  3 +--
 include/linux/skbuff.h    |  1 +
 net/can/af_can.c          |  6 +++---
 net/can/j1939/main.c      |  3 +--
 net/can/j1939/socket.c    |  3 +--
 net/can/j1939/transport.c |  4 +---
 net/core/skbuff.c         | 27 +++++++++++++++++++++++++++
 net/mac80211/tx.c         |  6 +-----
 8 files changed, 36 insertions(+), 17 deletions(-)

-- 
2.29.2

This code is trying to clone the skb with optional skb->sk. But this
will fail to clone the skb if socket was closed just after the skb was
pushed into the networking stack.

Fixes: a7528198add8 ("mac80211: support control port TX status reporting")
Signed-off-by: Oleksij Rempel <o.rempel@pengutronix.de>
---
 net/mac80211/tx.c | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/net/mac80211/tx.c b/net/mac80211/tx.c
index 5d06de61047a..c0dd326db10d 100644
--- a/net/mac80211/tx.c
+++ b/net/mac80211/tx.c

@@ -2439,11 +2439,7 @@ static u16 ieee80211_store_ack_skb(struct ieee80211_local *local,
 	struct sk_buff *ack_skb;
 	u16 info_id = 0;
 
-	if (skb->sk)
-		ack_skb = skb_clone_sk(skb);
-	else
-		ack_skb = skb_clone(skb, GFP_ATOMIC);
-
+	ack_skb = skb_clone_sk_optional(skb);
 	if (ack_skb) {
 		unsigned long flags;
 		int id;

-- 
2.29.2

There already the function skb_clone_sk(), which clones the skb, but
only if the sk is valid.

There are several users in the networking stack, which always need a
clone of a skb with the sk refcount incremented (but only if the sk is
valid). This patch adds such a function.

Signed-off-by: Oleksij Rempel <o.rempel@pengutronix.de>
---
 include/linux/skbuff.h |  1 +
 net/core/skbuff.c      | 27 +++++++++++++++++++++++++++
 2 files changed, 28 insertions(+)

diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index 6d0a33d1c0db..99d552017508 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h

@@ -3874,6 +3874,7 @@ static inline void skb_metadata_clear(struct sk_buff *skb)
 	skb_metadata_set(skb, 0);
 }
 
+struct sk_buff *skb_clone_sk_optional(struct sk_buff *skb);
 struct sk_buff *skb_clone_sk(struct sk_buff *skb);
 
 #ifdef CONFIG_NETWORK_PHY_TIMESTAMPING

diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 545a472273a5..97341f173fb0 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c

@@ -4671,6 +4671,33 @@ struct sk_buff *sock_dequeue_err_skb(struct sock *sk)
 }
 EXPORT_SYMBOL(sock_dequeue_err_skb);
 
+/**
+ * skb_clone_sk_optional - create clone of skb, and take reference to socket if
+ *                         socket is referenced in original skb
+ * @skb: the skb to clone
+ *
+ * This function always creates a clone of a buffer. If it that holds a valid
+ * reference on sk_refcnt this is increased.
+ */
+struct sk_buff *skb_clone_sk_optional(struct sk_buff *skb)
+{
+	struct sock *sk = skb->sk;
+	struct sk_buff *clone;
+
+	clone = skb_clone(skb, GFP_ATOMIC);
+	if (!clone)
+		return NULL;
+
+	if (!sk || !refcount_inc_not_zero(&sk->sk_refcnt))
+		return clone;
+
+	clone->sk = sk;
+	clone->destructor = sock_efree;
+
+	return clone;
+}
+EXPORT_SYMBOL(skb_clone_sk_optional);
+
 /**
  * skb_clone_sk - create clone of skb, and take reference to socket
  * @skb: the skb to clone

-- 
2.29.2

There are two ref count variables controlling the free()ing of a socket:
- struct sock::sk_refcnt - which is changed by sock_hold()/sock_put()
- struct sock::sk_wmem_alloc - which accounts the memory allocated by
  the skbs in the send path.

If the socket is closed the struct sock::sk_refcnt will finally reach 0
and sk_free() is called. Which then calls
refcount_dec_and_test(&sk->sk_wmem_alloc). If sk_wmem_alloc reaches 0
the socket is actually free()ed.

In case there are still TX skbs on the fly and the socket() is closed,
the struct sock::sk_refcnt reaches 0. In the TX-path the CAN stack
clones an "echo" skb, calls sock_hold() on the original socket and
references it. This produces the following back trace:

| WARNING: CPU: 0 PID: 280 at lib/refcount.c:25 refcount_warn_saturate+0x114/0x134
| refcount_t: addition on 0; use-after-free.
| Modules linked in: coda_vpu(E) v4l2_jpeg(E) videobuf2_vmalloc(E) imx_vdoa(E)
| CPU: 0 PID: 280 Comm: test_can.sh Tainted: G            E     5.11.0-04577-gf8ff6603c617 #203
| Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
| Backtrace:
| [<80bafea4>] (dump_backtrace) from [<80bb0280>] (show_stack+0x20/0x24) r7:00000000 r6:600f0113 r5:00000000 r4:81441220
| [<80bb0260>] (show_stack) from [<80bb593c>] (dump_stack+0xa0/0xc8)
| [<80bb589c>] (dump_stack) from [<8012b268>] (__warn+0xd4/0x114) r9:00000019 r8:80f4a8c2 r7:83e4150c r6:00000000 r5:00000009 r4:80528f90
| [<8012b194>] (__warn) from [<80bb09c4>] (warn_slowpath_fmt+0x88/0xc8) r9:83f26400 r8:80f4a8d1 r7:00000009 r6:80528f90 r5:00000019 r4:80f4a8c2
| [<80bb0940>] (warn_slowpath_fmt) from [<80528f90>] (refcount_warn_saturate+0x114/0x134) r8:00000000 r7:00000000 r6:82b44000 r5:834e5600 r4:83f4d540
| [<80528e7c>] (refcount_warn_saturate) from [<8079a4c8>] (__refcount_add.constprop.0+0x4c/0x50)
| [<8079a47c>] (__refcount_add.constprop.0) from [<8079a57c>] (can_put_echo_skb+0xb0/0x13c)
| [<8079a4cc>] (can_put_echo_skb) from [<8079ba98>] (flexcan_start_xmit+0x1c4/0x230) r9:00000010 r8:83f48610 r7:0fdc0000 r6:0c080000 r5:82b44000 r4:834e5600
| [<8079b8d4>] (flexcan_start_xmit) from [<80969078>] (netdev_start_xmit+0x44/0x70) r9:814c0ba0 r8:80c8790c r7:00000000 r6:834e5600 r5:82b44000 r4:82ab1f00
| [<80969034>] (netdev_start_xmit) from [<809725a4>] (dev_hard_start_xmit+0x19c/0x318) r9:814c0ba0 r8:00000000 r7:82ab1f00 r6:82b44000 r5:00000000 r4:834e5600
| [<80972408>] (dev_hard_start_xmit) from [<809c6584>] (sch_direct_xmit+0xcc/0x264) r10:834e5600 r9:00000000 r8:00000000 r7:82b44000 r6:82ab1f00 r5:834e5600 r4:83f27400
| [<809c64b8>] (sch_direct_xmit) from [<809c6c0c>] (__qdisc_run+0x4f0/0x534)

To fix this problem, we have to take into account, that the socket
technically still there but should not used (by any new skbs) any more.
The function skb_clone_sk_optional() (introduced in the previous patch)
takes care of this. It will only clone the skb, if the sk is set and the
refcount has not reached 0.

Cc: Oliver Hartkopp <socketcan@hartkopp.net>
Cc: Andre Naujoks <redacted>
Cc: Eric Dumazet <edumazet@google.com>
Fixes: 0ae89beb283a ("can: add destructor for self generated skbs")
Signed-off-by: Oleksij Rempel <o.rempel@pengutronix.de>
---
 include/linux/can/skb.h   | 3 +--
 net/can/af_can.c          | 6 +++---
 net/can/j1939/main.c      | 3 +--
 net/can/j1939/socket.c    | 3 +--
 net/can/j1939/transport.c | 4 +---
 5 files changed, 7 insertions(+), 12 deletions(-)

diff --git a/include/linux/can/skb.h b/include/linux/can/skb.h
index 685f34cfba20..bc1af38697a2 100644
--- a/include/linux/can/skb.h
+++ b/include/linux/can/skb.h

@@ -79,13 +79,12 @@ static inline struct sk_buff *can_create_echo_skb(struct sk_buff *skb)
 {
 	struct sk_buff *nskb;
 
-	nskb = skb_clone(skb, GFP_ATOMIC);
+	nskb = skb_clone_sk_optional(skb);
 	if (unlikely(!nskb)) {
 		kfree_skb(skb);
 		return NULL;
 	}
 
-	can_skb_set_owner(nskb, skb->sk);
 	consume_skb(skb);
 	return nskb;
 }

diff --git a/net/can/af_can.c b/net/can/af_can.c
index cce2af10eb3e..9e1bd60e7e1b 100644
--- a/net/can/af_can.c
+++ b/net/can/af_can.c

@@ -251,20 +251,20 @@ int can_send(struct sk_buff *skb, int loop)
 		 * its own. Example: can_raw sockopt CAN_RAW_RECV_OWN_MSGS
 		 * Therefore we have to ensure that skb->sk remains the
 		 * reference to the originating sock by restoring skb->sk
-		 * after each skb_clone() or skb_orphan() usage.
+		 * after each skb_clone() or skb_orphan() usage -
+		 * skb_clone_sk_optional() takes care of that.
 		 */
 
 		if (!(skb->dev->flags & IFF_ECHO)) {
 			/* If the interface is not capable to do loopback
 			 * itself, we do it here.
 			 */
-			newskb = skb_clone(skb, GFP_ATOMIC);
+			newskb = skb_clone_sk_optional(skb);
 			if (!newskb) {
 				kfree_skb(skb);
 				return -ENOMEM;
 			}
 
-			can_skb_set_owner(newskb, skb->sk);
 			newskb->ip_summed = CHECKSUM_UNNECESSARY;
 			newskb->pkt_type = PACKET_BROADCAST;
 		}

diff --git a/net/can/j1939/main.c b/net/can/j1939/main.c
index da3a7a7bcff2..4f6852d48077 100644
--- a/net/can/j1939/main.c
+++ b/net/can/j1939/main.c

@@ -47,12 +47,11 @@ static void j1939_can_recv(struct sk_buff *iskb, void *data)
 	 * the header goes into sockaddr.
 	 * j1939 may not touch the incoming skb in such way
 	 */
-	skb = skb_clone(iskb, GFP_ATOMIC);
+	skb = skb_clone_sk_optional(iskb);
 	if (!skb)
 		return;
 
 	j1939_priv_get(priv);
-	can_skb_set_owner(skb, iskb->sk);
 
 	/* get a pointer to the header of the skb
 	 * the skb payload (pointer) is moved, so that the next skb_data

diff --git a/net/can/j1939/socket.c b/net/can/j1939/socket.c
index 4e4a510d82f9..c1be6c26ff76 100644
--- a/net/can/j1939/socket.c
+++ b/net/can/j1939/socket.c

@@ -305,12 +305,11 @@ static void j1939_sk_recv_one(struct j1939_sock *jsk, struct sk_buff *oskb)
 	if (!j1939_sk_recv_match_one(jsk, oskcb))
 		return;
 
-	skb = skb_clone(oskb, GFP_ATOMIC);
+	skb = skb_clone_sk_optional(oskb);
 	if (!skb) {
 		pr_warn("skb clone failed\n");
 		return;
 	}
-	can_skb_set_owner(skb, oskb->sk);
 
 	skcb = j1939_skb_to_cb(skb);
 	skcb->msg_flags &= ~(MSG_DONTROUTE);

diff --git a/net/can/j1939/transport.c b/net/can/j1939/transport.c
index e09d087ba240..e902557bbe17 100644
--- a/net/can/j1939/transport.c
+++ b/net/can/j1939/transport.c

@@ -1014,12 +1014,10 @@ static int j1939_simple_txnext(struct j1939_session *session)
 	if (!se_skb)
 		return 0;
 
-	skb = skb_clone(se_skb, GFP_ATOMIC);
+	skb = skb_clone_sk_optional(se_skb);
 	if (!skb)
 		return -ENOMEM;
 
-	can_skb_set_owner(skb, se_skb->sk);
-
 	j1939_tp_set_rxtimeout(session, J1939_SIMPLE_ECHO_TIMEOUT_MS);
 
 	ret = j1939_send_one(priv, skb);

-- 
2.29.2

On Mon, 2021-02-22 at 16:12 +0100, Oleksij Rempel wrote:

This code is trying to clone the skb with optional skb->sk. But this
will fail to clone the skb if socket was closed just after the skb was
pushed into the networking stack.

Which IMHO is completely fine. If we then still clone the SKB we can't
do anything with it, since the point would be to ... send it back to the
socket, but it's gone.

Nothing to fix here, I'd think. If you wanted to get a copy back that
gives you the status of the SKB, it should not come as a huge surprise
that you have to keep the socket open for that :)

Having the ACK skb will just make us do more work by handing it back
to skb_complete_wifi_ack() at TX status time, which is supposed to put
it into the socket's error queue, but if the socket is closed ... no
point in that.

johannes

On 22.02.2021 17:30:59, Johannes Berg wrote:

On Mon, 2021-02-22 at 16:12 +0100, Oleksij Rempel wrote:

quoted

This code is trying to clone the skb with optional skb->sk. But this
will fail to clone the skb if socket was closed just after the skb was
pushed into the networking stack.

Which IMHO is completely fine. If we then still clone the SKB we can't
do anything with it, since the point would be to ... send it back to the
socket, but it's gone.

Ok, but why is the skb cloned if there is no socket linked in skb->sk?

| static u16 ieee80211_store_ack_skb(struct ieee80211_local *local,
| 				   struct sk_buff *skb,
| 				   u32 *info_flags,
| 				   u64 *cookie)
| {
| 	struct sk_buff *ack_skb;
| 	u16 info_id = 0;
| 
| 	if (skb->sk)
| 		ack_skb = skb_clone_sk(skb);
| 	else
| 		ack_skb = skb_clone(skb, GFP_ATOMIC);

Looks like this is dead code, since both callers of
ieee80211_store_ack_skb() first check if there is a skb->sk

| 	if (unlikely(!multicast && ((skb->sk &&
| 		     skb_shinfo(skb)->tx_flags & SKBTX_WIFI_STATUS) ||
| 		     ctrl_flags & IEEE80211_TX_CTL_REQ_TX_STATUS)))
| 		info_id = ieee80211_store_ack_skb(local, skb, &info_flags,
| 						  cookie);

Nothing to fix here, I'd think. If you wanted to get a copy back that
gives you the status of the SKB, it should not come as a huge surprise
that you have to keep the socket open for that :)

Having the ACK skb will just make us do more work by handing it back
to skb_complete_wifi_ack() at TX status time, which is supposed to put
it into the socket's error queue, but if the socket is closed ... no
point in that.

We haven't looked at the callers of ieee80211_store_ack_skb().

Marc

-- 
Pengutronix e.K.                 | Marc Kleine-Budde           |
Embedded Linux                   | https://www.pengutronix.de  |
Vertretung West/Dortmund         | Phone: +49-231-2826-924     |
Amtsgericht Hildesheim, HRA 2686 | Fax:   +49-5121-206917-5555 |

On Mon, 2021-02-22 at 19:51 +0100, Marc Kleine-Budde wrote:

On 22.02.2021 17:30:59, Johannes Berg wrote:

quoted

On Mon, 2021-02-22 at 16:12 +0100, Oleksij Rempel wrote:

quoted

This code is trying to clone the skb with optional skb->sk. But this
will fail to clone the skb if socket was closed just after the skb was
pushed into the networking stack.

Which IMHO is completely fine. If we then still clone the SKB we can't
do anything with it, since the point would be to ... send it back to the
socket, but it's gone.

Ok, but why is the skb cloned if there is no socket linked in skb->sk?

Hm? There are two different ways to get here, one with and one without a
socket.

johannes