cross namespace interface notification for tun devices

Hey guys,

It's possible to create a tun device in a process in namespace A and
then move that interface to namespace B. The controlling process in A
needs to receive notifications on when the interface is brought up or
down. It can receive these notifications via netlink while the
interface lives in A but not when it moves to B.

Any tricks or APIs to get around this?

The best I've come up with is, in a sleep loop, writing to the tun
device's fd something with a NULL or invalid payload. If the interface
is down, the kernel returns -EIO. If the interface is up, the kernel
returns -EFAULT. This seems to be a reliable distinguisher, but is a
pretty insane way of doing it. And sleep loops are somewhat different
from events too.

If there aren't any current APIs for receiving events directly from
the fd of a tun interface, would this list be happy with a patch that
adds one?

Thanks,
Jason

On Mon, Sep 18, 2017 at 8:47 PM, Jason A. Donenfeld [off-list ref] wrote:

The best I've come up with is, in a sleep loop, writing to the tun
device's fd something with a NULL or invalid payload. If the interface
is down, the kernel returns -EIO. If the interface is up, the kernel
returns -EINVAL. This seems to be a reliable distinguisher, but is a
pretty insane way of doing it. And sleep loops are somewhat different
from events too.

Specifically, I'm referring to the horrific hack exemplified in the
attached .c file, in case anybody is curious about the details of what
I'd rather not use.

On Mon, Sep 18, 2017 at 11:47 AM, Jason A. Donenfeld [off-list ref] wrote:

Hey guys,

It's possible to create a tun device in a process in namespace A and
then move that interface to namespace B. The controlling process in A
needs to receive notifications on when the interface is brought up or
down. It can receive these notifications via netlink while the
interface lives in A but not when it moves to B.

By "notification" I assume you mean netlink notification.

Any tricks or APIs to get around this?

The question is why does the process in A still care about
the device sitting in B?

Also, the process should be able to receive a last notification
on IFF_UP|IFF_RUNNING before device is finally moved to B.
After this point, it should not have any relation to netns A
any more, like the device were completely gone.

On Tue, Sep 19, 2017 at 10:40 PM, Cong Wang [off-list ref] wrote:

By "notification" I assume you mean netlink notification.

Yes, netlink notification.

The question is why does the process in A still care about
the device sitting in B?

Also, the process should be able to receive a last notification
on IFF_UP|IFF_RUNNING before device is finally moved to B.
After this point, it should not have any relation to netns A
any more, like the device were completely gone.

That's very clearly not the case with a tun device. Tun devices work
by letting a userspace process control the inputs (ndo_start_xmit) and
outputs (netif_rx) of the actual network device. This controlling
userspace process needs to know when its own interface that it
controls goes up and down. In the kernel, we can do this by just
checking dev->flags&IFF_UP, and receive notifications on ndo_open and
ndo_stop. In userspace, the controlling process looses the ability to
receive notifications like ndo_open/ndo_stop when the interface is
moved to a new namespace. After the interface is moved to a namespace,
the process will still control inputs and ouputs (ndo_start_xmit and
netif_rx), but it will no longer receive netlink notifications for the
equivalent of ndo_open and ndo_stop. This is problematic.

On Tue, Sep 19, 2017 at 2:02 PM, Jason A. Donenfeld [off-list ref] wrote:

On Tue, Sep 19, 2017 at 10:40 PM, Cong Wang [off-list ref] wrote:

quoted

By "notification" I assume you mean netlink notification.

Yes, netlink notification.

quoted

The question is why does the process in A still care about
the device sitting in B?

Also, the process should be able to receive a last notification
on IFF_UP|IFF_RUNNING before device is finally moved to B.
After this point, it should not have any relation to netns A
any more, like the device were completely gone.

That's very clearly not the case with a tun device. Tun devices work
by letting a userspace process control the inputs (ndo_start_xmit) and
outputs (netif_rx) of the actual network device. This controlling
userspace process needs to know when its own interface that it
controls goes up and down. In the kernel, we can do this by just
checking dev->flags&IFF_UP, and receive notifications on ndo_open and
ndo_stop. In userspace, the controlling process looses the ability to
receive notifications like ndo_open/ndo_stop when the interface is
moved to a new namespace. After the interface is moved to a namespace,
the process will still control inputs and ouputs (ndo_start_xmit and
netif_rx), but it will no longer receive netlink notifications for the
equivalent of ndo_open and ndo_stop. This is problematic.

Sounds like we should set NETIF_F_NETNS_LOCAL for tun
device.

What is your legitimate use case of send/receive packet to/from
a tun device in a different netns?

On Wed, 2017-09-20 at 11:29 -0700, Cong Wang wrote:

On Tue, Sep 19, 2017 at 2:02 PM, Jason A. Donenfeld [off-list ref]
wrote:

quoted

On Tue, Sep 19, 2017 at 10:40 PM, Cong Wang <xiyou.wangcong@gmail.c
om> wrote:

quoted

By "notification" I assume you mean netlink notification.

Yes, netlink notification.

quoted

The question is why does the process in A still care about
the device sitting in B?

Also, the process should be able to receive a last notification
on IFF_UP|IFF_RUNNING before device is finally moved to B.
After this point, it should not have any relation to netns A
any more, like the device were completely gone.

That's very clearly not the case with a tun device. Tun devices
work
by letting a userspace process control the inputs (ndo_start_xmit)
and
outputs (netif_rx) of the actual network device. This controlling
userspace process needs to know when its own interface that it
controls goes up and down. In the kernel, we can do this by just
checking dev->flags&IFF_UP, and receive notifications on ndo_open
and
ndo_stop. In userspace, the controlling process looses the ability
to
receive notifications like ndo_open/ndo_stop when the interface is
moved to a new namespace. After the interface is moved to a
namespace,
the process will still control inputs and ouputs (ndo_start_xmit
and
netif_rx), but it will no longer receive netlink notifications for
the
equivalent of ndo_open and ndo_stop. This is problematic.

Sounds like we should set NETIF_F_NETNS_LOCAL for tun
device.

What is your legitimate use case of send/receive packet to/from
a tun device in a different netns?

One thought: run openvpn in the master netns, but put its tun0
interface into an application's netns.  Per-application VPN,
essentially?  Or maybe that's not how people do this kind of thing, but
it's a thought.

Dan

On Wed, Sep 20, 2017 at 8:29 PM, Cong Wang [off-list ref] wrote:

Sounds like we should set NETIF_F_NETNS_LOCAL for tun
device.

Absolutely do not do this under any circumstances. This would be a
regression and would break API compatibility. As I wrote in my first
email, it's already possible to sleep-loop for that information using
the tun device's fd; I'm just looking for a better event-based
approach.

What is your legitimate use case of send/receive packet to/from
a tun device in a different netns?

Because sometimes it's very nice to be able to move network interfaces
that use tun devices into different namespaces, for some xnamespace
proxying.

What Dan described in the email he just sent is exactly this use case.

In WireGuard (a kernel thing), I have facilities for this --
https://www.wireguard.com/netns/ . Now I'm working on the userspace
version and would like to expose the same utility.

Anyway, the purpose of me sending this message to the list was not to
question the "legitimacy" of my application usage, but rather to
elicit feedback on two specific things:

1. to determine if there's already a mechanism in place for this that
I've overlooked; and
2. to determine particularities of me implementing a mechanism, if
it's not already there.

I'm slightly more convinced that there isn't currently a mechanism for
this. It seems like the easiest way, therefore, would be some kind of
control message that could be poll'd for, using the existing
per-process fd. That way there wouldn't be any violations of the
current namespace situation, yet processes could still get event
notifications as needed.

Le 18/09/2017 à 20:47, Jason A. Donenfeld a écrit :

Hey guys,

It's possible to create a tun device in a process in namespace A and
then move that interface to namespace B. The controlling process in A
needs to receive notifications on when the interface is brought up or
down. It can receive these notifications via netlink while the
interface lives in A but not when it moves to B.

Any tricks or APIs to get around this?

There are two options.

1. Move the process to netns B, open the netlink socket and move back the
process to netns A. The socket will remain in netns B and you will receive all
netlink messages related to netns B.

2. Assign a nsid to netns B in netns A and use NETLINK_LISTEN_ALL_NSID on your
netlink socket (see iproute2).


Regards,
Nicolas

On Mon, Oct 2, 2017 at 11:32 AM, Nicolas Dichtel
[off-list ref] wrote:

1. Move the process to netns B, open the netlink socket and move back the
process to netns A. The socket will remain in netns B and you will receive all
netlink messages related to netns B.

2. Assign a nsid to netns B in netns A and use NETLINK_LISTEN_ALL_NSID on your
netlink socket (see iproute2).

Both of these seem to rely on the process knowing where the device is
being moved and having access to that namespace. I don't think these
two things are a given though. Unless I'm missing something?

Jason

Le 02/10/2017 à 13:11, Jason A. Donenfeld a écrit :

On Mon, Oct 2, 2017 at 11:32 AM, Nicolas Dichtel
[off-list ref] wrote:

quoted

1. Move the process to netns B, open the netlink socket and move back the
process to netns A. The socket will remain in netns B and you will receive all
netlink messages related to netns B.

2. Assign a nsid to netns B in netns A and use NETLINK_LISTEN_ALL_NSID on your
netlink socket (see iproute2).

Both of these seem to rely on the process knowing where the device is
being moved and having access to that namespace. I don't think these
two things are a given though. Unless I'm missing something?

I didn't understand correctly.
Your control process cannot monitor or control an interface which is in a
unkown/hidden netns. But x-netns interfaces are special. We already add a way to
identify peer netns for this kind of interfaces.
If an handler get_link_net was added to the rtnl_link_ops of the tun driver, it
will help to identify netns A when you are in netns B. But you need the opposite.
I already try a patch to advertise via netlink the dst netns when an interface
moves to a new netns. I think that it is valid for x-netns interfaces.
As soon as you can identify the dst netns, your problem is solved, right?


Nicolas

x-netns interfaces are bound to two netns: the link netns and the upper
netns. Usually, this kind of interfaces is created in the link netns and
then moved to the upper netns. At the end, the interface is visible only
in the upper netns. The link nsid is advertised via netlink in the upper
netns, thus the user always knows where is the link part.

There is no such mechanism in the link netns. When the interface is moved
to another netns, the user cannot "follow" it.
This patch adds a new netlink attribute which helps to follow an interface
which moves to another netns. When the interface is unregistered, the new
nsid is advertised. If the interface is a x-netns interface (ie
rtnl_link_ops->get_link_net is defined), the nsid is allocated if needed.

CC: Jason A. Donenfeld <Jason@zx2c4.com>
Signed-off-by: Nicolas Dichtel <redacted>
---
 include/linux/rtnetlink.h    |  4 +++-
 include/uapi/linux/if_link.h |  1 +
 net/core/dev.c               | 11 ++++++++---
 net/core/rtnetlink.c         | 31 ++++++++++++++++++++++---------
 4 files changed, 34 insertions(+), 13 deletions(-)

diff --git a/include/linux/rtnetlink.h b/include/linux/rtnetlink.h
index dea59c8eec54..1251638e60d3 100644
--- a/include/linux/rtnetlink.h
+++ b/include/linux/rtnetlink.h

@@ -17,9 +17,11 @@ extern int rtnl_put_cacheinfo(struct sk_buff *skb, struct dst_entry *dst,
 			      u32 id, long expires, u32 error);
 
 void rtmsg_ifinfo(int type, struct net_device *dev, unsigned change, gfp_t flags);
+void rtmsg_ifinfo_newnet(int type, struct net_device *dev, unsigned int change,
+			 gfp_t flags, int *new_nsid);
 struct sk_buff *rtmsg_ifinfo_build_skb(int type, struct net_device *dev,
 				       unsigned change, u32 event,
-				       gfp_t flags);
+				       gfp_t flags, int *new_nsid);
 void rtmsg_ifinfo_send(struct sk_buff *skb, struct net_device *dev,
 		       gfp_t flags);
 

diff --git a/include/uapi/linux/if_link.h b/include/uapi/linux/if_link.h
index ea87bd708ee9..cd580fc0e58f 100644
--- a/include/uapi/linux/if_link.h
+++ b/include/uapi/linux/if_link.h

@@ -158,6 +158,7 @@ enum {
 	IFLA_PAD,
 	IFLA_XDP,
 	IFLA_EVENT,
+	IFLA_NEW_NETNSID,
 	__IFLA_MAX
 };
 

diff --git a/net/core/dev.c b/net/core/dev.c
index e350c768d4b5..2341e9d64e02 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c

@@ -145,6 +145,7 @@
 #include <linux/crash_dump.h>
 #include <linux/sctp.h>
 #include <net/udp_tunnel.h>
+#include <linux/net_namespace.h>
 
 #include "net-sysfs.h"
 

@@ -7178,7 +7179,7 @@ static void rollback_registered_many(struct list_head *head)
 		if (!dev->rtnl_link_ops ||
 		    dev->rtnl_link_state == RTNL_LINK_INITIALIZED)
 			skb = rtmsg_ifinfo_build_skb(RTM_DELLINK, dev, ~0U, 0,
-						     GFP_KERNEL);
+						     GFP_KERNEL, NULL);
 
 		/*
 		 *	Flush the unicast and multicast chains

@@ -8265,7 +8266,7 @@ EXPORT_SYMBOL(unregister_netdev);
 
 int dev_change_net_namespace(struct net_device *dev, struct net *net, const char *pat)
 {
-	int err;
+	int err, new_nsid;
 
 	ASSERT_RTNL();
 

@@ -8321,7 +8322,11 @@ int dev_change_net_namespace(struct net_device *dev, struct net *net, const char
 	call_netdevice_notifiers(NETDEV_UNREGISTER, dev);
 	rcu_barrier();
 	call_netdevice_notifiers(NETDEV_UNREGISTER_FINAL, dev);
-	rtmsg_ifinfo(RTM_DELLINK, dev, ~0U, GFP_KERNEL);
+	if (dev->rtnl_link_ops && dev->rtnl_link_ops->get_link_net)
+		new_nsid = peernet2id_alloc(dev_net(dev), net);
+	else
+		new_nsid = peernet2id(dev_net(dev), net);
+	rtmsg_ifinfo_newnet(RTM_DELLINK, dev, ~0U, GFP_KERNEL, &new_nsid);
 
 	/*
 	 *	Flush the unicast and multicast chains

diff --git a/net/core/rtnetlink.c b/net/core/rtnetlink.c
index e6955da0d58d..5bec24c348bf 100644
--- a/net/core/rtnetlink.c
+++ b/net/core/rtnetlink.c

@@ -927,6 +927,7 @@ static noinline size_t if_nlmsg_size(const struct net_device *dev,
 	       + nla_total_size(IFNAMSIZ) /* IFLA_PHYS_PORT_NAME */
 	       + rtnl_xdp_size() /* IFLA_XDP */
 	       + nla_total_size(4)  /* IFLA_EVENT */
+	       + nla_total_size(4)  /* IFLA_NEW_NETNSID */
 	       + nla_total_size(1); /* IFLA_PROTO_DOWN */
 
 }

@@ -1386,7 +1387,7 @@ static int rtnl_fill_link_netnsid(struct sk_buff *skb,
 static int rtnl_fill_ifinfo(struct sk_buff *skb, struct net_device *dev,
 			    int type, u32 pid, u32 seq, u32 change,
 			    unsigned int flags, u32 ext_filter_mask,
-			    u32 event)
+			    u32 event, int *new_nsid)
 {
 	struct ifinfomsg *ifm;
 	struct nlmsghdr *nlh;

@@ -1475,6 +1476,10 @@ static int rtnl_fill_ifinfo(struct sk_buff *skb, struct net_device *dev,
 	if (rtnl_fill_link_netnsid(skb, dev))
 		goto nla_put_failure;
 
+	if (new_nsid &&
+	    nla_put_s32(skb, IFLA_NEW_NETNSID, *new_nsid) < 0)
+		goto nla_put_failure;
+
 	if (!(af_spec = nla_nest_start(skb, IFLA_AF_SPEC)))
 		goto nla_put_failure;
 

@@ -1704,7 +1709,7 @@ static int rtnl_dump_ifinfo(struct sk_buff *skb, struct netlink_callback *cb)
 					       NETLINK_CB(cb->skb).portid,
 					       cb->nlh->nlmsg_seq, 0,
 					       flags,
-					       ext_filter_mask, 0);
+					       ext_filter_mask, 0, NULL);
 
 			if (err < 0) {
 				if (likely(skb->len))

@@ -2817,7 +2822,7 @@ static int rtnl_getlink(struct sk_buff *skb, struct nlmsghdr *nlh,
 		return -ENOBUFS;
 
 	err = rtnl_fill_ifinfo(nskb, dev, RTM_NEWLINK, NETLINK_CB(skb).portid,
-			       nlh->nlmsg_seq, 0, 0, ext_filter_mask, 0);
+			       nlh->nlmsg_seq, 0, 0, ext_filter_mask, 0, NULL);
 	if (err < 0) {
 		/* -EMSGSIZE implies BUG in if_nlmsg_size */
 		WARN_ON(err == -EMSGSIZE);

@@ -2902,7 +2907,7 @@ static int rtnl_dump_all(struct sk_buff *skb, struct netlink_callback *cb)
 
 struct sk_buff *rtmsg_ifinfo_build_skb(int type, struct net_device *dev,
 				       unsigned int change,
-				       u32 event, gfp_t flags)
+				       u32 event, gfp_t flags, int *new_nsid)
 {
 	struct net *net = dev_net(dev);
 	struct sk_buff *skb;

@@ -2913,7 +2918,8 @@ struct sk_buff *rtmsg_ifinfo_build_skb(int type, struct net_device *dev,
 	if (skb == NULL)
 		goto errout;
 
-	err = rtnl_fill_ifinfo(skb, dev, type, 0, 0, change, 0, 0, event);
+	err = rtnl_fill_ifinfo(skb, dev, type, 0, 0, change, 0, 0, event,
+			       new_nsid);
 	if (err < 0) {
 		/* -EMSGSIZE implies BUG in if_nlmsg_size() */
 		WARN_ON(err == -EMSGSIZE);

@@ -2936,14 +2942,14 @@ void rtmsg_ifinfo_send(struct sk_buff *skb, struct net_device *dev, gfp_t flags)
 
 static void rtmsg_ifinfo_event(int type, struct net_device *dev,
 			       unsigned int change, u32 event,
-			       gfp_t flags)
+			       gfp_t flags, int *new_nsid)
 {
 	struct sk_buff *skb;
 
 	if (dev->reg_state != NETREG_REGISTERED)
 		return;
 
-	skb = rtmsg_ifinfo_build_skb(type, dev, change, event, flags);
+	skb = rtmsg_ifinfo_build_skb(type, dev, change, event, flags, new_nsid);
 	if (skb)
 		rtmsg_ifinfo_send(skb, dev, flags);
 }

@@ -2951,10 +2957,17 @@ static void rtmsg_ifinfo_event(int type, struct net_device *dev,
 void rtmsg_ifinfo(int type, struct net_device *dev, unsigned int change,
 		  gfp_t flags)
 {
-	rtmsg_ifinfo_event(type, dev, change, rtnl_get_event(0), flags);
+	rtmsg_ifinfo_event(type, dev, change, rtnl_get_event(0), flags, NULL);
 }
 EXPORT_SYMBOL(rtmsg_ifinfo);
 
+void rtmsg_ifinfo_newnet(int type, struct net_device *dev, unsigned int change,
+			 gfp_t flags, int *new_nsid)
+{
+	rtmsg_ifinfo_event(type, dev, change, rtnl_get_event(0), flags,
+			   new_nsid);
+}
+
 static int nlmsg_populate_fdb_fill(struct sk_buff *skb,
 				   struct net_device *dev,
 				   u8 *addr, u16 vid, u32 pid, u32 seq,

@@ -4330,7 +4343,7 @@ static int rtnetlink_event(struct notifier_block *this, unsigned long event, voi
 	case NETDEV_RESEND_IGMP:
 	case NETDEV_CHANGEINFODATA:
 		rtmsg_ifinfo_event(RTM_NEWLINK, dev, 0, rtnl_get_event(event),
-				   GFP_KERNEL);
+				   GFP_KERNEL, NULL);
 		break;
 	default:
 		break;

-- 
2.13.2

From: Nicolas Dichtel <redacted>
Date: Tue,  3 Oct 2017 13:53:23 +0200

x-netns interfaces are bound to two netns: the link netns and the upper
netns. Usually, this kind of interfaces is created in the link netns and
then moved to the upper netns. At the end, the interface is visible only
in the upper netns. The link nsid is advertised via netlink in the upper
netns, thus the user always knows where is the link part.

There is no such mechanism in the link netns. When the interface is moved
to another netns, the user cannot "follow" it.
This patch adds a new netlink attribute which helps to follow an interface
which moves to another netns. When the interface is unregistered, the new
nsid is advertised. If the interface is a x-netns interface (ie
rtnl_link_ops->get_link_net is defined), the nsid is allocated if needed.

CC: Jason A. Donenfeld <Jason@zx2c4.com>
Signed-off-by: Nicolas Dichtel <redacted>

Ok, applied, thanks.