I'm trying to reduce system call overhead when reading/writing to/from a
tun device in userspace. For sockets, one can use sendmmsg()/recvmmsg(),
but a tun fd is not a socket fd, so this doesn't work. I'm see several
options to allow userspace to read/write multiple packets with one
syscall:
- Implement a TX/RX ring buffer that is mmap()ed, like with AF_PACKET
sockets.
- Implement a ioctl() to emulate sendmmsg()/recvmmsg().
- Add a flag that can be set using TUNSETIFF that makes regular
read()/write() calls handle multiple packets in one go.
- Expose a socket fd to userspace, so regular sendmmsg()/recvmmsg() can
be used. There is tun_get_socket() which is used internally in the
kernel, but this is not exposed to userspace, and doesn't look trivial
to do either.
What would be the right way to do this?
--
Met vriendelijke groet / with kind regards,
Guus Sliepen [off-list ref]
From: Tom Herbert <hidden> Date: 2016-03-31 21:18:49
On Tue, Mar 29, 2016 at 6:40 PM, Guus Sliepen [off-list ref] wrote:
I'm trying to reduce system call overhead when reading/writing to/from a
tun device in userspace. For sockets, one can use sendmmsg()/recvmmsg(),
but a tun fd is not a socket fd, so this doesn't work. I'm see several
options to allow userspace to read/write multiple packets with one
syscall:
- Implement a TX/RX ring buffer that is mmap()ed, like with AF_PACKET
sockets.
- Implement a ioctl() to emulate sendmmsg()/recvmmsg().
- Add a flag that can be set using TUNSETIFF that makes regular
read()/write() calls handle multiple packets in one go.
- Expose a socket fd to userspace, so regular sendmmsg()/recvmmsg() can
be used. There is tun_get_socket() which is used internally in the
kernel, but this is not exposed to userspace, and doesn't look trivial
to do either.
What would be the right way to do this?
Personally I think tun could benefit greatly if it were implemented as
a socket instead of character interface. One thing that could be much
better is sending/receiving of meta data attached to skbuf. For
instance GSO data could be in ancillary data in a socket instead of
inline with packet data as tun seems to be doing now.
Tom
--
Met vriendelijke groet / with kind regards,
Guus Sliepen [off-list ref]
From: David Miller <davem@davemloft.net> Date: 2016-03-31 21:20:52
From: Tom Herbert <redacted>
Date: Thu, 31 Mar 2016 17:18:48 -0400
On Tue, Mar 29, 2016 at 6:40 PM, Guus Sliepen [off-list ref] wrote:
quoted
I'm trying to reduce system call overhead when reading/writing to/from a
tun device in userspace. For sockets, one can use sendmmsg()/recvmmsg(),
but a tun fd is not a socket fd, so this doesn't work. I'm see several
options to allow userspace to read/write multiple packets with one
syscall:
- Implement a TX/RX ring buffer that is mmap()ed, like with AF_PACKET
sockets.
- Implement a ioctl() to emulate sendmmsg()/recvmmsg().
- Add a flag that can be set using TUNSETIFF that makes regular
read()/write() calls handle multiple packets in one go.
- Expose a socket fd to userspace, so regular sendmmsg()/recvmmsg() can
be used. There is tun_get_socket() which is used internally in the
kernel, but this is not exposed to userspace, and doesn't look trivial
to do either.
What would be the right way to do this?
Personally I think tun could benefit greatly if it were implemented as
a socket instead of character interface. One thing that could be much
better is sending/receiving of meta data attached to skbuf. For
instance GSO data could be in ancillary data in a socket instead of
inline with packet data as tun seems to be doing now.
On Thu, Mar 31, 2016 at 05:20:50PM -0400, David Miller wrote:
quoted
quoted
I'm trying to reduce system call overhead when reading/writing to/from a
tun device in userspace. [...] What would be the right way to do this?
Personally I think tun could benefit greatly if it were implemented as
a socket instead of character interface. One thing that could be much
better is sending/receiving of meta data attached to skbuf. For
instance GSO data could be in ancillary data in a socket instead of
inline with packet data as tun seems to be doing now.
Agreed.
Ok. So how should the userspace API work? Creating an AF_PACKET socket
and then using a tun ioctl to create a tun interface and bind it to the
socket?
int fd = socket(AF_PACKET, ...)
struct ifreq ifr = {...};
ioctl(fd, TUNSETIFF, &ifr);
--
Met vriendelijke groet / with kind regards,
Guus Sliepen [off-list ref]
On Thu, Mar 31, 2016 at 05:20:50PM -0400, David Miller wrote:
quoted
quoted
quoted
I'm trying to reduce system call overhead when reading/writing to/from a
tun device in userspace. [...] What would be the right way to do this?
Personally I think tun could benefit greatly if it were implemented as
a socket instead of character interface. One thing that could be much
better is sending/receiving of meta data attached to skbuf. For
instance GSO data could be in ancillary data in a socket instead of
inline with packet data as tun seems to be doing now.
Agreed.
Ok. So how should the userspace API work? Creating an AF_PACKET socket
and then using a tun ioctl to create a tun interface and bind it to the
socket?
int fd = socket(AF_PACKET, ...)
struct ifreq ifr = {...};
ioctl(fd, TUNSETIFF, &ifr);
Rather than bodge AF_PACKET onto TUN, why not just create a new device type
and control it from something modern like netlink.
On Thu, Mar 31, 2016 at 05:20:50PM -0400, David Miller wrote:
quoted
quoted
quoted
I'm trying to reduce system call overhead when reading/writing to/from a
tun device in userspace. [...] What would be the right way to do this?
Personally I think tun could benefit greatly if it were implemented as
a socket instead of character interface. One thing that could be much
better is sending/receiving of meta data attached to skbuf. For
instance GSO data could be in ancillary data in a socket instead of
inline with packet data as tun seems to be doing now.
Agreed.
Ok. So how should the userspace API work? Creating an AF_PACKET socket
and then using a tun ioctl to create a tun interface and bind it to the
socket?
int fd = socket(AF_PACKET, ...)
struct ifreq ifr = {...};
ioctl(fd, TUNSETIFF, &ifr);
Rather than bodge AF_PACKET onto TUN, why not just create a new device type
and control it from something modern like netlink.
Depending on the use-case, it may be sufficient to extend AF_PACKET
with limited tap functionality:
- add a po->xmit mode that reinjects into the kernel receive path,
analogous to pktgen's M_NETIF_RECEIVE mode.
- optionally drop packets in __netif_receive_skb_core and xmit_one
if any of the registered packet sockets accepted the packet and has
a new intercept feature flag enabled.
This can be applied to a dummy device, but much more interesting
is to interpose on the flow of a normal nic. It is clearly not a drop-in
replacement for a tap (let alone tun) device. I have some preliminary
code.
On Sun, Apr 03, 2016 at 07:03:09PM -0400, Willem de Bruijn wrote:
On Thu, Mar 31, 2016 at 7:39 PM, Stephen Hemminger [off-list ref] wrote:
quoted
Rather than bodge AF_PACKET onto TUN, why not just create a new device type
and control it from something modern like netlink.
Do we really want to introduce a whole new device type? The tun device
is working perfectly fine, except for the fact that there is no way to
send/receive multiple packets in one go.
Depending on the use-case, it may be sufficient to extend AF_PACKET
with limited tap functionality:
- add a po->xmit mode that reinjects into the kernel receive path,
analogous to pktgen's M_NETIF_RECEIVE mode.
- optionally drop packets in __netif_receive_skb_core and xmit_one
if any of the registered packet sockets accepted the packet and has
a new intercept feature flag enabled.
This can be applied to a dummy device, but much more interesting
is to interpose on the flow of a normal nic. It is clearly not a drop-in
replacement for a tap (let alone tun) device. I have some preliminary
code.
It's not really tinc's use case, but I did try using socket(AF_PACKET)
bound to a tun interface, just to see if sendmmsg()/recvmmsg() works
then. It does, but indeed for packets that are sent to the socket, they
need to be reinjected into the kernel receive path. So I'll be happy to
test out your preliminary code.
--
Met vriendelijke groet / with kind regards,
Guus Sliepen [off-list ref]