On Fri, 28 May 2021 17:02:01 +0800
Xuan Zhuo [off-list ref] wrote:
On Fri, 28 May 2021 10:55:58 +0200, Toke Høiland-Jørgensen [off-list ref] wrote:
quoted
Xuan Zhuo [off-list ref] writes:
quoted
In xsk mode, users cannot use AF_PACKET(tcpdump) to observe the current
rx/tx data packets. This feature is very important in many cases. So
this patch allows AF_PACKET to obtain xsk packages.
Yes, it is rather cool (credit to Eelco). Notice the extra info you
can capture from 'exit', like XDP return codes, if_index, rx_queue.
The tool uses the perf ring-buffer to send/copy data to userspace.
This is actually surprisingly fast, but I still think AF_XDP will be
faster (but it usually 'steals' the packet).
Another (crazy?) idea is to extend this (and xdpdump), is to leverage
Hangbin's recent XDP_REDIRECT extension e624d4ed4aa8 ("xdp: Extend
xdp_redirect_map with broadcast support"). We now have a
xdp_redirect_map flag BPF_F_BROADCAST, what if we create a
BPF_F_CLONE_PASS flag?
The semantic meaning of BPF_F_CLONE_PASS flag is to copy/clone the
packet for the specified map target index (e.g AF_XDP map), but
afterwards it does like veth/cpumap and creates an SKB from the
xdp_frame (see __xdp_build_skb_from_frame()) and send to netstack.
(Feel free to kick me if this doesn't make any sense)
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
From: Magnus Karlsson <hidden> Date: 2021-05-28 10:00:29
On Fri, May 28, 2021 at 11:52 AM Jesper Dangaard Brouer
[off-list ref] wrote:
On Fri, 28 May 2021 17:02:01 +0800
Xuan Zhuo [off-list ref] wrote:
quoted
On Fri, 28 May 2021 10:55:58 +0200, Toke Høiland-Jørgensen [off-list ref] wrote:
quoted
Xuan Zhuo [off-list ref] writes:
quoted
In xsk mode, users cannot use AF_PACKET(tcpdump) to observe the current
rx/tx data packets. This feature is very important in many cases. So
this patch allows AF_PACKET to obtain xsk packages.
Yes, it is rather cool (credit to Eelco). Notice the extra info you
can capture from 'exit', like XDP return codes, if_index, rx_queue.
The tool uses the perf ring-buffer to send/copy data to userspace.
This is actually surprisingly fast, but I still think AF_XDP will be
faster (but it usually 'steals' the packet).
Another (crazy?) idea is to extend this (and xdpdump), is to leverage
Hangbin's recent XDP_REDIRECT extension e624d4ed4aa8 ("xdp: Extend
xdp_redirect_map with broadcast support"). We now have a
xdp_redirect_map flag BPF_F_BROADCAST, what if we create a
BPF_F_CLONE_PASS flag?
The semantic meaning of BPF_F_CLONE_PASS flag is to copy/clone the
packet for the specified map target index (e.g AF_XDP map), but
afterwards it does like veth/cpumap and creates an SKB from the
xdp_frame (see __xdp_build_skb_from_frame()) and send to netstack.
(Feel free to kick me if this doesn't make any sense)
This would be a smooth way to implement clone support for AF_XDP. If
we had this and someone added AF_XDP support to libpcap, we could both
capture AF_XDP traffic with tcpdump (using this clone functionality in
the XDP program) and speed up tcpdump for dumping traffic destined for
regular sockets. Would that solve your use case Xuan? Note that I have
not looked into the BPF_F_CLONE_PASS code, so do not know at this
point what it would take to support this for XSKMAPs.
From: Daniel Borkmann <daniel@iogearbox.net> Date: 2021-05-28 10:43:49
On 5/28/21 12:00 PM, Magnus Karlsson wrote:
On Fri, May 28, 2021 at 11:52 AM Jesper Dangaard Brouer
[off-list ref] wrote:
quoted
On Fri, 28 May 2021 17:02:01 +0800
Xuan Zhuo [off-list ref] wrote:
quoted
On Fri, 28 May 2021 10:55:58 +0200, Toke Høiland-Jørgensen [off-list ref] wrote:
quoted
Xuan Zhuo [off-list ref] writes:
quoted
In xsk mode, users cannot use AF_PACKET(tcpdump) to observe the current
rx/tx data packets. This feature is very important in many cases. So
this patch allows AF_PACKET to obtain xsk packages.
Yes, it is rather cool (credit to Eelco). Notice the extra info you
can capture from 'exit', like XDP return codes, if_index, rx_queue.
The tool uses the perf ring-buffer to send/copy data to userspace.
This is actually surprisingly fast, but I still think AF_XDP will be
faster (but it usually 'steals' the packet).
Another (crazy?) idea is to extend this (and xdpdump), is to leverage
Hangbin's recent XDP_REDIRECT extension e624d4ed4aa8 ("xdp: Extend
xdp_redirect_map with broadcast support"). We now have a
xdp_redirect_map flag BPF_F_BROADCAST, what if we create a
BPF_F_CLONE_PASS flag?
The semantic meaning of BPF_F_CLONE_PASS flag is to copy/clone the
packet for the specified map target index (e.g AF_XDP map), but
afterwards it does like veth/cpumap and creates an SKB from the
xdp_frame (see __xdp_build_skb_from_frame()) and send to netstack.
(Feel free to kick me if this doesn't make any sense)
This would be a smooth way to implement clone support for AF_XDP. If
we had this and someone added AF_XDP support to libpcap, we could both
capture AF_XDP traffic with tcpdump (using this clone functionality in
the XDP program) and speed up tcpdump for dumping traffic destined for
regular sockets. Would that solve your use case Xuan? Note that I have
not looked into the BPF_F_CLONE_PASS code, so do not know at this
point what it would take to support this for XSKMAPs.
Recently also ended up with something similar for our XDP LB to record pcaps [0] ;)
My question is.. tcpdump doesn't really care where the packet data comes from,
so why not extending libpcap's Linux-related internals to either capture from
perf RB or BPF ringbuf rather than AF_PACKET sockets? Cloning is slow, and if
you need to end up creating an skb which is then cloned once again inside AF_PACKET
it's even worse. Just relying and reading out, say, perf RB you don't need any
clones at all.
Thanks,
Daniel
[0] https://cilium.io/blog/2021/05/20/cilium-110#pcap
On Fri, May 28, 2021 at 11:52 AM Jesper Dangaard Brouer
[off-list ref] wrote:
quoted
On Fri, 28 May 2021 17:02:01 +0800
Xuan Zhuo [off-list ref] wrote:
quoted
On Fri, 28 May 2021 10:55:58 +0200, Toke Høiland-Jørgensen [off-list ref] wrote:
quoted
Xuan Zhuo [off-list ref] writes:
quoted
In xsk mode, users cannot use AF_PACKET(tcpdump) to observe the current
rx/tx data packets. This feature is very important in many cases. So
this patch allows AF_PACKET to obtain xsk packages.
Yes, it is rather cool (credit to Eelco). Notice the extra info you
can capture from 'exit', like XDP return codes, if_index, rx_queue.
The tool uses the perf ring-buffer to send/copy data to userspace.
This is actually surprisingly fast, but I still think AF_XDP will be
faster (but it usually 'steals' the packet).
Another (crazy?) idea is to extend this (and xdpdump), is to leverage
Hangbin's recent XDP_REDIRECT extension e624d4ed4aa8 ("xdp: Extend
xdp_redirect_map with broadcast support"). We now have a
xdp_redirect_map flag BPF_F_BROADCAST, what if we create a
BPF_F_CLONE_PASS flag?
The semantic meaning of BPF_F_CLONE_PASS flag is to copy/clone the
packet for the specified map target index (e.g AF_XDP map), but
afterwards it does like veth/cpumap and creates an SKB from the
xdp_frame (see __xdp_build_skb_from_frame()) and send to netstack.
(Feel free to kick me if this doesn't make any sense)
This would be a smooth way to implement clone support for AF_XDP. If
we had this and someone added AF_XDP support to libpcap, we could both
capture AF_XDP traffic with tcpdump (using this clone functionality in
the XDP program) and speed up tcpdump for dumping traffic destined for
regular sockets. Would that solve your use case Xuan? Note that I have
not looked into the BPF_F_CLONE_PASS code, so do not know at this
point what it would take to support this for XSKMAPs.
Recently also ended up with something similar for our XDP LB to record pcaps [0] ;)
My question is.. tcpdump doesn't really care where the packet data comes from,
so why not extending libpcap's Linux-related internals to either capture from
perf RB or BPF ringbuf rather than AF_PACKET sockets? Cloning is slow, and if
you need to end up creating an skb which is then cloned once again inside AF_PACKET
it's even worse. Just relying and reading out, say, perf RB you don't need any
clones at all.
We discussed this when creating xdpdump and decided to keep it as a
separate tool for the time being. I forget the details of the
discussion, maybe Eelco remembers.
Anyway, xdpdump does have a "pipe pcap to stdout" feature so you can do
`xdpdump | tcpdump` and get the interactive output; and it will also
save pcap information to disk, of course (using pcap-ng so it can also
save metadata like XDP program name and return code).
-Toke
From: Daniel Borkmann <daniel@iogearbox.net> Date: 2021-05-28 12:14:15
On 5/28/21 12:54 PM, Toke Høiland-Jørgensen wrote:
Daniel Borkmann [off-list ref] writes:
quoted
On 5/28/21 12:00 PM, Magnus Karlsson wrote:
quoted
On Fri, May 28, 2021 at 11:52 AM Jesper Dangaard Brouer
[off-list ref] wrote:
quoted
On Fri, 28 May 2021 17:02:01 +0800
Xuan Zhuo [off-list ref] wrote:
quoted
On Fri, 28 May 2021 10:55:58 +0200, Toke Høiland-Jørgensen [off-list ref] wrote:
quoted
Xuan Zhuo [off-list ref] writes:
quoted
In xsk mode, users cannot use AF_PACKET(tcpdump) to observe the current
rx/tx data packets. This feature is very important in many cases. So
this patch allows AF_PACKET to obtain xsk packages.
Yes, it is rather cool (credit to Eelco). Notice the extra info you
can capture from 'exit', like XDP return codes, if_index, rx_queue.
The tool uses the perf ring-buffer to send/copy data to userspace.
This is actually surprisingly fast, but I still think AF_XDP will be
faster (but it usually 'steals' the packet).
Another (crazy?) idea is to extend this (and xdpdump), is to leverage
Hangbin's recent XDP_REDIRECT extension e624d4ed4aa8 ("xdp: Extend
xdp_redirect_map with broadcast support"). We now have a
xdp_redirect_map flag BPF_F_BROADCAST, what if we create a
BPF_F_CLONE_PASS flag?
The semantic meaning of BPF_F_CLONE_PASS flag is to copy/clone the
packet for the specified map target index (e.g AF_XDP map), but
afterwards it does like veth/cpumap and creates an SKB from the
xdp_frame (see __xdp_build_skb_from_frame()) and send to netstack.
(Feel free to kick me if this doesn't make any sense)
This would be a smooth way to implement clone support for AF_XDP. If
we had this and someone added AF_XDP support to libpcap, we could both
capture AF_XDP traffic with tcpdump (using this clone functionality in
the XDP program) and speed up tcpdump for dumping traffic destined for
regular sockets. Would that solve your use case Xuan? Note that I have
not looked into the BPF_F_CLONE_PASS code, so do not know at this
point what it would take to support this for XSKMAPs.
Recently also ended up with something similar for our XDP LB to record pcaps [0] ;)
My question is.. tcpdump doesn't really care where the packet data comes from,
so why not extending libpcap's Linux-related internals to either capture from
perf RB or BPF ringbuf rather than AF_PACKET sockets? Cloning is slow, and if
you need to end up creating an skb which is then cloned once again inside AF_PACKET
it's even worse. Just relying and reading out, say, perf RB you don't need any
clones at all.
We discussed this when creating xdpdump and decided to keep it as a
separate tool for the time being. I forget the details of the
discussion, maybe Eelco remembers.
Anyway, xdpdump does have a "pipe pcap to stdout" feature so you can do
`xdpdump | tcpdump` and get the interactive output; and it will also
save pcap information to disk, of course (using pcap-ng so it can also
save metadata like XDP program name and return code).
Right, and this should yield a significantly better performance compared to
cloning & pushing traffic into AF_PACKET. I presume not many folks are aware
of xdpdump (yet) which is probably why such patch was created here.. a native
libpcap implementation could solve that aspect fwiw and additionally hook at
the same points as AF_PACKET via BPF but without the hassle/overhead of things
like dev_queue_xmit_nit() in fast path. (Maybe another option could be to have
a drop-in replacement libpcap.so for tcpdump using it transparently.)
Thanks,
Daniel
On Fri, 28 May 2021 12:22:40 +0200
Daniel Borkmann [off-list ref] wrote:
On 5/28/21 12:00 PM, Magnus Karlsson wrote:
quoted
On Fri, May 28, 2021 at 11:52 AM Jesper Dangaard Brouer
[off-list ref] wrote:
quoted
On Fri, 28 May 2021 17:02:01 +0800
Xuan Zhuo [off-list ref] wrote:
quoted
On Fri, 28 May 2021 10:55:58 +0200, Toke Høiland-Jørgensen [off-list ref] wrote:
quoted
Xuan Zhuo [off-list ref] writes:
quoted
In xsk mode, users cannot use AF_PACKET(tcpdump) to observe the current
rx/tx data packets. This feature is very important in many cases. So
this patch allows AF_PACKET to obtain xsk packages.
Yes, it is rather cool (credit to Eelco). Notice the extra info you
can capture from 'exit', like XDP return codes, if_index, rx_queue.
The tool uses the perf ring-buffer to send/copy data to userspace.
This is actually surprisingly fast, but I still think AF_XDP will be
faster (but it usually 'steals' the packet).
Another (crazy?) idea is to extend this (and xdpdump), is to leverage
Hangbin's recent XDP_REDIRECT extension e624d4ed4aa8 ("xdp: Extend
xdp_redirect_map with broadcast support"). We now have a
xdp_redirect_map flag BPF_F_BROADCAST, what if we create a
BPF_F_CLONE_PASS flag?
The semantic meaning of BPF_F_CLONE_PASS flag is to copy/clone the
packet for the specified map target index (e.g AF_XDP map), but
afterwards it does like veth/cpumap and creates an SKB from the
xdp_frame (see __xdp_build_skb_from_frame()) and send to netstack.
(Feel free to kick me if this doesn't make any sense)
This would be a smooth way to implement clone support for AF_XDP. If
we had this and someone added AF_XDP support to libpcap, we could both
capture AF_XDP traffic with tcpdump (using this clone functionality in
the XDP program) and speed up tcpdump for dumping traffic destined for
regular sockets. Would that solve your use case Xuan? Note that I have
not looked into the BPF_F_CLONE_PASS code, so do not know at this
point what it would take to support this for XSKMAPs.
There is no spoon... the BPF_F_CLONE_PASS code is an idea.
Recently also ended up with something similar for our XDP LB to record pcaps [0] ;)
My question is.. tcpdump doesn't really care where the packet data comes from,
so why not extending libpcap's Linux-related internals to either capture from
perf RB or BPF ringbuf
Just want to first mention, that I do like adding a perf ring-buffer
(BPF ringbuf) interface to AF_PACKET. But this is basically what
xdpdump already does. The cool thing is that it is super flexible for
adding extra info like xdpdump does with XDP-return codes.
rather than AF_PACKET sockets? Cloning is slow, and if
you need to end up creating an skb which is then cloned once again inside AF_PACKET
it's even worse. Just relying and reading out, say, perf RB you don't need any
clones at all.
Well, this is exactly what we avoid with my idea of BPF_F_CLONE_PASS
when combined with AF_XDP.
I should explain this idea better. The trick is that AF_XDP have
preallocated all the packets it will every use (at setup time). Thus,
the AF_XDP copy-mode does no allocations, which is why it is fast
(of-cause ZC mode is faster, but copy-mode AF_XDP is also VERY fast!).
(Details and step with AF_XDP code notes:)
When the xdp_do_redirect happens with ri->flags BPF_F_CLONE_PASS, then
the map specific enqueue (e.g. __xsk_map_redirect), will do a copy of
the xdp_buff (AF_XDP calls xsk_copy_xdp()) and for AF_XDP we don't need
to do a (real) allocation. Instead of freeing the xdp_buff in xsk_rcv()
(see call to xdp_return_buff()) then we do the xdp_frame to SKB work.
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
On 5/28/21 12:54 PM, Toke Høiland-Jørgensen wrote:
quoted
Daniel Borkmann [off-list ref] writes:
quoted
On 5/28/21 12:00 PM, Magnus Karlsson wrote:
quoted
On Fri, May 28, 2021 at 11:52 AM Jesper Dangaard Brouer
[off-list ref] wrote:
quoted
On Fri, 28 May 2021 17:02:01 +0800
Xuan Zhuo [off-list ref] wrote:
quoted
On Fri, 28 May 2021 10:55:58 +0200, Toke Høiland-Jørgensen [off-list ref] wrote:
quoted
Xuan Zhuo [off-list ref] writes:
quoted
In xsk mode, users cannot use AF_PACKET(tcpdump) to observe the current
rx/tx data packets. This feature is very important in many cases. So
this patch allows AF_PACKET to obtain xsk packages.
Yes, it is rather cool (credit to Eelco). Notice the extra info you
can capture from 'exit', like XDP return codes, if_index, rx_queue.
The tool uses the perf ring-buffer to send/copy data to userspace.
This is actually surprisingly fast, but I still think AF_XDP will be
faster (but it usually 'steals' the packet).
Another (crazy?) idea is to extend this (and xdpdump), is to leverage
Hangbin's recent XDP_REDIRECT extension e624d4ed4aa8 ("xdp: Extend
xdp_redirect_map with broadcast support"). We now have a
xdp_redirect_map flag BPF_F_BROADCAST, what if we create a
BPF_F_CLONE_PASS flag?
The semantic meaning of BPF_F_CLONE_PASS flag is to copy/clone the
packet for the specified map target index (e.g AF_XDP map), but
afterwards it does like veth/cpumap and creates an SKB from the
xdp_frame (see __xdp_build_skb_from_frame()) and send to netstack.
(Feel free to kick me if this doesn't make any sense)
This would be a smooth way to implement clone support for AF_XDP. If
we had this and someone added AF_XDP support to libpcap, we could both
capture AF_XDP traffic with tcpdump (using this clone functionality in
the XDP program) and speed up tcpdump for dumping traffic destined for
regular sockets. Would that solve your use case Xuan? Note that I have
not looked into the BPF_F_CLONE_PASS code, so do not know at this
point what it would take to support this for XSKMAPs.
Recently also ended up with something similar for our XDP LB to record pcaps [0] ;)
My question is.. tcpdump doesn't really care where the packet data comes from,
so why not extending libpcap's Linux-related internals to either capture from
perf RB or BPF ringbuf rather than AF_PACKET sockets? Cloning is slow, and if
you need to end up creating an skb which is then cloned once again inside AF_PACKET
it's even worse. Just relying and reading out, say, perf RB you don't need any
clones at all.
We discussed this when creating xdpdump and decided to keep it as a
separate tool for the time being. I forget the details of the
discussion, maybe Eelco remembers.
Anyway, xdpdump does have a "pipe pcap to stdout" feature so you can do
`xdpdump | tcpdump` and get the interactive output; and it will also
save pcap information to disk, of course (using pcap-ng so it can also
save metadata like XDP program name and return code).
Right, and this should yield a significantly better performance compared to
cloning & pushing traffic into AF_PACKET. I presume not many folks are aware
of xdpdump (yet) which is probably why such patch was created here..
What, are you implying we haven't achieved world domination yet?
Inconceivable! ;)
a native libpcap implementation could solve that aspect fwiw and
additionally hook at the same points as AF_PACKET via BPF but without
the hassle/overhead of things like dev_queue_xmit_nit() in fast path.
(Maybe another option could be to have a drop-in replacement
libpcap.so for tcpdump using it transparently.)
I do believe that Michael was open to adding something like this to
tcpdump/libpcap when I last talked to him about it; and I'm certainly
not opposed to it either! Hooking up tcpdump like this may be a bit of a
firehose, though, so it would be nice to be able to carry over the
kernel-side filtering as well. I suppose it should be possible to write
an eBPF bytecode generator that does a bit of setup and then just
translates the cBPF packet filtering ops, no? This would be cool to have
in any case; IIRC Cloudflare did something like that but took a detour
through C code generation?
-Toke