Hi Eric,
I'm trying to use GRE with RSS, but it looks broken on net-next since:
60769a5dcd8755715c7143b4571d5c44f01796f1 is the first bad commit
commit 60769a5dcd8755715c7143b4571d5c44f01796f1
Author: Eric Dumazet [off-list ref]
Date: Thu Sep 27 02:48:50 2012 +0000
ipv4: gre: add GRO capability
Add GRO capability to IPv4 GRE tunnels, using the gro_cells
infrastructure.
Tested using IPv4 and IPv6 TCP traffic inside this tunnel, and
checking GRO is building large packets.
Signed-off-by: Eric Dumazet [off-list ref]
Signed-off-by: David S. Miller [off-list ref]
:040000 040000 8eb4f570181b6d72abe24f8c1123b7e49134e662 fa20194bb14d1745e9271c8a962d0f140801a226 M include
:040000 040000 6f605ade7fed9fbe5fd57d4a0c3a8dc687e64ed6 4c06b880a6a6068aa791decb900f7b449c6ec7b5 M net
Multiple TCP streams over the tunnel cause (almost) immediately GRE interface to drop any ingress packet.
Please note that at current net-next head behavior is different - I hit null pointer dereference, I will try to bisect this behavior too.
-----Original Message-----
From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org]
On Behalf Of Dmitry Kravkov
Sent: Sunday, December 09, 2012 12:35 AM
To: Eric Dumazet; netdev@vger.kernel.org
Subject: ipgre rss is broken since gro
Please note that at current net-next head behavior is different - I hit null pointer
dereference, I will try to bisect this behavior too.
For the GRO stuff and RSS, I wonder if skbs have a property that makes
them dropped somewhere, you might try drop_monitor / drop_watch
for this item: drop_watch does not show any drops (i've disable all
other interfaces for clear env)
I will explain a little bit more the setup:
bnx2x device (under testing) is configured for RSS for IPGRE packets.
Sending multiple (3) TCP_STREAM causes ip_gre interface to disappear
packets (even ICMP).
This is not happening with single TCP_STREAM, or before gro_cell
introduction.
i was searching for the drops by print-out in ip_gre.c but disappeared
packets completes this code:
static int ipgre_rcv(struct sk_buff *skb)
(snip)
printk("%s:%d\n", __FUNCTION__, __LINE__);
tstats = this_cpu_ptr(tunnel->dev->tstats);
u64_stats_update_begin(&tstats->syncp);
tstats->rx_packets++;
tstats->rx_bytes += skb->len;
u64_stats_update_end(&tstats->syncp);
gro_cells_receive(&tunnel->gro_cells, skb);
return 0;
From: Eric Dumazet <edumazet@google.com> Date: 2012-12-09 23:27:08
On Sun, Dec 9, 2012 at 12:49 PM, Dmitry Kravkov [off-list ref] wrote:
for this item: drop_watch does not show any drops (i've disable all
other interfaces for clear env)
I will explain a little bit more the setup:
bnx2x device (under testing) is configured for RSS for IPGRE packets.
Sending multiple (3) TCP_STREAM causes ip_gre interface to disappear
packets (even ICMP).
This is not happening with single TCP_STREAM, or before gro_cell
introduction.
i was searching for the drops by print-out in ip_gre.c but disappeared
packets completes this code:
static int ipgre_rcv(struct sk_buff *skb)
(snip)
printk("%s:%d\n", __FUNCTION__, __LINE__);
tstats = this_cpu_ptr(tunnel->dev->tstats);
u64_stats_update_begin(&tstats->syncp);
tstats->rx_packets++;
tstats->rx_bytes += skb->len;
u64_stats_update_end(&tstats->syncp);
gro_cells_receive(&tunnel->gro_cells, skb);
return 0;
I dont know, I tried a bnx2x setup, and 100 tcp flows, no special problem.
If you receive a lot of packets on a single RX queue, they might be
dropped because cpu cant cope with the load
(This has nothing to do with GRE or GRO )
cat /proc/net/softnet_stat
-----Original Message-----
From: Eric Dumazet [mailto:edumazet@google.com]
Sent: Monday, December 10, 2012 1:27 AM
To: Dmitry Kravkov
Cc: netdev@vger.kernel.org
Subject: Re: ipgre rss is broken since gro
On Sun, Dec 9, 2012 at 12:49 PM, Dmitry Kravkov [off-list ref]
wrote:
quoted
for this item: drop_watch does not show any drops (i've disable all
other interfaces for clear env)
I will explain a little bit more the setup:
bnx2x device (under testing) is configured for RSS for IPGRE packets.
Sending multiple (3) TCP_STREAM causes ip_gre interface to disappear
packets (even ICMP).
This is not happening with single TCP_STREAM, or before gro_cell
introduction.
I dont know, I tried a bnx2x setup, and 100 tcp flows, no special problem.
Current bnx2x do not apply RSS for GRE, non GRE RSS is working w/o problem.
If you receive a lot of packets on a single RX queue, they might be
dropped because cpu cant cope with the load
(This has nothing to do with GRE or GRO )
CPU is not loaded at all
cat /proc/net/softnet_stat
Please find attached.
For gre interface RX and DROP statistics are advancing simultaneously (by one each ICMP request):
[root@ ~]# ifconfig gre
gre Link encap:UNSPEC HWaddr C0-A8-0A-40-73-72-83-D2-00-00-00-00-00-00-00-00
inet addr:8.0.0.1 P-t-P:8.0.0.1 Mask:255.255.255.0
inet6 addr: fe80::5efe:c0a8:a40/64 Scope:Link
UP POINTOPOINT RUNNING NOARP MTU:1476 Metric:1
RX packets:1646824 errors:0 dropped:51610 overruns:0 frame:0
TX packets:140519 errors:1 dropped:0 overruns:0 carrier:1
collisions:0 txqueuelen:0
RX bytes:2357650904 (2.1 GiB) TX bytes:7309072 (6.9 MiB)
[root@ ~]# ifconfig gre
gre Link encap:UNSPEC HWaddr C0-A8-0A-40-73-72-83-82-00-00-00-00-00-00-00-00
inet addr:8.0.0.1 P-t-P:8.0.0.1 Mask:255.255.255.0
inet6 addr: fe80::5efe:c0a8:a40/64 Scope:Link
UP POINTOPOINT RUNNING NOARP MTU:1476 Metric:1
RX packets:1646826 errors:0 dropped:51612 overruns:0 frame:0
TX packets:140519 errors:1 dropped:0 overruns:0 carrier:1
collisions:0 txqueuelen:0
RX bytes:2357651072 (2.1 GiB) TX bytes:7309072 (6.9 MiB)
[root@ ~]# tcpdump -i gre
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on gre, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
2 packets dropped by interface
From: Eric Dumazet <hidden> Date: 2012-12-10 16:54:35
On Mon, 2012-12-10 at 11:32 +0000, Dmitry Kravkov wrote:
CPU is not loaded at all
quoted
cat /proc/net/softnet_stat
Please find attached.
For gre interface RX and DROP statistics are advancing simultaneously (by one each ICMP request):
[root@ ~]# ifconfig gre
gre Link encap:UNSPEC HWaddr C0-A8-0A-40-73-72-83-D2-00-00-00-00-00-00-00-00
inet addr:8.0.0.1 P-t-P:8.0.0.1 Mask:255.255.255.0
inet6 addr: fe80::5efe:c0a8:a40/64 Scope:Link
UP POINTOPOINT RUNNING NOARP MTU:1476 Metric:1
RX packets:1646824 errors:0 dropped:51610 overruns:0 frame:0
TX packets:140519 errors:1 dropped:0 overruns:0 carrier:1
collisions:0 txqueuelen:0
RX bytes:2357650904 (2.1 GiB) TX bytes:7309072 (6.9 MiB)
dropped:51610 so obviously one cpu is fully loaded.
I believe performance problem might come from the
skb_set_queue_mapping(skb, 0); in __skb_tunnel_rx()
So all packets are queued into a single GRO queue, instead of being
split as intended in multiple queues.
I cant find why we must clear queue_mapping, so could you try :
From: David Miller <davem@davemloft.net> Date: 2012-12-10 19:02:47
From: Eric Dumazet <redacted>
Date: Mon, 10 Dec 2012 08:54:31 -0800
I believe performance problem might come from the
skb_set_queue_mapping(skb, 0); in __skb_tunnel_rx()
So all packets are queued into a single GRO queue, instead of being
split as intended in multiple queues.
I cant find why we must clear queue_mapping, so could you try :
Tom says:
commit 693019e90ca45d881109d32c0c6d29adf03f6447
Author: Tom Herbert [off-list ref]
Date: Thu Sep 23 11:19:54 2010 +0000
net: reset skb queue mapping when rx'ing over tunnel
Reset queue mapping when an skb is reentering the stack via a tunnel.
On second pass, the queue mapping from the original device is no
longer valid.
Signed-off-by: Tom Herbert [off-list ref]
Acked-by: Eric Dumazet [off-list ref]
Signed-off-by: David S. Miller [off-list ref]
-----Original Message-----
From: Eric Dumazet [mailto:eric.dumazet@gmail.com]
Sent: Monday, December 10, 2012 6:55 PM
To: Dmitry Kravkov
Cc: Eric Dumazet; netdev@vger.kernel.org
Subject: RE: ipgre rss is broken since gro
On Mon, 2012-12-10 at 11:32 +0000, Dmitry Kravkov wrote:
dropped:51610 so obviously one cpu is fully loaded.
Link partner continued pushing data overnight
quoted hunk
I believe performance problem might come from the
skb_set_queue_mapping(skb, 0); in __skb_tunnel_rx()
So all packets are queued into a single GRO queue, instead of being
split as intended in multiple queues.
I cant find why we must clear queue_mapping, so could you try :
From: Eric Dumazet <hidden> Date: 2012-12-10 21:55:53
On Mon, 2012-12-10 at 19:20 +0000, Dmitry Kravkov wrote:
Yep, this resolved the issue - Interface is functional after 3 and 100 TCP connections. Thanks
I guess its only lowering probability of the race.
Could you try instead the following patch ?
I'll address the queue_mapping separately for net-next
Thanks
From: Eric Dumazet <hidden> Date: 2012-12-10 22:32:06
From: Eric Dumazet <edumazet@google.com>
Dmitry Kravkov reported packet drops for GRE packets since GRO support
was added.
There is a race in gro_cell_poll() because we call napi_complete()
without any synchronization with a concurrent gro_cells_receive()
Once bug was triggered, we queued packets but did not schedule NAPI
poll.
We can fix this issue using the spinlock protected the napi_skbs queue,
as we have to hold it to perform skb dequeue anyway.
As we open-code skb_dequeue(), we no longer need to mask IRQS, as both
producer and consumer run under BH context.
Bug added in commit c9e6bc644e (net: add gro_cells infrastructure)
Reported-by: Dmitry Kravkov <redacted>
Signed-off-by: Eric Dumazet <edumazet@google.com>
---
David: I could reproduce the bug Dmitry reported, and have
verified this patch fixes the issue.
include/net/gro_cells.h | 14 +++++++++-----
1 file changed, 9 insertions(+), 5 deletions(-)
-----Original Message-----
From: Eric Dumazet [mailto:eric.dumazet@gmail.com]
Sent: Tuesday, December 11, 2012 12:32 AM
To: Dmitry Kravkov; David Miller
Cc: netdev@vger.kernel.org
Subject: [PATCH] net: fix a race in gro_cell_poll()
From: Eric Dumazet <edumazet@google.com>
Dmitry Kravkov reported packet drops for GRE packets since GRO support
was added.
There is a race in gro_cell_poll() because we call napi_complete()
without any synchronization with a concurrent gro_cells_receive()
Once bug was triggered, we queued packets but did not schedule NAPI
poll.
We can fix this issue using the spinlock protected the napi_skbs queue,
as we have to hold it to perform skb dequeue anyway.
As we open-code skb_dequeue(), we no longer need to mask IRQS, as both
producer and consumer run under BH context.
Bug added in commit c9e6bc644e (net: add gro_cells infrastructure)
Reported-by: Dmitry Kravkov <redacted>
Signed-off-by: Eric Dumazet <edumazet@google.com>
---
David: I could reproduce the bug Dmitry reported, and have
verified this patch fixes the issue.
include/net/gro_cells.h | 14 +++++++++-----
1 file changed, 9 insertions(+), 5 deletions(-)
From: David Miller <davem@davemloft.net> Date: 2012-12-11 17:50:55
From: Eric Dumazet <redacted>
Date: Mon, 10 Dec 2012 14:32:03 -0800
From: Eric Dumazet <edumazet@google.com>
Dmitry Kravkov reported packet drops for GRE packets since GRO support
was added.
There is a race in gro_cell_poll() because we call napi_complete()
without any synchronization with a concurrent gro_cells_receive()
Once bug was triggered, we queued packets but did not schedule NAPI
poll.
We can fix this issue using the spinlock protected the napi_skbs queue,
as we have to hold it to perform skb dequeue anyway.
As we open-code skb_dequeue(), we no longer need to mask IRQS, as both
producer and consumer run under BH context.
Bug added in commit c9e6bc644e (net: add gro_cells infrastructure)
Reported-by: Dmitry Kravkov <redacted>
Signed-off-by: Eric Dumazet <edumazet@google.com>