PROBLEM: Bnx2x Checksum/Length Error Over GRE Tunnel

Hello,

I am having incorrect checksum and length calculation error when using
GRE tunnel and this leads to unstable/unreliable connections where even
simple nc connection cannot be made correctly due to a lot of TCP
retransmissions. The reported iperf output becomes 23Kbit/s.

When I disable tx checksumming with "ethtool -K eth0 tx off", the
problem seems to be solved but then I lose tcp segmentation offloading
support, without which adds an additional overhead on 10Gbit network
interface. With tx off, I get 4 to 7Gbit/s with parallel 40 connections
using iperf.

In tcpdump output (tcpdump -i em1 proto gre) when tx checksumming is
enabled, I see "checksum error" in every packet + "IP truncated-ip -
63631 bytes missing!" messages, and a lot of TCP
retransmissions. When tx is off, checksums are correct but I get 4-7Gbit
output. With GSO generalization code in [0] it is reported that
9.3Gbit/s is possible over GRE tunnel but I'm nowhere near these results.

Pcap file is attached when tx is on. I tried to contact the original
maintainers (broadcom) but I thought it would help to send it to netdev
mailing list as well.

Thank you for your time.

- Eren

[0] http://thread.gmane.org/gmane.linux.network/332194

Additional Information
======================
em1 is a 10G interface. Jumbo frames are enabled and MTU is set to 9000.
My GRE tunnel has 1500 MTU, and it is tunneled using em1.

Host OS: Ubuntu 14.04
Kernel: 3.18.0 (compiled from ubuntu-vivid repository)
Physical host: HP ProLiant BL460c G6
NIC: Broadcom Corporation NetXtreme II BCM57711E 10-Gigabit PCIe
Ethtool info:

259 more lines

On Tue, 2015-02-03 at 20:05 +0200, Eren Türkay wrote:

Hello,

I am having incorrect checksum and length calculation error when using
GRE tunnel and this leads to unstable/unreliable connections where even
simple nc connection cannot be made correctly due to a lot of TCP
retransmissions. The reported iperf output becomes 23Kbit/s.

When I disable tx checksumming with "ethtool -K eth0 tx off", the
problem seems to be solved but then I lose tcp segmentation offloading
support, without which adds an additional overhead on 10Gbit network
interface. With tx off, I get 4 to 7Gbit/s with parallel 40 connections
using iperf.

In tcpdump output (tcpdump -i em1 proto gre) when tx checksumming is
enabled, I see "checksum error" in every packet + "IP truncated-ip -
63631 bytes missing!" messages, and a lot of TCP
retransmissions. When tx is off, checksums are correct but I get 4-7Gbit
output. With GSO generalization code in [0] it is reported that
9.3Gbit/s is possible over GRE tunnel but I'm nowhere near these results.

Strange, I am using bnx2x with no such problem.

Please note that tcpdump can display "checksum error" because it doesn't
always know the NIC will generate the checksums.

Can you please send : ifconfig -a, as I suspect an MTU issue

Here I easily reach line rate with a single flow.

ip tunnel add gre1 mode gre remote 10.246.11.52 local 10.246.11.51
ip link set gre1 up
ip addr add 7.7.8.51/24 dev gre1


lpk51:~# ip ro get 7.7.8.52
7.7.8.52 dev gre1  src 7.7.8.51 
    cache  expires 557sec mtu 1476

lpk51:~# ./netperf -H 7.7.8.52 -Cc
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 7.7.8.52 () port 0 AF_INET
Recv   Send    Send                          Utilization       Service Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local   remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB   us/KB

 87380  16384  16384    10.00      9227.94   2.52     3.96     0.715   1.125  

On 02/03/2015 10:46 AM, Eric Dumazet wrote:

On Tue, 2015-02-03 at 20:05 +0200, Eren Türkay wrote:

quoted

Hello,

I am having incorrect checksum and length calculation error when using
GRE tunnel and this leads to unstable/unreliable connections where even
simple nc connection cannot be made correctly due to a lot of TCP
retransmissions. The reported iperf output becomes 23Kbit/s.

When I disable tx checksumming with "ethtool -K eth0 tx off", the
problem seems to be solved but then I lose tcp segmentation offloading
support, without which adds an additional overhead on 10Gbit network
interface. With tx off, I get 4 to 7Gbit/s with parallel 40 connections
using iperf.

In tcpdump output (tcpdump -i em1 proto gre) when tx checksumming is
enabled, I see "checksum error" in every packet + "IP truncated-ip -
63631 bytes missing!" messages, and a lot of TCP
retransmissions. When tx is off, checksums are correct but I get 4-7Gbit
output. With GSO generalization code in [0] it is reported that
9.3Gbit/s is possible over GRE tunnel but I'm nowhere near these results.

Strange, I am using bnx2x with no such problem.

A pedantic question, but do the driver and firmware versions match between your setup and his?

rick

On 03-02-2015 20:46, Eric Dumazet wrote:

Can you please send : ifconfig -a, as I suspect an MTU issue

You are right. Previously I set gre interface MTU to 1500 and NIC interface had 9000 MTU.
I was able to get 3Gbit in a single flow, and 5-6Gbit in parallel. When I recreated the interfaces,
the MTU of gre was set 8976 and I can get 9.5Gbit in a single flow. The reason why I set 1500 on
gre interface was to mimic openvswitch setup as the main problem I have been investigating is VM
network performance problem (they have 3Gbit throughput). I have OpenStack installation using OVS with GRE
and those interfaces are set 1500 MTU by default. I tried to test iproute2 implementation with 1500.

With correct MTU setting in gre interface, it seems to work without checksumming and offloading
but I haven't tested my original OVS setting. I will tweak OVS bridge MTU settings and try to get
it work. If baremetal-to-baremetal works, I suspect there is an issue with OVS setup.
However, when I enable tx on, netperf cannot even connect.

"ifconfig -a" and "ovs-vsctl show" result is attached.

I am not sure if tx offloading will affect the VM performance. Regardless of OVS problem, there seems to be a
bug with segmentation offloading as it's only possible to use the interface with "tx off". I have compiled the
kernel with debugging on and I can enable debugging on related parts in the kernel for further information.

Here I easily reach line rate with a single flow.

ip tunnel add gre1 mode gre remote 10.246.11.52 local 10.246.11.51
ip link set gre1 up
ip addr add 7.7.8.51/24 dev gre1

ip tunnel add gre1 mode gre remote 10.20.0.162 local 10.20.0.161
ip l set gre1 up
ip addr add 7.7.8.161/24 dev gre1

root@compute1vx:~# ip r get 7.7.8.162
7.7.8.162 dev gre1  src 7.7.8.161 
    cache  expires 539sec mtu 8976


root@compute1vx:~# ethtool -K em1 tx off
Actual changes:
tx-checksumming: off
        tx-checksum-ipv4: off
        tx-checksum-ipv6: off
tcp-segmentation-offload: off
        tx-tcp-segmentation: off [requested on]
        tx-tcp-ecn-segmentation: off [requested on]
        tx-tcp6-segmentation: off [requested on]

root@compute1vx:~# netperf -H 7.7.8.162 -Cc                                                                                                                                                                                                   
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 7.7.8.162 () port 0 AF_INET : demo
Recv   Send    Send                          Utilization       Service Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local   remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB   us/KB

 87380  16384  16384    10.00      9534.00   4.05     4.39     0.835   0.906  


root@compute1vx:~# ethtool -K em1 tx on
Actual changes:
tx-checksumming: on
        tx-checksum-ipv4: on
        tx-checksum-ipv6: on
tcp-segmentation-offload: on
        tx-tcp-segmentation: on
        tx-tcp-ecn-segmentation: on
        tx-tcp6-segmentation: on

10 more lines

I am not sure if tx offloading will affect the VM performance. Regardless of OVS
problem, there seems to be a bug with segmentation offloading as it's only
possible to use the interface with "tx off". I have compiled the kernel with
debugging on and I can enable debugging on related parts in the kernel for
further information.

quoted

Here I easily reach line rate with a single flow.

ip tunnel add gre1 mode gre remote 10.246.11.52 local 10.246.11.51 ip
link set gre1 up ip addr add 7.7.8.51/24 dev gre1

ip tunnel add gre1 mode gre remote 10.20.0.162 local 10.20.0.161 ip l set gre1
up ip addr add 7.7.8.161/24 dev gre1

Tried this on latest net-next [using iproute2] and I didn't manage to hit the
issue you've indicated.
I.e., traffic seemed to work fine on gre tunnels using both mtu sizes and
regardless of whether Tx-checksum-offload was enabled or disabled.

Is there anything else which is worth mentioning in your setup?

Thanks,
Yuval

On 05-02-2015 18:13, Yuval Mintz wrote:

Tried this on latest net-next [using iproute2] and I didn't manage to hit the
issue you've indicated.
I.e., traffic seemed to work fine on gre tunnels using both mtu sizes and
regardless of whether Tx-checksum-offload was enabled or disabled.

Interesting. Could it be a firmware issue?

To summarize my current situation, I cannot get the network work with tx on so I
need to disable tx. Using iproute2, tunnels with MTU 8976 works OK (getting
9.7Gbit/s in a single flow). When MTU is decreased to 1500, I can only get
3-4Gbit/s.

Being unable to use tx offloading is strange.

Is there anything else which is worth mentioning in your setup?

For iproute2, I don't have anything special. I just create a tunnel bridge and
test it. For OVS part, I have openstack installation using neutron openvswitch
plugin, which is irrelevant for now. Openvswitch kernel module is loaded but
does it have an effect on iproute2? Please inform me if you need anything else.

Just to inform other people having the same issue, by increasing the MTU of OVS
bridges (br-int, tap, qvo, qbr, etc.) to 8950, setting br-tun ovs bridge to
8976, and instructing VMs to use 8950 MTU, I can now get 7.5Gbit from
VM-to-network node using GRE tunnel in a single flow, and 9.5Gbit using 2 flows.
I should add that I needed to enable MQ in libvirt.xml (using 4 queues).

Thanks,
Yuval

Regards,
Eren

-- 
System Administrator
https://skyatlas.com/

quoted

Tried this on latest net-next [using iproute2] and I didn't manage to
hit the issue you've indicated.
I.e., traffic seemed to work fine on gre tunnels using both mtu sizes
and regardless of whether Tx-checksum-offload was enabled or disabled.

Interesting. Could it be a firmware issue?

All latest drivers use the same FW;  The difference is usually in management
FW, which should have no relevance to a feature such as this.

Care to share a dump [ethtool -d] and lspci [-vvv] outputs for the device?

Thanks,
Yuval