Hi!
I have a vxlan tunnel established between two network interfaces, which
both have a MTU of 1500. The vxlan interfaces have the same MTU.
With TSO enabled, I observe low throughput with TCP connections (<100 kB/s).
Disabling TSO works around this issue and throughput is as expected.
Can someone please explain how TSO is influencing the tunnel to cause
such a difference?
Kind regards,
Reiner
From: Tom Herbert <hidden> Date: 2015-02-03 16:26:28
On Tue, Feb 3, 2015 at 7:50 AM, Reiner Herrmann
[off-list ref] wrote:
Hi!
I have a vxlan tunnel established between two network interfaces, which
both have a MTU of 1500. The vxlan interfaces have the same MTU.
With TSO enabled, I observe low throughput with TCP connections (<100 kB/s).
Disabling TSO works around this issue and throughput is as expected.
Can someone please explain how TSO is influencing the tunnel to cause
such a difference?
Please provide more information. What NIC? What is the link rate? What
are you running to test. Also report 'ethtool -k' of both tunnel and
Ethernet interfaces, and tcpdump on the interfaces (to see GSO
packets).
Thanks
Kind regards,
Reiner
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Rick Jones <hidden> Date: 2015-02-03 16:36:18
On 02/03/2015 07:50 AM, Reiner Herrmann wrote:
I have a vxlan tunnel established between two network interfaces, which
both have a MTU of 1500. The vxlan interfaces have the same MTU.
With TSO enabled, I observe low throughput with TCP connections (<100
kB/s).
Disabling TSO works around this issue and throughput is as expected.
Can someone please explain how TSO is influencing the tunnel to cause
such a difference?
I've been under the impression that one generally wants the MTU of a tunnel interface to be no more than the MTU of the physical interface over which it runs, less the size of the encapsulation headers used by the tunnel. What happens when you make the MTU of the tunnel interface 1400 bytes instead of 1500?
rick jones
On Tue, Feb 3, 2015 at 7:50 AM, Reiner Herrmann
[off-list ref] wrote:
quoted
I have a vxlan tunnel established between two network interfaces, which
both have a MTU of 1500. The vxlan interfaces have the same MTU.
With TSO enabled, I observe low throughput with TCP connections (<100 kB/s).
Disabling TSO works around this issue and throughput is as expected.
Can someone please explain how TSO is influencing the tunnel to cause
such a difference?
Please provide more information. What NIC? What is the link rate? What
are you running to test. Also report 'ethtool -k' of both tunnel and
Ethernet interfaces, and tcpdump on the interfaces (to see GSO
packets).
Both are GBit NICs (it can be reproduced with different GBit NICs).
One has a link rate of 100 Mbps, the other 1000 Mbps.
To test the throughput, I'm using iperf (in TCP mode) between the two vxlan
interfaces.
I attached captures of the Ethernet interfaces from both devices, one with TSO
enabled on both vxlan interfaces (low throughput), and one with TSO disabled
on the vxlan interface of the iperf client (high throughput).
And below is the ethtool information you asked for.
Thank you for having a look.
Endpoint 1:
===========
Features for eth0:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
Features for vxlan0:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: on
tx-tcp6-segmentation: on
udp-fragmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: on [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: on
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
Endpoint 2:
===========
Features for eth0:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]
Features for vxlan0:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: on
tx-tcp6-segmentation: on
udp-fragmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: on
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]
I've been under the impression that one generally wants the MTU of a tunnel interface to be no more than the MTU of the physical interface over which it runs, less the size of the encapsulation headers used by the tunnel. What happens when you make the MTU of the tunnel interface 1400 bytes instead of 1500?
Lowering the MTU also brings throughput back to normal, but I'm interested in
why disabling TSO also seems to help (even with the same MTU).