linux-next network throughput performance regression

I compared the network throughput performance on SLES12 bare metal servers, between SLES12 default kernel and latest linux-next (2015-11-05) kernel, based on the test results, I suspect there is a network regression exists on Linux-Next over the 40G Ethernet network:
a) iperf3 reports 50% performance drop with single TCP stream on latest linux-next;
b) iperf3 reports 10% ~ 30% performance drop with 2 to 128 TCP streams on latest linux-next;
Another throughput benchmarking tool (ntttcp-for-linux) test result is also listed at the end of the email for reference.


Server configuration:
------------------------------
Two servers (one client and one server, cross linked by 40G Ethernet), which have:
a) CPU: Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz, 2 sockets, 16 CPUs, cache size : 20480 KB
b) Memory: 64 GB
c) Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro], 40G Ethernet, default driver


Test with iperf3:
------------------------------
iperf3: https://github.com/esnet/iperf

a) SLES12 default kernel, network throughput tested by iperf3:
Test Connections	1	2	4	8	16	32	64	128	
Throughput (G bps)	36.7	37.3	37.6	37.7	37.7	37.7	37.7	25.7		

b) SLES12 + Linux-Next 20151105, network throughput tested by iperf3:
Test Connections	1	2	4	8	16	32	64	128
Throughput (G bps)	18.2	32.2	34.6	32.8	27.6	32.0	27.0	21.3
Percentage dropped	-50%	-14%	-8%	-13%	-27%	-15%	-28%	-17%


Test with ntttcp-for-linux:
------------------------------
ntttcp-for-linux: https://github.com/Microsoft/ntttcp-for-linux 

a) SLES12 default kernel, network throughput tested by ntttcp-for-linux:
Test Connections	1	2	4	8	16	32	64	128	256	512
Throughput (Gbps)	36.19	37.29	37.67	37.68	37.7	37.72	37.74	37.76	37.81	37.9	

b) SLES12 + Linux-Next 20151105, network throughput tested by ntttcp-for-linux:
Test Connections	1	2	4	8	16	32	64	128	256	512	
Throughput (Gbps)	28.12	34.01	37.6	36.53	32.94	33.07	33.63	33.44	33.83	34.42	
Percentage dropped	-22%	-9%	0%	-3%	-13%	-12%	-11%	-11%	-11%	-9%

On 11/6/15 1:31 PM, Simon Xiao wrote:

I compared the network throughput performance on SLES12 bare metal servers, between SLES12 default kernel and latest linux-next (2015-11-05) kernel, based on the test results, I suspect there is a network regression exists on Linux-Next over the 40G Ethernet network:
a) iperf3 reports 50% performance drop with single TCP stream on latest linux-next;
b) iperf3 reports 10% ~ 30% performance drop with 2 to 128 TCP streams on latest linux-next;
Another throughput benchmarking tool (ntttcp-for-linux) test result is also listed at the end of the email for reference.

Can you post your kernel config file?

On 11/6/15 2:18 PM, Simon Xiao wrote:

The .config file used to build linux-next kernel is attached to this mail.

Thanks.

Failed to notice this on the first response; my brain filled in. Why 
linux-next tree? Can you try net-next which is more relevant for this 
mailing list, post the top commit id and config file used?

The .config file used to build linux-next kernel is attached to this mail.

-----Original Message-----
From: David Ahern [mailto:dsa@cumulusnetworks.com]
Sent: Friday, November 6, 2015 12:46 PM
To: Simon Xiao <redacted>; devel@linuxdriverproject.org;
netdev@vger.kernel.org; linux-kernel@vger.kernel.org
Cc: David Miller <davem@davemloft.net>; KY Srinivasan
[off-list ref]; Haiyang Zhang [off-list ref]
Subject: Re: linux-next network throughput performance regression

On 11/6/15 1:31 PM, Simon Xiao wrote:

quoted

I compared the network throughput performance on SLES12 bare metal

servers, between SLES12 default kernel and latest linux-next (2015-11-05)
kernel, based on the test results, I suspect there is a network regression
exists on Linux-Next over the 40G Ethernet network:

quoted

a) iperf3 reports 50% performance drop with single TCP stream on
latest linux-next;
b) iperf3 reports 10% ~ 30% performance drop with 2 to 128 TCP streams
on latest linux-next; Another throughput benchmarking tool (ntttcp-for-

linux) test result is also listed at the end of the email for reference.

quoted

Can you post your kernel config file?

On Fri, 2015-11-06 at 14:30 -0700, David Ahern wrote:

On 11/6/15 2:18 PM, Simon Xiao wrote:

quoted

The .config file used to build linux-next kernel is attached to this mail.

Thanks.

Failed to notice this on the first response; my brain filled in. Why 
linux-next tree? Can you try net-next which is more relevant for this 
mailing list, post the top commit id and config file used?

Throughput on a single TCP flow for a 40G NIC can be tricky to tune.

Make sure IRQ are properly setup/balanced, as I know that IRQ names were
changed recently and your scripts might have not noticed...

Also "ethtool -c eth0" might show very different interrupt coalescing
params ?

I too have a Mellanox 40Gb in my lab and saw no difference in
performance with recent kernels.

Of course, a simple "perf record -a -g sleep 4 ; perf report" might
point to some obvious issue. Like unexpected segmentation in case of
forwarding...

On Sat, 2015-11-07 at 11:35 -0800, Eric Dumazet wrote:

On Fri, 2015-11-06 at 14:30 -0700, David Ahern wrote:

quoted

On 11/6/15 2:18 PM, Simon Xiao wrote:

quoted

The .config file used to build linux-next kernel is attached to this mail.

Thanks.

Failed to notice this on the first response; my brain filled in. Why 
linux-next tree? Can you try net-next which is more relevant for this 
mailing list, post the top commit id and config file used?

Throughput on a single TCP flow for a 40G NIC can be tricky to tune.

Make sure IRQ are properly setup/balanced, as I know that IRQ names were
changed recently and your scripts might have not noticed...

Also "ethtool -c eth0" might show very different interrupt coalescing
params ?

I too have a Mellanox 40Gb in my lab and saw no difference in
performance with recent kernels.

Of course, a simple "perf record -a -g sleep 4 ; perf report" might
point to some obvious issue. Like unexpected segmentation in case of
forwarding...

I did a test with current net tree on both sender and receiver

lpaa23:~# ./netperf -H 10.246.7.152
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
10.246.7.152 () port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 87380  16384  16384    10.00    26864.98   
lpaa23:~# ethtool -c eth1
Coalesce parameters for eth1:
Adaptive RX: on  TX: off
stats-block-usecs: 0
sample-interval: 0
pkt-rate-low: 400000
pkt-rate-high: 450000

rx-usecs: 16
rx-frames: 44
rx-usecs-irq: 0
rx-frames-irq: 0

tx-usecs: 16
tx-frames: 16
tx-usecs-irq: 0
tx-frames-irq: 256

rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0

rx-usecs-high: 128
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0

lpaa23:~# ethtool -C eth1 tx-usecs 4 tx-frames 4
lpaa23:~# ./netperf -H 10.246.7.152
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
10.246.7.152 () port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 87380  16384  16384    10.00    30206.27   

From: Dexuan Cui <decui@microsoft.com>
Date: Mon, 9 Nov 2015 02:39:24 +0000

quoted

Throughput on a single TCP flow for a 40G NIC can be tricky to tune.

Why is a single TCP flow trickier than multiple TCP flows?
IMO it should be easier to analyze the issue of a single TCP flow?

Because a single TCP flow can only use one of the many TX queues
that such modern NICs have.

The single TX queue becomes the bottleneck.

Whereas if you have several TCP flows, all of them can use independant
TX queues on the NIC in parallel to fill the link with traffic.

That's why.

From: devel [mailto:driverdev-devel-bounces@linuxdriverproject.org] On Behalf
Of Eric Dumazet
Sent: Sunday, November 8, 2015 3:36
To: David Ahern <redacted>
Cc: netdev@vger.kernel.org; Haiyang Zhang <haiyangz@microsoft.com>; linux-
kernel@vger.kernel.org; devel@linuxdriverproject.org; David Miller
[off-list ref]
Subject: Re: linux-next network throughput performance regression

On Fri, 2015-11-06 at 14:30 -0700, David Ahern wrote:

quoted

On 11/6/15 2:18 PM, Simon Xiao wrote:

quoted

The .config file used to build linux-next kernel is attached to this mail.

Thanks.

Failed to notice this on the first response; my brain filled in. Why
linux-next tree? Can you try net-next which is more relevant for this
mailing list, post the top commit id and config file used?

Throughput on a single TCP flow for a 40G NIC can be tricky to tune.

Why is a single TCP flow trickier than multiple TCP flows?
IMO it should be easier to analyze the issue of a single TCP flow?

Here the perf drop in Simon's test is very obvious -- 50%, but it looks Eric
can't reproduce it, so I suppose some net-related kernel config options may
do the magic?

Maybe Simon can narrow the regression down by bisecting. :-)
 

Make sure IRQ are properly setup/balanced, as I know that IRQ names were
changed recently and your scripts might have not noticed...

Also "ethtool -c eth0" might show very different interrupt coalescing
params ?

I too have a Mellanox 40Gb in my lab and saw no difference in
performance with recent kernels.

Of course, a simple "perf record -a -g sleep 4 ; perf report" might
point to some obvious issue. Like unexpected segmentation in case of
forwarding...

Thanks,
-- Dexuan

From: Dexuan Cui <decui@microsoft.com>
Date: Mon, 9 Nov 2015 03:11:35 +0000

quoted

-----Original Message-----
From: David Miller [mailto:davem@davemloft.net]
Sent: Monday, November 9, 2015 10:53
To: Dexuan Cui <decui@microsoft.com>
Cc: eric.dumazet@gmail.com; dsa@cumulusnetworks.com; Simon Xiao
[off-list ref]; netdev@vger.kernel.org; Haiyang Zhang
[off-list ref]; linux-kernel@vger.kernel.org;
devel@linuxdriverproject.org
Subject: Re: linux-next network throughput performance regression

From: Dexuan Cui <decui@microsoft.com>
Date: Mon, 9 Nov 2015 02:39:24 +0000

quoted

quoted

Throughput on a single TCP flow for a 40G NIC can be tricky to tune.

Why is a single TCP flow trickier than multiple TCP flows?
IMO it should be easier to analyze the issue of a single TCP flow?

Because a single TCP flow can only use one of the many TX queues
that such modern NICs have.

The single TX queue becomes the bottleneck.

Whereas if you have several TCP flows, all of them can use independant
TX queues on the NIC in parallel to fill the link with traffic.

That's why.

Thanks, David!
I understand 1 TX queue is the bottleneck (however in Simon's
test, TX=1 => 36.7Gb/s, TX=8 => 37.7 Gb/s, so it looks the TX=1 bottleneck
is not so obvious).
I'm just wondering how the bottleneck became much narrower with
recent linux-next in Simon's result (36.7 Gb/s vs. 18.2 Gb/s). IMO there
must be some latency somewhere.

I think the whole thing here is that you misinterpreted what Eric said.

He is not arguing that some regression did, or did not, happen.

He instead was making the basic statement about the fact that due to
the lack of paralellness a single stream TCP case is harder to
optimize for high speed NICs.

That is all.

-----Original Message-----
From: David Miller [mailto:davem@davemloft.net]
Sent: Monday, November 9, 2015 10:53
To: Dexuan Cui <decui@microsoft.com>
Cc: eric.dumazet@gmail.com; dsa@cumulusnetworks.com; Simon Xiao
[off-list ref]; netdev@vger.kernel.org; Haiyang Zhang
[off-list ref]; linux-kernel@vger.kernel.org;
devel@linuxdriverproject.org
Subject: Re: linux-next network throughput performance regression

From: Dexuan Cui <decui@microsoft.com>
Date: Mon, 9 Nov 2015 02:39:24 +0000

quoted

quoted

Throughput on a single TCP flow for a 40G NIC can be tricky to tune.

Why is a single TCP flow trickier than multiple TCP flows?
IMO it should be easier to analyze the issue of a single TCP flow?

Because a single TCP flow can only use one of the many TX queues
that such modern NICs have.

The single TX queue becomes the bottleneck.

Whereas if you have several TCP flows, all of them can use independant
TX queues on the NIC in parallel to fill the link with traffic.

That's why.

Thanks, David!
I understand 1 TX queue is the bottleneck (however in Simon's
test, TX=1 => 36.7Gb/s, TX=8 => 37.7 Gb/s, so it looks the TX=1 bottleneck
is not so obvious).
I'm just wondering how the bottleneck became much narrower with
recent linux-next in Simon's result (36.7 Gb/s vs. 18.2 Gb/s). IMO there
must be some latency somewhere.

Thanks,
-- Dexuan

On 9 November 2015 at 13:23, David Miller [off-list ref] wrote:

From: Dexuan Cui <decui@microsoft.com>
Date: Mon, 9 Nov 2015 03:11:35 +0000

quoted

quoted

-----Original Message-----
From: David Miller [mailto:davem@davemloft.net]
Sent: Monday, November 9, 2015 10:53
To: Dexuan Cui <decui@microsoft.com>
Cc: eric.dumazet@gmail.com; dsa@cumulusnetworks.com; Simon Xiao
[off-list ref]; netdev@vger.kernel.org; Haiyang Zhang
[off-list ref]; linux-kernel@vger.kernel.org;
devel@linuxdriverproject.org
Subject: Re: linux-next network throughput performance regression

From: Dexuan Cui <decui@microsoft.com>
Date: Mon, 9 Nov 2015 02:39:24 +0000

quoted

quoted

Throughput on a single TCP flow for a 40G NIC can be tricky to tune.

Why is a single TCP flow trickier than multiple TCP flows?
IMO it should be easier to analyze the issue of a single TCP flow?

Because a single TCP flow can only use one of the many TX queues
that such modern NICs have.

The single TX queue becomes the bottleneck.

Whereas if you have several TCP flows, all of them can use independant
TX queues on the NIC in parallel to fill the link with traffic.

That's why.

Thanks, David!
I understand 1 TX queue is the bottleneck (however in Simon's
test, TX=1 => 36.7Gb/s, TX=8 => 37.7 Gb/s, so it looks the TX=1 bottleneck
is not so obvious).
I'm just wondering how the bottleneck became much narrower with
recent linux-next in Simon's result (36.7 Gb/s vs. 18.2 Gb/s). IMO there
must be some latency somewhere.

I think the whole thing here is that you misinterpreted what Eric said.

He is not arguing that some regression did, or did not, happen.

He instead was making the basic statement about the fact that due to
the lack of paralellness a single stream TCP case is harder to
optimize for high speed NICs.

That is all.

We recently had a regression tracked down in a similiar area that was
because of link order.

Dave.

From: David Miller [mailto:davem@davemloft.net]
Sent: Monday, November 9, 2015 11:24
...

quoted

Thanks, David!
I understand 1 TX queue is the bottleneck (however in Simon's
test, TX=1 => 36.7Gb/s, TX=8 => 37.7 Gb/s, so it looks the TX=1 bottleneck
is not so obvious).
I'm just wondering how the bottleneck became much narrower with
recent linux-next in Simon's result (36.7 Gb/s vs. 18.2 Gb/s). IMO there
must be some latency somewhere.

I think the whole thing here is that you misinterpreted what Eric said.

He is not arguing that some regression did, or did not, happen.

He instead was making the basic statement about the fact that due to
the lack of paralellness a single stream TCP case is harder to
optimize for high speed NICs.

That is all.

Thanks, I got it. 
I'm actually new to network performance tuning, trying to understand
all the related details. :-)

Thanks,
-- Dexuan

On Sun, Nov 8, 2015 at 7:31 PM, Dexuan Cui [off-list ref] wrote:

quoted

From: David Miller [mailto:davem@davemloft.net]
Sent: Monday, November 9, 2015 11:24
...

quoted

Thanks, David!
I understand 1 TX queue is the bottleneck (however in Simon's
test, TX=1 => 36.7Gb/s, TX=8 => 37.7 Gb/s, so it looks the TX=1 bottleneck
is not so obvious).
I'm just wondering how the bottleneck became much narrower with
recent linux-next in Simon's result (36.7 Gb/s vs. 18.2 Gb/s). IMO there
must be some latency somewhere.

I think the whole thing here is that you misinterpreted what Eric said.

He is not arguing that some regression did, or did not, happen.

He instead was making the basic statement about the fact that due to
the lack of paralellness a single stream TCP case is harder to
optimize for high speed NICs.

That is all.

Thanks, I got it.
I'm actually new to network performance tuning, trying to understand
all the related details. :-)

You might want to look at
https://www.kernel.org/doc/Documentation/networking/scaling.txt as an
introduction to the scaling capabilities of the stack.

Tom

Thanks,
-- Dexuan
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Thanks Eric to provide the data. I am looping Tom (as I am looking into his recent patches) and Olaf (from Suse).

So, if I understand it correctly, you are running netperf with single TCP connection, and you got ~26Gbps initially and got ~30Gbps after turning the tx-usecs and tx-frames.

Do you have a baseline on your environment for the best/max/or peak throughput?
Again, in my environment (SLES bare metal), if use SLES12 default kernel as a baseline, we can see significant performance drop (10% ~ 50%) on latest linux-next kernel. 
Absolutely I will try the same test on net-next soon and update the results to here later.

Thanks,
Simon

-----Original Message-----
From: Eric Dumazet [mailto:eric.dumazet@gmail.com]
Sent: Saturday, November 7, 2015 11:50 AM
To: David Ahern <redacted>
Cc: Simon Xiao <redacted>; devel@linuxdriverproject.org;
netdev@vger.kernel.org; linux-kernel@vger.kernel.org; David Miller
[off-list ref]; KY Srinivasan [off-list ref]; Haiyang
Zhang [off-list ref]
Subject: Re: linux-next network throughput performance regression

On Sat, 2015-11-07 at 11:35 -0800, Eric Dumazet wrote:

quoted

On Fri, 2015-11-06 at 14:30 -0700, David Ahern wrote:

quoted

On 11/6/15 2:18 PM, Simon Xiao wrote:

quoted

The .config file used to build linux-next kernel is attached to this mail.

Thanks.

Failed to notice this on the first response; my brain filled in. Why
linux-next tree? Can you try net-next which is more relevant for
this mailing list, post the top commit id and config file used?

Throughput on a single TCP flow for a 40G NIC can be tricky to tune.

Make sure IRQ are properly setup/balanced, as I know that IRQ names
were changed recently and your scripts might have not noticed...

Also "ethtool -c eth0" might show very different interrupt coalescing
params ?

I too have a Mellanox 40Gb in my lab and saw no difference in
performance with recent kernels.

Of course, a simple "perf record -a -g sleep 4 ; perf report" might
point to some obvious issue. Like unexpected segmentation in case of
forwarding...

I did a test with current net tree on both sender and receiver

lpaa23:~# ./netperf -H 10.246.7.152
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
10.246.7.152 () port 0 AF_INET
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

 87380  16384  16384    10.00    26864.98
lpaa23:~# ethtool -c eth1
Coalesce parameters for eth1:
Adaptive RX: on  TX: off
stats-block-usecs: 0
sample-interval: 0
pkt-rate-low: 400000
pkt-rate-high: 450000

rx-usecs: 16
rx-frames: 44
rx-usecs-irq: 0
rx-frames-irq: 0

tx-usecs: 16
tx-frames: 16
tx-usecs-irq: 0
tx-frames-irq: 256

rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0

rx-usecs-high: 128
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0

lpaa23:~# ethtool -C eth1 tx-usecs 4 tx-frames 4

lpaa23:~# ./netperf -H
10.246.7.152 MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0
AF_INET to
10.246.7.152 () port 0 AF_INET
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

 87380  16384  16384    10.00    30206.27

On Mon, 2015-11-09 at 20:23 +0000, Simon Xiao wrote:

Thanks Eric to provide the data. I am looping Tom (as I am looking into his recent patches) and Olaf (from Suse).

So, if I understand it correctly, you are running netperf with single
TCP connection, and you got ~26Gbps initially and got ~30Gbps after
turning the tx-usecs and tx-frames.

Do you have a baseline on your environment for the best/max/or peak
throughput?

The peak on my lab pair is about 34Gbits, usually I get this if I pin
the receiving thread on a cpu, otherwise process scheduler can really
hurt too much.

lpaa23:~# DUMP_TCP_INFO=1 ./netperf -H lpaa24 -l 20 -Cc -T ,1
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to lpaa24.prod.google.com () port 0 AF_INET : cpu bind
tcpi_rto 201000 tcpi_ato 0 tcpi_pmtu 1500 tcpi_rcv_ssthresh 29200
tcpi_rtt 101 tcpi_rttvar 15 tcpi_snd_ssthresh 289 tpci_snd_cwnd 289
tcpi_reordering 3 tcpi_total_retrans 453
Recv   Send    Send                          Utilization       Service Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local   remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB   us/KB

 87380  16384  16384    20.00      33975.99   1.27     3.36     0.147   0.389  

Not too bad, I don't recall reaching more than that ever.