17 messages, 2 authors, 2012-06-20 · open the first message on its own page
From: Stefan Priebe <hidden>
Date: 2012-06-19 21:08:07
Hello List, i'm testing 10GBE speed with tweo servers. One with 3.5-rc3 nd thoe other one whith RHEL 6 (2.6.32 kernel). I noticed that setting net.ipv4.tcp_timestamps=0 descreased the performance from 9,7 Full Duplex to 3-4Gb/s. Is this bahviour fine? What should / could i tet? Greets Stefan
From: Eric Dumazet <hidden>
Date: 2012-06-19 21:31:38
On Tue, 2012-06-19 at 23:08 +0200, Stefan Priebe wrote:
Hello List, i'm testing 10GBE speed with tweo servers. One with 3.5-rc3 nd thoe other one whith RHEL 6 (2.6.32 kernel). I noticed that setting net.ipv4.tcp_timestamps=0 descreased the performance from 9,7 Full Duplex to 3-4Gb/s. Is this bahviour fine? What should / could i tet?
Really, you should provide more input than that, if you really want us to help.
From: Stefan Priebe - Profihost AG <hidden>
Date: 2012-06-20 07:00:56
Am 19.06.2012 23:31, schrieb Eric Dumazet:
On Tue, 2012-06-19 at 23:08 +0200, Stefan Priebe wrote:quoted
i'm testing 10GBE speed with tweo servers. One with 3.5-rc3 nd thoe other one whith RHEL 6 (2.6.32 kernel). I noticed that setting net.ipv4.tcp_timestamps=0 descreased the performance from 9,7 Full Duplex to 3-4Gb/s. Is this bahviour fine? What should / could i tet?Really, you should provide more input than that, if you really want us to help.
*arg* forgot to add the pastebin links. Sorry. Speed degraded in this case from 9,88Gbit/s to 2,45Gbit/s. When i turn on timestamps it's perfect again. Server A has 3.5.0-rc3 and server B has an RHEL6 2.6.32 kernel. Before: http://pastebin.com/raw.php?i=1gVraWVc After: http://pastebin.com/raw.php?i=NSh8Y29s Thanks, Stefan
From: Eric Dumazet <hidden>
Date: 2012-06-20 07:22:25
On Wed, 2012-06-20 at 09:00 +0200, Stefan Priebe - Profihost AG wrote:
Am 19.06.2012 23:31, schrieb Eric Dumazet:quoted
On Tue, 2012-06-19 at 23:08 +0200, Stefan Priebe wrote:quoted
i'm testing 10GBE speed with tweo servers. One with 3.5-rc3 nd thoe other one whith RHEL 6 (2.6.32 kernel). I noticed that setting net.ipv4.tcp_timestamps=0 descreased the performance from 9,7 Full Duplex to 3-4Gb/s. Is this bahviour fine? What should / could i tet?Really, you should provide more input than that, if you really want us to help.*arg* forgot to add the pastebin links. Sorry. Speed degraded in this case from 9,88Gbit/s to 2,45Gbit/s. When i turn on timestamps it's perfect again. Server A has 3.5.0-rc3 and server B has an RHEL6 2.6.32 kernel. Before: http://pastebin.com/raw.php?i=1gVraWVc After: http://pastebin.com/raw.php?i=NSh8Y29s
You have a lot of packet losses add "tc -s -d qdisc" , "ifconfig -a " and "ethtool -S ethX" outputs for both servers
From: Stefan Priebe - Profihost AG <hidden>
Date: 2012-06-20 08:21:54
Am 20.06.2012 09:22, schrieb Eric Dumazet:
On Wed, 2012-06-20 at 09:00 +0200, Stefan Priebe - Profihost AG wrote:quoted
Am 19.06.2012 23:31, schrieb Eric Dumazet: Before: http://pastebin.com/raw.php?i=1gVraWVc After: http://pastebin.com/raw.php?i=NSh8Y29sYou have a lot of packet losses
But this ONLY happens with tcp_timestamps=0.
add "tc -s -d qdisc" , "ifconfig -a " and "ethtool -S ethX" outputs for both servers
eth2 is the 10Gb device on both systems server a has kernel 3.5 server b has rhel 6 kernel Server A: # tc -s -d qdisc RTNETLINK answers: Operation not supported Dump terminated Server B (eth2 is the 10GB/s device): # tc -s -d qdisc qdisc mq 0: dev eth0 root Sent 55151 bytes 555 pkt (dropped 0, overlimits 0 requeues 0) rate 0bit 0pps backlog 0b 0p requeues 0 qdisc mq 0: dev eth2 root Sent 38374148475 bytes 2774405 pkt (dropped 0, overlimits 0 requeues 4) rate 0bit 0pps backlog 0b 0p requeues 4 ifconfig -a: http://pastebin.com/raw.php?i=M3QHQjSU ethtool -S: http://pastebin.com/raw.php?i=Eap05xKc Thanks again, Stefan
From: Eric Dumazet <hidden>
Date: 2012-06-20 08:41:21
On Wed, 2012-06-20 at 10:21 +0200, Stefan Priebe - Profihost AG wrote:
Am 20.06.2012 09:22, schrieb Eric Dumazet:quoted
On Wed, 2012-06-20 at 09:00 +0200, Stefan Priebe - Profihost AG wrote:quoted
Am 19.06.2012 23:31, schrieb Eric Dumazet: Before: http://pastebin.com/raw.php?i=1gVraWVc After: http://pastebin.com/raw.php?i=NSh8Y29sYou have a lot of packet lossesBut this ONLY happens with tcp_timestamps=0.
Yes, you already told that in subject line. single tcp flow ? You seem to have a switch or something that drops packets in this case. You could try to rate limit to 9Gb/s and see if it is better. Here, I roughly have same bandwidth with tcp_timestamps on or off, with ixgbe cards and net-next kernels.
From: Stefan Priebe - Profihost AG <hidden>
Date: 2012-06-20 09:06:47
Am 20.06.2012 10:41, schrieb Eric Dumazet:
On Wed, 2012-06-20 at 10:21 +0200, Stefan Priebe - Profihost AG wrote:quoted
Am 20.06.2012 09:22, schrieb Eric Dumazet:Yes, you already told that in subject line. single tcp flow ?
I use iperf - i think it uses just a single tcp flow. But i'm not sure.
You seem to have a switch or something that drops packets in this case. You could try to rate limit to 9Gb/s and see if it is better.
Sadly i can't rate limit to 9Gbit/s on the switch.
Here, I roughly have same bandwidth with tcp_timestamps on or off, with ixgbe cards and net-next kernels.
Mhm strange. Do you have any vague idea what could cause this? Any wrong reordering of the packets without tcp_timestamps? Stefan
From: Stefan Priebe - Profihost AG <hidden>
Date: 2012-06-20 09:12:33
Am 20.06.2012 11:06, schrieb Stefan Priebe - Profihost AG:
quoted
You seem to have a switch or something that drops packets in this case. You could try to rate limit to 9Gb/s and see if it is better.Sadly i can't rate limit to 9Gbit/s on the switch.quoted
Here, I roughly have same bandwidth with tcp_timestamps on or off, with ixgbe cards and net-next kernels.Mhm strange. Do you have any vague idea what could cause this? Any wrong reordering of the packets without tcp_timestamps?
I've now done another test without the switch. So both systems where direct attached and still the same. Without tcp_timstamps speed drops to 2-4Gbit/s. Stefan
From: Eric Dumazet <hidden>
Date: 2012-06-20 09:16:38
On Wed, 2012-06-20 at 11:06 +0200, Stefan Priebe - Profihost AG wrote:
Am 20.06.2012 10:41, schrieb Eric Dumazet:quoted
On Wed, 2012-06-20 at 10:21 +0200, Stefan Priebe - Profihost AG wrote:quoted
Am 20.06.2012 09:22, schrieb Eric Dumazet:Yes, you already told that in subject line. single tcp flow ?I use iperf - i think it uses just a single tcp flow. But i'm not sure.quoted
You seem to have a switch or something that drops packets in this case. You could try to rate limit to 9Gb/s and see if it is better.Sadly i can't rate limit to 9Gbit/s on the switch.
I was suggesting rate limiting on your linux sender machine, just to verify if its indeed a problem on the path. Its a matter of a "tc qdisc ..." command ;) Or maybe iperf has an option for rate limiting.
From: Eric Dumazet <hidden>
Date: 2012-06-20 09:17:39
On Wed, 2012-06-20 at 11:12 +0200, Stefan Priebe - Profihost AG wrote:
Am 20.06.2012 11:06, schrieb Stefan Priebe - Profihost AG:quoted
quoted
You seem to have a switch or something that drops packets in this case. You could try to rate limit to 9Gb/s and see if it is better.Sadly i can't rate limit to 9Gbit/s on the switch.quoted
Here, I roughly have same bandwidth with tcp_timestamps on or off, with ixgbe cards and net-next kernels.Mhm strange. Do you have any vague idea what could cause this? Any wrong reordering of the packets without tcp_timestamps?I've now done another test without the switch. So both systems where direct attached and still the same. Without tcp_timstamps speed drops to 2-4Gbit/s. Stefan
If you exchange sender/receiver role between linux kernel versions, is it the same problem ?
From: Stefan Priebe - Profihost AG <hidden>
Date: 2012-06-20 09:25:15
Am 20.06.2012 11:17, schrieb Eric Dumazet:
On Wed, 2012-06-20 at 11:12 +0200, Stefan Priebe - Profihost AG wrote:quoted
Am 20.06.2012 11:06, schrieb Stefan Priebe - Profihost AG:quoted
quoted
You seem to have a switch or something that drops packets in this case. You could try to rate limit to 9Gb/s and see if it is better.Sadly i can't rate limit to 9Gbit/s on the switch.
If you exchange sender/receiver role between linux kernel versions, is it the same problem ?
I'm testing in both directions. So both are sending and receiving. I've now made tests with only one sending an the other receiving. When server B is the sender i get 4Gbit/s. When server A is the sender i get full 9,9Gbit/s. Stefan
From: Eric Dumazet <hidden>
Date: 2012-06-20 09:28:53
On Wed, 2012-06-20 at 11:25 +0200, Stefan Priebe - Profihost AG wrote:
Am 20.06.2012 11:17, schrieb Eric Dumazet:quoted
On Wed, 2012-06-20 at 11:12 +0200, Stefan Priebe - Profihost AG wrote:quoted
Am 20.06.2012 11:06, schrieb Stefan Priebe - Profihost AG:quoted
quoted
You seem to have a switch or something that drops packets in this case. You could try to rate limit to 9Gb/s and see if it is better.Sadly i can't rate limit to 9Gbit/s on the switch.quoted
If you exchange sender/receiver role between linux kernel versions, is it the same problem ?I'm testing in both directions. So both are sending and receiving. I've now made tests with only one sending an the other receiving. When server B is the sender i get 4Gbit/s. When server A is the sender i get full 9,9Gbit/s. Stefan
To rule out bad hardware, could you try a 3.5-rc3 kernel on B ?
From: Stefan Priebe - Profihost AG <hidden>
Date: 2012-06-20 09:33:56
Am 20.06.2012 11:28, schrieb Eric Dumazet:
On Wed, 2012-06-20 at 11:25 +0200, Stefan Priebe - Profihost AG wrote:quoted
Am 20.06.2012 11:17, schrieb Eric Dumazet:quoted
On Wed, 2012-06-20 at 11:12 +0200, Stefan Priebe - Profihost AG wrote:quoted
Am 20.06.2012 11:06, schrieb Stefan Priebe - Profihost AG:quoted
quoted
You seem to have a switch or something that drops packets in this case. You could try to rate limit to 9Gb/s and see if it is better.Sadly i can't rate limit to 9Gbit/s on the switch.quoted
If you exchange sender/receiver role between linux kernel versions, is it the same problem ?I'm testing in both directions. So both are sending and receiving. I've now made tests with only one sending an the other receiving. When server B is the sender i get 4Gbit/s. When server A is the sender i get full 9,9Gbit/s.To rule out bad hardware, could you try a 3.5-rc3 kernel on B ?
Sure. In that case i get 4Gbit/s in both variants. I also tried two other different machines same result. Stefan
From: Eric Dumazet <hidden>
Date: 2012-06-20 09:47:29
On Wed, 2012-06-20 at 11:33 +0200, Stefan Priebe - Profihost AG wrote:
Sure. In that case i get 4Gbit/s in both variants. I also tried two other different machines same result.
So 3.5 on receiver is the problem, it seems ? And you checked all the stuff about irq affinities, i presume, since a lot of things might have changed between 2.6.32 and 3.5 ? cat /proc/interrupts what kind of NIC it is ?
From: Stefan Priebe - Profihost AG <hidden>
Date: 2012-06-20 09:50:59
Am 20.06.2012 11:47, schrieb Eric Dumazet:
On Wed, 2012-06-20 at 11:33 +0200, Stefan Priebe - Profihost AG wrote:quoted
Sure. In that case i get 4Gbit/s in both variants. I also tried two other different machines same result.So 3.5 on receiver is the problem, it seems ?
Yes.
And you checked all the stuff about irq affinities, i presume, since a lot of things might have changed between 2.6.32 and 3.5 ?
It is a single core E5 Xeon - i've set the affinity like this: eth2 mask=1 for /proc/irq/83/smp_affinity eth2 mask=2 for /proc/irq/84/smp_affinity eth2 mask=4 for /proc/irq/85/smp_affinity eth2 mask=8 for /proc/irq/86/smp_affinity eth2 mask=10 for /proc/irq/87/smp_affinity eth2 mask=20 for /proc/irq/88/smp_affinity eth2 mask=40 for /proc/irq/89/smp_affinity eth2 mask=80 for /proc/irq/90/smp_affinity
cat /proc/interrupts
# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5
CPU6 CPU7
0: 141 0 0 0 0 0
0 0 IO-APIC-edge timer
1: 1 8 0 0 0 0
0 0 IO-APIC-edge i8042
9: 0 0 0 0 0 0
0 0 IO-APIC-fasteoi acpi
12: 0 3 0 0 0 0
0 0 IO-APIC-edge i8042
14: 0 0 0 0 0 0
0 0 IO-APIC-edge ide0
15: 0 0 0 0 0 0
0 0 IO-APIC-edge ide1
16: 0 0 26 0 0 0
0 0 IO-APIC-fasteoi ehci_hcd:usb1
23: 0 0 30 0 0 0
0 0 IO-APIC-fasteoi ehci_hcd:usb2
64: 0 0 0 81979 0 0
0 0 PCI-MSI-edge ahci
65: 0 0 0 1 0 0
0 0 PCI-MSI-edge eth0
66: 0 0 0 0 1090 0
0 0 PCI-MSI-edge eth0-TxRx-0
67: 0 0 0 0 411 0
0 0 PCI-MSI-edge eth0-TxRx-1
68: 0 0 0 0 592 0
0 0 PCI-MSI-edge eth0-TxRx-2
69: 0 0 0 0 472 0
0 0 PCI-MSI-edge eth0-TxRx-3
70: 0 0 0 0 0 1196
0 0 PCI-MSI-edge eth0-TxRx-4
71: 0 0 0 0 0 374
0 0 PCI-MSI-edge eth0-TxRx-5
72: 0 0 0 0 0 405
0 0 PCI-MSI-edge eth0-TxRx-6
73: 0 0 0 0 0 468
0 0 PCI-MSI-edge eth0-TxRx-7
83: 31278 0 0 65 0 0
0 0 PCI-MSI-edge eth2-TxRx-0
84: 0 36311 0 0 61 0
0 0 PCI-MSI-edge eth2-TxRx-1
85: 0 0 46189 0 61 0
0 0 PCI-MSI-edge eth2-TxRx-2
86: 0 0 0 28712 67 0
0 0 PCI-MSI-edge eth2-TxRx-3
87: 0 0 0 0 28089 0
0 0 PCI-MSI-edge eth2-TxRx-4
88: 0 0 0 0 0 34982
0 0 PCI-MSI-edge eth2-TxRx-5
89: 0 0 0 0 0 61
32420 0 PCI-MSI-edge eth2-TxRx-6
90: 0 0 0 0 0 61
0 25922 PCI-MSI-edge eth2-TxRx-7
91: 0 0 0 0 0 3
0 0 PCI-MSI-edge eth2
NMI: 13 12 15 22 5 5
5 5 Non-maskable interrupts
LOC: 58919 61420 65519 82647 35519 40489
27141 30228 Local timer interrupts
SPU: 0 0 0 0 0 0
0 0 Spurious interrupts
PMI: 13 12 15 22 5 5
5 5 Performance monitoring interrupts
IWI: 0 0 0 0 0 0
0 0 IRQ work interrupts
RTR: 6 0 0 0 0 0
0 0 APIC ICR read retries
RES: 15116 4521 2418 1814 2375 1615
1488 1367 Rescheduling interrupts
CAL: 134 148 100 162 170 172
172 172 Function call interrupts
TLB: 422 486 415 483 460 460
476 398 TLB shootdowns
TRM: 0 0 0 0 0 0
0 0 Thermal event interrupts
THR: 0 0 0 0 0 0
0 0 Threshold APIC interrupts
MCE: 0 0 0 0 0 0
0 0 Machine check exceptions
MCP: 4 4 4 4 4 4
4 4 Machine check polls
what kind of NIC it is ?
# lspci | grep 10-Giga 06:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) 06:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) Stefan
From: Eric Dumazet <hidden>
Date: 2012-06-20 10:06:34
On Wed, 2012-06-20 at 11:50 +0200, Stefan Priebe - Profihost AG wrote:
Am 20.06.2012 11:47, schrieb Eric Dumazet:quoted
On Wed, 2012-06-20 at 11:33 +0200, Stefan Priebe - Profihost AG wrote:quoted
Sure. In that case i get 4Gbit/s in both variants. I also tried two other different machines same result.So 3.5 on receiver is the problem, it seems ?Yes.quoted
And you checked all the stuff about irq affinities, i presume, since a lot of things might have changed between 2.6.32 and 3.5 ?It is a single core E5 Xeon - i've set the affinity like this:
And you still have the retransmits in "netstat -s" output ? Might be a firmware or pci issue, I have same cards but no problem here. Check LRO is on ? ethtool -k eth2
From: Eric Dumazet <hidden>
Date: 2012-06-20 11:08:40
On Wed, 2012-06-20 at 12:06 +0200, Eric Dumazet wrote:
On Wed, 2012-06-20 at 11:50 +0200, Stefan Priebe - Profihost AG wrote:quoted
Am 20.06.2012 11:47, schrieb Eric Dumazet:quoted
On Wed, 2012-06-20 at 11:33 +0200, Stefan Priebe - Profihost AG wrote:quoted
Sure. In that case i get 4Gbit/s in both variants. I also tried two other different machines same result.So 3.5 on receiver is the problem, it seems ?Yes.quoted
And you checked all the stuff about irq affinities, i presume, since a lot of things might have changed between 2.6.32 and 3.5 ?It is a single core E5 Xeon - i've set the affinity like this:And you still have the retransmits in "netstat -s" output ? Might be a firmware or pci issue, I have same cards but no problem here. Check LRO is on ? ethtool -k eth2
Ah, your ethtool -S gives strange fdir_miss counts, you should ask Intel guys help maybe...