From: Nieścierowicz Adam <hidden> Date: 2012-10-07 19:18:27
W dniu 06.07.2012 12:13, Eric Dumazet napisał(a):
On Fri, 2012-07-06 at 11:47 +0200, Nieścierowicz Adam wrote:
quoted
Hello, Can I send something that will help determine the cause of the
problem? W dniu 08.06.2012 11:41, Eric Dumazet napisał(a):
quoted
On Fri, 2012-06-08 at 10:58 +0200, Nieścierowicz Adam wrote:
quoted
Hello, recently we changed on the router kernel from 2.6.38.1 to
3.4.1 and noticed 30% packet loss when traffic increases up to
250MB / s. Similar is for kernel 3.5-rc1 Here a link to ifstat
http://wklej.org/id/767577/ [1] [2]
You should give as much as possible delails on your setup (hardware,
software) lspci cat /proc/cpuinfo cat /proc/interrupts ifconfig -a tc
-s -d qdisc dmesg netstat -s
Same infos of 3.5-rcX kernel would be nice.
What NIC is eth0 ? (dmesg please)
It seems all network traffic on 2.6.38 is handled by a single cpu (cpu0)
(seen in /proc/interrupts)
I suspect that with 3.4 or 3.5 kernels, traffic is handled by many cpus
and they hit false sharing and contention.
You probably get better performance doing some affinity tuning :
For example,
eth0 serviced by cpu0
eth2 serviced by cpu1
eth3 serviced by cpu2
eth5 serviced by cpu3
and so on...
check and/or set /proc/irq/${NUM}/smp_affinity
From: Eric Dumazet <hidden> Date: 2012-10-08 06:23:11
On Sun, 2012-10-07 at 21:18 +0200, Nieścierowicz Adam wrote:
W dniu 06.07.2012 12:13, Eric Dumazet napisał(a):
quoted
On Fri, 2012-07-06 at 11:47 +0200, Nieścierowicz Adam wrote:
quoted
Hello, Can I send something that will help determine the cause of
the
problem? W dniu 08.06.2012 11:41, Eric Dumazet napisał(a):
quoted
On Fri, 2012-06-08 at 10:58 +0200, Nieścierowicz Adam wrote:
quoted
Hello, recently we changed on the router kernel from 2.6.38.1 to
3.4.1 and noticed 30% packet loss when traffic increases up to
250MB / s. Similar is for kernel 3.5-rc1 Here a link to ifstat
http://wklej.org/id/767577/ [1] [2]
You should give as much as possible delails on your setup
(hardware,
software) lspci cat /proc/cpuinfo cat /proc/interrupts ifconfig -a
tc
-s -d qdisc dmesg netstat -s
Same infos of 3.5-rcX kernel would be nice.
What NIC is eth0 ? (dmesg please)
It seems all network traffic on 2.6.38 is handled by a single cpu
(cpu0)
(seen in /proc/interrupts)
I suspect that with 3.4 or 3.5 kernels, traffic is handled by many
cpus
and they hit false sharing and contention.
You probably get better performance doing some affinity tuning :
For example,
eth0 serviced by cpu0
eth2 serviced by cpu1
eth3 serviced by cpu2
eth5 serviced by cpu3
and so on...
check and/or set /proc/irq/${NUM}/smp_affinity
From: Nieścierowicz Adam <hidden> Date: 2012-10-08 09:29:55
You should use RPS on eth2/eth3 because they are non multi queue.
Documentation/networking/scaling.txt should give you all the needed info
I set processors for rps such as affinity, unfortunately it did not help
---
cat /sys/class/net/eth{2,3,4,5}/queues/rx-0/rps_cpus
0040
0080
0100
0200
---
CPU affinity http://wklej.org/id/843161/
From: Eric Dumazet <hidden> Date: 2012-10-08 09:47:24
On Mon, 2012-10-08 at 11:29 +0200, Nieścierowicz Adam wrote:
quoted
You should use RPS on eth2/eth3 because they are non multi queue.
Documentation/networking/scaling.txt should give you all the needed
info
I set processors for rps such as affinity, unfortunately it did not
help
---
cat /sys/class/net/eth{2,3,4,5}/queues/rx-0/rps_cpus
0040
0080
0100
0200
---
CPU affinity http://wklej.org/id/843161/
I said eth2 and eth3
And you should use cpu11->cpu15 instead of cpu6->cpu9 since they are in
use...
Anyway you dont say where are drops, (ifconfig give us very few drops)
Also your eth0 seems to have a strange balance :
RX interrupts seems to be well balanced on 4 queues :
76: 503 0 169271690 0 0 0 PCI-MSI-edge eth0-rx-0
77: 405 0 0 164532538 0 0 PCI-MSI-edge eth0-rx-1
78: 408 0 0 0 152778723 0 PCI-MSI-edge eth0-rx-2
79: 349 0 0 0 0 155011301 PCI-MSI-edge eth0-rx-3
80: 144 0 443432394 0 0 0 PCI-MSI-edge eth0-tx-0
81: 18 0 0 2043311 0 0 PCI-MSI-edge eth0-tx-1
82: 30 0 0 0 1934537 0 PCI-MSI-edge eth0-tx-2
83: 137 0 0 0 0 1968272 PCI-MSI-edge eth0-tx-3
But TX seems to mostly use queue 0
Packets sent to eth0 are coming from where ?
From: Nieścierowicz Adam <hidden> Date: 2012-10-08 10:50:05
W dniu 08.10.2012 11:47, Eric Dumazet napisał(a):
On Mon, 2012-10-08 at 11:29 +0200, Nieścierowicz Adam wrote:
quoted
quoted
You should use RPS on eth2/eth3 because they are non multi queue.
Documentation/networking/scaling.txt should give you all the needed
info
I set processors for rps such as affinity, unfortunately it did not
help --- cat /sys/class/net/eth{2,3,4,5}/queues/rx-0/rps_cpus 0040 0080
0100 0200 --- CPU affinity http://wklej.org/id/843161/ [1]
I said eth2 and eth3
eth2,eth3 and eth4,eth5 is the same card so I changed them as well, or made a mistake?
And you should use cpu11->cpu15 instead of cpu6->cpu9 since they are in
use...
I read in the file Documentation/networking/scaling.txt
---
if the rps_cpus for each queue are the ones that
share the same memory domain as the interrupting CPU for that queue
---
so i used the same CPU, I misunderstood?
Anyway you dont say where are drops, (ifconfig give us very few drops)
you can see no losses(drop), but a temporary decline in traffic on the interface to 0kb/s
Also your eth0 seems to have a strange balance :
RX interrupts seems to be well balanced on 4 queues :
76: 503 0 169271690 0 0 0 PCI-MSI-edge eth0-rx-0
77: 405 0 0 164532538 0 0 PCI-MSI-edge eth0-rx-1
78: 408 0 0 0 152778723 0 PCI-MSI-edge eth0-rx-2
79: 349 0 0 0 0 155011301 PCI-MSI-edge eth0-rx-3
80: 144 0 443432394 0 0 0 PCI-MSI-edge eth0-tx-0
81: 18 0 0 2043311 0 0 PCI-MSI-edge eth0-tx-1
82: 30 0 0 0 1934537 0 PCI-MSI-edge eth0-tx-2
83: 137 0 0 0 0 1968272 PCI-MSI-edge eth0-tx-3
But TX seems to mostly use queue 0
Packets sent to eth0 are coming from where ?
Packets come mainly from two routers(Edge BGP and local NAT)
Yes, but Adam says 3.4.1 already has a problem, while
commit 2cb7a9cc008c25dc03314de563c00c107b3e5432 is in 3.5 only.
Since Adam uses Intel e1000e, it could be the BQL related problem.
(Not sure if Intel guys finally fixed the problem, if not, its really
insane)
http://patchwork.ozlabs.org/patch/163298/
Yes, but Adam says 3.4.1 already has a problem, while
commit 2cb7a9cc008c25dc03314de563c00c107b3e5432 is in 3.5 only.
Since Adam uses Intel e1000e, it could be the BQL related problem.
The other chips have had DMA burst flag enabled for longer, so that he sees the same problem in 3.4 while I'm not makes sense. Hmm, as 3.4 is when BQL went in (IIRC) it seems very likely that this BQL issue is the problem for both of us.
Yes, but Adam says 3.4.1 already has a problem, while
commit 2cb7a9cc008c25dc03314de563c00c107b3e5432 is in 3.5 only.
>
quoted
Since Adam uses Intel e1000e, it could be the BQL related problem.
The other chips have had DMA burst flag enabled for longer, so that he
sees the same problem in 3.4 while I'm not makes sense. Hmm, as 3.4 is
when BQL went in (IIRC) it seems very likely that this BQL issue is the
problem for both of us.
To clarify; I think the DMA burst flag in the driver triggers the BQL related issue. Judging by the patchwork link for wthresh=1 this seems very related indeed.
Removing the FLAG2_DMA_BURST flag for 82574 in the driver works for me. Adam, it might be worth testing out a build on your system too with the flag removed. If you try the attached patch (for 3.6, probably OK for 3.5) and the problem dissapears, we are probably at least talking about the same bug.
Yes, but Adam says 3.4.1 already has a problem, while commit
2cb7a9cc008c25dc03314de563c00c107b3e5432 is in 3.5 only. Since Adam
uses Intel e1000e, it could be the BQL related problem.
The other chips have had DMA burst flag enabled for longer, so that he
sees the same problem in 3.4 while I'm not makes sense. Hmm, as 3.4 is
when BQL went in (IIRC) it seems very likely that this BQL issue is the
problem for both of us.
To clarify; I think the DMA burst flag in the driver triggers the BQL
related issue. Judging by the patchwork link for wthresh=1 this seems
very related indeed.
Removing the FLAG2_DMA_BURST flag for 82574 in the driver works for me.
Adam, it might be worth testing out a build on your system too with the
flag removed. If you try the attached patch (for 3.6, probably OK for
3.5) and the problem dissapears, we are probably at least talking about
the same bug.
after applying the patch everything looks good, no visible loss
Do you expect to correct the bug in mainline?
Yes, but Adam says 3.4.1 already has a problem, while commit
2cb7a9cc008c25dc03314de563c00c107b3e5432 is in 3.5 only. Since Adam
uses Intel e1000e, it could be the BQL related problem.
The other chips have had DMA burst flag enabled for longer, so that he
sees the same problem in 3.4 while I'm not makes sense. Hmm, as 3.4 is
when BQL went in (IIRC) it seems very likely that this BQL issue is the
problem for both of us.
To clarify; I think the DMA burst flag in the driver triggers the BQL
related issue. Judging by the patchwork link for wthresh=1 this seems
very related indeed.
Removing the FLAG2_DMA_BURST flag for 82574 in the driver works for me.
Adam, it might be worth testing out a build on your system too with the
flag removed. If you try the attached patch (for 3.6, probably OK for
3.5) and the problem dissapears, we are probably at least talking about
the same bug.
after applying the patch everything looks good, no visible loss
Do you expect to correct the bug in mainline? Jesse Brandenburg is working on a patch for upstream currently to fix the issue.