From: Timo Teras <hidden> Date: 2012-03-14 17:02:37
Hi,
I have a router box running linux-3.0.18 (with grsec patches).
with the NIC hardware:
r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
r8169 0000:00:09.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
r8169 0000:00:09.0: (unregistered net_device): no PCI Express capability
r8169 0000:00:09.0: eth0: RTL8169sc/8110sc at 0xf82f8000, 00:30:18:ab:6b:54, XID 18000000 IRQ 18
r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
r8169 0000:00:0b.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19
r8169 0000:00:0b.0: (unregistered net_device): no PCI Express capability
r8169 0000:00:0b.0: eth1: RTL8169sc/8110sc at 0xf82fa000, 00:30:18:ab:6b:55, XID 18000000 IRQ 19
r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
r8169 0000:00:0c.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
r8169 0000:00:0c.0: (unregistered net_device): no PCI Express capability
r8169 0000:00:0c.0: eth2: RTL8169sc/8110sc at 0xf82fc000, 00:30:18:ab:6b:56, XID 18000000 IRQ 16
This box is working just as a plain IPv4 router (internal RFC1918
address space) forwarding packets.
It routes basically from eth2 to multiple vlans over bond0 consisting of eth0 and eth1.
I have most hw accel stuff turned off, and "ethtool -k eth0" says:
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: off
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off
The same applies for all interfaces (except lo).
However, tcpdump on this box indicates that I'm receiving very
long (tcp length more than mtu) incoming packets on eth2 implying that
gso/tso got turned on somehow. eth2 is connected with cross-over cable
to similar box running a bit older linux box; but gso/tso is turned off
there too. When dumping simultaneously on the other side, it indicates
that all packets sent are normal length, and no merging was performed
earlier (fits mtu 1500).
So it would appear that the router box somehow insists on doing gso/tso,
and sadly it will also mess up on the send path (the incoming merged
packet is forwarded, but sent out short) causing lost segments and
serious performance degration.
Any pointers how to next debug/fix/workaround this issue?
-Timo
From: Eric Dumazet <hidden> Date: 2012-03-14 17:15:18
On Wed, 2012-03-14 at 19:01 +0200, Timo Teras wrote:
Hi,
I have a router box running linux-3.0.18 (with grsec patches).
with the NIC hardware:
r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
r8169 0000:00:09.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
r8169 0000:00:09.0: (unregistered net_device): no PCI Express capability
r8169 0000:00:09.0: eth0: RTL8169sc/8110sc at 0xf82f8000, 00:30:18:ab:6b:54, XID 18000000 IRQ 18
r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
r8169 0000:00:0b.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19
r8169 0000:00:0b.0: (unregistered net_device): no PCI Express capability
r8169 0000:00:0b.0: eth1: RTL8169sc/8110sc at 0xf82fa000, 00:30:18:ab:6b:55, XID 18000000 IRQ 19
r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
r8169 0000:00:0c.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
r8169 0000:00:0c.0: (unregistered net_device): no PCI Express capability
r8169 0000:00:0c.0: eth2: RTL8169sc/8110sc at 0xf82fc000, 00:30:18:ab:6b:56, XID 18000000 IRQ 16
This box is working just as a plain IPv4 router (internal RFC1918
address space) forwarding packets.
It routes basically from eth2 to multiple vlans over bond0 consisting of eth0 and eth1.
I have most hw accel stuff turned off, and "ethtool -k eth0" says:
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: off
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off
The same applies for all interfaces (except lo).
However, tcpdump on this box indicates that I'm receiving very
long (tcp length more than mtu) incoming packets on eth2 implying that
gso/tso got turned on somehow. eth2 is connected with cross-over cable
to similar box running a bit older linux box; but gso/tso is turned off
there too. When dumping simultaneously on the other side, it indicates
that all packets sent are normal length, and no merging was performed
earlier (fits mtu 1500).
So it would appear that the router box somehow insists on doing gso/tso,
and sadly it will also mess up on the send path (the incoming merged
packet is forwarded, but sent out short) causing lost segments and
serious performance degration.
Any pointers how to next debug/fix/workaround this issue?
You are fighting the wrong side ;)
Here, its GRO doing the aggregation on receiver.
What kind of problems do you experiment because of this ?
ethtool -k eth2
From: Timo Teras <hidden> Date: 2012-03-14 17:30:22
On Wed, 14 Mar 2012 10:15:14 -0700 Eric Dumazet
[off-list ref] wrote:
On Wed, 2012-03-14 at 19:01 +0200, Timo Teras wrote:
quoted
Hi,
I have a router box running linux-3.0.18 (with grsec patches).
with the NIC hardware:
r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
r8169 0000:00:09.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
r8169 0000:00:09.0: (unregistered net_device): no PCI Express
capability r8169 0000:00:09.0: eth0: RTL8169sc/8110sc at
0xf82f8000, 00:30:18:ab:6b:54, XID 18000000 IRQ 18 r8169 Gigabit
Ethernet driver 2.3LK-NAPI loaded r8169 0000:00:0b.0: PCI INT A ->
GSI 19 (level, low) -> IRQ 19 r8169 0000:00:0b.0: (unregistered
net_device): no PCI Express capability r8169 0000:00:0b.0: eth1:
RTL8169sc/8110sc at 0xf82fa000, 00:30:18:ab:6b:55, XID 18000000 IRQ
19 r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded r8169
0000:00:0c.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 r8169
0000:00:0c.0: (unregistered net_device): no PCI Express capability
r8169 0000:00:0c.0: eth2: RTL8169sc/8110sc at 0xf82fc000,
00:30:18:ab:6b:56, XID 18000000 IRQ 16
This box is working just as a plain IPv4 router (internal RFC1918
address space) forwarding packets.
It routes basically from eth2 to multiple vlans over bond0
consisting of eth0 and eth1.
I have most hw accel stuff turned off, and "ethtool -k eth0" says:
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: off
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off
The same applies for all interfaces (except lo).
However, tcpdump on this box indicates that I'm receiving very
long (tcp length more than mtu) incoming packets on eth2 implying
that gso/tso got turned on somehow. eth2 is connected with
cross-over cable to similar box running a bit older linux box; but
gso/tso is turned off there too. When dumping simultaneously on the
other side, it indicates that all packets sent are normal length,
and no merging was performed earlier (fits mtu 1500).
So it would appear that the router box somehow insists on doing
gso/tso, and sadly it will also mess up on the send path (the
incoming merged packet is forwarded, but sent out short) causing
lost segments and serious performance degration.
Any pointers how to next debug/fix/workaround this issue?
You are fighting the wrong side ;)
Here, its GRO doing the aggregation on receiver.
Yes, I figured this much. But I have explictly turned GRO off and it's
still happening.
What kind of problems do you experiment because of this ?
I'm getting lost packets (the non-first TCP segments off the GRO merged
packet). This causes serious TCP speed degration (should get 10MB/s
through 100mbit/s link; but I'm getting only 2-3MB/s). Doing the same
transfer on the next hop router gives full speed, so the problem is
definitely on this router and due to GRO badness.
I also remember this working before, so this seems a regression from
upgrading 2.6.35.x kernel or something like that.
ethtool -k eth2
gro off. I am even trying now with:
Offload parameters for eth2:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off
Additionally, I'm looking at my other router boxes with same hardware
but different kernel versions. Looks that all of them are acting as GRO
is enabled, even though it's turned off by ethtool.
I can verify that 2.6.35.8, 2.6.38.8, and 3.0.18 (all of these with
grsec patch) are doing GRO for this r8169 hardware, even though it's
configured OFF on all boxes.
There seems to be no performance issues in 2.6.35.8 kernel. This would
indicate that the incoming GRO packets are properly handled and
segmented (likely by software) on the path out. However, I'm also
having issues with the 2.6.38.8 box, and badness on GRO send path
seems to be the cause. And of course to mention that GRO is happening
even though it's turned off.
Additionally, it seems that at the 2.6.38.8 and 3.0.18 kernels are
having the performance issues even if it's locally terminated TCP
connection. So it's not limited to the forward path. The latest good
kernel I can verify is 2.6.35.x.
- Timo
From: Eric Dumazet <hidden> Date: 2012-03-14 18:25:38
On Wed, 2012-03-14 at 19:29 +0200, Timo Teras wrote:
On Wed, 14 Mar 2012 10:15:14 -0700 Eric Dumazet
[off-list ref] wrote:
quoted
On Wed, 2012-03-14 at 19:01 +0200, Timo Teras wrote:
quoted
Hi,
I have a router box running linux-3.0.18 (with grsec patches).
with the NIC hardware:
r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
r8169 0000:00:09.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
r8169 0000:00:09.0: (unregistered net_device): no PCI Express
capability r8169 0000:00:09.0: eth0: RTL8169sc/8110sc at
0xf82f8000, 00:30:18:ab:6b:54, XID 18000000 IRQ 18 r8169 Gigabit
Ethernet driver 2.3LK-NAPI loaded r8169 0000:00:0b.0: PCI INT A ->
GSI 19 (level, low) -> IRQ 19 r8169 0000:00:0b.0: (unregistered
net_device): no PCI Express capability r8169 0000:00:0b.0: eth1:
RTL8169sc/8110sc at 0xf82fa000, 00:30:18:ab:6b:55, XID 18000000 IRQ
19 r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded r8169
0000:00:0c.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 r8169
0000:00:0c.0: (unregistered net_device): no PCI Express capability
r8169 0000:00:0c.0: eth2: RTL8169sc/8110sc at 0xf82fc000,
00:30:18:ab:6b:56, XID 18000000 IRQ 16
This box is working just as a plain IPv4 router (internal RFC1918
address space) forwarding packets.
It routes basically from eth2 to multiple vlans over bond0
consisting of eth0 and eth1.
I have most hw accel stuff turned off, and "ethtool -k eth0" says:
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: off
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off
The same applies for all interfaces (except lo).
However, tcpdump on this box indicates that I'm receiving very
long (tcp length more than mtu) incoming packets on eth2 implying
that gso/tso got turned on somehow. eth2 is connected with
cross-over cable to similar box running a bit older linux box; but
gso/tso is turned off there too. When dumping simultaneously on the
other side, it indicates that all packets sent are normal length,
and no merging was performed earlier (fits mtu 1500).
So it would appear that the router box somehow insists on doing
gso/tso, and sadly it will also mess up on the send path (the
incoming merged packet is forwarded, but sent out short) causing
lost segments and serious performance degration.
Any pointers how to next debug/fix/workaround this issue?
You are fighting the wrong side ;)
Here, its GRO doing the aggregation on receiver.
Yes, I figured this much. But I have explictly turned GRO off and it's
still happening.
quoted
What kind of problems do you experiment because of this ?
I'm getting lost packets (the non-first TCP segments off the GRO merged
packet). This causes serious TCP speed degration (should get 10MB/s
through 100mbit/s link; but I'm getting only 2-3MB/s). Doing the same
transfer on the next hop router gives full speed, so the problem is
definitely on this router and due to GRO badness.
There is something completely unrelated to GRO then. 2-3 MB/s sound more
a tcp issue.
I also remember this working before, so this seems a regression from
upgrading 2.6.35.x kernel or something like that.
quoted
ethtool -k eth2
gro off. I am even trying now with:
Offload parameters for eth2:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off
I cant see how you can then receive tcp frames bigger than MTU.
Additionally, I'm looking at my other router boxes with same hardware
but different kernel versions. Looks that all of them are acting as GRO
is enabled, even though it's turned off by ethtool.
I can verify that 2.6.35.8, 2.6.38.8, and 3.0.18 (all of these with
grsec patch) are doing GRO for this r8169 hardware, even though it's
configured OFF on all boxes.
There seems to be no performance issues in 2.6.35.8 kernel. This would
indicate that the incoming GRO packets are properly handled and
segmented (likely by software) on the path out. However, I'm also
having issues with the 2.6.38.8 box, and badness on GRO send path
seems to be the cause. And of course to mention that GRO is happening
even though it's turned off.
Additionally, it seems that at the 2.6.38.8 and 3.0.18 kernels are
having the performance issues even if it's locally terminated TCP
connection. So it's not limited to the forward path. The latest good
kernel I can verify is 2.6.35.x.
- Timo
If trafic is localy terminated :
netstat -s
should give us some input.
From: Ben Hutchings <hidden> Date: 2012-03-14 19:29:17
On Wed, 2012-03-14 at 19:29 +0200, Timo Teras wrote:
[...]
gro off. I am even trying now with:
Offload parameters for eth2:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off
[...]
GRO isn't even reported there! Apparently you need a newer version of
ethtool.
Ben.
--
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
From: Timo Teras <hidden> Date: 2012-03-14 19:52:19
On Wed, 14 Mar 2012 19:29:14 +0000 Ben Hutchings
[off-list ref] wrote:
On Wed, 2012-03-14 at 19:29 +0200, Timo Teras wrote:
[...]
quoted
gro off. I am even trying now with:
Offload parameters for eth2:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp segmentation offload: off
udp fragmentation offload: off
generic segmentation offload: off
[...]
GRO isn't even reported there! Apparently you need a newer version of
ethtool.
Very good point. I thought gso also enabled gro, but seems that my
ethtool was old.
And GRO was enabled along with some other stuff. Turning GRO off made
my tcp performance immediately a lot better; jumped from 2MB/s to 8MB/s
or so (not ideal yet, though; but the remainder of the difference could
be related to other issue).
So something is definitely broke in 3.0.x with GRO enabled, but GSO off.
From: Eric Dumazet <hidden> Date: 2012-03-14 20:12:48
On Wed, 2012-03-14 at 21:51 +0200, Timo Teras wrote:
Very good point. I thought gso also enabled gro, but seems that my
ethtool was old.
And GRO was enabled along with some other stuff. Turning GRO off made
my tcp performance immediately a lot better; jumped from 2MB/s to 8MB/s
or so (not ideal yet, though; but the remainder of the difference could
be related to other issue).
So something is definitely broke in 3.0.x with GRO enabled, but GSO off.
"ifconfig eth2 ; netstat -s" can really help, I suspect tcp stack drops
From: Timo Teras <hidden> Date: 2012-03-14 20:34:18
On Wed, 14 Mar 2012 13:12:45 -0700 Eric Dumazet
[off-list ref] wrote:
On Wed, 2012-03-14 at 21:51 +0200, Timo Teras wrote:
quoted
Very good point. I thought gso also enabled gro, but seems that my
ethtool was old.
And GRO was enabled along with some other stuff. Turning GRO off
made my tcp performance immediately a lot better; jumped from 2MB/s
to 8MB/s or so (not ideal yet, though; but the remainder of the
difference could be related to other issue).
So something is definitely broke in 3.0.x with GRO enabled, but GSO
off.
"ifconfig eth2 ; netstat -s" can really help, I suspect tcp stack
drops
It does not look like stuff for the higher layers guys.
Can you tshark -w foobar on the sender side and
'while : ; do sleep 1; ethtool -S eth2 >> glop; done' on the receiver
during a bad wget (a big zero filled file should compress well).
--
Ueimor
It does not look like stuff for the higher layers guys.
Can you tshark -w foobar on the sender side and
'while : ; do sleep 1; ethtool -S eth2 >> glop; done' on the receiver
during a bad wget (a big zero filled file should compress well).
Indeed.
It seems that my earlier test about the "GRO off" effect were mistaken
(I used accidentally proxy, and that gave the illusion that things are
working. Whoops.)
So far I changed the cross-over cable and it didn't help. However,
forcing the NIC to 100mbit/full-duplex mode fixes the rx_errors. It
seems that something bad is happening in the gigabit mode.
# ethtool eth2
Settings for eth2:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes: 100baseT/Full
Advertised pause frame use: Symmetric Receive-only
Advertised auto-negotiation: Yes
Link partner advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
Link partner advertised pause frame use: Symmetric Receive-only
Link partner advertised auto-negotiation: Yes
Speed: 100Mb/s
Duplex: Full
Port: MII
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000033 (51)
drv probe ifdown ifup
Link detected: yes
I wonder if it's using pause frames and that's messing things up. Seems
that I can't turn it off, though.
I can also double check my cables, though it is factory made Cat-5E
cross-over cable; and happens with two different cables.
-Timo
It does not look like stuff for the higher layers guys.
Can you tshark -w foobar on the sender side and
'while : ; do sleep 1; ethtool -S eth2 >> glop; done' on the
receiver during a bad wget (a big zero filled file should compress
well).
Indeed.
It seems that my earlier test about the "GRO off" effect were mistaken
(I used accidentally proxy, and that gave the illusion that things are
working. Whoops.)
So far I changed the cross-over cable and it didn't help. However,
forcing the NIC to 100mbit/full-duplex mode fixes the rx_errors. It
seems that something bad is happening in the gigabit mode.
I wonder if it's using pause frames and that's messing things up.
Seems that I can't turn it off, though.
I can also double check my cables, though it is factory made Cat-5E
cross-over cable; and happens with two different cables.
Ok. So far I have two of these boxes with same r8169 hardware. Both
generate bad packets on transmit only; and on both 3 nic systems it's
the middle eth1 nic. The symptoms are identical: in 1GB mode I have
minor packet loss, where as 100Mbit/s mode seems to work just fine.
The first box, that I've been talking so far about, is as mentioned
connected to another similar box. The r8169 there reports rx_errors.
The cable is ok; I've tried with two different ones.
The other broken box is connected to a HP ProCurve 4202vl-48G, and the
switch is reporting drops due to FCS Rx errors.
So I have two broken pieces of hardware, or there is a driver bug.
I'll try upgrading my kernel to 3.0.x series on the sender box and see
if it's fixing anything. Suggestions for further testing would be
appreciated.
It does not look like stuff for the higher layers guys.
Can you tshark -w foobar on the sender side and
'while : ; do sleep 1; ethtool -S eth2 >> glop; done' on the
receiver during a bad wget (a big zero filled file should compress
well).
Indeed.
It seems that my earlier test about the "GRO off" effect were mistaken
(I used accidentally proxy, and that gave the illusion that things are
working. Whoops.)
So far I changed the cross-over cable and it didn't help. However,
forcing the NIC to 100mbit/full-duplex mode fixes the rx_errors. It
seems that something bad is happening in the gigabit mode.
I wonder if it's using pause frames and that's messing things up.
Seems that I can't turn it off, though.
I can also double check my cables, though it is factory made Cat-5E
cross-over cable; and happens with two different cables.
Ok. So far I have two of these boxes with same r8169 hardware. Both
generate bad packets on transmit only; and on both 3 nic systems it's
the middle eth1 nic. The symptoms are identical: in 1GB mode I have
minor packet loss, where as 100Mbit/s mode seems to work just fine.
The first box, that I've been talking so far about, is as mentioned
connected to another similar box. The r8169 there reports rx_errors.
The cable is ok; I've tried with two different ones.
The other broken box is connected to a HP ProCurve 4202vl-48G, and the
switch is reporting drops due to FCS Rx errors.
So I have two broken pieces of hardware, or there is a driver bug.
I'll try upgrading my kernel to 3.0.x series on the sender box and see
if it's fixing anything. Suggestions for further testing would be
appreciated.
r8169 has to make an additional copy of incoming frames, because of
hardware flaw and security requirements.
This was added in 2.6.37 or 2.6.38, dont remember exactly.
So your cpu might be to slow to handle the load at 1Gb speed.
If you have one flow, there is nothing to do, but if your workload has
several flows and your machine is SMP, you can try RPS/RFS as documented
in Documentation/networking/scaling.txt
It does not look like stuff for the higher layers guys.
Can you tshark -w foobar on the sender side and
'while : ; do sleep 1; ethtool -S eth2 >> glop; done' on the
receiver during a bad wget (a big zero filled file should
compress well).
Indeed.
It seems that my earlier test about the "GRO off" effect were
mistaken (I used accidentally proxy, and that gave the illusion
that things are working. Whoops.)
So far I changed the cross-over cable and it didn't help. However,
forcing the NIC to 100mbit/full-duplex mode fixes the rx_errors.
It seems that something bad is happening in the gigabit mode.
I wonder if it's using pause frames and that's messing things up.
Seems that I can't turn it off, though.
I can also double check my cables, though it is factory made
Cat-5E cross-over cable; and happens with two different cables.
Ok. So far I have two of these boxes with same r8169 hardware. Both
generate bad packets on transmit only; and on both 3 nic systems
it's the middle eth1 nic. The symptoms are identical: in 1GB mode I
have minor packet loss, where as 100Mbit/s mode seems to work just
fine.
The first box, that I've been talking so far about, is as mentioned
connected to another similar box. The r8169 there reports rx_errors.
The cable is ok; I've tried with two different ones.
The other broken box is connected to a HP ProCurve 4202vl-48G, and
the switch is reporting drops due to FCS Rx errors.
So I have two broken pieces of hardware, or there is a driver bug.
I'll try upgrading my kernel to 3.0.x series on the sender box and
see if it's fixing anything. Suggestions for further testing would
be appreciated.
r8169 has to make an additional copy of incoming frames, because of
hardware flaw and security requirements.
This was added in 2.6.37 or 2.6.38, dont remember exactly.
So your cpu might be to slow to handle the load at 1Gb speed.
If you have one flow, there is nothing to do, but if your workload has
several flows and your machine is SMP, you can try RPS/RFS as
documented in Documentation/networking/scaling.txt
No. It's exactly the same amount of traffic on link: approx
50-80mbit/s. If link is in 100mbit/s mode, everything is perfect. But
if link is in 1gbit/s mode (but having only the 50-80mbit/s in average),
it's getting packet loss (and kills TCP performance).
There is definitely a hardware or a driver issue.
The other broken box is connected to a HP ProCurve 4202vl-48G, and the
switch is reporting drops due to FCS Rx errors.
[...]
So I have two broken pieces of hardware, or there is a driver bug.
I'll take blame for any bug in the driver. However many ethernet controllers
are and the PCI 8169 is no exception.
I'll try upgrading my kernel to 3.0.x series on the sender box and see
if it's fixing anything. Suggestions for further testing would be
appreciated.
Please check you are using nothing but SLAB.
If you have not done so, you may then disable Tx checksumming.
If it does not change anything, you may consider using the r8169 from
David Miller's -next branch (backported ? no, no, the real thing). If it
still does not change anything and you are interested in new experiences,
please confirm you are above 18 and we may use Ben Grear's bad rx packets
capture (available in -next) and the port mirroring feature of your switch
to see what the corrupted tx frames look like. Before that, I would
welcome a short description of the router boxes (lspci, proc, etc) and
overall traffic / irq.
--
Ueimor
From: Timo Teras <hidden> Date: 2012-03-16 20:16:37
On Thu, 15 Mar 2012 20:11:18 +0100 Francois Romieu
[off-list ref] wrote:
Timo Teras [off-list ref] :
[...]
quoted
The other broken box is connected to a HP ProCurve 4202vl-48G, and
the switch is reporting drops due to FCS Rx errors.
[...]
quoted
So I have two broken pieces of hardware, or there is a driver bug.
I'll take blame for any bug in the driver. However many ethernet
controllers are and the PCI 8169 is no exception.
Ok.
As a side though, all these devices suffered from the bug I fixed
earlier. See commit 024a07bac (r8169: fix random mdio_write failures).
Also, all these devices probably got garbage written to their PHY. So
I'm wondering if it is possible that it caused some permanent damage?
Would it be possible to dump/compare the related things?
Additional pointer to this direction is that one of the "broken" boxes
has different PCI ID for the "broken NIC" of the three. The hardware is
Jetway daughter board with the three NICs on single board. So it sounds
really weird that one of those NICs chips would be from different
series. I wonder if the PCI ID and other stuff could have got corrupted
in EEPROM or something similar.
quoted
I'll try upgrading my kernel to 3.0.x series on the sender box and
see if it's fixing anything. Suggestions for further testing would
be appreciated.
Please check you are using nothing but SLAB.
Using SLUB, the current kernel default. Can retry with SLAB later.
If you have not done so, you may then disable Tx checksumming.
Tx checksumming is off.
If it does not change anything, you may consider using the r8169 from
David Miller's -next branch (backported ? no, no, the real thing). If
it still does not change anything and you are interested in new
experiences, please confirm you are above 18 and we may use Ben
Grear's bad rx packets capture (available in -next) and the port
mirroring feature of your switch to see what the corrupted tx frames
look like. Before that, I would welcome a short description of the
router boxes (lspci, proc, etc) and overall traffic / irq.
Ah, I see the good stuff. Will try to do capture of the FCS on broken
link. And I'll try to relocate the broken hardware to lab environment
where this can be easier reproduced and debugged.
From the system with one NIC showing wrong PCI id (but XID and
boottime detection is identical for all these):
# lspci -nn
00:00.0 Host bridge [0600]: VIA Technologies, Inc. CN700/VN800/P4M800CE/Pro Host Bridge [1106:0314]
00:00.1 Host bridge [0600]: VIA Technologies, Inc. CN700/VN800/P4M800CE/Pro Host Bridge [1106:1314]
00:00.2 Host bridge [0600]: VIA Technologies, Inc. CN700/VN800/P4M800CE/Pro Host Bridge [1106:2314]
00:00.3 Host bridge [0600]: VIA Technologies, Inc. PT890 Host Bridge [1106:3208]
00:00.4 Host bridge [0600]: VIA Technologies, Inc. CN700/VN800/P4M800CE/Pro Host Bridge [1106:4314]
00:00.7 Host bridge [0600]: VIA Technologies, Inc. CN700/VN800/P4M800CE/Pro Host Bridge [1106:7314]
00:01.0 PCI bridge [0604]: VIA Technologies, Inc. VT8237/VX700 PCI Bridge [1106:b198]
00:09.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL-8110SC/8169SC Gigabit Ethernet [10ec:8167] (rev 10)
00:0a.0 FireWire (IEEE 1394) [0c00]: VIA Technologies, Inc. VT6306 Fire II IEEE 1394 OHCI Link Layer Controller [1106:3044] (rev 80)
00:0b.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet [10ec:8169] (rev 10)
00:0c.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL-8110SC/8169SC Gigabit Ethernet [10ec:8167] (rev 10)
00:0f.0 IDE interface [0101]: VIA Technologies, Inc. VIA VT6420 SATA RAID Controller [1106:3149] (rev 80)
00:0f.1 IDE interface [0101]: VIA Technologies, Inc. VT82C586A/B/VT82C686/A/B/VT823x/A/C PIPC Bus Master IDE [1106:0571] (rev 06)
00:10.0 USB Controller [0c03]: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller [1106:3038] (rev 81)
00:10.1 USB Controller [0c03]: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller [1106:3038] (rev 81)
00:10.2 USB Controller [0c03]: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller [1106:3038] (rev 81)
00:10.3 USB Controller [0c03]: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller [1106:3038] (rev 81)
00:10.4 USB Controller [0c03]: VIA Technologies, Inc. USB 2.0 [1106:3104] (rev 86)
00:11.0 ISA bridge [0601]: VIA Technologies, Inc. VT8237 ISA bridge [KT600/K8T800/K8T890 South] [1106:3227]
00:11.5 Multimedia audio controller [0401]: VIA Technologies, Inc. VT8233/A/8235/8237 AC97 Audio Controller [1106:3059] (rev 60)
00:12.0 Ethernet controller [0200]: VIA Technologies, Inc. VT6102 [Rhine-II] [1106:3065] (rev 78)
01:00.0 VGA compatible controller [0300]: VIA Technologies, Inc. CN700/P4M800 Pro/P4M800 CE/VN800 [S3 UniChrome Pro] [1106:3344] (rev 01)
# grep eth /var/log/dmesg
r8169 0000:00:09.0: eth0: RTL8169sc/8110sc at 0xf81fe000, 00:30:18:a8:14:ac, XID 18000000 IRQ 18
r8169 0000:00:0b.0: eth1: RTL8169sc/8110sc at 0xf8202000, 00:30:18:ab:69:4b, XID 18000000 IRQ 19
r8169 0000:00:0c.0: eth2: RTL8169sc/8110sc at 0xf8206000, 00:30:18:a8:14:ad, XID 18000000 IRQ 16
eth3: VIA Rhine II at 0x1e800, 00:30:18:a0:d5:53, IRQ 23.
eth3: MII PHY found at address 1, status 0x7849 advertising 05e1 Link 0000.
# cat /proc/cpuinfo
processor : 0
vendor_id : CentaurHauls
cpu family : 6
model : 13
model name : VIA Eden Processor 1200MHz
stepping : 0
cpu MHz : 1199.906
cache size : 128 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce apic mtrr pge cmov pat clflush acpi mmx fxsr sse sse2 tm nx up pni est tm2 xtpr rng rng_en ace ace_en ace2 ace2_en phe phe_en pmm pmm_en
bogomips : 2400.80
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 32 bits virtual
power management:
From: Timo Teras <hidden> Date: 2012-03-17 09:57:10
On Fri, 16 Mar 2012 22:15:57 +0200 Timo Teras [off-list ref] wrote:
On Thu, 15 Mar 2012 20:11:18 +0100 Francois Romieu
[off-list ref] wrote:
quoted
Timo Teras [off-list ref] :
[...]
quoted
The other broken box is connected to a HP ProCurve 4202vl-48G, and
the switch is reporting drops due to FCS Rx errors.
[...]
quoted
So I have two broken pieces of hardware, or there is a driver bug.
I'll take blame for any bug in the driver. However many ethernet
controllers are and the PCI 8169 is no exception.
Ok.
As a side though, all these devices suffered from the bug I fixed
earlier. See commit 024a07bac (r8169: fix random mdio_write failures).
Also, all these devices probably got garbage written to their PHY. So
I'm wondering if it is possible that it caused some permanent damage?
Would it be possible to dump/compare the related things?
Additional pointer to this direction is that one of the "broken" boxes
has different PCI ID for the "broken NIC" of the three. The hardware
is Jetway daughter board with the three NICs on single board. So it
sounds really weird that one of those NICs chips would be from
different series. I wonder if the PCI ID and other stuff could have
got corrupted in EEPROM or something similar.
It seems that we have working eeprom reading code in commit 6709fe9a27e4
"r8169: read MAC address from EEPROM on init (2nd attempt)" which later
got reverted due to problems. I'm now wondering if those problems were
actually caused by unrelated issues that got later fixed in 78f1cd02457
"r8169: fix broken register writes".
I wonder if it'd be worth to do the eeprom reading and expose it via
ethtool so I can compare those. Or as easy alternative, enabling the VPD
bit in Config1 should allow me to read the EEPROM contents using the PCI
/sys/.../vpd interface, right?
And maybe re-introduce the reading of the MAC from there on reboot. Or
if could just do:
- Cfg9346_Lock = 0x00,
+ Cfg9346_Lock = 0x40,
The 0x40 apparently means "Auto-load: the EEPROM contents will be
reloaded when PCI RSTB signal is asserted, and will automatically
resume to normal 0x00 mode after the load".
On Fri, 16 Mar 2012 22:15:57 +0200 Timo Teras [off-list ref] wrote:
[...]
quoted
Additional pointer to this direction is that one of the "broken" boxes
has different PCI ID for the "broken NIC" of the three. The hardware
is Jetway daughter board with the three NICs on single board. So it
sounds really weird that one of those NICs chips would be from
different series. I wonder if the PCI ID and other stuff could have
got corrupted in EEPROM or something similar.
Some of my old PCI 8169 show an unpleasant trend to lose config bits and
they can turn into unavailable devices (i.e. all 0xff registers) when
things go really wrong.
It seems that we have working eeprom reading code in commit 6709fe9a27e4
"r8169: read MAC address from EEPROM on init (2nd attempt)" which later
got reverted due to problems. I'm now wondering if those problems were
actually caused by unrelated issues that got later fixed in 78f1cd02457
"r8169: fix broken register writes".
I have not tried working with the eeprom again since 024a07bac.
I wonder if it'd be worth to do the eeprom reading and expose it via
ethtool so I can compare those.
Yes.
Or as easy alternative, enabling the VPD bit in Config1 should allow me
to read the EEPROM contents using the PCI /sys/.../vpd interface, right?
In theory, yes. I have not tested it. Imho both access methods will be
useful.
I should have some unfinished VPD stuff somewhere. Will have to dig...
And maybe re-introduce the reading of the MAC from there on reboot. Or
if could just do:
- Cfg9346_Lock = 0x00,
+ Cfg9346_Lock = 0x40,
The 0x40 apparently means "Auto-load: the EEPROM contents will be
reloaded when PCI RSTB signal is asserted, and will automatically
resume to normal 0x00 mode after the load".
@@ -450,9 +452,16 @@ enum rtl_register_content {NPQ=0x40,/* Poll cmd on the low prio queue */FSWInt=0x01,/* Forced software interrupt */-/* Cfg9346Bits */-Cfg9346_Lock=0x00,+/* Cfg9346 operating mode register p.23 */Cfg9346_Unlock=0xc0,+Cfg9346_Prog=0x80,+Cfg9346_Auto=0x80,+Cfg9346_Lock=0x00,+/* Sub-mode bits in Programming or Auto-load mode. */+Cfg9346_CS=0x08,/* Chip Select */+Cfg9346_SK=0x04,/* Serial Data Clock */+Cfg9346_DI=0x02,/* Data In (going into the eeprom) */+Cfg9346_DO=0x01,/* Data Out (coming from the eeprom) *//* rx_mode_bits */AcceptErr=0x20,
@@ -751,6 +760,8 @@ struct rtl8169_private {}phy_action;}*rtl_fw;#define RTL_FIRMWARE_UNKNOWN ERR_PTR(-EAGAIN)++structeeprom_93cx6eeprom;};MODULE_AUTHOR("Realtek and the Linux r8169 crew <netdev@vger.kernel.org>");
@@ -5803,6 +5868,63 @@ static int rtl8169_suspend(struct device *device)return0;}+staticvoidrtl_93cx6_register_read(structeeprom_93cx6*eeprom)+{+structnet_device*dev=eeprom->data;+structrtl8169_private*tp=netdev_priv(dev);+void__iomem*ioaddr=tp->mmio_addr;+u8reg;++reg=RTL_R8(Cfg9346);++eeprom->reg_data_in=reg&Cfg9346_DI;+eeprom->reg_data_out=reg&Cfg9346_DO;+eeprom->reg_data_clock=reg&Cfg9346_SK;+eeprom->reg_chip_select=reg&Cfg9346_CS;+}++staticvoidrtl_93cx6_register_write(structeeprom_93cx6*eeprom)+{+structnet_device*dev=eeprom->data;+structrtl8169_private*tp=netdev_priv(dev);+void__iomem*ioaddr=tp->mmio_addr;+u8reg=Cfg9346_Prog;++if(eeprom->reg_data_in)+reg|=Cfg9346_DI;+if(eeprom->reg_data_out)+reg|=Cfg9346_DO;+if(eeprom->reg_data_clock)+reg|=Cfg9346_SK;+if(eeprom->reg_chip_select)+reg|=Cfg9346_CS;++RTL_W8(Cfg9346,reg);+/* PCI commit */+RTL_R8(ChipCmd);+/* This is not a posting bug band-aid: the eeprom wants ~250 ns. */+ndelay(250);+}++staticvoidrtl_init_eeprom(structnet_device*dev,structrtl8169_private*tp)+{+structeeprom_93cx6*eeprom=&tp->eeprom;+void__iomem*ioaddr=tp->mmio_addr;++eeprom->data=dev;++eeprom->register_read=rtl_93cx6_register_read;+eeprom->register_write=rtl_93cx6_register_write;++if(RTL_R32(RxConfig)&EEPROM_9356_SELECT){+eeprom->width=PCI_EEPROM_WIDTH_93C56;+eeprom->size=256;+}else{+eeprom->width=PCI_EEPROM_WIDTH_93C46;+eeprom->size=128;+}+}+staticvoid__rtl8169_resume(structnet_device*dev){structrtl8169_private*tp=netdev_priv(dev);
Ok. Thanks. I should be able to swap my broken box tomorrow or the day
after, and then start doing expirements and compare the eeprom.
I also found one more weirdly broken box. One of my similar boxes fails
to LACP bonding with a HP ProCurve switches. The switch just says LACP
failed, but linux thinks it's ok and starts aggregation, and this
results in broken traffic for all flows using the link that was not
accepted as part of the aggregation by the switch.
To only explanation I've found so far is: the switch reports other link
as MDI and the other as MDI-X. And this is enough to fail the
aggregation (links are not identical), but the linux bonding does not
notice this. I have no idea why the other link is MDI-X - it's a
straight cable. I wonder if it could be eeprom related too... or maybe
it's just another broken NIC.
Oh well - I'll get back to this after few days when I have the broken
box on my table for playing.
Ok. I now have the box that was sending faulty packets and has weird
PCI ID on my desk for playing. I swapped it for identical box - the
software and configuration are identical - and now the 1 gig mode errors
are gone on the setup. So the problem is caused by hardware; either due
to bad eeprom/firmware in the rtl8110sc or some other issue.
I also built net-next and took the ehttool -e dumps of the eeproms.
From a working eth0:
Offset Values
------ ------
0x0000 29 81 ec 10 67 81 f3 16 ec 10 20 40 00 a1 00 30
0x0010 18 a8 14 ac 15 0d c2 f7 00 80 00 00 00 00 00 13
0x0020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0030 00 00 82 c7 00 00 00 00 00 00 00 00 00 00 00 20
0x0040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
And the "broken" eth1:
Offset Values
------ ------
0x0000 29 81 ec 10 69 81 f3 16 ec 10 20 40 00 a1 00 30
0x0010 18 ab 69 4b 14 0d c2 f7 00 80 00 00 00 00 00 13
0x0020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0030 00 00 2c 25 00 00 00 00 00 00 00 00 00 00 00 20
0x0040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
The only differences seem to be the PCI ID field at byte offset 4-5,
Config 0 field at byte offset 0x14, and the checksum at 0x32-0x33.
If I understand correctly, the Config 0 bit 0 affects Boot ROM size. It
affects only the PXE boot sequence?
Additionally, I can verify that all the chips have "RTL8110SC 67233S1
G28B" on them. So the differing PCI IDs is an oddity.
I can do some additional tests, and test if the bad packets can be
reproduced against a switch and captured.
But other than that, I'm wondering if the failed mdio writing could have
caused permanent damage in the PHY.
The only differences seem to be the PCI ID field at byte offset 4-5,
Config 0 field at byte offset 0x14, and the checksum at 0x32-0x33.
If I understand correctly, the Config 0 bit 0 affects Boot ROM size. It
affects only the PXE boot sequence?
I have never played with it. I can only tell that it has a different value
for my motherboard included 8168b (see previous messages).
[...]
But other than that, I'm wondering if the failed mdio writing could have
caused permanent damage in the PHY.
It's hard to tell and it wouldn't explain the corrupted eeprom.
--
Ueimor