sky2 (was Re: 2.6.18-mm2)

Andrew Morton wrote:

Another customer..

Begin forwarded message:

Date: Fri, 29 Sep 2006 00:44:01 +0200
From: Matthias Hentges <redacted>
To: Andrew Morton <redacted>
Cc: linux-kernel@vger.kernel.org
Subject: Re: 2.6.18-mm2


Hello all,

I've just tested -mm2 on my C2D system and I'm getting a lot of these
messages:

"[  139.143807] printk: 131 messages suppressed.
[  139.148235] sky2 0000:03:00.0: pci express error (0x500547)"

Please note that the "sky2" driver has always been the black sheep on
that system due to regular full lock-ups of the driver, requiring a
rmmod sky2 + modprobe sky2 cycle.

This happens often enough to warrant writing a cronjob checking the
network and auto-rmmod'ing the module.....

While the above is bloody annoying at times (heh), the driver never
caused any messages like the ones I now get with -mm2 .

sky2 just turned on PCI Express error reporting, so it makes sense that 
messages would appear.  The better question is whether this is a driver 
problem, or a hardware problem.  With your "black sheep" comment, I 
wonder if it isn't a hardware problem that's been hidden.

	Jeff

On Thu, 28 Sep 2006 19:07:05 -0400
Jeff Garzik [off-list ref] wrote:

Andrew Morton wrote:

quoted

Another customer..

Begin forwarded message:

Date: Fri, 29 Sep 2006 00:44:01 +0200
From: Matthias Hentges <redacted>
To: Andrew Morton <redacted>
Cc: linux-kernel@vger.kernel.org
Subject: Re: 2.6.18-mm2


Hello all,

I've just tested -mm2 on my C2D system and I'm getting a lot of these
messages:

"[  139.143807] printk: 131 messages suppressed.
[  139.148235] sky2 0000:03:00.0: pci express error (0x500547)"

Please note that the "sky2" driver has always been the black sheep on
that system due to regular full lock-ups of the driver, requiring a
rmmod sky2 + modprobe sky2 cycle.

This happens often enough to warrant writing a cronjob checking the
network and auto-rmmod'ing the module.....

While the above is bloody annoying at times (heh), the driver never
caused any messages like the ones I now get with -mm2 .

sky2 just turned on PCI Express error reporting, so it makes sense that 
messages would appear.  The better question is whether this is a driver 
problem, or a hardware problem.  With your "black sheep" comment, I 
wonder if it isn't a hardware problem that's been hidden.

Here is the debug patch I sent to the first reporter of the problem.
I know what the offset is supposed to be, so if the PCI subsystem is
wrong, this will show. 

--- sky2.orig/drivers/net/sky2.c	2006-09-28 08:45:27.000000000 -0700
+++ sky2/drivers/net/sky2.c	2006-09-28 08:51:24.000000000 -0700

@@ -2463,6 +2463,7 @@
 
 	sky2_write8(hw, B0_CTST, CS_MRST_CLR);
 
+#define PEX_UNC_ERR_STAT 0x104		/* PCI extended error capablity */
 	/* clear any PEX errors */
 	if (pci_find_capability(hw->pdev, PCI_CAP_ID_EXP)) {
 		hw->err_cap = pci_find_ext_capability(hw->pdev, PCI_EXT_CAP_ID_ERR);

@@ -2470,6 +2471,15 @@
 			sky2_pci_write32(hw,
 					 hw->err_cap + PCI_ERR_UNCOR_STATUS,
 					 0xffffffffUL);
+		else
+			printk(KERN_ERR PFX "pci express found but not extended error support?\n");
+		
+		if (hw->err_cap + PCI_ERR_UNCOR_STATUS != PEX_UNC_ERR_STAT) {
+			
+			printk(KERN_ERR PFX "pci express error status register fixed from %#x to %#x\n",
+			       hw->err_cap, PEX_UNC_ERR_STAT - PCI_ERR_UNCOR_STATUS);
+			hw->err_cap = PEX_UNC_ERR_STAT - PCI_ERR_UNCOR_STATUS;
+		}
 	}
 
 	hw->pmd_type = sky2_read8(hw, B2_PMD_TYP);

On Thu, 28 Sep 2006 19:07:05 -0400
Jeff Garzik [off-list ref] wrote:

Andrew Morton wrote:

quoted

Another customer..

Begin forwarded message:

Date: Fri, 29 Sep 2006 00:44:01 +0200
From: Matthias Hentges <redacted>
To: Andrew Morton <redacted>
Cc: linux-kernel@vger.kernel.org
Subject: Re: 2.6.18-mm2


Hello all,

I've just tested -mm2 on my C2D system and I'm getting a lot of these
messages:

"[  139.143807] printk: 131 messages suppressed.
[  139.148235] sky2 0000:03:00.0: pci express error (0x500547)"

Please note that the "sky2" driver has always been the black sheep on
that system due to regular full lock-ups of the driver, requiring a
rmmod sky2 + modprobe sky2 cycle.

This happens often enough to warrant writing a cronjob checking the
network and auto-rmmod'ing the module.....

While the above is bloody annoying at times (heh), the driver never
caused any messages like the ones I now get with -mm2 .

sky2 just turned on PCI Express error reporting, so it makes sense that 
messages would appear.  The better question is whether this is a driver 
problem, or a hardware problem.  With your "black sheep" comment, I 
wonder if it isn't a hardware problem that's been hidden.

See also http://bugzilla.kernel.org/show_bug.cgi?id=7222

That's two reports in 18 hours, from amongst the presumably-small population
of sky2-owning -mm testers.

See also http://bugzilla.kernel.org/show_bug.cgi?id=7222

That's two reports in 18 hours, from amongst the presumably-small population
of sky2-owning -mm testers.

I'll back it out if we don't get a simple resolution. It was just trying to
use the pci facilities as intended.

Note: I know what is causing all the sky2 problems, there is something wrong that
is causing flow control negotiation not to propagate back to all the multiple levels
of the chip. Unclear how to fix it, the documentation is not helpful on this.
If not resolved soon, I'll just force Tx flow control off for now.


-- 
Stephen Hemminger [off-list ref]

On Thu, Sep 28, 2006 at 04:30:23PM -0700, Stephen Hemminger wrote:

Note: I know what is causing all the sky2 problems, there is something wrong that
is causing flow control negotiation not to propagate back to all the multiple levels
of the chip. Unclear how to fix it, the documentation is not helpful on this.
If not resolved soon, I'll just force Tx flow control off for now.

just for the record, same problem here.
I mean, with my Sony Vaio VGN-SZ1VP (here[1] you can find all hardware
details), it's enough some mega of udp traffic, usually nfs, to "freeze"
the network. Well, no complain from the kernel. It's enough to rmmod and
modprobe sky2 to fix the problem. I already tried -mm1, but nothing
changed. In the meanwhile I will continue to use my usb network card.

Thanks a lot for your time,
Andrea Gelmini

-------
[1] http://groups.google.it/group/linux.kernel/msg/ceff3014c410bea6 

Hello Stephen,

Am Donnerstag, den 28.09.2006, 16:19 -0700 schrieb Stephen Hemminger:

[...]

[...]

Thanks for the patch:

+root@mhcln01:~ >dmesg |grep -i expres
[   36.814288] sky2 pci express found but not extended error support?
[   36.820494] sky2 pci express error status register fixed from 0x0 to
0x100
[   36.833769] sky2 0000:04:00.0: pci express error (0x100000)
[   36.851296] sky2 pci express found but not extended error support?
[   36.857499] sky2 pci express error status register fixed from 0x0 to
0x100
[   36.870756] sky2 0000:03:00.0: pci express error (0x100000)
+root@mhcln01:~ >

The annoying messages are gone =)
Attached is a complete dmesg of a fresh boot.

Thanks for your time.
-- 
Matthias Hentges 

My OS: Debian SID. Geek by Nature, Linux by Choice

Hello Stephen,

Am Donnerstag, den 28.09.2006, 16:19 -0700 schrieb Stephen Hemminger:

while the above patch indeed removes the error messages from my previous
mail, I have since seen random but reproduceable  freezes of the box in
question. I believe they are sky2 related since the freeze can be
triggered by continuous network traffic (like playing a movie over NFS
etc.).

The freezes only happen with 2.6.18-mm2 and 2.6.18-mm3. 2.6.18-mm1 works
perfectly fine.
I've hooked up the box to my laptop via a serial cable and captured all
kernel messages from booting up the machine to the freeze. You'll note
that the last messages are from the sky2 driver ;)

Once frozen the network is dead, the screen won't wake up from suspend
and CAPSLOCK can not be toggled. SYSRQ (sp?) still works tho.

Any help in debugging this problem would be appreciated =)
-- 
Matthias 'CoreDump' Hentges 

My OS: Debian SID. Geek by Nature, Linux by Choice

On Wed, 04 Oct 2006 04:57:08 +0200
Matthias Hentges [off-list ref] wrote:

Hello Stephen,

Am Donnerstag, den 28.09.2006, 16:19 -0700 schrieb Stephen Hemminger:

quoted

Here is the debug patch I sent to the first reporter of the problem.
I know what the offset is supposed to be, so if the PCI subsystem is
wrong, this will show. 

--- sky2.orig/drivers/net/sky2.c	2006-09-28 08:45:27.000000000 -0700
+++ sky2/drivers/net/sky2.c	2006-09-28 08:51:24.000000000 -0700

@@ -2463,6 +2463,7 @@
 
 	sky2_write8(hw, B0_CTST, CS_MRST_CLR);
 
+#define PEX_UNC_ERR_STAT 0x104		/* PCI extended error capablity */
 	/* clear any PEX errors */
 	if (pci_find_capability(hw->pdev, PCI_CAP_ID_EXP)) {
 		hw->err_cap = pci_find_ext_capability(hw->pdev, PCI_EXT_CAP_ID_ERR);

@@ -2470,6 +2471,15 @@
 			sky2_pci_write32(hw,
 					 hw->err_cap + PCI_ERR_UNCOR_STATUS,
 					 0xffffffffUL);
+		else
+			printk(KERN_ERR PFX "pci express found but not extended error support?\n");
+		
+		if (hw->err_cap + PCI_ERR_UNCOR_STATUS != PEX_UNC_ERR_STAT) {
+			
+			printk(KERN_ERR PFX "pci express error status register fixed from %#x to %#x\n",
+			       hw->err_cap, PEX_UNC_ERR_STAT - PCI_ERR_UNCOR_STATUS);
+			hw->err_cap = PEX_UNC_ERR_STAT - PCI_ERR_UNCOR_STATUS;
+		}
 	}
 
 	hw->pmd_type = sky2_read8(hw, B2_PMD_TYP);

while the above patch indeed removes the error messages from my previous
mail, I have since seen random but reproduceable  freezes of the box in
question. I believe they are sky2 related since the freeze can be
triggered by continuous network traffic (like playing a movie over NFS
etc.).

When it fixes what does the log say. I'm probably going to back out
the PCI express extended error using the pci_XXX functions.

The freezes only happen with 2.6.18-mm2 and 2.6.18-mm3. 2.6.18-mm1 works
perfectly fine.
I've hooked up the box to my laptop via a serial cable and captured all
kernel messages from booting up the machine to the freeze. You'll note
that the last messages are from the sky2 driver ;)

Does it still happen with linus git tree. If so, a git bisect might
help. It might not be sky2 related at all, there has been lots of changes.

Once frozen the network is dead, the screen won't wake up from suspend
and CAPSLOCK can not be toggled. SYSRQ (sp?) still works tho.

Any help in debugging this problem would be appreciated =)

The TX timeout is a symptom of a common bug still not fixed where
the transmitter stops. I'm working on reproducing it on my hardware and switches,
because without a reproducible test, its just shooting in the dark and
that isn't working.

Hello Stephen,

Am Dienstag, den 03.10.2006, 20:26 -0700 schrieb Stephen Hemminger:

[...]

quoted

while the above patch indeed removes the error messages from my previous
mail, I have since seen random but reproduceable  freezes of the box in
question. I believe they are sky2 related since the freeze can be
triggered by continuous network traffic (like playing a movie over NFS
etc.).

When it fixes what does the log say. I'm probably going to back out
the PCI express extended error using the pci_XXX functions.

quoted

The freezes only happen with 2.6.18-mm2 and 2.6.18-mm3. 2.6.18-mm1 works
perfectly fine.
I've hooked up the box to my laptop via a serial cable and captured all
kernel messages from booting up the machine to the freeze. You'll note
that the last messages are from the sky2 driver ;)

Does it still happen with linus git tree. If so, a git bisect might
help. It might not be sky2 related at all, there has been lots of changes.

I am doing a bisect right now which is kind of a PITA as the freeze is
completely random.
I have since noticed that the freeze happens shortly after the network
dies, possibly during the "rrmod sky2 / moprobe sky2" my script is
doing.

quoted

Once frozen the network is dead, the screen won't wake up from suspend
and CAPSLOCK can not be toggled. SYSRQ (sp?) still works tho.

Any help in debugging this problem would be appreciated =)

The TX timeout is a symptom of a common bug still not fixed where
the transmitter stops. I'm working on reproducing it on my hardware and switches,
because without a reproducible test, its just shooting in the dark and
that isn't working.

I'd be happy to assist with that as I have his bug up to 5 times a day :
\
-- 
Matthias 'CoreDump' Hentges 

Webmaster of hentges.net and OpenZaurus developer.
You can reach me in #openzaurus on Freenode.

My OS: Debian SID. Geek by Nature, Linux by Choice

Hi Stephen,

I believe I have identified the problem. The freeze only happens when
your debug patch to work around sky2 PCIe error messages is applied.
Without your patch (attached) I get _tons_ of error messages and the NIC
dies every few seconds / minutes (reproduceable!), but the system
recovers just fine from a NIC crash.

I have verified this behavior (works fine w/o debug patch, freezes with
patch applied) with:
- 2.6.19-rc1-git4 
- 2.6.18-git something 
- 2.6.18-mm3
  
-- 
Matthias 'CoreDump' Hentges 

My OS: Debian SID. Geek by Nature, Linux by Choice

On Sun, 08 Oct 2006 15:41:45 +0200
Matthias Hentges [off-list ref] wrote:

Hi Stephen,

I believe I have identified the problem. The freeze only happens when
your debug patch to work around sky2 PCIe error messages is applied.
Without your patch (attached) I get _tons_ of error messages and the NIC
dies every few seconds / minutes (reproduceable!), but the system
recovers just fine from a NIC crash.

I have verified this behavior (works fine w/o debug patch, freezes with
patch applied) with:
- 2.6.19-rc1-git4 
- 2.6.18-git something 
- 2.6.18-mm3
  

Does 2.6.18 work?

What is the PCI config of the device (lspci -vvvx)?

What is the chip version (dmesg | grep sky2)?

Am Sonntag, den 08.10.2006, 09:20 -0700 schrieb Stephen Hemminger:

On Sun, 08 Oct 2006 15:41:45 +0200
Matthias Hentges [off-list ref] wrote:

[...]

quoted

I have verified this behavior (works fine w/o debug patch, freezes with
patch applied) with:
- 2.6.19-rc1-git4 
- 2.6.18-git something 
- 2.6.18-mm3
  

Does 2.6.18 work?

2.6.18 does not require the patch ( in fact, the patch won't apply at
all) and thus does not freeze when the NIC goes down.

What is the PCI config of the device (lspci -vvvx)?

I've attached the lspci dump of both onboard sky2 NICs.

What is the chip version (dmesg | grep sky2)?

+root@mhcln01:..patches/generic/coretemp >dmesg |grep sky2
[   34.833453] sky2 v1.7 addr 0xfa9fc000 irq 17 Yukon-EC (0xb6) rev 2
[   34.833527] sky2 eth0: addr 00:17:31:f4:f1:8c
[   34.833663] sky2 v1.7 addr 0xfa8fc000 irq 16 Yukon-EC (0xb6) rev 2
[   34.833729] sky2 eth1: addr 00:17:31:f4:f7:cc
[   42.165687] sky2 eth1: enabling interface
[   43.847972] sky2 eth1: Link is up at 100 Mbps, full duplex, flow
control both

-- 
Matthias 'CoreDump' Hentges 

My OS: Debian SID. Geek by Nature, Linux by Choice

Le Sun, 8 Oct 2006 09:20:01 -0700, Stephen Hemminger
[off-list ref] a écrit :

On Sun, 08 Oct 2006 15:41:45 +0200
Matthias Hentges [off-list ref] wrote:

quoted

Hi Stephen,

I believe I have identified the problem. The freeze only happens when
your debug patch to work around sky2 PCIe error messages is applied.
Without your patch (attached) I get _tons_ of error messages and the NIC
dies every few seconds / minutes (reproduceable!), but the system
recovers just fine from a NIC crash.

I have verified this behavior (works fine w/o debug patch, freezes with
patch applied) with:
- 2.6.19-rc1-git4 
- 2.6.18-git something 
- 2.6.18-mm3
  

Does 2.6.18 work?

What is the PCI config of the device (lspci -vvvx)?

What is the chip version (dmesg | grep sky2)?

I've got the same issue here since a while (See Message-ID:
[off-list ref] from 18 Apr 2006)

I steel get freeze (with non tainted kernel) and getting these messages
at boot :

~ % dmesg | grep -i sky
sky2 v1.7 addr 0xff3fc000 irq 233 Yukon-EC (0xb6) rev 2
sky2 eth0: addr 00:15:f2:a9:6a:65
sky2 0000:02:00.0: No interrupt was generated using MSI, switching to INTx mode. Please report this failure to the PCI maintainer and include system chipset information.
sky2 eth0: enabling interface
sky2 eth0: disabling interface

lspci -vvvx attached

-- 
Beber - E-Mail / Jabber (+GMail) : beber_AT_meleeweb.net
http://www.meleeweb.net