From: Thomas Glanzmann <hidden> Date: 2007-02-01 19:22:42
Hello,
I have a sky2 network card in my intel mac mini. It stops working when I
do havy network load like watching a divx over http/sshfs. However if I
remove the driver module and load it again it works and even the tcp
connection doesn't get shutdown. I automated the above procedure using
a userland watchdog which basically does the same thing and is written
entirely by me, because the traditional watchdog wasn't that reliable
and did a lot of false positives:
* Look every ten seconds if my default router is pingable (3
pings, one has to get back).
If it isn't the case I call network_fix script (it calls the
script only once after a ping gets lost. To run the script again at least one
ping has to arrive again)
(mini) [~] cat /usr/local/sbin/fix_network
#!/bin/bash
export PATH=/bin:/usr/bin:/usr/sbin:/sbin
rmmod sky2
modprobe sky2
ifdown eth0
ifup eth0
If after that no ping is received from the default
router for another 90 seconds I tell init to reboot and
stop feeding the kernel software watchdog.
* My watchdog also checks if sshd process is running. If it is
down for more than 100 seconds it reboots the machine, too.
Jan 27 22:35:35 mini watchdog-tg[4146]: No PONG received from 192.168.0.3 (failure 1 of 10)
Jan 27 22:35:35 mini watchdog-tg[4146]: Running fix_network script.
Jan 27 22:38:46 mini watchdog-tg[4146]: No PONG received from 192.168.0.3 (failure 1 of 10)
Jan 27 22:38:46 mini watchdog-tg[4146]: Running fix_network script.
Jan 27 22:44:17 mini watchdog-tg[4146]: No PONG received from 192.168.0.3 (failure 1 of 10)
Jan 27 22:44:17 mini watchdog-tg[4146]: Running fix_network script.
Jan 29 12:00:13 mini watchdog-tg[4146]: No PONG received from 192.168.0.3 (failure 1 of 10)
Jan 29 12:00:13 mini watchdog-tg[4146]: Running fix_network script.
Jan 29 19:18:59 mini watchdog-tg[4146]: No PONG received from 192.168.0.3 (failure 1 of 10)
Jan 29 19:18:59 mini watchdog-tg[4146]: Running fix_network script.
Jan 31 15:56:29 mini watchdog-tg[4146]: No PONG received from 192.168.0.3 (failure 1 of 10)
Jan 31 15:56:29 mini watchdog-tg[4146]: Running fix_network script.
Feb 1 08:56:57 mini watchdog-tg[4146]: No PONG received from 192.168.0.3 (failure 1 of 10)
Feb 1 08:56:57 mini watchdog-tg[4146]: Running fix_network script.
I have a question to this: I wonder why the Linux Kernel (no longer?)
increments the use counter of an ethernet driver (I saw it on sky2 and
e1000) when the interface is up, running and configured? I can unload
the sky2 driver without doing a 'ifconfig eth0 down' beforehand. Could
somone provide me with background on this fact?
With that everything works. If somone is interested in my userland
watchdog, just send me an E-Mail.
@Sam: I can provide you access to my hardware including root access via
the wifi driver so that you can debug this network driver lockup, if you
want to.
Thomas
From: Stephen Hemminger <hidden> Date: 2007-02-01 19:09:16
On Thu, 1 Feb 2007 19:55:32 +0100
Thomas Glanzmann [off-list ref] wrote:
Hello,
I have a sky2 network card in my intel mac mini. It stops working when I
do havy network load like watching a divx over http/sshfs. However if I
remove the driver module and load it again it works and even the tcp
connection doesn't get shutdown. I automated the above procedure using
a userland watchdog which basically does the same thing and is written
entirely by me, because the traditional watchdog wasn't that reliable
and did a lot of false positives:
* Look every ten seconds if my default router is pingable (3
pings, one has to get back).
If it isn't the case I call network_fix script (it calls the
script only once after a ping gets lost. To run the script again at least one
ping has to arrive again)
(mini) [~] cat /usr/local/sbin/fix_network
#!/bin/bash
export PATH=/bin:/usr/bin:/usr/sbin:/sbin
rmmod sky2
modprobe sky2
ifdown eth0
ifup eth0
If after that no ping is received from the default
router for another 90 seconds I tell init to reboot and
stop feeding the kernel software watchdog.
* My watchdog also checks if sshd process is running. If it is
down for more than 100 seconds it reboots the machine, too.
Jan 27 22:35:35 mini watchdog-tg[4146]: No PONG received from 192.168.0.3 (failure 1 of 10)
Jan 27 22:35:35 mini watchdog-tg[4146]: Running fix_network script.
Jan 27 22:38:46 mini watchdog-tg[4146]: No PONG received from 192.168.0.3 (failure 1 of 10)
Jan 27 22:38:46 mini watchdog-tg[4146]: Running fix_network script.
Jan 27 22:44:17 mini watchdog-tg[4146]: No PONG received from 192.168.0.3 (failure 1 of 10)
Jan 27 22:44:17 mini watchdog-tg[4146]: Running fix_network script.
Jan 29 12:00:13 mini watchdog-tg[4146]: No PONG received from 192.168.0.3 (failure 1 of 10)
Jan 29 12:00:13 mini watchdog-tg[4146]: Running fix_network script.
Jan 29 19:18:59 mini watchdog-tg[4146]: No PONG received from 192.168.0.3 (failure 1 of 10)
Jan 29 19:18:59 mini watchdog-tg[4146]: Running fix_network script.
Jan 31 15:56:29 mini watchdog-tg[4146]: No PONG received from 192.168.0.3 (failure 1 of 10)
Jan 31 15:56:29 mini watchdog-tg[4146]: Running fix_network script.
Feb 1 08:56:57 mini watchdog-tg[4146]: No PONG received from 192.168.0.3 (failure 1 of 10)
Feb 1 08:56:57 mini watchdog-tg[4146]: Running fix_network script.
I have a question to this: I wonder why the Linux Kernel (no longer?)
increments the use counter of an ethernet driver (I saw it on sky2 and
e1000) when the interface is up, running and configured? I can unload
the sky2 driver without doing a 'ifconfig eth0 down' beforehand. Could
somone provide me with background on this fact?
It was intentional in 2.6 to allow interfaces to be hot-removed.
Remember with Internet protocols there is no hard binding (normally)
between address and device and connections should not go down
if link fails.
With that everything works. If somone is interested in my userland
watchdog, just send me an E-Mail.
From: Thomas Glanzmann <hidden> Date: 2007-02-01 19:19:05
Hello Stephen,
It was intentional in 2.6 to allow interfaces to be hot-removed.
Remember with Internet protocols there is no hard binding (normally)
between address and device and connections should not go down if link
fails.
of course. That makes sense. I just wondered when the change in mind
happened. And actually I like this behaviour.
quoted
With that everything works. If somone is interested in my userland
watchdog, just send me an E-Mail.
From: Stephen Hemminger <hidden> Date: 2007-02-01 23:01:15
I can reproduce the problem now (on mac mini). Interestingly it seems to whack
the whole ethernet switch when it happens.
- a previously suggested fix - passing idle=poll to the kernel - did not
work for me at the end
It is not an MSI or IRQ problem. It is a phy problem (see below).
- the locks I have happen very periodically (somewhere around every 22-28
hours), as if the chip would die after a given amount of data transferred;
I know this looks stupid but I thought I might mention it
- I have about 1Mbit/s of (incoming) traffic on this interface: with
short, very high peaks, as there is a MySQL server on the other end,
receiving about 100 queries per second
- unloading the sky2 module totally freezes the computer for me
If you do:
ethtool -r eth0
it cause a PHY reset (renegotiation) and clears the problem.
--
Stephen Hemminger [off-list ref]
Hello,
I have a sky2 network card in my intel mac mini. It stops working when I
do havy network load like watching a divx over http/sshfs. However if I
remove the driver module and load it again it works and even the tcp
connection doesn't get shutdown. I automated the above procedure using a
userland watchdog which basically does the same thing and is written
entirely by me, because the traditional watchdog wasn't that reliable
and did a lot of false positives:
* Look every ten seconds if my default router is pingable (3
pings, one has to get back).
If it isn't the case I call network_fix script (it calls
the script only once after a ping gets lost. To run the
script again at least one ping has to arrive again)
(mini) [~] cat /usr/local/sbin/fix_network
#!/bin/bash
export PATH=/bin:/usr/bin:/usr/sbin:/sbin
rmmod sky2
modprobe sky2
ifdown eth0
ifup eth0
If after that no ping is received from the default
router for another 90 seconds I tell init to reboot and
stop feeding the kernel software watchdog.
* My watchdog also checks if sshd process is running. If it is
down for more than 100 seconds it reboots the machine, too.
Jan 27 22:35:35 mini watchdog-tg[4146]: No PONG received from
192.168.0.3 (failure 1 of 10) Jan 27 22:35:35 mini watchdog-tg[4146]:
Running fix_network script. Jan 27 22:38:46 mini watchdog-tg[4146]: No
PONG received from 192.168.0.3 (failure 1 of 10) Jan 27 22:38:46 mini
watchdog-tg[4146]: Running fix_network script. Jan 27 22:44:17 mini
watchdog-tg[4146]: No PONG received from 192.168.0.3 (failure 1 of 10)
Jan 27 22:44:17 mini watchdog-tg[4146]: Running fix_network script. Jan
29 12:00:13 mini watchdog-tg[4146]: No PONG received from 192.168.0.3
(failure 1 of 10) Jan 29 12:00:13 mini watchdog-tg[4146]: Running
fix_network script. Jan 29 19:18:59 mini watchdog-tg[4146]: No PONG
received from 192.168.0.3 (failure 1 of 10) Jan 29 19:18:59 mini
watchdog-tg[4146]: Running fix_network script. Jan 31 15:56:29 mini
watchdog-tg[4146]: No PONG received from 192.168.0.3 (failure 1 of 10)
Jan 31 15:56:29 mini watchdog-tg[4146]: Running fix_network script. Feb
1 08:56:57 mini watchdog-tg[4146]: No PONG received from 192.168.0.3
(failure 1 of 10) Feb 1 08:56:57 mini watchdog-tg[4146]: Running
fix_network script.
[...]
I would like to add a few things:
- a previously suggested fix - passing idle=poll to the kernel - did not
work for me at the end
- the locks I have happen very periodically (somewhere around every 22-28
hours), as if the chip would die after a given amount of data transferred;
I know this looks stupid but I thought I might mention it
- I have about 1Mbit/s of (incoming) traffic on this interface: with
short, very high peaks, as there is a MySQL server on the other end,
receiving about 100 queries per second
- unloading the sky2 module totally freezes the computer for me
- Fagzal
From: Thomas Glanzmann <hidden> Date: 2007-02-02 06:27:25
Hello Fagyal,
- a previously suggested fix - passing idle=poll to the kernel - did not
work for me at the end
same for me. I tried the two module parameters and the kernel parameter:
pci=nomsi sky2.disable_msi=1 sky2.idle_timeout=1000
- the locks I have happen very periodically (somewhere around every 22-28
hours), as if the chip would die after a given amount of data transferred;
I know this looks stupid but I thought I might mention it
I had a dedicated server with sky2 which had the same symptoms but I
disabled the onboard sky2 to and added a e100. On my mac mini I can
reproduce it nearly immediately. Just have to scp a kernel tar tree over
and it hangs.
- unloading the sky2 module totally freezes the computer for me
From: Thomas Glanzmann <hidden> Date: 2007-02-02 06:31:08
Hello Stephen,
I can reproduce the problem now (on mac mini). Interestingly it seems
to whack the whole ethernet switch when it happens.
wow. I have Linksys wrt54g has 'ethernet switch' and my Snom 320 VoIP
phone still works when the mini network card goes down. On the other
side the wrt54g isn't exactly a switch but more like a bunch of network
cards which use the linux bridging code IIRC.
If you do:
ethtool -r eth0
it cause a PHY reset (renegotiation) and clears the problem.
But this isn't related to my problen (on mac mini), is it?
Thomas