From: Nick Bowler <hidden> Date: 2012-02-02 19:21:38
Hi,
I just saw this panic on 3.3-rc2 with skge. I don't know whether it's
reproducible yet -- the machine crashed while I was not actively using
it. We've had this type of card for a few years and I've never seen this
before so it may be a regression, but admittedly we don't use them all
that often.
At the time of the crash, the network interface in question was up, but
not configured with any addresses; mtu configured to (the default) 1500
bytes. It was used for packet capture (tcpdump) shortly beforehand,
although this was not running at the time of the crash. It's a PCI
gigabit ethernet card:
03:01.0 Ethernet controller: D-Link System Inc DGE-530T Gigabit Ethernet Adapter (rev 11) (rev 11)
In case it matters, the primary network interface of the system is an
onboard device using the sky2 driver:
02:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8057 PCI-E Gigabit Ethernet Controller (rev 10Ethernet controller: Marvell Technology Group Ltd. 88E8057 PCI-E Gigabit Ethernet Controller (rev 10)
I took a photo of the screen, but unfortuately the top few lines are
lost forever. Posted at imgur because it seems a bit big for the
mailing lists: <http://i.imgur.com/xklGU.jpg>. Apologies for the
terrible quality; the best camera I could find around was in someone's
mobile phone. I'll see about configuring a serial console in case it
crashes again.
For convenience (hopefully), here's the call trace retyped in plain text
(addresses elided, see photo for them). The full code line is
reproduced by hand because it's almost unreadable in the photo (and
truncated to boot).
Call Trace:
<IRQ>
[...] net_rx_action+0xaa/0x1c0
[...] __do_softirq+0x7e/0x125
[...] ? _raw_spin_unlock+0x26/0x31
[...] call_softirq+0x1c/0x30
[...] do_softirq+0x33/0x68
[...] irq_exit+0x3f/0xb9
[...] do_IRQ+0x97/0xae
[...] common_interrupt+0x6b/0x6b
<EOI>
[...] ? hrtimer_start+0x13/0x15
[...] ? mwait_idle+0x6e/0x80
[...] ? mwait_idle+0x61/0x80
[...] cpu_idle+0x61/0xbd
[...] rest_init+0x8d/0x91
[...] start_kernel+0x338/0x343
[...] x86_64_start_reservations+0xb8/0xbd
[...] x86_64_start_kernel+0xed/0xf4
Code: 48 8b 40 30 48 85 c0 74 0a b9 02 00 00 00 4c 89 fa ff d0 49 8b 86 d0 00 00 00 49 8b 4d b4 48 89 c7 48 8b b2 d0 00 00 00 <f3> a4 31 ff 48 8b 03 49 8b 75 18 48 8b 40 08 48 85 c0 74 13 48
RIP [...] skge_poll+0x367/0x5cd [skge]
Let me know if you need any more info,
--
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)
From: Stephen Hemminger <hidden> Date: 2012-02-02 20:45:33
On Thu, 2 Feb 2012 14:21:15 -0500
Nick Bowler [off-list ref] wrote:
Hi,
I just saw this panic on 3.3-rc2 with skge. I don't know whether it's
reproducible yet -- the machine crashed while I was not actively using
it. We've had this type of card for a few years and I've never seen this
before so it may be a regression, but admittedly we don't use them all
that often.
At the time of the crash, the network interface in question was up, but
not configured with any addresses; mtu configured to (the default) 1500
bytes. It was used for packet capture (tcpdump) shortly beforehand,
although this was not running at the time of the crash. It's a PCI
gigabit ethernet card:
03:01.0 Ethernet controller: D-Link System Inc DGE-530T Gigabit Ethernet Adapter (rev 11) (rev 11)
In case it matters, the primary network interface of the system is an
onboard device using the sky2 driver:
02:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8057 PCI-E Gigabit Ethernet Controller (rev 10Ethernet controller: Marvell Technology Group Ltd. 88E8057 PCI-E Gigabit Ethernet Controller (rev 10)
I took a photo of the screen, but unfortuately the top few lines are
lost forever. Posted at imgur because it seems a bit big for the
mailing lists: <http://i.imgur.com/xklGU.jpg>. Apologies for the
terrible quality; the best camera I could find around was in someone's
mobile phone. I'll see about configuring a serial console in case it
crashes again.
For convenience (hopefully), here's the call trace retyped in plain text
(addresses elided, see photo for them). The full code line is
reproduced by hand because it's almost unreadable in the photo (and
truncated to boot).
Call Trace:
<IRQ>
[...] net_rx_action+0xaa/0x1c0
[...] __do_softirq+0x7e/0x125
[...] ? _raw_spin_unlock+0x26/0x31
[...] call_softirq+0x1c/0x30
[...] do_softirq+0x33/0x68
[...] irq_exit+0x3f/0xb9
[...] do_IRQ+0x97/0xae
[...] common_interrupt+0x6b/0x6b
<EOI>
[...] ? hrtimer_start+0x13/0x15
[...] ? mwait_idle+0x6e/0x80
[...] ? mwait_idle+0x61/0x80
[...] cpu_idle+0x61/0xbd
[...] rest_init+0x8d/0x91
[...] start_kernel+0x338/0x343
[...] x86_64_start_reservations+0xb8/0xbd
[...] x86_64_start_kernel+0xed/0xf4
Code: 48 8b 40 30 48 85 c0 74 0a b9 02 00 00 00 4c 89 fa ff d0 49 8b 86 d0 00 00 00 49 8b 4d b4 48 89 c7 48 8b b2 d0 00 00 00 <f3> a4 31 ff 48 8b 03 49 8b 75 18 48 8b 40 08 48 85 c0 74 13 48
RIP [...] skge_poll+0x367/0x5cd [skge]
Let me know if you need any more info,
Try reverting this commit, it seems problematic
commit d0249e44432aa0ffcf710b64449b8eaa3722547e
Author: stephen hemminger [off-list ref]
Date: Thu Jan 19 14:37:18 2012 +0000
skge: check for PCI dma mapping errors
From: Nick Bowler <hidden> Date: 2012-02-03 19:28:37
Hi Stephen,
On 2012-02-02 12:45 -0800, Stephen Hemminger wrote:
On Thu, 2 Feb 2012 14:21:15 -0500
Nick Bowler [off-list ref] wrote:
quoted
I just saw this panic on 3.3-rc2 with skge. I don't know whether it's
reproducible yet -- the machine crashed while I was not actively using
it. We've had this type of card for a few years and I've never seen this
before so it may be a regression, but admittedly we don't use them all
that often.
[...]
Try reverting this commit, it seems problematic
commit d0249e44432aa0ffcf710b64449b8eaa3722547e
Author: stephen hemminger [off-list ref]
Date: Thu Jan 19 14:37:18 2012 +0000
skge: check for PCI dma mapping errors
Thanks for the pointer, I'll try that. Unfortunately some other stuff
has come up so I probably won't be able to test it until next week.
Cheers,
--
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)
On Thu, Feb 2, 2012 at 9:45 PM, Stephen Hemminger [off-list ref] wrote:
Try reverting this commit, it seems problematic
commit d0249e44432aa0ffcf710b64449b8eaa3722547e
Reverting this commit fixed skge for me. The NIC stopped working after
a kernel upgrade with nothing interesting in dmesg:
skge: 1.14 addr 0xfbffc000 irq 18 chip Yukon-Lite rev 9
skge 0000:03:00.0: eth0: addr 00:15:f2:46:d3:3b
skge 0000:03:00.0: eth0: enabling interface
ADDRCONF(NETDEV_UP): eth0: link is not ready
skge 0000:03:00.0: eth0: Link is up at 100 Mbps, full duplex, flow control both
ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
eth0: no IPv6 routers present
NetworkManager would Oops during shutting down: http://i.imgur.com/eAFgM.jpg
--
Guillaume
From: Stephen Hemminger <hidden> Date: 2012-02-07 00:18:01
On Mon, 6 Feb 2012 19:11:27 -0500
Paul Gortmaker [off-list ref] wrote:
On Thu, Feb 2, 2012 at 3:45 PM, Stephen Hemminger [off-list ref] wrote:
[...]
quoted
Try reverting this commit, it seems problematic
commit d0249e44432aa0ffcf710b64449b8eaa3722547e
Author: stephen hemminger [off-list ref]
Date: Thu Jan 19 14:37:18 2012 +0000
skge: check for PCI dma mapping errors
I'm seeing similar issues, and a revert of the above caused the
problems to go away. I'm testing on a baseline of net-next
as of today (3238a9be4d7a) plus some TIPC patches I was
trying to test (which are 99.9% unrelated to this, I'm sure).
Details captured from serial console are below. 100% reproducible.
I can probably try a test/debug patch for you if need be.
Paul.
There is a simple bug in the cleanup code reordering. And it is
reproducible here. Working on a better solution.
From: Nick Bowler <hidden> Date: 2012-02-08 16:32:46
On 2012-02-03 14:28 -0500, Nick Bowler wrote:
On 2012-02-02 12:45 -0800, Stephen Hemminger wrote:
quoted
On Thu, 2 Feb 2012 14:21:15 -0500
Nick Bowler [off-list ref] wrote:
quoted
I just saw this panic on 3.3-rc2 with skge. I don't know whether it's
reproducible yet -- the machine crashed while I was not actively using
it. We've had this type of card for a few years and I've never seen this
before so it may be a regression, but admittedly we don't use them all
that often.
[...]
quoted
Try reverting this commit, it seems problematic
commit d0249e44432aa0ffcf710b64449b8eaa3722547e
Author: stephen hemminger [off-list ref]
Date: Thu Jan 19 14:37:18 2012 +0000
skge: check for PCI dma mapping errors
Thanks for the pointer, I'll try that. Unfortunately some other stuff
has come up so I probably won't be able to test it until next week.
Just to confirm: I can reliably reproduce the crash and reverting that
commit fixes it.
For reference, I captured the full trace over serial console:
skge 0000:03:01.0: eth1: enabling interface
ADDRCONF(NETDEV_UP): eth1: link is not ready
skge 0000:03:01.0: eth1: Link is up at 1000 Mbps, full duplex, flow control none
ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
device eth1 entered promiscuous mode
BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffffa001826e>] skge_poll+0x367/0x5cd [skge]
PGD 0
Oops: 0000 [#1] PREEMPT SMP
CPU 0
Modules linked in: nfs lockd auth_rpcgss nfs_acl sunrpc autofs4 acpi_cpufreq mperf deflate zlib_deflate ctr aes_x86_64 aes_generic des_generic cbc sha512_generic sha256_generic sha1_ssse3 sha1_generic md5 hmac crypto_null af_key ipv6 loop snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_pcm snd_seq snd_timer snd_seq_device snd soundcore skge snd_page_alloc sky2 evdev i2c_i801
Pid: 10, comm: kworker/0:1 Not tainted 3.3.0-rc2+ #10 LENOVO 0841A5U/LENOVO
RIP: 0010:[<ffffffffa001826e>] [<ffffffffa001826e>] skge_poll+0x367/0x5cd [skge]
RSP: 0018:ffff88007f403e00 EFLAGS: 00010246
RAX: ffff880079e3bc40 RBX: ffff88007baf3600 RCX: 0000000000000046
RDX: ffff88007bddaf00 RSI: 0000000000000000 RDI: ffff880079e3bc40
RBP: ffff88007f403e70 R08: 0000000000000300 R09: ffffffff812d7e11
R10: ffff880079eb7200 R11: ffff88007baf3600 R12: ffff88007baf3000
R13: ffff88007ae98208 R14: ffff880079eb7200 R15: 0000000000000046
FS: 0000000000000000(0000) GS:ffff88007f400000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 000000007ae4d000 CR4: 00000000000406f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/0:1 (pid: 10, threadinfo ffff88007bca8000, task ffff88007c87c260)
Stack:
ffff88007c870600 ffff88007f403e18 ffff88007c870600 897488007f403e58
0000004600000040 ffff88007baf3600 0046000000000001 ffff88007baf3610
0000000000000000 ffff88007baf3610 ffff88007f411380 0000000000000000
Call Trace:
<IRQ>
[<ffffffff812e432c>] net_rx_action+0xaa/0x1c0
[<ffffffff8102fe91>] __do_softirq+0x7e/0x125
[<ffffffff8135ecb7>] ? _raw_spin_unlock+0x26/0x31
[<ffffffff8136092c>] call_softirq+0x1c/0x30
[<ffffffff8100411b>] do_softirq+0x33/0x68
[<ffffffff8102fc7f>] irq_exit+0x3f/0xb9
[<ffffffff81003a20>] do_IRQ+0x97/0xae
[<ffffffff8135f02b>] common_interrupt+0x6b/0x6b
<EOI>
[<ffffffff8135ed02>] ? _raw_spin_unlock_irq+0xd/0x32
[<ffffffff8103ea56>] worker_thread+0x24b/0x255
[<ffffffff8103e80b>] ? manage_workers+0x190/0x190
[<ffffffff81041f31>] kthread+0x84/0x8c
[<ffffffff81360834>] kernel_thread_helper+0x4/0x10
[<ffffffff81041ead>] ? kthread_freezable_should_stop+0x6b/0x6b
[<ffffffff81360830>] ? gs_change+0xb/0xb
Code: 48 8b 40 30 48 85 c0 74 0a b9 02 00 00 00 4c 89 fa ff d0 49 8b 86 d0 00 00 00 49 8b 55 10 8b 4d b4 48 89 c7 48 8b b2 d0 00 00 00 <f3> a4 31 ff 48 8b 03 49 8b 75 18 48 8b 40 08 48 85 c0 74 13 48
RIP [<ffffffffa001826e>] skge_poll+0x367/0x5cd [skge]
RSP <ffff88007f403e00>
CR2: 0000000000000000
---[ end trace 13c07164f6f205a2 ]---
Kernel panic - not syncing: Fatal exception in interrupt
panic occurred, switching back to text console
Cheers,
--
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)