From: Peter Zijlstra <hidden> Date: 2011-05-26 10:12:55
On Thu, 2011-05-26 at 11:21 +0200, Laurent Riffard wrote:
Hi,
Recently, netconsole was broken by some scheduler updates. Kernel hangs
on boot near the network card initialization. I noticed that it does
hang just where a "inconsistent lock state" message normally appears.
I did a bisection : e4a52bcb9a18142d79e231b6733cabdbf2e67c1f is the
first bad commit.
commit e4a52bcb9a18142d79e231b6733cabdbf2e67c1f
Author: Peter Zijlstra [off-list ref]
Date: Tue Apr 5 17:23:54 2011 +0200
sched: Remove rq->lock from the first half of ttwu()
Before this commit, kernel was booting succesfully despite the
"inconsistent lock state" message. After this commit, the kernel does
hang on boot, I have to push the reset button.
Do you have CONFIG_HARDLOCKUP_DETECTOR=y, and do you get an NMI splat
after some 10 seconds? It would be interesting to see where its stuck.
Also, are you very _very_ sure that lockdep message is a false positive?
On Thu, 2011-05-26 at 11:21 +0200, Laurent Riffard wrote:
quoted
Hi,
Recently, netconsole was broken by some scheduler updates. Kernel hangs
on boot near the network card initialization. I noticed that it does
hang just where a "inconsistent lock state" message normally appears.
I did a bisection : e4a52bcb9a18142d79e231b6733cabdbf2e67c1f is the
first bad commit.
commit e4a52bcb9a18142d79e231b6733cabdbf2e67c1f
Author: Peter Zijlstra [off-list ref]
Date: Tue Apr 5 17:23:54 2011 +0200
sched: Remove rq->lock from the first half of ttwu()
Before this commit, kernel was booting succesfully despite the
"inconsistent lock state" message. After this commit, the kernel does
hang on boot, I have to push the reset button.
Do you have CONFIG_HARDLOCKUP_DETECTOR=y, and do you get an NMI splat
after some 10 seconds? It would be interesting to see where its stuck.
CONFIG_HARDLOCKUP_DETECTOR is not set for now. I'll give it a try.
Also, are you very _very_ sure that lockdep message is a false positive?
I can't answer this question. I just know that before commit e4a52bcb9,
the kernel was able to boot with netconsole despite this message.
Maybe some netconsole developper knows.
~~
laurent
On Thu, 2011-05-26 at 11:21 +0200, Laurent Riffard wrote:
quoted
Hi,
Recently, netconsole was broken by some scheduler updates. Kernel hangs
on boot near the network card initialization. I noticed that it does
hang just where a "inconsistent lock state" message normally appears.
I did a bisection : e4a52bcb9a18142d79e231b6733cabdbf2e67c1f is the
first bad commit.
commit e4a52bcb9a18142d79e231b6733cabdbf2e67c1f
Author: Peter Zijlstra [off-list ref]
Date: Tue Apr 5 17:23:54 2011 +0200
sched: Remove rq->lock from the first half of ttwu()
Before this commit, kernel was booting succesfully despite the
"inconsistent lock state" message. After this commit, the kernel does
hang on boot, I have to push the reset button.
Do you have CONFIG_HARDLOCKUP_DETECTOR=y, and do you get an NMI splat
after some 10 seconds? It would be interesting to see where its stuck.
CONFIG_HARDLOCKUP_DETECTOR is not set for now. I'll give it a try.
I activated CONFIG_HARDLOCKUP_DETECTOR, but failed to obtain more
information. During the boot, the kernel hangs just after bringing up
the NIC. I waited 2 minutes, without getting any more messages.
~~
laurent
From: Peter Zijlstra <hidden> Date: 2011-05-26 16:14:33
On Thu, 2011-05-26 at 18:12 +0200, Laurent Riffard wrote:
Le 26/05/2011 12:35, Laurent Riffard a écrit :
quoted
On Thu, 26 May 2011, Peter Zijlstra wrote:
quoted
On Thu, 2011-05-26 at 11:21 +0200, Laurent Riffard wrote:
quoted
Hi,
Recently, netconsole was broken by some scheduler updates. Kernel hangs
on boot near the network card initialization. I noticed that it does
hang just where a "inconsistent lock state" message normally appears.
I did a bisection : e4a52bcb9a18142d79e231b6733cabdbf2e67c1f is the
first bad commit.
commit e4a52bcb9a18142d79e231b6733cabdbf2e67c1f
Author: Peter Zijlstra [off-list ref]
Date: Tue Apr 5 17:23:54 2011 +0200
sched: Remove rq->lock from the first half of ttwu()
Before this commit, kernel was booting succesfully despite the
"inconsistent lock state" message. After this commit, the kernel does
hang on boot, I have to push the reset button.
Do you have CONFIG_HARDLOCKUP_DETECTOR=y, and do you get an NMI splat
after some 10 seconds? It would be interesting to see where its stuck.
CONFIG_HARDLOCKUP_DETECTOR is not set for now. I'll give it a try.
I activated CONFIG_HARDLOCKUP_DETECTOR, but failed to obtain more
information. During the boot, the kernel hangs just after bringing up
the NIC. I waited 2 minutes, without getting any more messages.
OK, thanks for trying, I shall try and reproduce locally.
From: David Miller <davem@davemloft.net> Date: 2011-05-26 20:18:34
From: Peter Zijlstra <redacted>
Date: Thu, 26 May 2011 11:48:28 +0200
Also, are you very _very_ sure that lockdep message is a false positive?
I suspect these warnings are entirely specific to using netconsole
with the forcedeth driver.
There is some really scary stuff in there, to say the least.
Just have a look at nv_do_nic_poll(). That's the first time I've
seen a networking driver use the {disable,enable}_irq_lockdep()
interfaces.
It's also using netif_tx_unlock_bh() in the "recovery" case which
in netpoll will cause a local_bh_enable() with cpu interrupts
disabled. Which is not allowed.
Much of the forcedeth driver's locking seems suspect to me.
On Thu, 2011-05-26 at 18:12 +0200, Laurent Riffard wrote:
quoted
Le 26/05/2011 12:35, Laurent Riffard a écrit :
quoted
On Thu, 26 May 2011, Peter Zijlstra wrote:
quoted
On Thu, 2011-05-26 at 11:21 +0200, Laurent Riffard wrote:
quoted
Hi,
Recently, netconsole was broken by some scheduler updates. Kernel hangs
on boot near the network card initialization. I noticed that it does
hang just where a "inconsistent lock state" message normally appears.
I did a bisection : e4a52bcb9a18142d79e231b6733cabdbf2e67c1f is the
first bad commit.
commit e4a52bcb9a18142d79e231b6733cabdbf2e67c1f
Author: Peter Zijlstra [off-list ref]
Date: Tue Apr 5 17:23:54 2011 +0200
sched: Remove rq->lock from the first half of ttwu()
Before this commit, kernel was booting succesfully despite the
"inconsistent lock state" message. After this commit, the kernel does
hang on boot, I have to push the reset button.
Do you have CONFIG_HARDLOCKUP_DETECTOR=y, and do you get an NMI splat
after some 10 seconds? It would be interesting to see where its stuck.
CONFIG_HARDLOCKUP_DETECTOR is not set for now. I'll give it a try.
I activated CONFIG_HARDLOCKUP_DETECTOR, but failed to obtain more
information. During the boot, the kernel hangs just after bringing up
the NIC. I waited 2 minutes, without getting any more messages.
OK, thanks for trying, I shall try and reproduce locally.