Next March 25: Boot failure on powerpc [recursive locking detected]

Today's next failed to boot on a powerpc box
(Power6 blade IBM,7998-61X) with following recursive locking message.

=============================================
[ INFO: possible recursive locking detected ]
2.6.29-next-20090325 #1
---------------------------------------------
khelper/1202 is trying to acquire lock:
 (&q->__queue_lock){..-...}, at: [<c0000000002f44d0>] .blk_end_io+0x88/0xd8

but task is already holding lock:
 (&q->__queue_lock){..-...}, at: [<c000000000407874>] .__scsi_queue_insert+0xc4/
0x128

other info that might help us debug this:
1 lock held by khelper/1202:
 #0:  (&q->__queue_lock){..-...}, at: [<c000000000407874>] .__scsi_queue_insert+
0xc4/0x128

stack backtrace:
Call Trace:
[c00000000ffff940] [c0000000000115f4] .show_stack+0x70/0x184 (unreliable)
[c00000000ffff9f0] [c0000000000964c0] .validate_chain+0x6a8/0xe64
[c00000000ffffab0] [c000000000097494] .__lock_acquire+0x818/0x8e0
[c00000000ffffba0] [c000000000097664] .lock_acquire+0x108/0x154
[c00000000ffffc60] [c00000000058d2e4] ._spin_lock_irqsave+0x54/0x84
[c00000000ffffd00] [c0000000002f44d0] .blk_end_io+0x88/0xd8
[c00000000ffffda0] [c000000000407888] .__scsi_queue_insert+0xd8/0x128
[c00000000ffffe50] [c0000000002fa248] .blk_done_softirq+0xb0/0xe0
[c00000000ffffee0] [c00000000006f824] .__do_softirq+0x120/0x298
[c00000000fffff90] [c00000000002ba94] .call_do_softirq+0x14/0x24
[c000000044453330] [c00000000000d674] .do_softirq+0x94/0x114
[c0000000444533d0] [c00000000006fab0] .irq_exit+0x70/0x88
[c000000044453450] [c00000000000db2c] .do_IRQ+0x1c8/0x210
[c000000044453500] [c000000000004814] hardware_interrupt_entry+0x1c/0x20

--- Exception: 501 at .raw_local_irq_restore+0x3c/0x40
    LR = .kmem_cache_alloc+0xf4/0x14c

[c0000000444537f0] [c000000000112038] .kmem_cache_alloc+0xe8/0x14c (unreliable)
[c0000000444538a0] [c00000000001154c] .alloc_thread_info+0x28/0x60
[c000000044453920] [c0000000000663b0] .copy_process+0xe4/0x1168
[c000000044453a10] [c000000000067804] .do_fork+0x194/0x438
[c000000044453b30] [c000000000011a18] .sys_clone+0x5c/0x74
[c000000044453ba0] [c000000000008788] .ppc_clone+0x8/0xc

--- Exception: c00 at .kernel_thread+0x28/0x70
    LR = .wait_for_helper+0x38/0xb0

[c000000044453e90] [0000000000000078] 0x78 (unreliable)
[c000000044453f00] [c00000000007f200] .wait_for_helper+0x24/0xb0
[c000000044453f90] [c00000000002bd9c] .kernel_thread+0x54/0x70
BUG: spinlock lockup on CPU#0, khelper/1202, c0000000449b0368
Call Trace:
[c00000000ffffb10] [c0000000000115f4] .show_stack+0x70/0x184 (unreliable)
[c00000000ffffbc0] [c000000000316710] ._raw_spin_lock+0x140/0x17c
[c00000000ffffc60] [c00000000058d2f0] ._spin_lock_irqsave+0x60/0x84
[c00000000ffffd00] [c0000000002f44d0] .blk_end_io+0x88/0xd8
[c00000000ffffda0] [c000000000407888] .__scsi_queue_insert+0xd8/0x128
[c00000000ffffe50] [c0000000002fa248] .blk_done_softirq+0xb0/0xe0
[c00000000ffffee0] [c00000000006f824] .__do_softirq+0x120/0x298
[c00000000fffff90] [c00000000002ba94] .call_do_softirq+0x14/0x24
[c000000044453330] [c00000000000d674] .do_softirq+0x94/0x114
[c0000000444533d0] [c00000000006fab0] .irq_exit+0x70/0x88
[c000000044453450] [c00000000000db2c] .do_IRQ+0x1c8/0x210
[c000000044453500] [c000000000004814] hardware_interrupt_entry+0x1c/0x20

--- Exception: 501 at .raw_local_irq_restore+0x3c/0x40
    LR = .kmem_cache_alloc+0xf4/0x14c

[c0000000444537f0] [c000000000112038] .kmem_cache_alloc+0xe8/0x14c (unreliable)
[c0000000444538a0] [c00000000001154c] .alloc_thread_info+0x28/0x60
[c000000044453920] [c0000000000663b0] .copy_process+0xe4/0x1168
[c000000044453a10] [c000000000067804] .do_fork+0x194/0x438
[c000000044453b30] [c000000000011a18] .sys_clone+0x5c/0x74
[c000000044453ba0] [c000000000008788] .ppc_clone+0x8/0xc

--- Exception: c00 at .kernel_thread+0x28/0x70
    LR = .wait_for_helper+0x38/0xb0

[c000000044453e90] [0000000000000078] 0x78 (unreliable)
[c000000044453f00] [c00000000007f200] .wait_for_helper+0x24/0xb0
[c000000044453f90] [c00000000002bd9c] .kernel_thread+0x54/0x70

I could boot Next 24 on the same machine.

Attached here is the .config and complete dmesg.

Thanks
-Sachin


-- 

---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------

Sachin Sant wrote:

Today's next failed to boot on a powerpc box
(Power6 blade IBM,7998-61X) with following recursive locking message.

=============================================
[ INFO: possible recursive locking detected ]
2.6.29-next-20090325 #1

After bisecting the failure seems to be because of the following
patch from James ( block: move SCSI timeout check into block )

http://patchwork.kernel.org/patch/8017/

If i back out the above mentioned patch, the machine boots fine
without any problems.

Thanks
-Sachin


-- 

---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------

On Thu, 2009-03-26 at 12:04 +0530, Sachin Sant wrote:

Sachin Sant wrote:

quoted

Today's next failed to boot on a powerpc box
(Power6 blade IBM,7998-61X) with following recursive locking message.

=============================================
[ INFO: possible recursive locking detected ]
2.6.29-next-20090325 #1

After bisecting the failure seems to be because of the following
patch from James ( block: move SCSI timeout check into block )

http://patchwork.kernel.org/patch/8017/

If i back out the above mentioned patch, the machine boots fine
without any problems.

Yes, that patch already got dropped for other reasons:

http://marc.info/?t=123740773700002

I'm going to see if I can redo it in a better way, since moving this
type of timeout checking from scsi to block is a useful generalisation.

James

Hi all,

On Thu, 26 Mar 2009 08:50:03 -0500 James Bottomley [off-list ref] wrote:

On Thu, 2009-03-26 at 12:04 +0530, Sachin Sant wrote:

quoted

Sachin Sant wrote:

quoted

Today's next failed to boot on a powerpc box
(Power6 blade IBM,7998-61X) with following recursive locking message.

=============================================
[ INFO: possible recursive locking detected ]
2.6.29-next-20090325 #1

After bisecting the failure seems to be because of the following
patch from James ( block: move SCSI timeout check into block )

http://patchwork.kernel.org/patch/8017/

If i back out the above mentioned patch, the machine boots fine
without any problems.

Yes, that patch already got dropped for other reasons:

http://marc.info/?t=123740773700002

I'm going to see if I can redo it in a better way, since moving this
type of timeout checking from scsi to block is a useful generalisation.

I will revert it from next-20090327 as well as it is still in the
for-next branch of the block tree.

-- 
Cheers,
Stephen Rothwell                    sfr@canb.auug.org.au
http://www.canb.auug.org.au/~sfr/

On Fri, Mar 27 2009, Stephen Rothwell wrote:

Hi all,

On Thu, 26 Mar 2009 08:50:03 -0500 James Bottomley [off-list ref] wrote:

quoted

On Thu, 2009-03-26 at 12:04 +0530, Sachin Sant wrote:

quoted

Sachin Sant wrote:

quoted

Today's next failed to boot on a powerpc box
(Power6 blade IBM,7998-61X) with following recursive locking message.

=============================================
[ INFO: possible recursive locking detected ]
2.6.29-next-20090325 #1

After bisecting the failure seems to be because of the following
patch from James ( block: move SCSI timeout check into block )

http://patchwork.kernel.org/patch/8017/

If i back out the above mentioned patch, the machine boots fine
without any problems.

Yes, that patch already got dropped for other reasons:

http://marc.info/?t=123740773700002

I'm going to see if I can redo it in a better way, since moving this
type of timeout checking from scsi to block is a useful generalisation.

I will revert it from next-20090327 as well as it is still in the
for-next branch of the block tree.

I'll update for-next, sorry about that. I had dropped it from
for-2.6.30, but forgot to update akpm/next branches.

-- 
Jens Axboe

Hi Jens,

On Fri, 27 Mar 2009 07:42:08 +0100 Jens Axboe [off-list ref] wrote:

I'll update for-next, sorry about that. I had dropped it from
for-2.6.30, but forgot to update akpm/next branches.

Thanks.  Its OK for today.

-- 
Cheers,
Stephen Rothwell                    sfr@canb.auug.org.au
http://www.canb.auug.org.au/~sfr/