[rhashtable] [ INFO: possible recursive locking detected ]
From: Fengguang Wu <hidden>
Date: 2015-03-12 03:29:46
Greetings,
0day kernel testing robot got the below dmesg and the first bad commit is
git://git.kernel.org/pub/scm/linux/kernel/git/jkirsher/net-next.git master
commit aa34a6cb0478842452bac58edb50d3ef9e178c92
Author: Herbert Xu [off-list ref]
AuthorDate: Wed Mar 11 09:43:48 2015 +1100
Commit: David S. Miller [off-list ref]
CommitDate: Wed Mar 11 16:36:21 2015 -0400
rhashtable: Add arbitrary rehash function
This patch adds a rehash function that supports the use of any
hash function for the new table. This is needed to support changing
the random seed value during the lifetime of the hash table.
However for now the random seed value is still constant and the
rehash function is simply used to replace the existing expand/shrink
functions.
[ ASSERT_BUCKET_LOCK() and thus debug_dump_table() +
debug_dump_buckets() are not longer used, so delete them
entirely. -DaveM ]
Signed-off-by: Herbert Xu [off-list ref]
Signed-off-by: David S. Miller [off-list ref]
+------------------------------------------+------------+------------+------------+
| | 988dfbd795 | aa34a6cb04 | 61f0d861fc |
+------------------------------------------+------------+------------+------------+
| boot_successes | 60 | 0 | 0 |
| boot_failures | 0 | 20 | 14 |
| INFO:possible_recursive_locking_detected | 0 | 20 | 14 |
| backtrace:test_rht_init | 0 | 20 | 14 |
| backtrace:rhashtable_expand | 0 | 20 | 14 |
| backtrace:kernel_init_freeable | 0 | 20 | 14 |
+------------------------------------------+------------+------------+------------+
[ 2.307176]
[ 2.307922] =============================================
[ 2.307922] =============================================
[ 2.310446] [ INFO: possible recursive locking detected ]
[ 2.310446] [ INFO: possible recursive locking detected ]
[ 2.311089] 4.0.0-rc3-00505-gaa34a6c #7 Not tainted
[ 2.311089] 4.0.0-rc3-00505-gaa34a6c #7 Not tainted
[ 2.311089] ---------------------------------------------
[ 2.311089] ---------------------------------------------
[ 2.311089] swapper/1 is trying to acquire lock:
[ 2.311089] swapper/1 is trying to acquire lock:
[ 2.311089] (
[ 2.311089] (&(&tbl->locks[i])->rlock&(&tbl->locks[i])->rlock){+.....}){+.....}, at: , at: [<ffffffff813ef48d>] rhashtable_rehash+0xfd/0x1a0
[<ffffffff813ef48d>] rhashtable_rehash+0xfd/0x1a0
[ 2.311089]
[ 2.311089] but task is already holding lock:
[ 2.311089]
[ 2.311089] but task is already holding lock:
[ 2.311089] (
[ 2.311089] (&(&tbl->locks[i])->rlock&(&tbl->locks[i])->rlock){+.....}){+.....}, at: , at: [<ffffffff813ef401>] rhashtable_rehash+0x71/0x1a0
[<ffffffff813ef401>] rhashtable_rehash+0x71/0x1a0
[ 2.311089]
[ 2.311089] other info that might help us debug this:
[ 2.311089]
[ 2.311089] other info that might help us debug this:
[ 2.311089] Possible unsafe locking scenario:
[ 2.311089]
[ 2.311089] Possible unsafe locking scenario:
[ 2.311089]
[ 2.311089] CPU0
[ 2.311089] CPU0
[ 2.311089] ----
[ 2.311089] ----
[ 2.311089] lock(
[ 2.311089] lock(&(&tbl->locks[i])->rlock&(&tbl->locks[i])->rlock);
);
[ 2.311089] lock(
[ 2.311089] lock(&(&tbl->locks[i])->rlock&(&tbl->locks[i])->rlock);
);
[ 2.311089]
[ 2.311089] *** DEADLOCK ***
[ 2.311089]
[ 2.311089]
[ 2.311089] *** DEADLOCK ***
[ 2.311089]
[ 2.311089] May be due to missing lock nesting notation
[ 2.311089]
[ 2.311089] May be due to missing lock nesting notation
[ 2.311089]
[ 2.311089] 2 locks held by swapper/1:
[ 2.311089] 2 locks held by swapper/1:
[ 2.311089] #0:
[ 2.311089] #0: ( (&ht->mutex&ht->mutex){+.+.+.}){+.+.+.}, at: , at: [<ffffffff81eb1d2e>] test_rht_init+0x15c/0x37d
[<ffffffff81eb1d2e>] test_rht_init+0x15c/0x37d
[ 2.311089] #1:
[ 2.311089] #1: ( (&(&tbl->locks[i])->rlock&(&tbl->locks[i])->rlock){+.....}){+.....}, at: , at: [<ffffffff813ef401>] rhashtable_rehash+0x71/0x1a0
[<ffffffff813ef401>] rhashtable_rehash+0x71/0x1a0
[ 2.311089]
[ 2.311089] stack backtrace:
[ 2.311089]
[ 2.311089] stack backtrace:
[ 2.311089] CPU: 0 PID: 1 Comm: swapper Not tainted 4.0.0-rc3-00505-gaa34a6c #7
[ 2.311089] CPU: 0 PID: 1 Comm: swapper Not tainted 4.0.0-rc3-00505-gaa34a6c #7
[ 2.311089] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014
[ 2.311089] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.7.5-20140531_083030-gandalf 04/01/2014
[ 2.311089] ffffffff826cedb0
[ 2.311089] ffffffff826cedb0 ffff88001246fc18 ffff88001246fc18 ffffffff817c123e ffffffff817c123e ffff88001246fcf8 ffff88001246fcf8
[ 2.311089] ffffffff81098da8
[ 2.311089] ffffffff81098da8 0000000000000292 0000000000000292 ffffffff81e1d7b8 ffffffff81e1d7b8 ffff88001246fc48 ffff88001246fc48
[ 2.311089] ffffffff00000000
[ 2.311089] ffffffff00000000 000000001246fc68 000000001246fc68 000000040c102081 000000040c102081 0000000000000002 0000000000000002
[ 2.311089] Call Trace:
[ 2.311089] Call Trace:
[ 2.311089] [<ffffffff817c123e>] dump_stack+0x19/0x1b
[ 2.311089] [<ffffffff817c123e>] dump_stack+0x19/0x1b
[ 2.311089] [<ffffffff81098da8>] __lock_acquire+0x17a8/0x2180
[ 2.311089] [<ffffffff81098da8>] __lock_acquire+0x17a8/0x2180
[ 2.311089] [<ffffffff81099fda>] lock_acquire+0x6a/0x90
[ 2.311089] [<ffffffff81099fda>] lock_acquire+0x6a/0x90
[ 2.311089] [<ffffffff813ef48d>] ? rhashtable_rehash+0xfd/0x1a0
[ 2.311089] [<ffffffff813ef48d>] ? rhashtable_rehash+0xfd/0x1a0
[ 2.311089] [<ffffffff81eb1cba>] ? test_rht_init+0xe8/0x37d
[ 2.311089] [<ffffffff81eb1cba>] ? test_rht_init+0xe8/0x37d
[ 2.311089] [<ffffffff817c7377>] _raw_spin_lock+0x37/0x70
[ 2.311089] [<ffffffff817c7377>] _raw_spin_lock+0x37/0x70
[ 2.311089] [<ffffffff813ef48d>] ? rhashtable_rehash+0xfd/0x1a0
[ 2.311089] [<ffffffff813ef48d>] ? rhashtable_rehash+0xfd/0x1a0
[ 2.311089] [<ffffffff813ef48d>] rhashtable_rehash+0xfd/0x1a0
[ 2.311089] [<ffffffff813ef48d>] rhashtable_rehash+0xfd/0x1a0
[ 2.311089] [<ffffffff81eb1cba>] ? test_rht_init+0xe8/0x37d
[ 2.311089] [<ffffffff81eb1cba>] ? test_rht_init+0xe8/0x37d
[ 2.311089] [<ffffffff81eb1bd2>] ? test_rht_lookup+0xca/0xca
[ 2.311089] [<ffffffff81eb1bd2>] ? test_rht_lookup+0xca/0xca
[ 2.311089] [<ffffffff813eff12>] rhashtable_expand+0x42/0x70
[ 2.311089] [<ffffffff813eff12>] rhashtable_expand+0x42/0x70
[ 2.311089] [<ffffffff81eb1d3a>] test_rht_init+0x168/0x37d
[ 2.311089] [<ffffffff81eb1d3a>] test_rht_init+0x168/0x37d
[ 2.311089] [<ffffffff81124838>] ? kfree+0xa8/0x110
[ 2.311089] [<ffffffff81124838>] ? kfree+0xa8/0x110
[ 2.311089] [<ffffffff813f19c0>] ? test_fw_misc_read+0x60/0x60
[ 2.311089] [<ffffffff813f19c0>] ? test_fw_misc_read+0x60/0x60
[ 2.311089] [<ffffffff810002a4>] do_one_initcall+0x84/0x1c0
[ 2.311089] [<ffffffff810002a4>] do_one_initcall+0x84/0x1c0
[ 2.311089] [<ffffffff81e930c4>] kernel_init_freeable+0x1eb/0x273
[ 2.311089] [<ffffffff81e930c4>] kernel_init_freeable+0x1eb/0x273
[ 2.311089] [<ffffffff817bcef0>] ? rest_init+0xc0/0xc0
[ 2.311089] [<ffffffff817bcef0>] ? rest_init+0xc0/0xc0
[ 2.311089] [<ffffffff817bcef9>] kernel_init+0x9/0xf0
[ 2.311089] [<ffffffff817bcef9>] kernel_init+0x9/0xf0
[ 2.311089] [<ffffffff817c8297>] ret_from_fork+0x57/0x90
[ 2.311089] [<ffffffff817c8297>] ret_from_fork+0x57/0x90
[ 2.311089] [<ffffffff817bcef0>] ? rest_init+0xc0/0xc0
[ 2.311089] [<ffffffff817bcef0>] ? rest_init+0xc0/0xc0
[ 2.566279] Verifying lookups...
[ 2.566279] Verifying lookups...
[ 2.570044] Table expansion iteration 1...
git bisect start 61f0d861fc6924fa673ecf1128a911d49cb10dc8 2c90331cf5ed1d648a711b9483e173aaaf2c4a9b --
git bisect good 843c2fdf7a12951815d981fa431d7e04d8259332 # 07:17 20+ 0 net: dctcp: loosen requirement to assert ECT(0) during 3WHS
git bisect good 723b8e460d87e957f251dc5764f4ab86af6ab44e # 07:45 20+ 0 udp: In udp_flow_src_port use random hash value if skb_get_hash fails
git bisect good 496127290f298d839918a14728b653b43ef02708 # 07:53 20+ 0 inet_diag: remove duplicate code from inet_twsk_diag_dump()
git bisect good cbe21d92e4d501e4895ef668b43fd8998c9b3b02 # 08:10 20+ 0 net: stmmac: make reset control an optional requirement
git bisect good bf0b211256be342e92d3ee5c5a1fb8d42f3928bb # 08:33 20+ 0 Merge branch 'dsa-next'
git bisect good ddb4b9a1328ea89733133e86cf1972d23891abfc # 08:44 20+ 0 fib_trie: Address possible NULL pointer dereference in resize
git bisect bad aa34a6cb0478842452bac58edb50d3ef9e178c92 # 09:02 0- 1 rhashtable: Add arbitrary rehash function
git bisect good 0ddcf43d5d4a03ded1ee3f6b3b72a0cbed4e90b1 # 09:09 20+ 0 ipv4: FIB Local/MAIN table collapse
git bisect good 988dfbd795cf08b00576c1ced4210281b2bccffc # 09:23 20+ 0 rhashtable: Move hash_rnd into bucket_table
# first bad commit: [aa34a6cb0478842452bac58edb50d3ef9e178c92] rhashtable: Add arbitrary rehash function
git bisect good 988dfbd795cf08b00576c1ced4210281b2bccffc # 09:26 60+ 0 rhashtable: Move hash_rnd into bucket_table
# extra tests with DEBUG_INFO
git bisect bad aa34a6cb0478842452bac58edb50d3ef9e178c92 # 09:35 0- 2 rhashtable: Add arbitrary rehash function
# extra tests on HEAD of net-next/master
git bisect bad 61f0d861fc6924fa673ecf1128a911d49cb10dc8 # 09:35 0- 14 fib_trie: Fix uninitialized variable warning
# extra tests with first bad commit reverted
# extra tests on tree/branch linus/master
git bisect good cca28a5fda5d8fa69982bdb54341eeeb3eab215a # 09:54 60+ 1 Merge tag 'for-linus-20150310' of git://git.infradead.org/linux-mtd
# extra tests on tree/branch next/master
git bisect good 23bd64da26b62ee37a637be015149b703e6c3d04 # 10:08 60+ 0 Add linux-next specific files for 20150311
This script may reproduce the error.
----------------------------------------------------------------------------
#!/bin/bash
kernel=$1
initrd=quantal-core-x86_64.cgz
wget --no-clobber https://github.com/fengguang/reproduce-kernel-bug/raw/master/initrd/$initrd
kvm=(
qemu-system-x86_64
-cpu kvm64
-enable-kvm
-kernel $kernel
-initrd $initrd
-m 320
-smp 2
-net nic,vlan=1,model=e1000
-net user,vlan=1
-boot order=nc
-no-reboot
-watchdog i6300esb
-rtc base=localtime
-serial stdio
-display none
-monitor null
)
append=(
hung_task_panic=1
earlyprintk=ttyS0,115200
rd.udev.log-priority=err
systemd.log_target=journal
systemd.log_level=warning
debug
apic=debug
sysrq_always_enabled
rcupdate.rcu_cpu_stall_timeout=100
panic=-1
softlockup_panic=1
nmi_watchdog=panic
oops=panic
load_ramdisk=2
prompt_ramdisk=0
console=ttyS0,115200
console=tty0
vga=normal
root=/dev/ram0
rw
drbd.minor_count=8
)
"${kvm[@]}" --append "${append[*]}"
----------------------------------------------------------------------------
Thanks,
Fengguang