From: Arjan van de Ven <hidden> Date: 2006-07-07 18:48:41
On Fri, 2006-07-07 at 12:13 -0400, Dave Jones wrote:
With 2.6.18rc1 + a selection of the lockdep tweaks found so far,
midnight commander makes the kernel unhappy.
Dave
=======================================================
[ INFO: possible circular locking dependency detected ]
-------------------------------------------------------
mc/4913 is trying to acquire lock:
(sk_lock-AF_INET){--..}, at: [<ffffffff8022800c>] tcp_sendmsg+0x1f/0xb1a
but task is already holding lock:
(&inode->i_mutex){--..}, at: [<ffffffff802692e0>] mutex_lock+0x2a/0x2e
which lock already depends on the new lock.
ok this looks like a fun three-way variant of the traditional AB-BA
deadlock... an ABCA deadlock :)
Lock A: rtnl_mutex
Lock B: i_mutex
Lock C: sk_lock for AF_INET
i_mutex is taken within rtln_mutex like this:
[<ffffffff8030f4a0>] create_dir+0x2c/0x1e2
[<ffffffff8030fa5b>] sysfs_create_dir+0x59/0x78
[<ffffffff8034d2e2>] kobject_add+0x114/0x1d8
[<ffffffff803bb1e7>] class_device_add+0xb5/0x49d
[<ffffffff804300b1>] netdev_register_sysfs+0x98/0xa2
[<ffffffff80426f58>] register_netdevice+0x28c/0x376
[<ffffffff8042709c>] register_netdev+0x5a/0x69
creating the AB dependency
now for the second part:
if you use the IP_DROP_MEMBERSHIP socket option, you end up going into
the function do_ip_setsockopt(), which first does a lock_sock(), and
then a call to ip_mc_leave_group(), which calls rtnl_lock()
So we have the CA dependency right here.
now for the third part, which involves the nfs client:
stat on an nfs file, which ends up taken the i_mutex of a directory in
the path (obvious), and then does
[<ffffffff8022800b>] tcp_sendmsg+0x1e/0xb1a
[<ffffffff80248f4b>] inet_sendmsg+0x45/0x53
[<ffffffff80259d25>] sock_sendmsg+0x110/0x130
[<ffffffff8041f462>] kernel_sendmsg+0x3c/0x52
[<ffffffff885399e9>] xs_tcp_send_request+0x117/0x320 [sunrpc]
[<ffffffff885388d5>] xprt_transmit+0x105/0x21e [sunrpc]
[<ffffffff8853771e>] call_transmit+0x1f4/0x239 [sunrpc]
[<ffffffff8853c06e>] __rpc_execute+0x9b/0x1e6 [sunrpc]
[<ffffffff8853c1de>] rpc_execute+0x1a/0x1d [sunrpc]
[<ffffffff885364ad>] rpc_call_sync+0x87/0xb9 [sunrpc]
[<ffffffff885a2587>] nfs3_rpc_wrapper+0x2e/0x74 [nfs]
[<ffffffff885a2a14>] nfs3_proc_lookup+0xe0/0x163 [nfs]
where tcp_sendmsg calls lock_sock. So this is the BC dependency.
In summary, we have an AB dependency, a BC dependency and a CA
dependency. This is reason for lockdep to conclude that there is a
circular dependency happening, which, looking at my analysis, seems to
be at least reasonably right...
From: Herbert Xu <herbert@gondor.apana.org.au> Date: 2006-07-08 03:00:55
Arjan van de Ven [off-list ref] wrote:
i_mutex is taken within rtln_mutex like this:
[<ffffffff8030f4a0>] create_dir+0x2c/0x1e2
[<ffffffff8030fa5b>] sysfs_create_dir+0x59/0x78
[<ffffffff8034d2e2>] kobject_add+0x114/0x1d8
[<ffffffff803bb1e7>] class_device_add+0xb5/0x49d
[<ffffffff804300b1>] netdev_register_sysfs+0x98/0xa2
[<ffffffff80426f58>] register_netdevice+0x28c/0x376
[<ffffffff8042709c>] register_netdev+0x5a/0x69
creating the AB dependency
This is a sysfs inode.
now for the third part, which involves the nfs client:
stat on an nfs file, which ends up taken the i_mutex of a directory in
the path (obvious), and then does
[<ffffffff8022800b>] tcp_sendmsg+0x1e/0xb1a
[<ffffffff80248f4b>] inet_sendmsg+0x45/0x53
[<ffffffff80259d25>] sock_sendmsg+0x110/0x130
[<ffffffff8041f462>] kernel_sendmsg+0x3c/0x52
[<ffffffff885399e9>] xs_tcp_send_request+0x117/0x320 [sunrpc]
[<ffffffff885388d5>] xprt_transmit+0x105/0x21e [sunrpc]
[<ffffffff8853771e>] call_transmit+0x1f4/0x239 [sunrpc]
[<ffffffff8853c06e>] __rpc_execute+0x9b/0x1e6 [sunrpc]
[<ffffffff8853c1de>] rpc_execute+0x1a/0x1d [sunrpc]
[<ffffffff885364ad>] rpc_call_sync+0x87/0xb9 [sunrpc]
[<ffffffff885a2587>] nfs3_rpc_wrapper+0x2e/0x74 [nfs]
[<ffffffff885a2a14>] nfs3_proc_lookup+0xe0/0x163 [nfs]
where tcp_sendmsg calls lock_sock. So this is the BC dependency.
From: Arjan van de Ven <hidden> Date: 2006-07-08 09:53:37
On Sat, 2006-07-08 at 13:00 +1000, Herbert Xu wrote:
Arjan van de Ven [off-list ref] wrote:
quoted
i_mutex is taken within rtln_mutex like this:
[<ffffffff8030f4a0>] create_dir+0x2c/0x1e2
[<ffffffff8030fa5b>] sysfs_create_dir+0x59/0x78
[<ffffffff8034d2e2>] kobject_add+0x114/0x1d8
[<ffffffff803bb1e7>] class_device_add+0xb5/0x49d
[<ffffffff804300b1>] netdev_register_sysfs+0x98/0xa2
[<ffffffff80426f58>] register_netdevice+0x28c/0x376
[<ffffffff8042709c>] register_netdev+0x5a/0x69
creating the AB dependency
This is a sysfs inode.
quoted
now for the third part, which involves the nfs client:
stat on an nfs file, which ends up taken the i_mutex of a directory in
the path (obvious), and then does
[<ffffffff8022800b>] tcp_sendmsg+0x1e/0xb1a
[<ffffffff80248f4b>] inet_sendmsg+0x45/0x53
[<ffffffff80259d25>] sock_sendmsg+0x110/0x130
[<ffffffff8041f462>] kernel_sendmsg+0x3c/0x52
[<ffffffff885399e9>] xs_tcp_send_request+0x117/0x320 [sunrpc]
[<ffffffff885388d5>] xprt_transmit+0x105/0x21e [sunrpc]
[<ffffffff8853771e>] call_transmit+0x1f4/0x239 [sunrpc]
[<ffffffff8853c06e>] __rpc_execute+0x9b/0x1e6 [sunrpc]
[<ffffffff8853c1de>] rpc_execute+0x1a/0x1d [sunrpc]
[<ffffffff885364ad>] rpc_call_sync+0x87/0xb9 [sunrpc]
[<ffffffff885a2587>] nfs3_rpc_wrapper+0x2e/0x74 [nfs]
[<ffffffff885a2a14>] nfs3_proc_lookup+0xe0/0x163 [nfs]
where tcp_sendmsg calls lock_sock. So this is the BC dependency.
This is an nfs inode.
Did I miss something?
is it not possible to nfs export /sys, and then mount it over loopback?
From: Herbert Xu <herbert@gondor.apana.org.au> Date: 2006-07-08 11:13:38
On Sat, Jul 08, 2006 at 11:53:20AM +0200, Arjan van de Ven wrote:
quoted
quoted
now for the third part, which involves the nfs client:
stat on an nfs file, which ends up taken the i_mutex of a directory in
the path (obvious), and then does
[<ffffffff8022800b>] tcp_sendmsg+0x1e/0xb1a
[<ffffffff80248f4b>] inet_sendmsg+0x45/0x53
[<ffffffff80259d25>] sock_sendmsg+0x110/0x130
[<ffffffff8041f462>] kernel_sendmsg+0x3c/0x52
[<ffffffff885399e9>] xs_tcp_send_request+0x117/0x320 [sunrpc]
[<ffffffff885388d5>] xprt_transmit+0x105/0x21e [sunrpc]
[<ffffffff8853771e>] call_transmit+0x1f4/0x239 [sunrpc]
[<ffffffff8853c06e>] __rpc_execute+0x9b/0x1e6 [sunrpc]
[<ffffffff8853c1de>] rpc_execute+0x1a/0x1d [sunrpc]
[<ffffffff885364ad>] rpc_call_sync+0x87/0xb9 [sunrpc]
[<ffffffff885a2587>] nfs3_rpc_wrapper+0x2e/0x74 [nfs]
[<ffffffff885a2a14>] nfs3_proc_lookup+0xe0/0x163 [nfs]
where tcp_sendmsg calls lock_sock. So this is the BC dependency.
This is an nfs inode.
Did I miss something?
is it not possible to nfs export /sys, and then mount it over loopback?
Possibly, but not with the backtrace above. You'd need an nfs server
backtrace to get the real sysfs inode.
In any case, the sock lock from the other backtrace that you had
(udp setsockopt) cannot be held by the kernel nfs client or server
since the kernel nfs sockets are not visible to user space.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} [off-list ref]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt