From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-09-27 16:38:29
This is a follow up to my v7 series of fixes for the zram driver [0]
which ended up uncovering a generic deadlock issue with sysfs and module
removal. I've reported this issue and proposed a few patches first since
March 2021 [1]. At the end of this email you will find an itemized list
of changes since that v1 series, you can also find these changes on my
branch 20210927-sysfs-generic-deadlock-fix [4] which is based on
linux-next tag next-20210927.
Just a heads up, I'm goin on vacation in two days, won't be back until
Monday October 11th.
On this v8 I incorporate feedback from the v7 series, namely:
- Tejun requested I move the struct module to the last attribute when
extending functions
- As per discussion with Tejun, trimmed and clarified the commit log
and documentation on the generic fix on patch 7
- As requested by Bart Van Assche, I simplied the setting of the
struct test_config *config into one line instead of two on many
places on patch 3 which adds the new sysfs selftest
- Dan Williams had some questions about patch 7, and so clarified these
questions using a more elaborate example on the commit log to show
where the lock call was happening.
- Trimmed the Cc list considerably as it was way too long before
- Rebased onto linux-next tag next-20210927
Below a list of changes of this patch set since its inception:
On v1:
- Open coded the sysfs deadlock race to only be localized by the zram
driver
Changes on v2:
- used bdgrab() as well for another race which was speculated by
Minchan
- improved documentation of fixes
Changes on v3:
- used a localized zram macros for the sysfs attributes instead of
open coding on each routine
- replaced bdget() stuff for a generic get_device() and bus_get() on
dev_attr_show() / dev_attr_store() for the issue speculated by
Michan
Changes on v4:
- Cosmetic fixes on the zram fixes as requested by Greg
- Split out the driver core fix as requested by Greg for the
issue speculated by Michan. This fix ended up getting up to its 4th
patch iteration [2] and eventually hit linux-next. We got a 0day
0day suspend stres fail for this patch [3]
Changes on v5:
- I ended up writing a test_sysfs driver and with it I ended up
proving that the issue speculated by Michen was not possible and
so I asked Greg to drop the patch from his queue titled
"sysfs: fix kobject refcount to address races with kobject removal"
- checkpatch fixes for the zram changes
Changes on v6:
- I submitted my test_sysfs driver for inclusion upstream which easily
abstracted the deadlock issue in a driver generically [4]
- I rebased the zram fixes and added also a new patch for zram to use
ATTRIBUTE_GROUPS As per Minchen I sent the patches to be merged
through Andrew Morton.
- Greg ended up NACK'ing the patchset because he was not sure the fix
was correct still
Changes on v7:
- Formalizes the original proposed generic sysfs fix intead of using
macro helpers to work around the issue
- I decided it is best to merge all the effort together into
one patch set because communication was being lost when I split the
patches up. This was not helping in any way to either fix the zram
issues or come to consensus on a generic solution. The patches are
also merged now because they are all related now.
- Running checkpatch exposed that S_IRWXUGO and S_IRWXU|S_IRUGO|S_IXUGO
should be replaced, so I did that in this series in two new patches
- Adds a try_module_get() documentation extension with tribal
knowledge and new information I don't think some folks still believe
in. The new test_sysfs selftest however proves this information to
be correct, the same selftest can be used to try to prove that
documentation incorrect
- Because the fix is now generic zram's deadlock can easily be fixed
now by just making it use ATTRIBUTE_GROUPS().
[0] https://lkml.kernel.org/r/YUjLAbnEB5qPfnL8@slm.duckdns.org
[1] https://lkml.kernel.org/r/20210306022035.11266-1-mcgrof@kernel.org
[2] https://lkml.kernel.org/r/20210623215007.862787-1-mcgrof@kernel.org
[3] https://lkml.kernel.org/r/20210701022737.GC21279@xsang-OptiPlex-9020
[4] https://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux-next.git/log/?h=20210927-sysfs-generic-deadlock-fix
Luis Chamberlain (12):
LICENSES: Add the copyleft-next-0.3.1 license
testing: use the copyleft-next-0.3.1 SPDX tag
selftests: add tests_sysfs module
kernfs: add initial failure injection support
test_sysfs: add support to use kernfs failure injection
kernel/module: add documentation for try_module_get()
fs/kernfs/symlink.c: replace S_IRWXUGO with 0777 on
kernfs_create_link()
fs/sysfs/dir.c: replace S_IRWXU|S_IRUGO|S_IXUGO with 0755
sysfs_create_dir_ns()
sysfs: fix deadlock race with module removal
test_sysfs: enable deadlock tests by default
zram: fix crashes with cpu hotplug multistate
zram: use ATTRIBUTE_GROUPS to fix sysfs deadlock module removal
.../fault-injection/fault-injection.rst | 22 +
LICENSES/dual/copyleft-next-0.3.1 | 237 +++
MAINTAINERS | 9 +-
arch/x86/kernel/cpu/resctrl/rdtgroup.c | 4 +-
drivers/block/zram/zram_drv.c | 74 +-
fs/kernfs/Makefile | 1 +
fs/kernfs/dir.c | 44 +-
fs/kernfs/failure-injection.c | 91 ++
fs/kernfs/file.c | 19 +-
fs/kernfs/kernfs-internal.h | 75 +-
fs/kernfs/symlink.c | 4 +-
fs/sysfs/dir.c | 5 +-
fs/sysfs/file.c | 6 +-
fs/sysfs/group.c | 3 +-
include/linux/kernfs.h | 19 +-
include/linux/module.h | 34 +-
include/linux/sysfs.h | 52 +-
kernel/cgroup/cgroup.c | 2 +-
lib/Kconfig.debug | 25 +
lib/Makefile | 1 +
lib/test_kmod.c | 12 +-
lib/test_sysctl.c | 12 +-
lib/test_sysfs.c | 952 ++++++++++++
tools/testing/selftests/kmod/kmod.sh | 13 +-
tools/testing/selftests/sysctl/sysctl.sh | 12 +-
tools/testing/selftests/sysfs/Makefile | 12 +
tools/testing/selftests/sysfs/config | 5 +
tools/testing/selftests/sysfs/sysfs.sh | 1383 +++++++++++++++++
28 files changed, 3026 insertions(+), 102 deletions(-)
create mode 100644 LICENSES/dual/copyleft-next-0.3.1
create mode 100644 fs/kernfs/failure-injection.c
create mode 100644 lib/test_sysfs.c
create mode 100644 tools/testing/selftests/sysfs/Makefile
create mode 100644 tools/testing/selftests/sysfs/config
create mode 100755 tools/testing/selftests/sysfs/sysfs.sh
--
2.30.2
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-09-27 16:38:14
Provide a simple state machine to fix races with driver exit where we
remove the CPU multistate callbacks and re-initialization / creation of
new per CPU instances which should be managed by these callbacks.
The zram driver makes use of cpu hotplug multistate support, whereby it
associates a struct zcomp per CPU. Each struct zcomp represents a
compression algorithm in charge of managing compression streams per
CPU. Although a compiled zram driver only supports a fixed set of
compression algorithms, each zram device gets a struct zcomp allocated
per CPU. The "multi" in CPU hotplug multstate refers to these per
cpu struct zcomp instances. Each of these will have the CPU hotplug
callback called for it on CPU plug / unplug. The kernel's CPU hotplug
multistate keeps a linked list of these different structures so that
it will iterate over them on CPU transitions.
By default at driver initialization we will create just one zram device
(num_devices=1) and a zcomp structure then set for the now default
lzo-rle comrpession algorithm. At driver removal we first remove each
zram device, and so we destroy the associated struct zcomp per CPU. But
since we expose sysfs attributes to create new devices or reset /
initialize existing zram devices, we can easily end up re-initializing
a struct zcomp for a zram device before the exit routine of the module
removes the cpu hotplug callback. When this happens the kernel's CPU
hotplug will detect that at least one instance (struct zcomp for us)
exists. This can happen in the following situation:
CPU 1 CPU 2
disksize_store(...);
class_unregister(...);
idr_for_each(...);
zram_debugfs_destroy();
idr_destroy(...);
unregister_blkdev(...);
cpuhp_remove_multi_state(...);
The warning comes up on cpuhp_remove_multi_state() when it sees that the
state for CPUHP_ZCOMP_PREPARE does not have an empty instance linked list.
In this case, that a struct zcom still exists, the driver allowed its
creation per CPU even though we could have just freed them per CPU
though a call on another CPU, and we are then later trying to remove the
hotplug callback.
Fix all this by providing a zram initialization boolean
protected the shared in the driver zram_index_mutex, which we
can use to annotate when sysfs attributes are safe to use or
not -- once the driver is properly initialized. When the driver
is going down we also are sure to not let userspace muck with
attributes which may affect each per cpu struct zcomp.
This also fixes a series of possible memory leaks. The
crashes and memory leaks can easily be caused by issuing
the zram02.sh script from the LTP project [0] in a loop
in two separate windows:
cd testcases/kernel/device-drivers/zram
while true; do PATH=$PATH:$PWD:$PWD/../../../lib/ ./zram02.sh; done
You end up with a splat as follows:
kernel: zram: Removed device: zram0
kernel: zram: Added device: zram0
kernel: zram0: detected capacity change from 0 to 209715200
kernel: Adding 104857596k swap on /dev/zram0. <etc>
kernel: zram0: detected capacitky change from 209715200 to 0
kernel: zram0: detected capacity change from 0 to 209715200
kernel: ------------[ cut here ]------------
kernel: Error: Removing state 63 which has instances left.
kernel: WARNING: CPU: 7 PID: 70457 at \
kernel/cpu.c:2069 __cpuhp_remove_state_cpuslocked+0xf9/0x100
kernel: Modules linked in: zram(E-) zsmalloc(E) <etc>
kernel: CPU: 7 PID: 70457 Comm: rmmod Tainted: G \
E 5.12.0-rc1-next-20210304 #3
kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), \
BIOS 1.14.0-2 04/01/2014
kernel: RIP: 0010:__cpuhp_remove_state_cpuslocked+0xf9/0x100
kernel: Code: <etc>
kernel: RSP: 0018:ffffa800c139be98 EFLAGS: 00010282
kernel: RAX: 0000000000000000 RBX: ffffffff9083db58 RCX: ffff9609f7dd86d8
kernel: RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffff9609f7dd86d0
kernel: RBP: 0000000000000000i R08: 0000000000000000 R09: ffffa800c139bcb8
kernel: R10: ffffa800c139bcb0 R11: ffffffff908bea40 R12: 000000000000003f
kernel: R13: 00000000000009d8 R14: 0000000000000000 R15: 0000000000000000
kernel: FS: 00007f1b075a7540(0000) GS:ffff9609f7dc0000(0000) knlGS:<etc>
kernel: CS: 0010 DS: 0000 ES 0000 CR0: 0000000080050033
kernel: CR2: 00007f1b07610490 CR3: 00000001bd04e000 CR4: 0000000000350ee0
kernel: Call Trace:
kernel: __cpuhp_remove_state+0x2e/0x80
kernel: __do_sys_delete_module+0x190/0x2a0
kernel: do_syscall_64+0x33/0x80
kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae
The "Error: Removing state 63 which has instances left" refers
to the zram per CPU struct zcomp instances left.
[0] https://github.com/linux-test-project/ltp.git
Acked-by: Minchan Kim <minchan@kernel.org>
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
drivers/block/zram/zram_drv.c | 63 ++++++++++++++++++++++++++++++-----
1 file changed, 55 insertions(+), 8 deletions(-)
@@ -1785,7 +1810,8 @@ static ssize_t reset_store(struct device *dev,/* Do not reset an active device or claimed device */if(bdev->bd_openers||zram->claim){mutex_unlock(&bdev->bd_disk->open_mutex);-return-EBUSY;+len=-EBUSY;+gotoout;}/* From now on, anyone can't open /dev/zram[0-9] */
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-09-27 16:38:16
Now that sysfs has the deadlock race fixed with module removal,
enable the deadlock tests module removal tests. They were left
disabled by default as otherwise you would deadlock your system
./tools/testing/selftests/sysfs/sysfs.sh -t 0027
Running test: sysfs_test_0027 - run #0
Test for possible rmmod deadlock while writing x ... ok
./tools/testing/selftests/sysfs/sysfs.sh -t 0028
Running test: sysfs_test_0028 - run #0
Test for possible rmmod deadlock using rtnl_lock while writing x ... ok
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
tools/testing/selftests/sysfs/sysfs.sh | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-09-27 16:38:20
The ATTRIBUTE_GROUPS is typically used to avoid boiler plate
code which is used in many drivers. Embracing ATTRIBUTE_GROUPS was
long due on the zram driver, however a recent fix for sysfs allows
users of ATTRIBUTE_GROUPS to also associate a module to the group
attribute.
In zram's case this also means it allows us to fix a race which triggers
a deadlock on the zram driver. This deadlock happens when a sysfs attribute
use a lock also used on module removal. This happens when for instance a
sysfs file on a driver is used, then at the same time we have module
removal call trigger. The module removal call code holds a lock, and then
the sysfs file entry waits for the same lock. While holding the lock the
module removal tries to remove the sysfs entries, but these cannot be
removed yet as one is waiting for a lock. This won't complete as the lock
is already held. Likewise module removal cannot complete, and so we
deadlock.
Sysfs fixes this when the group attributes have a module associated to
it, sysfs will *try* to get a refcount to the module when a shared
lock is used, prior to mucking with a sysfs attribute. If this fails we
just give up right away.
This deadlock was first reported with the zram driver, a sketch of how
this can happen follows:
CPU A CPU B
whatever_store()
module_unload
mutex_lock(foo)
mutex_lock(foo)
del_gendisk(zram->disk);
device_del()
device_remove_groups()
In this situation whatever_store() is waiting for the mutex foo to
become unlocked, but that won't happen until module removal is complete.
But module removal won't complete until the sysfs file being poked
completes which is waiting for a lock already held.
This issue can be reproduced easily on the zram driver as follows:
Loop 1 on one terminal:
while true;
do modprobe zram;
modprobe -r zram;
done
Loop 2 on a second terminal:
while true; do
echo 1024 > /sys/block/zram0/disksize;
echo 1 > /sys/block/zram0/reset;
done
Without this patch we end up in a deadlock, and the following
stack trace is produced which hints to us what the issue was:
INFO: task bash:888 blocked for more than 120 seconds.
Tainted: G E 5.12.0-rc1-next-20210304+ #4
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:bash state:D stack: 0 pid: 888 ppid: 887 flags:<etc>
Call Trace:
__schedule+0x2e4/0x900
schedule+0x46/0xb0
schedule_preempt_disabled+0xa/0x10
__mutex_lock.constprop.0+0x2c3/0x490
? _kstrtoull+0x35/0xd0
reset_store+0x6c/0x160 [zram]
kernfs_fop_write_iter+0x124/0x1b0
new_sync_write+0x11c/0x1b0
vfs_write+0x1c2/0x260
ksys_write+0x5f/0xe0
do_syscall_64+0x33/0x80
entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x7f34f2c3df33
RSP: 002b:00007ffe751df6e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f34f2c3df33
RDX: 0000000000000002 RSI: 0000561ccb06ec10 RDI: 0000000000000001
RBP: 0000561ccb06ec10 R08: 000000000000000a R09: 0000000000000001
R10: 0000561ccb157590 R11: 0000000000000246 R12: 0000000000000002
R13: 00007f34f2d0e6a0 R14: 0000000000000002 R15: 00007f34f2d0e8a0
INFO: task modprobe:1104 can't die for more than 120 seconds.
task:modprobe state:D stack: 0 pid: 1104 ppid: 916 flags:<etc>
Call Trace:
__schedule+0x2e4/0x900
schedule+0x46/0xb0
__kernfs_remove.part.0+0x228/0x2b0
? finish_wait+0x80/0x80
kernfs_remove_by_name_ns+0x50/0x90
remove_files+0x2b/0x60
sysfs_remove_group+0x38/0x80
sysfs_remove_groups+0x29/0x40
device_remove_attrs+0x4a/0x80
device_del+0x183/0x3e0
? mutex_lock+0xe/0x30
del_gendisk+0x27a/0x2d0
zram_remove+0x8a/0xb0 [zram]
? hot_remove_store+0xf0/0xf0 [zram]
zram_remove_cb+0xd/0x10 [zram]
idr_for_each+0x5e/0xd0
destroy_devices+0x39/0x6f [zram]
__do_sys_delete_module+0x190/0x2a0
do_syscall_64+0x33/0x80
entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x7f32adf727d7
RSP: 002b:00007ffc08bb38a8 EFLAGS: 00000206 ORIG_RAX: 00000000000000b0
RAX: ffffffffffffffda RBX: 000055eea23cbb10 RCX: 00007f32adf727d7
RDX: 0000000000000000 RSI: 0000000000000800 RDI: 000055eea23cbb78
RBP: 000055eea23cbb10 R08: 0000000000000000 R09: 0000000000000000
R10: 00007f32adfe5ac0 R11: 0000000000000206 R12: 000055eea23cbb78
R13: 0000000000000000 R14: 0000000000000000 R15: 000055eea23cbc20
[0] https://lkml.kernel.org/r/20210401235925.GR4332@42.do-not-panic.com
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
drivers/block/zram/zram_drv.c | 11 ++---------
1 file changed, 2 insertions(+), 9 deletions(-)
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-09-27 16:38:26
There is quite a bit of tribal knowledge around proper use of
try_module_get() and that it must be used only in a context which
can ensure the module won't be gone during the operation. Document
this little bit of tribal knowledge.
I'm extending this tribal knowledge with new developments which it
seems some folks do not yet believe to be true: we can be sure a
module will exist during the lifetime of a sysfs file operation.
For proof, refer to test_sysfs test #32:
./tools/testing/selftests/sysfs/sysfs.sh -t 0032
Without this being true, the write would fail or worse,
a crash would happen, in this test. It does not.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
include/linux/module.h | 34 ++++++++++++++++++++++++++++++++--
1 file changed, 32 insertions(+), 2 deletions(-)
@@ -609,10 +609,40 @@ void symbol_put_addr(void *addr);tohandletheerrorcase(whichonlyhappenswithrmmod--wait).*/externvoid__module_get(structmodule*module);-/* This is the Right Way to get a module: if it fails, it's being removed,-*sopretendit'snotthere.*/+/**+*try_module_get()-yieldstomoduleremovalandbumpsrefcntotherwise+*@module:themoduleweshouldcheckfor+*+*Thiscanbeusedtotrytobumpthereferencecountofamodule,soto+*preventmoduleremoval.Thereferencecountofamoduleisnotallowed+*tobeincrementedifthemoduleisalreadybeingremoved.+*+*Caremustbetakentoensurethemodulecannotberemovedduringthecallto+*try_module_get().Thiscanbedonebyhavinganotherentityotherthanthe+*moduleitselfincrementthemodulereferencecount,orthroughsomeother+*meanswhichguaranteesthemodulecouldnotberemovedduringanoperation.+*Anexampleofthislatercaseisusingtry_module_get()inasysfsfile+*whichthemodulecreated.Thesysfsstore/readfileoperationsare+*gauranteedtoexistthroughtheuseofkernfs'sactivereference(see+*kernfs_active()).Ifasysfsfileoperationisbeingrun,themodulewhich+*createditmuststillexistasthemoduleisinchargeofremovingthesame+*sysfsfilebeingread.Also,asysfs/kernfsfileremovalcannothappen+*unlessthesamefileisnotactive.+*+*Oneoftherealvaluestotry_module_get()isthemodule_is_live()check+*whichensuresthisthecalleroftry_module_get()canyieldtouserspace+*moduleremovalrequestsandfailwhateveritwasabouttoprocess.+*/externbooltry_module_get(structmodule*module);+/**+*module_put()-releaseareferencecounttoamodule+*@module:themoduleweshouldreleaseareferencecountfor+*+*Ifyousuccessfullybumpareferencecounttoamodulewithtry_module_get(),+*whenyouarefinishedyoumustcallmodule_put()toreleasethatreference+*count.+*/externvoidmodule_put(structmodule*module);#else /*!CONFIG_MODULE_UNLOAD*/
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-09-27 16:38:26
Two selftests drivers exist under the copyleft-next license.
These drivers were added prior to SPDX practice taking full swing
in the kernel. Now that we have an SPDX tag for copylef-next-0.3.1
documented, embrace it and remove the boiler plate.
Cc: Goldwyn Rodrigues <redacted>
Cc: Kuno Woudt <redacted>
Cc: Richard Fontana <redacted>
Cc: copyleft-next@lists.fedorahosted.org
Cc: Ciaran Farrell <redacted>
Cc: Christopher De Nicolo <redacted>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Thomas Gleixner <redacted>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Thorsten Leemhuis <linux@leemhuis.info>
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
lib/test_kmod.c | 12 +-----------
lib/test_sysctl.c | 12 +-----------
tools/testing/selftests/kmod/kmod.sh | 13 +------------
tools/testing/selftests/sysctl/sysctl.sh | 12 +-----------
4 files changed, 4 insertions(+), 45 deletions(-)
@@ -1,18 +1,7 @@#!/bin/bash-#+# SPDX-License-Identifier: GPL-2.0-or-later OR copyleft-next-0.3.1# Copyright (C) 2017 Luis R. Rodriguez <mcgrof@kernel.org>#-# This program is free software; you can redistribute it and/or modify it-# under the terms of the GNU General Public License as published by the Free-# Software Foundation; either version 2 of the License, or at your option any-# later version; or, when distributed separately from the Linux kernel or-# when incorporated into other software packages, subject to the following-# license:-#-# This program is free software; you can redistribute it and/or modify it-# under the terms of copyleft-next (version 0.3.1 or later) as published-# at http://copyleft-next.org/.-# This is a stress test script for kmod, the kernel module loader. It uses# test_kmod which exposes a series of knobs for the API for us so we can# tweak each test in userspace rather than in kernelspace.
@@ -1,16 +1,6 @@#!/bin/bash+# SPDX-License-Identifier: GPL-2.0-or-later OR copyleft-next-0.3.1# Copyright (C) 2017 Luis R. Rodriguez <mcgrof@kernel.org>-#-# This program is free software; you can redistribute it and/or modify it-# under the terms of the GNU General Public License as published by the Free-# Software Foundation; either version 2 of the License, or at your option any-# later version; or, when distributed separately from the Linux kernel or-# when incorporated into other software packages, subject to the following-# license:-#-# This program is free software; you can redistribute it and/or modify it-# under the terms of copyleft-next (version 0.3.1 or later) as published-# at http://copyleft-next.org/.# This performs a series tests against the proc sysctl interface.
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-09-27 16:38:33
If one ends up extending this line checkpatch will complain about the
use of S_IRWXUGO suggesting it is not preferred and that 0777
should be used instead. Take the tip from checkpatch and do that
change before we do our subsequent changes.
This makes no functional changes.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
fs/kernfs/symlink.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-09-27 16:38:34
When driver sysfs attributes use a lock also used on module removal we
can race to deadlock. This happens when for instance a sysfs file on
a driver is used, then at the same time we have module removal call
trigger. The module removal call code holds a lock, and then the
driver's sysfs file entry waits for the same lock. While holding the
lock the module removal tries to remove the sysfs entries, but these
cannot be removed yet as one is waiting for a lock. This won't complete
as the lock is already held. Likewise module removal cannot complete,
and so we deadlock.
This can now be easily reproducible with our sysfs selftest as follows:
./tools/testing/selftests/sysfs/sysfs.sh -t 0027
This uses a local driver lock. Test 0028 can also be used, that uses
the rtnl_lock():
./tools/testing/selftests/sysfs/sysfs.sh -t 0028
To fix this we extend the struct kernfs_node with a module reference
and use the try_module_get() after kernfs_get_active() is called. As
documented in the prior patch, we now know that once kernfs_get_active()
is called the module is implicitly guarded to exist and cannot be removed.
This is because the module is the one in charge of removing the same
sysfs file it created, and removal of sysfs files on module exit will wait
until they don't have any active references. By using a try_module_get()
after kernfs_get_active() we yield to let module removal trump calls to
process a sysfs operation, while also preventing module removal if a sysfs
operation is in already progress. This prevents the deadlock.
This deadlock was first reported with the zram driver, however the live
patching folks have acknowledged they have observed this as well with
live patching, when a live patch is removed. I was then able to
reproduce easily by creating a dedicated selftest for it.
A sketch of how this can happen follows, consider foo a local mutex
part of a driver, and used on the driver's module exit routine and
on one of its sysfs ops:
foo.c:
static DEFINE_MUTEX(foo);
static ssize_t foo_store(struct device *dev,
struct device_attribute *attr,
const char *buf, size_t count)
{
...
mutex_lock(&foo);
...
mutex_lock(&foo);
...
}
static DEVICE_ATTR_RW(foo);
...
void foo_exit(void)
{
mutex_lock(&foo);
...
mutex_unlock(&foo);
}
module_exit(foo_exit);
And this can lead to this condition:
CPU A CPU B
foo_store()
foo_exit()
mutex_lock(&foo)
mutex_lock(&foo)
del_gendisk(some_struct->disk);
device_del()
device_remove_groups()
In this situation foo_store() is waiting for the mutex foo to
become unlocked, but that won't happen until module removal is complete.
But module removal won't complete until the sysfs file being poked at
completes which is waiting for a lock already held.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
arch/x86/kernel/cpu/resctrl/rdtgroup.c | 4 +-
fs/kernfs/dir.c | 44 ++++++++++++++++++----
fs/kernfs/file.c | 6 ++-
fs/kernfs/kernfs-internal.h | 3 +-
fs/kernfs/symlink.c | 3 +-
fs/sysfs/dir.c | 2 +-
fs/sysfs/file.c | 6 ++-
fs/sysfs/group.c | 3 +-
include/linux/kernfs.h | 14 ++++---
include/linux/sysfs.h | 52 ++++++++++++++++++++------
kernel/cgroup/cgroup.c | 2 +-
11 files changed, 105 insertions(+), 34 deletions(-)
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-09-27 16:38:35
If one ends up expanding on this line checkpatch will complain that the
combination S_IRWXU|S_IRUGO|S_IXUGO should just be replaced with the
octal 0755. Do that.
This makes no functional changes.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
fs/sysfs/dir.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-09-27 16:38:51
Add the full text of the copyleft-next-0.3.1 license to the kernel
tree as well as the required tags for reference and tooling.
The license text was copied directly from the copyleft-next project's
git tree [0].
Discussion of using copyleft-next-0.3.1 on Linux started since June,
2016 [1]. In the end Linus' preference was to have drivers use
MODULE_LICENSE("GPL") to make it clear that the GPL applies when it
comes to Linux [2]. Additionally, even though copyleft-next-0.3.1 has
been found to be to be GPLv2 compatible by three attorneys at SUSE and
Redhat [3], to err on the side of caution we simply recommend to
always use the "OR" language for this license [4].
Even though it has been a goal of the project to be GPL-v2 compatible
to be certain in 2016 I asked for a clarification about what makes
copyleft-next GPLv2 compatible and also asked for a summary of
benefits. This prompted some small minor changes to make compatibility
even further clear and as of copyleft 0.3.1 compatibility should
be crystal clear [5].
The summary of why copyleft-next 0.3.1 is compatible with GPLv2
is explained as follows:
Like GPLv2, copyleft-next requires distribution of derivative works
("Derived Works" in copyleft-next 0.3.x) to be under the same license.
Ordinarily this would make the two licenses incompatible. However,
copyleft-next 0.3.1 says: "If the Derived Work includes material
licensed under the GPL, You may instead license the Derived Work under
the GPL." "GPL" is defined to include GPLv2.
In practice this means copyleft-next code in Linux may be licensed
under the GPL2, however there are additional obvious gains for
bringing contributions from Linux outbound where copyleft-next is
preferred. A summary of benefits why projects outside of Linux might
prefer to use copyleft-next >= 0.3.1 over GPLv2:
o It is much shorter and simpler
o It has an explicit patent license grant, unlike GPLv2
o Its notice preservation conditions are clearer
o More free software/open source licenses are compatible
with it (via section 4)
o The source code requirement triggered by binary distribution
is much simpler in a procedural sense
o Recipients potentially have a contract claim against distributors
who are noncompliant with the source code requirement
o There is a built-in inbound=outbound policy for upstream
contributions (cf. Apache License 2.0 section 5)
o There are disincentives to engage in the controversial practice
of copyleft/ proprietary dual-licensing
o In 15 years copyleft expires, which can be advantageous
for legacy code
o There are explicit disincentives to bringing patent infringement
claims accusing the licensed work of infringement (see 10b)
o There is a cure period for licensees who are not compliant
with the license (there is no cure opportunity in GPLv2)
o copyleft-next has a 'built-in or-later' provision
The first driver submission to Linux under this dual strategy was
lib/test_sysctl.c through commit 9308f2f9e7f05 ("test_sysctl: add
dedicated proc sysctl test driver") merged in July 2017. Shortly after
that I also added test_kmod through commit d9c6a72d6fa29 ("kmod: add
test driver to stress test the module loader") in the same month. These
two drivers went in just a few months before the SPDX license practice
kicked in. In 2018 Kuno Woudt went through the process to get SPDX
identifiers for copyleft-next [6] [7]. Although there are SPDX tags
for copyleft-next-0.3.0, we only document use in Linux starting from
copyleft-next-0.3.1 which makes GPLv2 compatibility crystal clear.
This patch will let us update the two Linux selftest drivers in
subsequent patches with their respective SPDX license identifiers and
let us remove repetitive license boiler plate.
[0] https://github.com/copyleft-next/copyleft-next/blob/master/Releases/copyleft-next-0.3.1
[1] https://lore.kernel.org/lkml/1465929311-13509-1-git-send-email-mcgrof@kernel.org/
[2] https://lore.kernel.org/lkml/CA+55aFyhxcvD+q7tp+-yrSFDKfR0mOHgyEAe=f_94aKLsOu0Og@mail.gmail.com/
[3] https://lore.kernel.org/lkml/20170516232702.GL17314@wotan.suse.de/
[4] https://lkml.kernel.org/r/1495234558.7848.122.camel@linux.intel.com
[5] https://lists.fedorahosted.org/archives/list/copyleft-next@lists.fedorahosted.org/thread/JTGV56DDADWGKU7ZKTZA4DLXTGTLNJ57/#SQMDIKBRAVDOCT4UVNOOCRGBN2UJIKHZ
[6] https://spdx.org/licenses/copyleft-next-0.3.0.html
[7] https://spdx.org/licenses/copyleft-next-0.3.1.html
Cc: Goldwyn Rodrigues <redacted>
Cc: Kuno Woudt <redacted>
Cc: Richard Fontana <redacted>
Cc: copyleft-next@lists.fedorahosted.org
Cc: Ciaran Farrell <redacted>
Cc: Christopher De Nicolo <redacted>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Thomas Gleixner <redacted>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Thorsten Leemhuis <linux@leemhuis.info>
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
LICENSES/dual/copyleft-next-0.3.1 | 237 ++++++++++++++++++++++++++++++
1 file changed, 237 insertions(+)
create mode 100644 LICENSES/dual/copyleft-next-0.3.1
@@ -0,0 +1,237 @@+Valid-License-Identifier: copyleft-next-0.3.1+SPDX-URL: https://spdx.org/licenses/copyleft-next-0.3.1+Usage-Guide:+ This license can be used in code, it has been found to be GPLv2 compatible+ by attorneys at Redhat and SUSE, however to air on the side of caution,+ it's best to only use it together with a GPL2 compatible license using "OR".+ To use the copyleft-next-0.3.1 license put the following SPDX tag/value+ pair into a comment according to the placement guidelines in the+ licensing rules documentation:+ SPDX-License-Identifier: GPL-2.0 OR copyleft-next-0.3.1+ SPDX-License-Identifier: GPL-2.0-only OR copyleft-next 0.3.1+ SPDX-License-Identifier: GPL-2.0+ OR copyleft-next-0.3.1+ SPDX-License-Identifier: GPL-2.0-or-later OR copyleft-next-0.3.1+License-Text:++=======================================================================++ copyleft-next 0.3.1 ("this License")+ Release date: 2016-04-29++1. License Grants; No Trademark License++ Subject to the terms of this License, I grant You:++ a) A non-exclusive, worldwide, perpetual, royalty-free, irrevocable+ copyright license, to reproduce, Distribute, prepare derivative works+ of, publicly perform and publicly display My Work.++ b) A non-exclusive, worldwide, perpetual, royalty-free, irrevocable+ patent license under Licensed Patents to make, have made, use, sell,+ offer for sale, and import Covered Works.++ This License does not grant any rights in My name, trademarks, service+ marks, or logos.++2. Distribution: General Conditions++ You may Distribute Covered Works, provided that You (i) inform+ recipients how they can obtain a copy of this License; (ii) satisfy the+ applicable conditions of sections 3 through 6; and (iii) preserve all+ Legal Notices contained in My Work (to the extent they remain+ pertinent). "Legal Notices" means copyright notices, license notices,+ license texts, and author attributions, but does not include logos,+ other graphical images, trademarks or trademark legends.++3. Conditions for Distributing Derived Works; Outbound GPL Compatibility++ If You Distribute a Derived Work, You must license the entire Derived+ Work as a whole under this License, with prominent notice of such+ licensing. This condition may not be avoided through such means as+ separate Distribution of portions of the Derived Work.++ If the Derived Work includes material licensed under the GPL, You may+ instead license the Derived Work under the GPL.++4. Condition Against Further Restrictions; Inbound License Compatibility++ When Distributing a Covered Work, You may not impose further+ restrictions on the exercise of rights in the Covered Work granted under+ this License. This condition is not excused merely because such+ restrictions result from Your compliance with conditions or obligations+ extrinsic to this License (such as a court order or an agreement with a+ third party).++ However, You may Distribute a Covered Work incorporating material+ governed by a license that is both OSI-Approved and FSF-Free as of the+ release date of this License, provided that compliance with such+ other license would not conflict with any conditions stated in other+ sections of this License.++5. Conditions for Distributing Object Code++ You may Distribute an Object Code form of a Covered Work, provided that+ you accompany the Object Code with a URL through which the Corresponding+ Source is made available, at no charge, by some standard or customary+ means of providing network access to source code.++ If you Distribute the Object Code in a physical product or tangible+ storage medium ("Product"), the Corresponding Source must be available+ through such URL for two years from the date of Your most recent+ Distribution of the Object Code in the Product. However, if the Product+ itself contains or is accompanied by the Corresponding Source (made+ available in a customarily accessible manner), You need not also comply+ with the first paragraph of this section.++ Each direct and indirect recipient of the Covered Work from You is an+ intended third-party beneficiary of this License solely as to this+ section 5, with the right to enforce its terms.++6. Symmetrical Licensing Condition for Upstream Contributions++ If You Distribute a work to Me specifically for inclusion in or+ modification of a Covered Work (a "Patch"), and no explicit licensing+ terms apply to the Patch, You license the Patch under this License, to+ the extent of Your copyright in the Patch. This condition does not+ negate the other conditions of this License, if applicable to the Patch.++7. Nullification of Copyleft/Proprietary Dual Licensing++ If I offer to license, for a fee, a Covered Work under terms other than+ a license that is OSI-Approved or FSF-Free as of the release date of this+ License or a numbered version of copyleft-next released by the+ Copyleft-Next Project, then the license I grant You under section 1 is no+ longer subject to the conditions in sections 3 through 5.++8. Copyleft Sunset++ The conditions in sections 3 through 5 no longer apply once fifteen+ years have elapsed from the date of My first Distribution of My Work+ under this License.++9. Pass-Through++ When You Distribute a Covered Work, the recipient automatically receives+ a license to My Work from Me, subject to the terms of this License.++10. Termination++ Your license grants under section 1 are automatically terminated if You++ a) fail to comply with the conditions of this License, unless You cure+ such noncompliance within thirty days after becoming aware of it, or++ b) initiate a patent infringement litigation claim (excluding+ declaratory judgment actions, counterclaims, and cross-claims)+ alleging that any part of My Work directly or indirectly infringes+ any patent.++ Termination of Your license grants extends to all copies of Covered+ Works You subsequently obtain. Termination does not terminate the+ rights of those who have received copies or rights from You subject to+ this License.++ To the extent permission to make copies of a Covered Work is necessary+ merely for running it, such permission is not terminable.++11. Later License Versions++ The Copyleft-Next Project may release new versions of copyleft-next,+ designated by a distinguishing version number ("Later Versions").+ Unless I explicitly remove the option of Distributing Covered Works+ under Later Versions, You may Distribute Covered Works under any Later+ Version.++** 12. No Warranty **+** **+** My Work is provided "as-is", without warranty. You bear the risk **+** of using it. To the extent permitted by applicable law, each **+** Distributor of My Work excludes the implied warranties of title, **+** merchantability, fitness for a particular purpose and **+** non-infringement. **++** 13. Limitation of Liability **+** **+** To the extent permitted by applicable law, in no event will any **+** Distributor of My Work be liable to You for any damages **+** whatsoever, whether direct, indirect, special, incidental, or **+** consequential damages, whether arising under contract, tort **+** (including negligence), or otherwise, even where the Distributor **+** knew or should have known about the possibility of such damages. **++14. Severability++ The invalidity or unenforceability of any provision of this License+ does not affect the validity or enforceability of the remainder of+ this License. Such provision is to be reformed to the minimum extent+ necessary to make it valid and enforceable.++15. Definitions++ "Copyleft-Next Project" means the project that maintains the source+ code repository at <https://github.com/copyleft-next/copyleft-next.git/>+ as of the release date of this License.++ "Corresponding Source" of a Covered Work in Object Code form means (i)+ the Source Code form of the Covered Work; (ii) all scripts,+ instructions and similar information that are reasonably necessary for+ a skilled developer to generate such Object Code from the Source Code+ provided under (i); and (iii) a list clearly identifying all Separate+ Works (other than those provided in compliance with (ii)) that were+ specifically used in building and (if applicable) installing the+ Covered Work (for example, a specified proprietary compiler including+ its version number). Corresponding Source must be machine-readable.++ "Covered Work" means My Work or a Derived Work.++ "Derived Work" means a work of authorship that copies from, modifies,+ adapts, is based on, is a derivative work of, transforms, translates or+ contains all or part of My Work, such that copyright permission is+ required. The following are not Derived Works: (i) Mere Aggregation;+ (ii) a mere reproduction of My Work; and (iii) if My Work fails to+ explicitly state an expectation otherwise, a work that merely makes+ reference to My Work.++ "Distribute" means to distribute, transfer or make a copy available to+ someone else, such that copyright permission is required.++ "Distributor" means Me and anyone else who Distributes a Covered Work.++ "FSF-Free" means classified as 'free' by the Free Software Foundation.++ "GPL" means a version of the GNU General Public License or the GNU+ Affero General Public License.++ "I"/"Me"/"My" refers to the individual or legal entity that places My+ Work under this License. "You"/"Your" refers to the individual or legal+ entity exercising rights in My Work under this License. A legal entity+ includes each entity that controls, is controlled by, or is under+ common control with such legal entity. "Control" means (a) the power to+ direct the actions of such legal entity, whether by contract or+ otherwise, or (b) ownership of more than fifty percent of the+ outstanding shares or beneficial ownership of such legal entity.++ "Licensed Patents" means all patent claims licensable royalty-free by+ Me, now or in the future, that are necessarily infringed by making,+ using, or selling My Work, and excludes claims that would be infringed+ only as a consequence of further modification of My Work.++ "Mere Aggregation" means an aggregation of a Covered Work with a+ Separate Work.++ "My Work" means the particular work of authorship I license to You+ under this License.++ "Object Code" means any form of a work that is not Source Code.++ "OSI-Approved" means approved as 'Open Source' by the Open Source+ Initiative.++ "Separate Work" means a work that is separate from and independent of a+ particular Covered Work and is not by its nature an extension or+ enhancement of the Covered Work, and/or a runtime library, standard+ library or similar component that is used to generate an Object Code+ form of a Covered Work.++ "Source Code" means the preferred form of a work for making+ modifications to it.
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-09-27 16:38:52
This adds initial failure injection support to kernfs. We start
off with debug knobs which when enabled allow test drivers, such as
test_sysfs, to then make use of these to try to force certain
difficult races to take place with a high degree of certainty.
This only adds runtime code *iff* the new bool CONFIG_FAIL_KERNFS_KNOBS is
enabled in your kernel. If you don't have this enabled this provides
no new functional. When CONFIG_FAIL_KERNFS_KNOBS is disabled the new
routine kernfs_debug_should_wait() ends up being transformed to if
(false), and so the compiler should optimize these out as dead code
producing no new effective binary changes.
We start off with enabling failure injections in kernfs by allowing us to
alter the way kernfs_fop_write_iter() behaves. We allow for the routine
kernfs_fop_write_iter() to wait for a certain condition in the kernel to
occur, after which it will sleep a predefined amount of time. This lets
kernfs users to time exactly when it want kernfs_fop_write_iter() to
complete, allowing for developing race conditions and test for correctness
in kernfs.
You'd boot with this enabled on your kernel command line:
fail_kernfs_fop_write_iter=1,100,0,1
The values are <interval,probability,size,times>, we don't care for
size, so for now we ignore it. The above ensures a failure will trigger
only once.
*How* we allow for this routine to change behaviour is left to knobs we
expose under debugfs:
# ls -1 /sys/kernel/debug/kernfs/config_fail_kernfs_fop_write_iter/
wait_after_active
wait_after_mutex
wait_at_start
wait_before_mutex
A debugfs entry also exists to allow us to sleep a configurabler amount
of time after the completion:
/sys/kernel/debug/kernfs/sleep_after_wait_ms
These two sets of knobs allow us to construct races and demonstrate
how the kernfs active reference should suffice to project against
races.
Enabling CONFIG_FAULT_INJECTION_DEBUG_FS enables us to configure the
differnt fault injection parametres for the new fail_kernfs_fop_write_iter
fault injection at run time:
ls -1 /sys/kernel/debug/kernfs/fail_kernfs_fop_write_iter/
interval
probability
space
task-filter
times
verbose
verbose_ratelimit_burst
verbose_ratelimit_interval_ms
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
.../fault-injection/fault-injection.rst | 22 +++++
MAINTAINERS | 2 +-
fs/kernfs/Makefile | 1 +
fs/kernfs/failure-injection.c | 91 +++++++++++++++++++
fs/kernfs/file.c | 13 +++
fs/kernfs/kernfs-internal.h | 72 +++++++++++++++
include/linux/kernfs.h | 5 +
lib/Kconfig.debug | 10 ++
8 files changed, 215 insertions(+), 1 deletion(-)
create mode 100644 fs/kernfs/failure-injection.c
@@ -28,6 +28,28 @@ Available fault injection capabilities injects kernel RPC client and server failures.+- fail_kernfs_fop_write_iter++ Allows for failures to be enabled inside kernfs_fop_write_iter(). Enabling+ this does not immediately enable any errors to occur. You must configure+ how you want this routine to fail or change behaviour by using the debugfs+ knobs for it:++ # ls -1 /sys/kernel/debug/kernfs/config_fail_kernfs_fop_write_iter/+ wait_after_active+ wait_after_mutex+ wait_at_start+ wait_before_mutex++ You can also configure how long to sleep after a wait under++ /sys/kernel/debug/kernfs/sleep_after_wait_ms++ If you enable CONFIG_FAULT_INJECTION_DEBUG_FS the fail_add_disk failure+ injection parameters are placed under:++ /sys/kernel/debug/kernfs/fail_kernfs_fop_write_iter/+- fail_make_request injects disk IO errors on devices permitted by setting
@@ -0,0 +1,91 @@+// SPDX-License-Identifier: GPL-2.0++#include<linux/fault-inject.h>+#include<linux/delay.h>++#include"kernfs-internal.h"++staticDECLARE_FAULT_ATTR(fail_kernfs_fop_write_iter);+structkernfs_config_failkernfs_config_fail;++#define kernfs_config_fail(when) \+kernfs_config_fail.kernfs_fop_write_iter_fail.wait_##when++#define kernfs_config_fail(when) \+kernfs_config_fail.kernfs_fop_write_iter_fail.wait_##when++staticint__initsetup_fail_kernfs_fop_write_iter(char*str)+{+returnsetup_fault_attr(&fail_kernfs_fop_write_iter,str);+}++__setup("fail_kernfs_fop_write_iter=",setup_fail_kernfs_fop_write_iter);++structdentry*kernfs_debugfs_root;+structdentry*config_fail_kernfs_fop_write_iter;++staticint__initkernfs_init_failure_injection(void)+{+kernfs_config_fail.sleep_after_wait_ms=100;+kernfs_debugfs_root=debugfs_create_dir("kernfs",NULL);++fault_create_debugfs_attr("fail_kernfs_fop_write_iter",+kernfs_debugfs_root,&fail_kernfs_fop_write_iter);++config_fail_kernfs_fop_write_iter=+debugfs_create_dir("config_fail_kernfs_fop_write_iter",+kernfs_debugfs_root);++debugfs_create_u32("sleep_after_wait_ms",0600,+kernfs_debugfs_root,+&kernfs_config_fail.sleep_after_wait_ms);++debugfs_create_bool("wait_at_start",0600,+config_fail_kernfs_fop_write_iter,+&kernfs_config_fail(at_start));+debugfs_create_bool("wait_before_mutex",0600,+config_fail_kernfs_fop_write_iter,+&kernfs_config_fail(before_mutex));+debugfs_create_bool("wait_after_mutex",0600,+config_fail_kernfs_fop_write_iter,+&kernfs_config_fail(after_mutex));+debugfs_create_bool("wait_after_active",0600,+config_fail_kernfs_fop_write_iter,+&kernfs_config_fail(after_active));+return0;+}+late_initcall(kernfs_init_failure_injection);++int__kernfs_debug_should_wait_kernfs_fop_write_iter(boolevaluate)+{+if(!evaluate)+return0;++returnshould_fail(&fail_kernfs_fop_write_iter,0);+}++DECLARE_COMPLETION(kernfs_debug_wait_completion);+EXPORT_SYMBOL_NS_GPL(kernfs_debug_wait_completion,KERNFS_DEBUG_PRIVATE);++voidkernfs_debug_wait(void)+{+unsignedlongtimeout;++timeout=wait_for_completion_timeout(&kernfs_debug_wait_completion,+msecs_to_jiffies(3000));+if(!timeout)+pr_info("%s waiting for kernfs_debug_wait_completion timed out\n",+__func__);+else+pr_info("%s received completion with time left on timeout %u ms\n",+__func__,jiffies_to_msecs(timeout));++/**+*Thegoaliswaitforanevent,and*then*oncewehave+*reachedit,theothersidewilltrytodosomethingwhich+*itthinkswillbreak.Sowemustgiveitsometimetodo+*that.Theamountoftimeisconfigurable.+*/+msleep(kernfs_config_fail.sleep_after_wait_ms);+pr_info("%s ended\n",__func__);+}
@@ -1902,6 +1902,16 @@ config FAULT_INJECTION_USERCOPYProvidesfault-injectioncapabilitytoinjectfailuresinusercopyfunctions(copy_from_user(),get_user(),...).+configFAIL_KERNFS_KNOBS+bool"Fault-injection support in kernfs"+depends onFAULT_INJECTION+help+Providefault-injectioncapabilityforkernfs.Thisonlyenables+theerrorinjectionfunctionality.Touseityoumustconfigurewhich+whichpathyouwanttotriggeronerroronusingdebugfsunder+/sys/kernel/debug/kernfs/config_fail_kernfs_fop_write_iter/.By+defaultallofthesearedisabled.+configFAIL_MAKE_REQUESTbool"Fault-injection capability for disk IO"depends onFAULT_INJECTION&&BLOCK
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-09-27 16:38:54
This extends test_sysfs with support for using the failure injection
wait completion and knobs to force a few race conditions which
demonstrates that kernfs active reference protection is sufficient
for kobject / device protection at higher layers.
This adds 4 new tests which tries to remove the device attribute
store operation in 4 different situations:
1) at the start of kernfs_kernfs_fop_write_iter()
2) before the of->mutex is held in kernfs_kernfs_fop_write_iter()
3) after the of->mutex is held in kernfs_kernfs_fop_write_iter()
4) after the kernfs node active reference is taken
A write fails in call cases except the last one, test number #32. There
is a good explanation for this: *once* kernfs_get_active() gets called
we have a guarantee that the kernfs entry cannot be removed. If
kernfs_get_active() succeeds that entry cannot be removed and so
anything trying to remove that entry will have to wait. It is perhaps
not obvious but since a sysfs write will trigger eventually a
kernfs_get_active() call, and *only* if this succeeds will the sysfs
op be called, this and the fact that you cannot remove the kernfs
entry while the kenfs entry is active implies that a module that
created the respective sysfs / kernfs entry *cannot* possibly be
removed during a sysfs operation. And test number 32 provides us with
proof of this. If it were not true test #32 should crash.
No null dereferences are reproduced, even though this has been observed
in some complex testing cases [0]. If this issue really exists we should
have enough tools on the sysfs_test toolbox now to try to reproduce
this easily without having to poke around other drivers. It very likley
was the case that the issue reported [0] was possibly a side issue after
the first bug which was zram specific. This is why it is important to
isolate the issue and try to reproduce it in a generic form using the
test_sysfs driver.
[0] https://lkml.kernel.org/r/20210623215007.862787-1-mcgrof@kernel.org
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
lib/Kconfig.debug | 3 +
lib/test_sysfs.c | 31 +++++
tools/testing/selftests/sysfs/config | 3 +
tools/testing/selftests/sysfs/sysfs.sh | 175 +++++++++++++++++++++++++
4 files changed, 212 insertions(+)
@@ -904,10 +922,23 @@ static int __init test_sysfs_init(void)}module_init(test_sysfs_init);+#ifdef CONFIG_FAIL_KERNFS_KNOBS+/* The goal is to race our device removal with a pending kernfs -> store call */+staticvoidtest_sysfs_kernfs_send_completion_rmmod(void)+{+if(!enable_completion_on_rmmod)+return;+complete(&kernfs_debug_wait_completion);+}+#else+staticinlinevoidtest_sysfs_kernfs_send_completion_rmmod(void){}+#endif+staticvoid__exittest_sysfs_exit(void){if(enable_debugfs)debugfs_remove(debugfs_dir);+test_sysfs_kernfs_send_completion_rmmod();if(delay_rmmod_ms)msleep(delay_rmmod_ms);unregister_test_dev_sysfs(first_test_dev);
@@ -197,6 +212,63 @@ debugfs_reset_first_test_dev_ignore_errors()echo-n"1">"$SYSFS_DEBUGFS_DIR"/reset_first_test_dev}+debugfs_kernfs_kernfs_fop_write_iter_exists()+{+KNOB_DIR="${KERNFS_DEBUGFS_DIR}/config_fail_kernfs_fop_write_iter"+if[[!-d$KNOB_DIR]];then+echo"kernfs debugfs does not exist $KNOB_DIR"+return0;+fi+KNOB_DEBUGFS="${KERNFS_DEBUGFS_DIR}/fail_kernfs_fop_write_iter"+if[[!-d$KNOB_DEBUGFS]];then+echo-n"kernfs debugfs for coniguring fail_kernfs_fop_write_iter "+echo"does not exist $KNOB_DIR"+return0;+fi+return1+}++debugfs_kernfs_kernfs_fop_write_iter_set_fail_once()+{+KNOB_DEBUGFS="${KERNFS_DEBUGFS_DIR}/fail_kernfs_fop_write_iter"+echo1>$KNOB_DEBUGFS/interval+echo100>$KNOB_DEBUGFS/probability+echo0>$KNOB_DEBUGFS/space+# Disable verbose messages on the kernel ring buffer which may+# confuse developers with a kernel panic.+echo0>$KNOB_DEBUGFS/verbose++# Fail only once+echo1>$KNOB_DEBUGFS/times+}++debugfs_kernfs_kernfs_fop_write_iter_set_fail_never()+{+KNOB_DEBUGFS="${KERNFS_DEBUGFS_DIR}/fail_kernfs_fop_write_iter"+echo0>$KNOB_DEBUGFS/times+}++debugfs_kernfs_set_wait_ms()+{+SLEEP_AFTER_WAIT_MS="${KERNFS_DEBUGFS_DIR}/sleep_after_wait_ms"+echo$1>$SLEEP_AFTER_WAIT_MS+}++debugfs_kernfs_disable_wait_kernfs_fop_write_iter()+{+ENABLE_WAIT_KNOB="${KERNFS_DEBUGFS_DIR}/config_fail_kernfs_fop_write_iter/wait_"+forKNOBin${ENABLE_WAIT_KNOB}*;do+echo0>$KNOB+done+}++debugfs_kernfs_enable_wait_kernfs_fop_write_iter()+{+ENABLE_WAIT_KNOB="${KERNFS_DEBUGFS_DIR}/config_fail_kernfs_fop_write_iter/wait_$1"+echo-n"1">$ENABLE_WAIT_KNOB+return$?+}+ set_orig(){if[[!-z$TARGET]]&&[[!-z$ORIG]];then
@@ -972,6 +1044,105 @@ sysfs_test_0028()fi}+sysfs_race_kernfs_kernfs_fop_write_iter()+{+TARGET="${DIR}/$(get_test_target$1)"+WAIT_AT=$2+EXPECT_WRITE_RETURNS=$3+MSDELAY=$4++modprobe_reset_enable_completion+ORIG=$(cat"${TARGET}")+TEST_STR=$(($ORIG+1))++echo-n"Test racing removal of sysfs store op with kernfs $WAIT_AT ... "++ifdebugfs_kernfs_kernfs_fop_write_iter_exists;then+echo-n"skipping test as CONFIG_FAIL_KERNFS_KNOBS "+echo" or CONFIG_FAULT_INJECTION_DEBUG_FS is disabled"+return$ksft_skip+fi++# Allow for failing the kernfs_kernfs_fop_write_iter call once,+# we'll provide exact context shortly afterwards.+debugfs_kernfs_kernfs_fop_write_iter_set_fail_once++# First disable all waits+debugfs_kernfs_disable_wait_kernfs_fop_write_iter++# Enable a wait_for_completion(&kernfs_debug_wait_completion) at the+# specified location inside the kernfs_fop_write_iter() routine+debugfs_kernfs_enable_wait_kernfs_fop_write_iter$WAIT_AT++# Configure kernfs so that after its wait_for_completion() it+# will msleep() this amount of time and schedule(). We figure this+# will be sufficient time to allow for our module removal to complete.+debugfs_kernfs_set_wait_ms$MSDELAY++# Now we trigger a kernfs write op, which will run kernfs_fop_write_iter,+# but will wait until our driver sends a respective completion+set_test_ignore_errors&+write_pid=$!++# At this point kernfs_fop_write_iter() hasn't run our op, its+# waiting for our completion at the specified time $WAIT_AT.+# We now remove our module which will send a+# complete(&kernfs_debug_wait_completion) right before we deregister+# our device and the sysfs device attributes are removed.+#+# After the completion is sent, the test_sysfs driver races with+# kernfs to do the device deregistration with the kernfs msleep+# and schedule(). This should mean we've forced trying to remove the+# module prior to allowing kernfs to run our store operation. If the+# race did happen we'll panic with a null dereference on the store op.+#+# If no race happens we should see no write operation triggered.+modprobe-r$TEST_DRIVER>/dev/null2>&1++debugfs_kernfs_kernfs_fop_write_iter_set_fail_never++wait$write_pid+if[[$?-eq$EXPECT_WRITE_RETURNS]];then+echo"ok"+else+echo"FAIL">&2+fi+}++sysfs_test_0029()+{+fordelayin02481632641282465121024;do+echo"Using delay-after-completion: $delay"+sysfs_race_kernfs_kernfs_fop_write_iter0029at_start1$delay+done+}++sysfs_test_0030()+{+fordelayin02481632641282465121024;do+echo"Using delay-after-completion: $delay"+sysfs_race_kernfs_kernfs_fop_write_iter0030before_mutex1$delay+done+}++sysfs_test_0031()+{+fordelayin02481632641282465121024;do+echo"Using delay-after-completion: $delay"+sysfs_race_kernfs_kernfs_fop_write_iter0031after_mutex1$delay+done+}++# A write only succeeds *iff* a module removal happens *after* the+# kernfs active reference is obtained with kernfs_get_active().+sysfs_test_0032()+{+fordelayin02481632641282465121024;do+echo"Using delay-after-completion: $delay"+sysfs_race_kernfs_kernfs_fop_write_iter0032after_active0$delay+done+}+ test_gen_desc(){echo-n"$1 x $(get_test_count$1)"
@@ -1013,6 +1184,10 @@ list_tests()echo"$(test_gen_desc0026) - block test writing y larger delay and resetting device"echo"$(test_gen_desc0027) - test rmmod deadlock while writing x ... "echo"$(test_gen_desc0028) - test rmmod deadlock using rtnl_lock while writing x ..."+echo"$(test_gen_desc0029) - racing removal of store op with kernfs at start"+echo"$(test_gen_desc0030) - racing removal of store op with kernfs before mutex"+echo"$(test_gen_desc0031) - racing removal of store op with kernfs after mutex"+echo"$(test_gen_desc0032) - racing removal of store op with kernfs after active"} usage()
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-09-27 16:38:54
This adds a new selftest module which can be used to test sysfs, which
would otherwise require using an existing driver. This lets us muck
with a template driver to test breaking things without affecting
system behaviour or requiring the dependencies of a real device
driver.
A series of 28 tests are added. Support for using two device types are
supported:
* misc
* block
Contrary to sysctls, sysfs requires a full write to happen at once, and
so we reduce the digit tests to single writes. Two main sysfs knobs are
provided for testing reading/storing, one which doesn't inclur any
delays and another which can incur programmed delays. What locks are
held, if any, are configurable, at module load time, or through dynamic
configuration at run time.
Since sysfs is a technically filesystem, but a pseudo one, which
requires a kernel user, our test_sysfs module and respective test script
embraces fstests format for tests in the kernel ring bufffer. Likewise,
a scraper for kernel crashes is provided which matches what fstests does
as well.
Two tests are kept disabled as they currently cause a deadlock, and so
this provides a mechanism to easily show proof and demo how the deadlock
can happen:
Demos the deadlock with a device specific lock
./tools/testing/selftests/sysfs/sysfs.sh -t 0027
Demos the deadlock with rtnl_lock()
./tools/testing/selftests/sysfs/sysfs.sh -t 0028
Two separate solutions to the deadlock issue have been proposed,
and so now its a matter of either documenting this limitation or
eventually adopting a generic fix.
This selftests will shortly be expanded upon with more tests which
require further kernel changes in order to provide better test
coverage.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
MAINTAINERS | 7 +
lib/Kconfig.debug | 12 +
lib/Makefile | 1 +
lib/test_sysfs.c | 921 ++++++++++++++++++
tools/testing/selftests/sysfs/Makefile | 12 +
tools/testing/selftests/sysfs/config | 2 +
tools/testing/selftests/sysfs/sysfs.sh | 1208 ++++++++++++++++++++++++
7 files changed, 2163 insertions(+)
create mode 100644 lib/test_sysfs.c
create mode 100644 tools/testing/selftests/sysfs/Makefile
create mode 100644 tools/testing/selftests/sysfs/config
create mode 100755 tools/testing/selftests/sysfs/sysfs.sh
@@ -0,0 +1,921 @@+// SPDX-License-Identifier: GPL-2.0-or-later OR copyleft-next-0.3.1+/*+*sysfstestdriver+*+*Copyright(C)2021LuisChamberlain<mcgrof@kernel.org>+*+*Thisprogramisfreesoftware;youcanredistributeitand/ormodifyit+*underthetermsoftheGNUGeneralPublicLicenseaspublishedbytheFree+*SoftwareFoundation;eitherversion2oftheLicense,oratyouroptionany+*laterversion;or,whendistributedseparatelyfromtheLinuxkernelor+*whenincorporatedintoothersoftwarepackages,subjecttothefollowing+*license:+*+*Thisprogramisfreesoftware;youcanredistributeitand/ormodifyit+*underthetermsofcopyleft-next(version0.3.1orlater)aspublished+*athttp://copyleft-next.org/.+*/++/*+*Thismoduleallowsustoaddraceconditionswhichwecantestfor+*againstthesysfsfilesystem.+*/++#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt++#include<linux/init.h>+#include<linux/list.h>+#include<linux/module.h>+#include<linux/printk.h>+#include<linux/fs.h>+#include<linux/miscdevice.h>+#include<linux/slab.h>+#include<linux/uaccess.h>+#include<linux/async.h>+#include<linux/delay.h>+#include<linux/vmalloc.h>+#include<linux/debugfs.h>+#include<linux/rtnetlink.h>+#include<linux/genhd.h>+#include<linux/blkdev.h>++staticboolenable_lock;+module_param(enable_lock,bool_enable_only,0644);+MODULE_PARM_DESC(enable_lock,+"enable locking on reads / stores from the start");++staticboolenable_lock_on_rmmod;+module_param(enable_lock_on_rmmod,bool_enable_only,0644);+MODULE_PARM_DESC(enable_lock_on_rmmod,+"enable locking on rmmod");++staticbooluse_rtnl_lock;+module_param(use_rtnl_lock,bool_enable_only,0644);+MODULE_PARM_DESC(use_rtnl_lock,+"use an rtnl_lock instead of the device mutex_lock");++staticunsignedintwrite_delay_msec_y=500;+module_param_named(write_delay_msec_y,write_delay_msec_y,uint,0644);+MODULE_PARM_DESC(write_delay_msec_y,"msec write delay for writes to y");++staticunsignedinttest_devtype;+module_param_named(devtype,test_devtype,uint,0644);+MODULE_PARM_DESC(devtype,"device type to register");++staticboolenable_busy_alloc;+module_param(enable_busy_alloc,bool_enable_only,0644);+MODULE_PARM_DESC(enable_busy_alloc,"do a fake allocation during writes");++staticboolenable_debugfs;+module_param(enable_debugfs,bool_enable_only,0644);+MODULE_PARM_DESC(enable_debugfs,"enable a few debugfs files");++staticboolenable_verbose_writes;+module_param(enable_verbose_writes,bool_enable_only,0644);+MODULE_PARM_DESC(enable_debugfs,"enable stores to print verbose information");++staticunsignedintdelay_rmmod_ms;+module_param_named(delay_rmmod_ms,delay_rmmod_ms,uint,0644);+MODULE_PARM_DESC(delay_rmmod_ms,"if set how many ms to delay rmmod before device deletion");++staticboolenable_verbose_rmmod;+module_param(enable_verbose_rmmod,bool_enable_only,0644);+MODULE_PARM_DESC(enable_verbose_rmmod,"enable verbose print messages on rmmod");++staticintsysfs_test_major;++/**+*test_config-usedforconfiguringhowthesysfstestdevicewillbehave+*+*@enable_lock:ifenabledalockwillbeusedwhenreading/storingvariables+*@enable_lock_on_rmmod:ifenabledalockwillbeusedwhenreading/storing+*sysfsattributes,butitwillalsobeusedtolockonrmmod.Thisis+*usefultotestforadeadlock.+*@use_rtnl_lock:ifenabledinsteadofconfigurationspecificmutex,we'll+*usethertnl_lock.Ifyourtestcaseismodifyingthisonthefly+*whiledoingotherstores/reads,thingswillbreakasalockcanbe+*leftcontending.Bestisthattestsusethisknobserially,without+*allowinguserspacetomodifyotherknobswhilethisonechanges.+*@write_delay_msec_y:theamountofdelaytousewhenwritingtoy+*@enable_busy_alloc:ifenabledwe'lldoalargeallocationbetween+*writes.Weimmediatelyfreerightaway.Wealsoscheduletogivethe+*kernelsometimetore-useanymemorywedon'tneed.Thisisintened+*tomimictypicaldriverbehaviour.+*/+structtest_config{+boolenable_lock;+boolenable_lock_on_rmmod;+booluse_rtnl_lock;+unsignedintwrite_delay_msec_y;+boolenable_busy_alloc;+};++/**+*enumsysfs_test_devtype-sysfsdevicetype+*@TESTDEV_TYPE_MISC:miscdevicetype+*@TESTDEV_TYPE_BLOCK:useablockdeviceforthesysfstestdevice.+*/+enumsysfs_test_devtype{+TESTDEV_TYPE_MISC=0,+TESTDEV_TYPE_BLOCK,+};++/**+*sysfs_test_device-testdevicetohelptestsysfs+*+*@devtype:thetypeofdevicetouse+*@config:configurationforthetest+*@config_mutex:protectsconfigurationoftest+*@misc_dev:weuseamiscdeviceunderthehood+*@disk:representsadiskwhenusedasablockdevice+*@dev:pointertomisc_dev'sownstructdevice+*@dev_idx:uniqueIDfortestdevice+*@x:variablewecanusetotestread/store+*@y:slowvariablewecanusetotestread/store+*/+structsysfs_test_device{+enumsysfs_test_devtypedevtype;+structtest_configconfig;+structmutexconfig_mutex;+structmiscdevicemisc_dev;+structgendisk*disk;+structdevice*dev;+intdev_idx;+intx;+inty;+};++staticstructsysfs_test_device*first_test_dev;++staticstructmiscdevice*dev_to_misc_dev(structdevice*dev)+{+returndev_get_drvdata(dev);+}++staticstructsysfs_test_device*misc_dev_to_test_dev(structmiscdevice*misc_dev)+{+returncontainer_of(misc_dev,structsysfs_test_device,misc_dev);+}++staticstructsysfs_test_device*devblock_to_test_dev(structdevice*dev)+{+return(structsysfs_test_device*)dev_to_disk(dev)->private_data;+}++staticstructsysfs_test_device*devmisc_to_testdev(structdevice*dev)+{+structmiscdevice*misc_dev;++misc_dev=dev_to_misc_dev(dev);+returnmisc_dev_to_test_dev(misc_dev);+}++staticstructsysfs_test_device*dev_to_test_dev(structdevice*dev)+{+if(test_devtype==TESTDEV_TYPE_MISC)+returndevmisc_to_testdev(dev);+elseif(test_devtype==TESTDEV_TYPE_BLOCK)+returndevblock_to_test_dev(dev);+returnNULL;+}++staticvoidtest_dev_config_lock(structsysfs_test_device*test_dev)+{+structtest_config*config=&test_dev->config;++if(config->enable_lock){+if(config->use_rtnl_lock)+rtnl_lock();+else+mutex_lock(&test_dev->config_mutex);+}+}++staticvoidtest_dev_config_unlock(structsysfs_test_device*test_dev)+{+structtest_config*config=&test_dev->config;++if(config->enable_lock){+if(config->use_rtnl_lock)+rtnl_unlock();+else+mutex_unlock(&test_dev->config_mutex);+}+}++staticvoidtest_dev_config_lock_rmmod(structsysfs_test_device*test_dev)+{+structtest_config*config=&test_dev->config;++if(config->enable_lock_on_rmmod)+test_dev_config_lock(test_dev);+}++staticvoidtest_dev_config_unlock_rmmod(structsysfs_test_device*test_dev)+{+structtest_config*config=&test_dev->config;++if(config->enable_lock_on_rmmod)+test_dev_config_unlock(test_dev);+}++staticvoidfree_test_dev_sysfs(structsysfs_test_device*test_dev)+{+if(test_dev){+kfree_const(test_dev->misc_dev.name);+test_dev->misc_dev.name=NULL;+kfree(test_dev);+test_dev=NULL;+}+}++staticvoidtest_sysfs_reset_vals(structsysfs_test_device*test_dev)+{+test_dev->x=3;+test_dev->y=4;+}++staticssize_tconfig_show(structdevice*dev,+structdevice_attribute*attr,+char*buf)+{+structsysfs_test_device*test_dev=dev_to_test_dev(dev);+structtest_config*config=&test_dev->config;+intlen=0;++test_dev_config_lock(test_dev);++len+=snprintf(buf,PAGE_SIZE,+"Configuration for: %s\n",+dev_name(dev));++len+=snprintf(buf+len,PAGE_SIZE-len,+"x:\t%d\n",+test_dev->x);++len+=snprintf(buf+len,PAGE_SIZE-len,+"y:\t%d\n",+test_dev->y);++len+=snprintf(buf+len,PAGE_SIZE-len,+"enable_lock:\t%s\n",+config->enable_lock?"true":"false");++len+=snprintf(buf+len,PAGE_SIZE-len,+"enable_lock_on_rmmmod:\t%s\n",+config->enable_lock_on_rmmod?"true":"false");++len+=snprintf(buf+len,PAGE_SIZE-len,+"use_rtnl_lock:\t%s\n",+config->use_rtnl_lock?"true":"false");++len+=snprintf(buf+len,PAGE_SIZE-len,+"write_delay_msec_y:\t%d\n",+config->write_delay_msec_y);++len+=snprintf(buf+len,PAGE_SIZE-len,+"enable_busy_alloc:\t%s\n",+config->enable_busy_alloc?"true":"false");++len+=snprintf(buf+len,PAGE_SIZE-len,+"enable_debugfs:\t%s\n",+enable_debugfs?"true":"false");++len+=snprintf(buf+len,PAGE_SIZE-len,+"enable_verbose_writes:\t%s\n",+enable_verbose_writes?"true":"false");++test_dev_config_unlock(test_dev);++returnlen;+}+staticDEVICE_ATTR_RO(config);++staticssize_treset_store(structdevice*dev,+structdevice_attribute*attr,+constchar*buf,size_tcount)+{+structsysfs_test_device*test_dev=dev_to_test_dev(dev);+structtest_config*config=&test_dev->config;++/*+*Wecompromiseandsimplifythisconditionanddonotusealock+*hereasthelocktypecanchange.+*/+config->enable_lock=false;+config->enable_lock_on_rmmod=false;+config->use_rtnl_lock=false;+config->enable_busy_alloc=false;+test_sysfs_reset_vals(test_dev);++dev_info(dev,"reset\n");++returncount;+}+staticDEVICE_ATTR_WO(reset);++staticvoidtest_dev_busy_alloc(structsysfs_test_device*test_dev)+{+structtest_config*config=&test_dev->config;+char*ignore;++if(!config->enable_busy_alloc)+return;++ignore=kzalloc(sizeof(structsysfs_test_device)*10,GFP_KERNEL);+kfree(ignore);++schedule();+}++staticssize_ttest_dev_x_store(structdevice*dev,+structdevice_attribute*attr,+constchar*buf,size_tcount)+{+structsysfs_test_device*test_dev=dev_to_test_dev(dev);+intret;++test_dev_busy_alloc(test_dev);+test_dev_config_lock(test_dev);++ret=kstrtoint(buf,10,&test_dev->x);+if(ret)+count=ret;++if(enable_verbose_writes)+dev_info(test_dev->dev,"wrote x = %d\n",test_dev->x);++test_dev_config_unlock(test_dev);++returncount;+}++staticssize_ttest_dev_x_show(structdevice*dev,+structdevice_attribute*attr,+char*buf)+{+structsysfs_test_device*test_dev=dev_to_test_dev(dev);+intret;++test_dev_config_lock(test_dev);+ret=snprintf(buf,PAGE_SIZE,"%d\n",test_dev->x);+test_dev_config_unlock(test_dev);++returnret;+}+staticDEVICE_ATTR_RW(test_dev_x);++staticssize_ttest_dev_y_store(structdevice*dev,+structdevice_attribute*attr,+constchar*buf,size_tcount)+{+structsysfs_test_device*test_dev=dev_to_test_dev(dev);+structtest_config*config;+inty;+intret;++test_dev_busy_alloc(test_dev);+test_dev_config_lock(test_dev);++config=&test_dev->config;++ret=kstrtoint(buf,10,&y);+if(ret)+count=ret;++msleep(config->write_delay_msec_y);+test_dev->y=test_dev->x+y+7;++if(enable_verbose_writes)+dev_info(test_dev->dev,"wrote y = %d\n",test_dev->y);++test_dev_config_unlock(test_dev);++returncount;+}++staticssize_ttest_dev_y_show(structdevice*dev,+structdevice_attribute*attr,+char*buf)+{+structsysfs_test_device*test_dev=dev_to_test_dev(dev);+intret;++test_dev_config_lock(test_dev);+ret=snprintf(buf,PAGE_SIZE,"%d\n",test_dev->y);+test_dev_config_unlock(test_dev);++returnret;+}+staticDEVICE_ATTR_RW(test_dev_y);++staticssize_tconfig_enable_lock_store(structdevice*dev,+structdevice_attribute*attr,+constchar*buf,size_tcount)+{+structsysfs_test_device*test_dev=dev_to_test_dev(dev);+structtest_config*config=&test_dev->config;+intret;+intval;++ret=kstrtoint(buf,10,&val);+if(ret)+returnret;++/*+*Wecompromiseforsimplictyanddonotlockwhenchanging+*lockingconfiguration,withtheassumptionuserspacetests+*willknowthis.+*/+if(val)+config->enable_lock=true;+else+config->enable_lock=false;++returncount;+}++staticssize_tconfig_enable_lock_show(structdevice*dev,+structdevice_attribute*attr,+char*buf)+{+structsysfs_test_device*test_dev=dev_to_test_dev(dev);+structtest_config*config=&test_dev->config;+ssize_tret;++test_dev_config_lock(test_dev);+ret=snprintf(buf,PAGE_SIZE,"%d\n",config->enable_lock);+test_dev_config_unlock(test_dev);++returnret;+}+staticDEVICE_ATTR_RW(config_enable_lock);++staticssize_tconfig_enable_lock_on_rmmod_store(structdevice*dev,+structdevice_attribute*attr,+constchar*buf,size_tcount)+{+structsysfs_test_device*test_dev=dev_to_test_dev(dev);+structtest_config*config=&test_dev->config;+intret;+intval;++ret=kstrtoint(buf,10,&val);+if(ret)+returnret;++test_dev_config_lock(test_dev);+if(val)+config->enable_lock_on_rmmod=true;+else+config->enable_lock_on_rmmod=false;+test_dev_config_unlock(test_dev);++returncount;+}++staticssize_tconfig_enable_lock_on_rmmod_show(structdevice*dev,+structdevice_attribute*attr,+char*buf)+{+structsysfs_test_device*test_dev=dev_to_test_dev(dev);+structtest_config*config=&test_dev->config;+ssize_tret;++test_dev_config_lock(test_dev);+ret=snprintf(buf,PAGE_SIZE,"%d\n",config->enable_lock_on_rmmod);+test_dev_config_unlock(test_dev);++returnret;+}+staticDEVICE_ATTR_RW(config_enable_lock_on_rmmod);++staticssize_tconfig_use_rtnl_lock_store(structdevice*dev,+structdevice_attribute*attr,+constchar*buf,size_tcount)+{+structsysfs_test_device*test_dev=dev_to_test_dev(dev);+structtest_config*config=&test_dev->config;+intret;+intval;++ret=kstrtoint(buf,10,&val);+if(ret)+returnret;++/*+*Wecompromiseandsimplifythisconditionanddonotusealock+*hereasthelocktypecanchange.+*/+if(val)+config->use_rtnl_lock=true;+else+config->use_rtnl_lock=false;++returncount;+}++staticssize_tconfig_use_rtnl_lock_show(structdevice*dev,+structdevice_attribute*attr,+char*buf)+{+structsysfs_test_device*test_dev=dev_to_test_dev(dev);+structtest_config*config=&test_dev->config;++returnsnprintf(buf,PAGE_SIZE,"%d\n",config->use_rtnl_lock);+}+staticDEVICE_ATTR_RW(config_use_rtnl_lock);++staticssize_tconfig_write_delay_msec_y_store(structdevice*dev,+structdevice_attribute*attr,+constchar*buf,size_tcount)+{+structsysfs_test_device*test_dev=dev_to_test_dev(dev);+structtest_config*config=&test_dev->config;+intret;+intval;++ret=kstrtoint(buf,10,&val);+if(ret)+returnret;++test_dev_config_lock(test_dev);+config->write_delay_msec_y=val;+test_dev_config_unlock(test_dev);++returncount;+}++staticssize_tconfig_write_delay_msec_y_show(structdevice*dev,+structdevice_attribute*attr,+char*buf)+{+structsysfs_test_device*test_dev=dev_to_test_dev(dev);+structtest_config*config=&test_dev->config;++returnsnprintf(buf,PAGE_SIZE,"%d\n",config->write_delay_msec_y);+}+staticDEVICE_ATTR_RW(config_write_delay_msec_y);++staticssize_tconfig_enable_busy_alloc_store(structdevice*dev,+structdevice_attribute*attr,+constchar*buf,size_tcount)+{+structsysfs_test_device*test_dev=dev_to_test_dev(dev);+structtest_config*config=&test_dev->config;+intret;+intval;++ret=kstrtoint(buf,10,&val);+if(ret)+returnret;++test_dev_config_lock(test_dev);+config->enable_busy_alloc=val;+test_dev_config_unlock(test_dev);++returncount;+}++staticssize_tconfig_enable_busy_alloc_show(structdevice*dev,+structdevice_attribute*attr,+char*buf)+{+structsysfs_test_device*test_dev=dev_to_test_dev(dev);+structtest_config*config=&test_dev->config;++returnsnprintf(buf,PAGE_SIZE,"%d\n",config->enable_busy_alloc);+}+staticDEVICE_ATTR_RW(config_enable_busy_alloc);++#define TEST_SYSFS_DEV_ATTR(name) (&dev_attr_##name.attr)++staticstructattribute*test_dev_attrs[]={+/* Generic driver knobs go here */+TEST_SYSFS_DEV_ATTR(config),+TEST_SYSFS_DEV_ATTR(reset),++/* These are used to test sysfs */+TEST_SYSFS_DEV_ATTR(test_dev_x),+TEST_SYSFS_DEV_ATTR(test_dev_y),++/*+*Theseareconfigurationknobstomodifyhowwetestsysfswhen+*doingreads/stores.+*/+TEST_SYSFS_DEV_ATTR(config_enable_lock),+TEST_SYSFS_DEV_ATTR(config_enable_lock_on_rmmod),+TEST_SYSFS_DEV_ATTR(config_use_rtnl_lock),+TEST_SYSFS_DEV_ATTR(config_write_delay_msec_y),+TEST_SYSFS_DEV_ATTR(config_enable_busy_alloc),++NULL,+};++ATTRIBUTE_GROUPS(test_dev);++staticintsysfs_test_dev_alloc_miscdev(structsysfs_test_device*test_dev)+{+structmiscdevice*misc_dev;++misc_dev=&test_dev->misc_dev;+misc_dev->minor=MISC_DYNAMIC_MINOR;+misc_dev->name=kasprintf(GFP_KERNEL,"test_sysfs%d",test_dev->dev_idx);+if(!misc_dev->name){+pr_err("Cannot alloc misc_dev->name\n");+return-ENOMEM;+}+misc_dev->groups=test_dev_groups;++return0;+}++staticinttestdev_open(structblock_device*bdev,fmode_tmode)+{+return-EINVAL;+}++staticblk_qc_ttestdev_submit_bio(structbio*bio)+{+returnBLK_QC_T_NONE;+}++staticvoidtestdev_slot_free_notify(structblock_device*bdev,+unsignedlongindex)+{+}++staticinttestdev_rw_page(structblock_device*bdev,sector_tsector,+structpage*page,unsignedintop)+{+return-EOPNOTSUPP;+}++staticconststructblock_device_operationssysfs_testdev_ops={+.open=testdev_open,+.submit_bio=testdev_submit_bio,+.swap_slot_free_notify=testdev_slot_free_notify,+.rw_page=testdev_rw_page,+.owner=THIS_MODULE+};++staticintsysfs_test_dev_alloc_blockdev(structsysfs_test_device*test_dev)+{+intret=-ENOMEM;++test_dev->disk=blk_alloc_disk(NUMA_NO_NODE);+if(!test_dev->disk){+pr_err("Error allocating disk structure for device %d\n",+test_dev->dev_idx);+gotoout;+}++test_dev->disk->major=sysfs_test_major;+test_dev->disk->first_minor=test_dev->dev_idx+1;+test_dev->disk->fops=&sysfs_testdev_ops;+test_dev->disk->private_data=test_dev;+snprintf(test_dev->disk->disk_name,16,"test_sysfs%d",+test_dev->dev_idx);+set_capacity(test_dev->disk,0);+blk_queue_flag_set(QUEUE_FLAG_NONROT,test_dev->disk->queue);+blk_queue_flag_clear(QUEUE_FLAG_ADD_RANDOM,test_dev->disk->queue);+blk_queue_physical_block_size(test_dev->disk->queue,PAGE_SIZE);+blk_queue_max_discard_sectors(test_dev->disk->queue,UINT_MAX);+blk_queue_flag_set(QUEUE_FLAG_DISCARD,test_dev->disk->queue);++return0;+out:+returnret;+}++staticstructsysfs_test_device*alloc_test_dev_sysfs(intidx)+{+structsysfs_test_device*test_dev;+intret;++switch(test_devtype){+caseTESTDEV_TYPE_MISC:+fallthrough;+caseTESTDEV_TYPE_BLOCK:+break;+default:+returnNULL;+}++test_dev=kzalloc(sizeof(structsysfs_test_device),GFP_KERNEL);+if(!test_dev)+gotoerr_out;++mutex_init(&test_dev->config_mutex);+test_dev->dev_idx=idx;+test_dev->devtype=test_devtype;++if(test_dev->devtype==TESTDEV_TYPE_MISC){+ret=sysfs_test_dev_alloc_miscdev(test_dev);+if(ret)+gotoerr_out_free;+}elseif(test_dev->devtype==TESTDEV_TYPE_BLOCK){+ret=sysfs_test_dev_alloc_blockdev(test_dev);+if(ret)+gotoerr_out_free;+}+returntest_dev;++err_out_free:+kfree(test_dev);+test_dev=NULL;+err_out:+returnNULL;+}++staticintregister_test_dev_sysfs_misc(structsysfs_test_device*test_dev)+{+intret;++ret=misc_register(&test_dev->misc_dev);+if(ret)+returnret;++test_dev->dev=test_dev->misc_dev.this_device;++return0;+}++staticintregister_test_dev_sysfs_block(structsysfs_test_device*test_dev)+{+device_add_disk(NULL,test_dev->disk,test_dev_groups);+test_dev->dev=disk_to_dev(test_dev->disk);++return0;+}++staticstructsysfs_test_device*register_test_dev_sysfs(void)+{+structsysfs_test_device*test_dev=NULL;+intret;++test_dev=alloc_test_dev_sysfs(0);+if(!test_dev)+gotoout;++if(test_dev->devtype==TESTDEV_TYPE_MISC){+ret=register_test_dev_sysfs_misc(test_dev);+if(ret){+pr_err("could not register misc device: %d\n",ret);+gotoout_free_dev;+}+}elseif(test_dev->devtype==TESTDEV_TYPE_BLOCK){+ret=register_test_dev_sysfs_block(test_dev);+if(ret){+pr_err("could not register block device: %d\n",ret);+gotoout_free_dev;+}+}++dev_info(test_dev->dev,"interface ready\n");++out:+returntest_dev;+out_free_dev:+free_test_dev_sysfs(test_dev);+returnNULL;+}++staticstructsysfs_test_device*register_test_dev_set_config(void)+{+structsysfs_test_device*test_dev;+structtest_config*config;++test_dev=register_test_dev_sysfs();+if(!test_dev)+returnNULL;++config=&test_dev->config;++if(enable_lock)+config->enable_lock=true;+if(enable_lock_on_rmmod)+config->enable_lock_on_rmmod=true;+if(use_rtnl_lock)+config->use_rtnl_lock=true;+if(enable_busy_alloc)+config->enable_busy_alloc=true;++config->write_delay_msec_y=write_delay_msec_y;+test_sysfs_reset_vals(test_dev);++returntest_dev;+}++staticvoidunregister_test_dev_sysfs_misc(structsysfs_test_device*test_dev)+{+misc_deregister(&test_dev->misc_dev);+}++staticvoidunregister_test_dev_sysfs_block(structsysfs_test_device*test_dev)+{+del_gendisk(test_dev->disk);+blk_cleanup_disk(test_dev->disk);+}++staticvoidunregister_test_dev_sysfs(structsysfs_test_device*test_dev)+{+test_dev_config_lock_rmmod(test_dev);++dev_info(test_dev->dev,"removing interface\n");++if(test_dev->devtype==TESTDEV_TYPE_MISC)+unregister_test_dev_sysfs_misc(test_dev);+elseif(test_dev->devtype==TESTDEV_TYPE_BLOCK)+unregister_test_dev_sysfs_block(test_dev);++test_dev_config_unlock_rmmod(test_dev);++free_test_dev_sysfs(test_dev);+}++staticstructdentry*debugfs_dir;++/* When read represents how many times we have reset the first_test_dev */+staticu8reset_first_test_dev;++staticssize_tread_reset_first_test_dev(structfile*file,+char__user*user_buf,+size_tcount,loff_t*ppos)+{+ssize_tlen;+charbuf[32];++reset_first_test_dev++;+len=sprintf(buf,"%d\n",reset_first_test_dev);+returnsimple_read_from_buffer(user_buf,count,ppos,buf,len);+}++staticssize_twrite_reset_first_test_dev(structfile*file,+constchar__user*user_buf,+size_tcount,loff_t*ppos)+{+if(!try_module_get(THIS_MODULE))+return-ENODEV;++if(!first_test_dev){+module_put(THIS_MODULE);+return-ENODEV;+}++dev_info(first_test_dev->dev,"going to reset first interface ...\n");++unregister_test_dev_sysfs(first_test_dev);+first_test_dev=register_test_dev_set_config();++dev_info(first_test_dev->dev,"first interface reset complete\n");++module_put(THIS_MODULE);++returncount;+}++staticconststructfile_operationsfops_reset_first_test_dev={+.read=read_reset_first_test_dev,+.write=write_reset_first_test_dev,+.open=simple_open,+.owner=THIS_MODULE,+.llseek=default_llseek,+};++staticint__inittest_sysfs_init(void)+{+first_test_dev=register_test_dev_set_config();+if(!first_test_dev)+return-ENOMEM;++if(!enable_debugfs)+return0;++debugfs_dir=debugfs_create_dir("test_sysfs",NULL);+if(!debugfs_dir){+unregister_test_dev_sysfs(first_test_dev);+return-ENOMEM;+}++debugfs_create_file("reset_first_test_dev",0600,debugfs_dir,+NULL,&fops_reset_first_test_dev);+return0;+}+module_init(test_sysfs_init);++staticvoid__exittest_sysfs_exit(void)+{+if(enable_debugfs)+debugfs_remove(debugfs_dir);+if(delay_rmmod_ms)+msleep(delay_rmmod_ms);+unregister_test_dev_sysfs(first_test_dev);+if(enable_verbose_rmmod)+pr_info("unregister_test_dev_sysfs() completed\n");+first_test_dev=NULL;+}+module_exit(test_sysfs_exit);++MODULE_AUTHOR("Luis Chamberlain <mcgrof@kernel.org>");+MODULE_LICENSE("GPL");
@@ -0,0 +1,12 @@+# SPDX-License-Identifier: GPL-2.0-only+# Makefile for sysfs selftests.++# No binaries, but make sure arg-less "make" doesn't trigger "run_tests".+all:++TEST_PROGS:=sysfs.sh++include ../lib.mk++# Nothing to clean up.+clean:
@@ -0,0 +1,1208 @@+#!/bin/bash+# SPDX-License-Identifier: GPL-2.0-or-later+# Copyright (C) 2021 Luis Chamberlain <mcgrof@kernel.org>+#+# This program is free software; you can redistribute it and/or modify it+# under the terms of the GNU General Public License as published by the Free+# Software Foundation; either version 2 of the License, or at your option any+# later version; or, when distributed separately from the Linux kernel or+# when incorporated into other software packages, subject to the following+# license:+#+# This program is free software; you can redistribute it and/or modify it+# under the terms of copyleft-next (version 0.3.1 or later) as published+# at http://copyleft-next.org/.++# This performs a series tests against the sysfs filesystem.++# Kselftest framework requirement - SKIP code is 4.+ksft_skip=4++TEST_NAME="sysfs"+TEST_DRIVER="test_${TEST_NAME}"+TEST_DIR=$(dirname$0)+TEST_FILE=$(mktemp)++# This represents+#+# TEST_ID:TEST_COUNT:ENABLED:TARGET+#+# TEST_ID: is the test id number+# TEST_COUNT: number of times we should run the test+# ENABLED: 1 if enabled, 0 otherwise+# TARGET: test target file required on the test_sysfs module+#+# Once these are enabled please leave them as-is. Write your own test,+# we have tons of space.+ALL_TESTS="0001:3:1:test_dev_x:misc"+ALL_TESTS="$ALL_TESTS 0002:3:1:test_dev_y:misc"+ALL_TESTS="$ALL_TESTS 0003:3:1:test_dev_x:misc"+ALL_TESTS="$ALL_TESTS 0004:3:1:test_dev_y:misc"+ALL_TESTS="$ALL_TESTS 0005:1:1:test_dev_x:misc"+ALL_TESTS="$ALL_TESTS 0006:1:1:test_dev_y:misc"+ALL_TESTS="$ALL_TESTS 0007:1:1:test_dev_y:misc"+ALL_TESTS="$ALL_TESTS 0008:1:1:test_dev_x:misc"+ALL_TESTS="$ALL_TESTS 0009:1:1:test_dev_y:misc"+ALL_TESTS="$ALL_TESTS 0010:1:1:test_dev_y:misc"+ALL_TESTS="$ALL_TESTS 0011:1:1:test_dev_x:misc"+ALL_TESTS="$ALL_TESTS 0012:1:1:test_dev_y:misc"+ALL_TESTS="$ALL_TESTS 0013:1:1:test_dev_y:misc"+ALL_TESTS="$ALL_TESTS 0014:3:1:test_dev_x:block"# block equivalent set+ALL_TESTS="$ALL_TESTS 0015:3:1:test_dev_x:block"+ALL_TESTS="$ALL_TESTS 0016:3:1:test_dev_x:block"+ALL_TESTS="$ALL_TESTS 0017:3:1:test_dev_y:block"+ALL_TESTS="$ALL_TESTS 0018:1:1:test_dev_x:block"+ALL_TESTS="$ALL_TESTS 0019:1:1:test_dev_y:block"+ALL_TESTS="$ALL_TESTS 0020:1:1:test_dev_y:block"+ALL_TESTS="$ALL_TESTS 0021:1:1:test_dev_x:block"+ALL_TESTS="$ALL_TESTS 0022:1:1:test_dev_y:block"+ALL_TESTS="$ALL_TESTS 0023:1:1:test_dev_y:block"+ALL_TESTS="$ALL_TESTS 0024:1:1:test_dev_x:block"+ALL_TESTS="$ALL_TESTS 0025:1:1:test_dev_y:block"+ALL_TESTS="$ALL_TESTS 0026:1:1:test_dev_y:block"+ALL_TESTS="$ALL_TESTS 0027:1:0:test_dev_x:block"# deadlock test+ALL_TESTS="$ALL_TESTS 0028:1:0:test_dev_x:block"# deadlock test with rntl_lock++allow_user_defaults()+{+if[-z$DIR];then+case$TEST_DEV_TYPEin+misc)+DIR="/sys/devices/virtual/misc/${TEST_DRIVER}0"+;;+block)+DIR="/sys/devices/virtual/block/${TEST_DRIVER}0"+;;+*)+DIR="/sys/devices/virtual/misc/${TEST_DRIVER}0"+;;+esac+fi+case$TEST_DEV_TYPEin+misc)+MODPROBE_TESTDEV_TYPE=""+;;+block)+MODPROBE_TESTDEV_TYPE="devtype=1"+;;+*)+MODPROBE_TESTDEV_TYPE=""+;;+esac+if[-z$SYSFS_DEBUGFS_DIR];then+SYSFS_DEBUGFS_DIR="/sys/kernel/debug/test_sysfs"+fi+if[-z$PAGE_SIZE];then+PAGE_SIZE=$(getconfPAGESIZE)+fi+if[-z$MAX_DIGITS];then+MAX_DIGITS=$(($PAGE_SIZE/8))+fi+if[-z$INT_MAX];then+INT_MAX=$(getconfINT_MAX)+fi+if[-z$UINT_MAX];then+UINT_MAX=$(getconfUINT_MAX)+fi+}++test_reqs()+{+uid=$(id-u)+if[$uid-ne0];then+echo$msgmustberunasroot>&2+exit$ksft_skip+fi++if!whichmodprobe2>/dev/null>/dev/null;then+echo"$0: You need modprobe installed">&2+exit$ksft_skip+fi+if!whichgetconf2>/dev/null>/dev/null;then+echo"$0: You need getconf installed"+exit$ksft_skip+fi+if!whichdiff2>/dev/null>/dev/null;then+echo"$0: You need diff installed"+exit$ksft_skip+fi+if!whichperl2>/dev/null>/dev/null;then+echo"$0: You need perl installed"+exit$ksft_skip+fi+}++call_modprobe()+{+modprobe$TEST_DRIVER$MODPROBE_TESTDEV_TYPE$FIRST_MODPROBE_ARGS$MODPROBE_ARGS+return$?+}++modprobe_reset()+{+modprobe-q-r$TEST_DRIVER+call_modprobe+return$?+}++modprobe_reset_enable_debugfs()+{+FIRST_MODPROBE_ARGS="enable_debugfs=1"+modprobe_reset+unsetFIRST_MODPROBE_ARGS+}++modprobe_reset_enable_lock_on_rmmod()+{+FIRST_MODPROBE_ARGS="enable_lock=1 enable_lock_on_rmmod=1 enable_verbose_writes=1"+modprobe_reset+unsetFIRST_MODPROBE_ARGS+}++modprobe_reset_enable_rtnl_lock_on_rmmod()+{+FIRST_MODPROBE_ARGS="enable_lock=1 use_rtnl_lock=1 enable_lock_on_rmmod=1"+FIRST_MODPROBE_ARGS="$FIRST_MODPROBE_ARGS enable_verbose_writes=1"+modprobe_reset+unsetFIRST_MODPROBE_ARGS+}++load_req_mod()+{+modprobe_reset+if[!-d$DIR];then+if!modprobe-q-n$TEST_DRIVER;then+echo"$0: module $TEST_DRIVER not found [SKIP]"+echo"You must set CONFIG_TEST_SYSFS=m in your kernel">&2+exit$ksft_skip+fi+call_modprobe+if[$?-ne0];then+echo"$0: modprobe $TEST_DRIVER failed."+exit+fi+fi+}++config_reset()+{+if!echo-n"1">"$DIR"/reset;then+echo"$0: reset should have worked">&2+exit1+fi+}++debugfs_reset_first_test_dev_ignore_errors()+{+echo-n"1">"$SYSFS_DEBUGFS_DIR"/reset_first_test_dev+}++set_orig()+{+if[[!-z$TARGET]]&&[[!-z$ORIG]];then+if[-f${TARGET}];then+echo"${ORIG}">"${TARGET}"+fi+fi+}++set_test()+{+echo"${TEST_STR}">"${TARGET}"+}++set_test_ignore_errors()+{+echo"${TEST_STR}">"${TARGET}"2>/dev/null+}++verify()+{+localseen+seen=$(cat"$1")+target_short=$(basename$TARGET)+case$target_shortin+test_dev_x)+if["${seen}"!="${TEST_STR}"];then+return1+fi+;;+test_dev_y)+DIRNAME=$(dirname$1)+EXPECTED_RESULT=""+# If our target was the test file then what we write to it+# is the same as what that we expect when we read from it.+# When we write to test_dev_y directly though we expect+# a computed value which is driver specific.+if[["$DIRNAME"=="/tmp"]];then+letEXPECTED_RESULT="${TEST_STR}"+else+x=$(cat${DIR}/test_dev_x)+letEXPECTED_RESULT="$x+${TEST_STR}+7"+fi++if[["${seen}"!="${EXPECTED_RESULT}"]];then+return1+fi+;;+*)+echo"Unsupported target type update test script: $target_short"+exit1+esac+return0+}++verify_diff_w()+{+echo"$TEST_STR"|diff-q-w-u-$1>/dev/null+return$?+}++test_rc()+{+if[[$rc!=0]];then+echo"Failed test, return value: $rc">&2+exit$rc+fi+}++test_finish()+{+set_orig+rm-f"${TEST_FILE}"++if[!-z${old_strict}];then+echo${old_strict}>${WRITES_STRICT}+fi+exit$rc+}++# kernfs requires us to write everything we want in one shot because+# There is no easy way for us to know if userspace is only doing a partial+# write, so we don't support them. We expect the entire buffer to come on+# the first write. If you're writing a value, first read the file,+# modify only the value you're changing, then write entire buffer back.+# Since we are only testing digits we just full single writes and old stuff.+# For more details, refer to kernfs_fop_write_iter().+run_numerictests_single_write()+{+echo"== Testing sysfs behavior against ${TARGET} =="++rc=0++echo-n"Writing test file ... "+echo"${TEST_STR}">"${TEST_FILE}"+if!verify"${TEST_FILE}";then+echo"FAIL">&2+exit1+else+echo"ok"+fi++echo-n"Checking the sysfs file is not set to test value ... "+ifverify"${TARGET}";then+echo"FAIL">&2+exit1+else+echo"ok"+fi++echo-n"Writing to sysfs file from shell ... "+set_test+if!verify"${TARGET}";then+echo"FAIL">&2+exit1+else+echo"ok"+fi++echo-n"Resetting sysfs file to original value ... "+set_orig+ifverify"${TARGET}";then+echo"FAIL">&2+exit1+else+echo"ok"+fi++# Now that we've validated the sanity of "set_test" and "set_orig",+# we can use those functions to set starting states before running+# specific behavioral tests.++echo-n"Writing to the entire sysfs file in a single write ... "+set_orig+ddif="${TEST_FILE}"of="${TARGET}"bs=40962>/dev/null+if!verify"${TARGET}";then+echo"FAIL">&2+rc=1+else+echo"ok"+fi++echo-n"Writing to the sysfs file with multiple long writes ... "+set_orig+(perl-e'print "A" x 50;';echo"${TEST_STR}")|\+ddof="${TARGET}"bs=502>/dev/null+ifverify"${TARGET}";then+echo"FAIL">&2+rc=1+else+echo"ok"+fi+test_rc+}++reset_vals()+{+echo-n3>$DIR/test_dev_x+echo-n4>$DIR/test_dev_x+}++check_failure()+{+echo-n"Testing that $1 fails as expected..."+reset_vals+TEST_STR="$1"+orig="$(cat$TARGET)"+echo-n"$TEST_STR">$TARGET2>/dev/null++# write should fail and $TARGET should retain its original value+if[$?=0]||["$(cat$TARGET)"!="$orig"];then+echo"FAIL">&2+rc=1+else+echo"ok"+fi+test_rc+}++load_modreqs()+{+exportTEST_DEV_TYPE=$(get_test_type$1)+unsetDIR+allow_user_defaults+load_req_mod+}++target_exists()+{+TARGET="${DIR}/$1"+TEST_ID="$2"++if[!-f${TARGET}];then+echo"Target for test $TEST_ID: $TARGET does not exist, skipping test ..."+return0+fi+return1+}++config_enable_lock()+{+if!echo-n1>$DIR/config_enable_lock;then+echo"$0: Unable to enable locks">&2+exit1+fi+}++config_write_delay_msec_y()+{+if!echo-n$1>$DIR/config_write_delay_msec_y;then+echo"$0: Unable to set write_delay_msec_y to $1">&2+exit1+fi+}++# Default filter for dmesg scanning.+# Ignore lockdep complaining about its own bugginess when scanning dmesg+# output, because we shouldn't be failing filesystem tests on account of+# lockdep.+_check_dmesg_filter()+{+egrep-v-e"BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low"\+-e"BUG: MAX_STACK_TRACE_ENTRIES too low"+}++check_dmesg()+{+# filter out intentional WARNINGs or Oopses+localfilter=${1:-_check_dmesg_filter}++_dmesg_since_test_start|$filter>$seqres.dmesg+egrep-q-e"kernel BUG at"\+-e"WARNING:"\+-e"\bBUG:"\+-e"Oops:"\+-e"possible recursive locking detected"\+-e"Internal error"\+-e"(INFO|ERR): suspicious RCU usage"\+-e"INFO: possible circular locking dependency detected"\+-e"general protection fault:"\+-e"BUG .* remaining"\+-e"UBSAN:"\+$seqres.dmesg+if[$?-eq0];then+echo"something found in dmesg (see $seqres.dmesg)"+return1+else+if["$KEEP_DMESG"!="yes"];then+rm-f$seqres.dmesg+fi+return0+fi+}++log_kernel_fstest_dmesg()+{+exportFSTYP="$1"+exportseqnum="$FSTYP/$2"+exportdate_time=$(date+"%F %T")+echo"run fstests $seqnum at $date_time">/dev/kmsg+}++modprobe_loop()+{+whiletrue;do+call_modprobe>/dev/null2>&1+modprobe-r$TEST_DRIVER>/dev/null2>&1+done>/dev/null2>&1+}++write_loop()+{+whiletrue;do+set_test_ignore_errors>/dev/null2>&1+TEST_STR=$(($TEST_STR+1))+done>/dev/null2>&1+}++write_loop_reset()+{+whiletrue;do+set_test_ignore_errors>/dev/null2>&1+debugfs_reset_first_test_dev_ignore_errors>/dev/null2>&1+done>/dev/null2>&1+}++write_loop_bg()+{+BG_WRITES=1000>/dev/null2>&1+whiletrue;do+foriin$(seq1$BG_WRITES);do+set_test_ignore_errors>/dev/null2>&1&+TEST_STR=$(($TEST_STR+1))+done>/dev/null2>&1+wait+done>/dev/null2>&1+wait+}++reset_loop()+{+whiletrue;do+debugfs_reset_first_test_dev_ignore_errors>/dev/null2>&1+done>/dev/null2>&1+}++kill_trigger_loop()+{++localmy_first_loop_pid=$1+localmy_second_loop_pid=$2+localmy_sleep_max=$3+localmy_loop=0++whiletrue;do+sleep1+if[[$my_loop-ge$my_sleep_max]];then+break+fi+letmy_loop=$my_loop+1+done++kill-sTERM$my_first_loop_pid2>&1>/dev/null+kill-sTERM$my_second_loop_pid2>&1>/dev/null+}++_dmesg_since_test_start()+{+# search the dmesg log of last run of $seqnum for possible failures+# use sed \cregexpc address type, since $seqnum contains "/"+dmesg|tac|sed-ne"0,\#run fstests $seqnum at $date_time#p"|tac+}++sysfs_test_0001()+{+TARGET="${DIR}/$(get_test_target0001)"+config_reset+reset_vals+ORIG=$(cat"${TARGET}")+TEST_STR=$(($ORIG+1))++run_numerictests_single_write+}++sysfs_test_0002()+{+TARGET="${DIR}/$(get_test_target0002)"+config_reset+ORIG=$(cat"${TARGET}")+TEST_STR=$(($ORIG+1))++run_numerictests_single_write+}++sysfs_test_0003()+{+TARGET="${DIR}/$(get_test_target0003)"+config_reset+ORIG=$(cat"${TARGET}")+TEST_STR=$(($ORIG+1))++config_enable_lock++run_numerictests_single_write+}++sysfs_test_0004()+{+TARGET="${DIR}/$(get_test_target0004)"+config_reset+ORIG=$(cat"${TARGET}")+TEST_STR=$(($ORIG+1))++config_enable_lock++run_numerictests_single_write+}++sysfs_test_0005()+{+TARGET="${DIR}/$(get_test_target0005)"+modprobe_reset+config_reset+ORIG=$(cat"${TARGET}")+TEST_STR=$(($ORIG+1))+WAIT_TIME=2++echo-n"Loop writing x while loading/unloading the module... "++modprobe_loop&+modprobe_pid=$!++write_loop&+write_pid=$!++kill_trigger_loop$modprobe_pid$write_pid$WAIT_TIME>/dev/null2>&1&+kill_pid=$!++wait$kill_pid>/dev/null2>&1++if[[$?-eq0]];then+echo"ok"+else+echo"FAIL">&2+fi+}++sysfs_test_0006()+{+TARGET="${DIR}/$(get_test_target0006)"+modprobe_reset+config_reset+ORIG=$(cat"${TARGET}")+TEST_STR=$(($ORIG+1))+WAIT_TIME=2++echo-n"Loop writing y while loading/unloading the module... "+modprobe_loop&+modprobe_pid=$!++write_loop&+write_pid=$!++kill_trigger_loop$modprobe_pid$write_pid$WAIT_TIME>/dev/null2>&1&+kill_pid=$!++wait$kill_pid>/dev/null2>&1++if[[$?-eq0]];then+echo"ok"+else+echo"FAIL">&2+fi+}++sysfs_test_0007()+{+TARGET="${DIR}/$(get_test_target0007)"+modprobe_reset+config_reset+ORIG=$(cat"${TARGET}")+TEST_STR=$(($ORIG+1))+WAIT_TIME=2++echo-n"Loop writing y with a larger delay while loading/unloading the module... "++MODPROBE_ARGS="write_delay_msec_y=1500"+modprobe_loop>/dev/null2>&1&+modprobe_pid=$!+unsetMODPROBE_ARGS++write_loop&+write_pid=$!++kill_trigger_loop$modprobe_pid$write_pid$WAIT_TIME>/dev/null2>&1&+kill_pid=$!++wait$kill_pid>/dev/null2>&1++if[[$?-eq0]];then+echo"ok"+else+echo"FAIL">&2+fi+}++sysfs_test_0008()+{+TARGET="${DIR}/$(get_test_target0008)"+modprobe_reset+config_reset+reset_vals+ORIG=$(cat"${TARGET}")+TEST_STR=$(($ORIG+1))+WAIT_TIME=2++echo-n"Loop busy writing x while loading/unloading the module... "++modprobe_loop>/dev/null2>&1&+modprobe_pid=$!++write_loop_bg>/dev/null2>&1&+write_pid=$!++kill_trigger_loop$modprobe_pid$write_pid$WAIT_TIME>/dev/null2>&1&+kill_pid=$!++wait$kill_pid>/dev/null2>&1++if[[$?-eq0]];then+echo"ok"+else+echo"FAIL">&2+fi+}++sysfs_test_0009()+{+TARGET="${DIR}/$(get_test_target0009)"+modprobe_reset+config_reset+reset_vals+ORIG=$(cat"${TARGET}")+TEST_STR=$(($ORIG+1))+WAIT_TIME=2++echo-n"Loop busy writing y while loading/unloading the module... "++modprobe_loop>/dev/null2>&1&+modprobe_pid=$!++write_loop_bg>/dev/null2>&1&+write_pid=$!++kill_trigger_loop$modprobe_pid$write_pid$WAIT_TIME>/dev/null2>&1&+kill_pid=$!++wait$kill_pid>/dev/null2>&1++if[[$?-eq0]];then+echo"ok"+else+echo"FAIL">&2+fi+}++sysfs_test_0010()+{+TARGET="${DIR}/$(get_test_target0010)"+modprobe_reset+config_reset+reset_vals+ORIG=$(cat"${TARGET}")+TEST_STR=$(($ORIG+1))+WAIT_TIME=2++echo-n"Loop busy writing y with a larger delay while loading/unloading the module... "+modprobe-q-r$TEST_DRIVER>/dev/null2>&1++MODPROBE_ARGS="write_delay_msec_y=1500"+modprobe_loop>/dev/null2>&1&+modprobe_pid=$!+unsetMODPROBE_ARGS++write_loop_bg>/dev/null2>&1&+write_pid=$!++kill_trigger_loop$modprobe_pid$write_pid$WAIT_TIME>/dev/null2>&1&+kill_pid=$!++wait$kill_pid>/dev/null2>&1++if[[$?-eq0]];then+echo"ok"+else+echo"FAIL">&2+fi+}++sysfs_test_0011()+{+TARGET="${DIR}/$(get_test_target0011)"+modprobe_reset_enable_debugfs+config_reset+reset_vals+ORIG=$(cat"${TARGET}")+TEST_STR=$(($ORIG+1))+WAIT_TIME=2++echo-n"Loop writing x and resetting ... "++write_loop>/dev/null2>&1&+write_pid=$!++reset_loop>/dev/null2>&1&+reset_pid=$!++kill_trigger_loop$write_pid$reset_pid$WAIT_TIME>/dev/null2>&1&+kill_pid=$!++wait$kill_pid>/dev/null2>&1++if[[$?-eq0]];then+echo"ok"+else+echo"FAIL">&2+fi+}++sysfs_test_0012()+{+TARGET="${DIR}/$(get_test_target0012)"+modprobe_reset_enable_debugfs+config_reset+reset_vals+ORIG=$(cat"${TARGET}")+TEST_STR=$(($ORIG+1))+WAIT_TIME=2++echo-n"Loop writing y and resetting ... "++write_loop>/dev/null2>&1&+write_pid=$!++reset_loop>/dev/null2>&1&+reset_pid=$!++kill_trigger_loop$write_pid$reset_pid$WAIT_TIME>/dev/null2>&1&+kill_pid=$!++wait$kill_pid>/dev/null2>&1++if[[$?-eq0]];then+echo"ok"+else+echo"FAIL">&2+fi+}++sysfs_test_0013()+{+TARGET="${DIR}/$(get_test_target0013)"+modprobe_reset_enable_debugfs+config_reset+reset_vals+config_write_delay_msec_y1500+ORIG=$(cat"${TARGET}")+TEST_STR=$(($ORIG+1))+WAIT_TIME=2++echo-n"Loop writing y with a larger delay and resetting ... "++write_loop>/dev/null2>&1&+write_pid=$!++reset_loop>/dev/null2>&1&+reset_pid=$!++kill_trigger_loop$write_pid$reset_pid$WAIT_TIME>/dev/null2>&1&+kill_pid=$!++wait$kill_pid>/dev/null2>&1++if[[$?-eq0]];then+echo"ok"+else+echo"FAIL">&2+fi+}++sysfs_test_0014()+{+sysfs_test_0001+}++sysfs_test_0015()+{+sysfs_test_0002+}++sysfs_test_0016()+{+sysfs_test_0003+}++sysfs_test_0017()+{+sysfs_test_0004+}++sysfs_test_0018()+{+sysfs_test_0005+}++sysfs_test_0019()+{+sysfs_test_0006+}++sysfs_test_0020()+{+sysfs_test_0007+}++sysfs_test_0021()+{+sysfs_test_0008+}++sysfs_test_0022()+{+sysfs_test_0009+}++sysfs_test_0023()+{+sysfs_test_0010+}++sysfs_test_0024()+{+sysfs_test_0011+}++sysfs_test_0025()+{+sysfs_test_0012+}++sysfs_test_0026()+{+sysfs_test_0013+}++sysfs_test_0027()+{+TARGET="${DIR}/$(get_test_target0027)"+modprobe_reset_enable_lock_on_rmmod+ORIG=$(cat"${TARGET}")+TEST_STR=$(($ORIG+1))+WAIT_TIME=2++echo-n"Test for possible rmmod deadlock while writing x ... "++write_loop>/dev/null2>&1&+write_pid=$!++MODPROBE_ARGS="enable_lock=1 enable_lock_on_rmmod=1 enable_verbose_writes=1"+modprobe_loop>/dev/null2>&1&+modprobe_pid=$!+unsetMODPROBE_ARGS++kill_trigger_loop$modprobe_pid$write_pid$WAIT_TIME>/dev/null2>&1&+kill_pid=$!++wait$kill_pid>/dev/null2>&1++if[[$?-eq0]];then+echo"ok"+else+echo"FAIL">&2+fi+}++sysfs_test_0028()+{+TARGET="${DIR}/$(get_test_target0028)"+modprobe_reset_enable_lock_on_rmmod+ORIG=$(cat"${TARGET}")+TEST_STR=$(($ORIG+1))+WAIT_TIME=2++echo-n"Test for possible rmmod deadlock using rtnl_lock while writing x ... "++write_loop>/dev/null2>&1&+write_pid=$!++MODPROBE_ARGS="enable_lock=1 enable_lock_on_rmmod=1 use_rtnl_lock=1 enable_verbose_writes=1"+modprobe_loop>/dev/null2>&1&+modprobe_pid=$!+unsetMODPROBE_ARGS++kill_trigger_loop$modprobe_pid$write_pid$WAIT_TIME>/dev/null2>&1&+kill_pid=$!++wait$kill_pid>/dev/null2>&1++if[[$?-eq0]];then+echo"ok"+else+echo"FAIL">&2+fi+}++test_gen_desc()+{+echo-n"$1 x $(get_test_count$1)"+}++list_tests()+{+echo"Test ID list:"+echo+echo"TEST_ID x NUM_TEST"+echo"TEST_ID: Test ID"+echo"NUM_TESTS: Number of recommended times to run the test"+echo+echo"$(test_gen_desc0001) - misc test writing x in different ways"+echo"$(test_gen_desc0002) - misc test writing y in different ways"+echo"$(test_gen_desc0003) - misc test writing x in different ways using a mutex lock"+echo"$(test_gen_desc0004) - misc test writing y in different ways using a mutex lock"+echo"$(test_gen_desc0005) - misc test writing x load and remove the test_sysfs module"+echo"$(test_gen_desc0006) - misc writing y load and remove the test_sysfs module"+echo"$(test_gen_desc0007) - misc test writing y larger delay, load, remove test_sysfs"+echo"$(test_gen_desc0008) - misc test busy writing x remove test_sysfs module"+echo"$(test_gen_desc0009) - misc test busy writing y remove the test_sysfs module"+echo"$(test_gen_desc0010) - misc test busy writing y larger delay, remove test_sysfs"+echo"$(test_gen_desc0011) - misc test writing x and resetting device"+echo"$(test_gen_desc0012) - misc test writing y and resetting device"+echo"$(test_gen_desc0013) - misc test writing y with a larger delay and resetting device"+echo"$(test_gen_desc0014) - block test writing x in different ways"+echo"$(test_gen_desc0015) - block test writing y in different ways"+echo"$(test_gen_desc0016) - block test writing x in different ways using a mutex lock"+echo"$(test_gen_desc0017) - block test writing y in different ways using a mutex lock"+echo"$(test_gen_desc0018) - block test writing x load and remove the test_sysfs module"+echo"$(test_gen_desc0019) - block test writing y load and remove the test_sysfs module"+echo"$(test_gen_desc0020) - block test writing y larger delay, load, remove test_sysfs"+echo"$(test_gen_desc0021) - block test busy writing x remove the test_sysfs module"+echo"$(test_gen_desc0022) - block test busy writing y remove the test_sysfs module"+echo"$(test_gen_desc0023) - block test busy writing y larger delay, remove test_sysfs"+echo"$(test_gen_desc0024) - block test writing x and resetting device"+echo"$(test_gen_desc0025) - block test writing y and resetting device"+echo"$(test_gen_desc0026) - block test writing y larger delay and resetting device"+echo"$(test_gen_desc0027) - test rmmod deadlock while writing x ... "+echo"$(test_gen_desc0028) - test rmmod deadlock using rtnl_lock while writing x ..."+}++usage()+{+NUM_TESTS=$(grep-o' '<<<"$ALL_TESTS"|grep-c.)+letNUM_TESTS=$NUM_TESTS+1+MAX_TEST=$(printf"%04d\n"$NUM_TESTS)+echo"Usage: $0 [ -t <4-number-digit> ] | [ -w <4-number-digit> ] |"+echo" [ -s <4-number-digit> ] | [ -c <4-number-digit> <test- count>"+echo" [ all ] [ -h | --help ] [ -l ]"+echo""+echo"Valid tests: 0001-$MAX_TEST"+echo""+echo" all Runs all tests (default)"+echo" -t Run test ID the number amount of times is recommended"+echo" -w Watch test ID run until it runs into an error"+echo" -c Run test ID once"+echo" -s Run test ID x test-count number of times"+echo" -l List all test ID list"+echo" -h|--help Help"+echo+echo"If an error every occurs execution will immediately terminate."+echo"If you are adding a new test try using -w <test-ID> first to"+echo"make sure the test passes a series of tests."+echo+echoExampleuses:+echo+echo"$TEST_NAME.sh -- executes all tests"+echo"$TEST_NAME.sh -t 0002 -- Executes test ID 0002 number of times is recomended"+echo"$TEST_NAME.sh -w 0002 -- Watch test ID 0002 run until an error occurs"+echo"$TEST_NAME.sh -s 0002 -- Run test ID 0002 once"+echo"$TEST_NAME.sh -c 0002 3 -- Run test ID 0002 three times"+echo+list_tests+exit1+}++test_num()+{+re='^[0-9]+$'+if![[$1=~$re]];then+usage+fi+}++get_test_count()+{+test_num$1+TEST_NUM=$(echo$1|sed's/^0*//')+TEST_DATA=$(echo$ALL_TESTS|awk'{print $'$TEST_NUM'}')+echo${TEST_DATA}|awk-F":"'{print $2}'+}++get_test_enabled()+{+test_num$1+TEST_NUM=$(echo$1|sed's/^0*//')+TEST_DATA=$(echo$ALL_TESTS|awk'{print $'$TEST_NUM'}')+echo${TEST_DATA}|awk-F":"'{print $3}'+}++get_test_target()+{+test_num$1+TEST_NUM=$(echo$1|sed's/^0*//')+TEST_DATA=$(echo$ALL_TESTS|awk'{print $'$TEST_NUM'}')+echo${TEST_DATA}|awk-F":"'{print $4}'+}++get_test_type()+{+test_num$1+TEST_NUM=$(echo$1|sed's/^0*//')+TEST_DATA=$(echo$ALL_TESTS|awk'{print $'$TEST_NUM'}')+echo${TEST_DATA}|awk-F":"'{print $5}'+}++run_all_tests()+{+foriin$ALL_TESTS;do+TEST_ID=$(echo$i|awk-F":"'{print $1}')+ENABLED=$(get_test_enabled$TEST_ID)+TEST_COUNT=$(get_test_count$TEST_ID)+TEST_TARGET=$(get_test_target$TEST_ID)+if[[$ENABLED-eq"1"]];then+test_case$TEST_ID$TEST_COUNT$TEST_TARGET+else+echo-n"Skipping test $TEST_ID as its disabled, likely "+echo"could crash your system ..."+fi+done+}++watch_log()+{+if[$#-ne3];then+clear+fi+echo"Running test: $2 - run #$1"+}++watch_case()+{+i=0+while[1];do+if[$#-eq1];then+test_num$1+watch_log$i${TEST_NAME}_test_$1+log_kernel_fstest_dmesgsysfs$1+RUN_TEST=${TEST_NAME}_test_$1+$RUN_TEST+check_dmesg+if[[$?-ne0]];then+exit1+fi+else+watch_log$iall+run_all_tests+fi+leti=$i+1+done+}++test_case()+{+NUM_TESTS=$2++i=0++load_modreqs$1+iftarget_exists$3$1;then+return+fi++while[[$i-lt$NUM_TESTS]];do+test_num$1+watch_log$i${TEST_NAME}_test_$1noclear+log_kernel_fstest_dmesgsysfs$1+RUN_TEST=${TEST_NAME}_test_$1+$RUN_TEST+leti=$i+1+done+check_dmesg+if[[$?-ne0]];then+exit1+fi+}++parse_args()+{+if[$#-eq0];then+run_all_tests+else+if[["$1"="all"]];then+run_all_tests+elif[["$1"="-w"]];then+shift+watch_case$@+elif[["$1"="-t"]];then+shift+test_num$1+test_case$1$(get_test_count$1)$(get_test_target$1)+shift+elif[["$1"="-c"]];then+shift+test_num$1+test_num$2+test_case$1$2$(get_test_target$1)+shift+shift+elif[["$1"="-s"]];then+shift+test_case$11$(get_test_target$1)+shift+elif[["$1"="-l"]];then+list_tests+shift+elif[["$1"="-h"||"$1"="--help"]];then+usage+else+usage+fi+fi+}++test_reqs+allow_user_defaults++trap"test_finish"EXIT++parse_args$@++exit0
On Mon, Sep 27, 2021 at 09:38:02AM -0700, Luis Chamberlain wrote:
When driver sysfs attributes use a lock also used on module removal we
can race to deadlock. This happens when for instance a sysfs file on
a driver is used, then at the same time we have module removal call
trigger. The module removal call code holds a lock, and then the
driver's sysfs file entry waits for the same lock. While holding the
lock the module removal tries to remove the sysfs entries, but these
cannot be removed yet as one is waiting for a lock. This won't complete
as the lock is already held. Likewise module removal cannot complete,
and so we deadlock.
This can now be easily reproducible with our sysfs selftest as follows:
./tools/testing/selftests/sysfs/sysfs.sh -t 0027
This uses a local driver lock. Test 0028 can also be used, that uses
the rtnl_lock():
./tools/testing/selftests/sysfs/sysfs.sh -t 0028
To fix this we extend the struct kernfs_node with a module reference
and use the try_module_get() after kernfs_get_active() is called. As
documented in the prior patch, we now know that once kernfs_get_active()
is called the module is implicitly guarded to exist and cannot be removed.
This is because the module is the one in charge of removing the same
sysfs file it created, and removal of sysfs files on module exit will wait
until they don't have any active references. By using a try_module_get()
after kernfs_get_active() we yield to let module removal trump calls to
process a sysfs operation, while also preventing module removal if a sysfs
operation is in already progress. This prevents the deadlock.
This deadlock was first reported with the zram driver, however the live
Looks not see the lock pattern you mentioned in zram driver, can you
share the related zram code?
patching folks have acknowledged they have observed this as well with
live patching, when a live patch is removed. I was then able to
reproduce easily by creating a dedicated selftest for it.
A sketch of how this can happen follows, consider foo a local mutex
part of a driver, and used on the driver's module exit routine and
on one of its sysfs ops:
foo.c:
static DEFINE_MUTEX(foo);
static ssize_t foo_store(struct device *dev,
struct device_attribute *attr,
const char *buf, size_t count)
{
...
mutex_lock(&foo);
...
mutex_lock(&foo);
...
}
static DEVICE_ATTR_RW(foo);
...
void foo_exit(void)
{
mutex_lock(&foo);
...
mutex_unlock(&foo);
}
module_exit(foo_exit);
And this can lead to this condition:
CPU A CPU B
foo_store()
foo_exit()
mutex_lock(&foo)
mutex_lock(&foo)
del_gendisk(some_struct->disk);
device_del()
device_remove_groups()
I guess the deadlock exists if foo_exit() is called anywhere. If yes,
look the issue may not be related with removing module directly, right?
Thanks,
Ming
On Mon, Sep 27, 2021 at 09:37:56AM -0700, Luis Chamberlain wrote:
quoted hunk
--- /dev/null+++ b/lib/test_sysfs.c
@@ -0,0 +1,921 @@+// SPDX-License-Identifier: GPL-2.0-or-later OR copyleft-next-0.3.1+/*+*sysfstestdriver+*+*Copyright(C)2021LuisChamberlain<mcgrof@kernel.org>+*+*Thisprogramisfreesoftware;youcanredistributeitand/ormodifyit+*underthetermsoftheGNUGeneralPublicLicenseaspublishedbytheFree+*SoftwareFoundation;eitherversion2oftheLicense,oratyouroptionany+*laterversion;or,whendistributedseparatelyfromtheLinuxkernelor+*whenincorporatedintoothersoftwarepackages,subjecttothefollowing+*license:+*+*Thisprogramisfreesoftware;youcanredistributeitand/ormodifyit+*underthetermsofcopyleft-next(version0.3.1orlater)aspublished+*athttp://copyleft-next.org/.
Independant of the fact that I don't like sysfs code attempting to be
accessed in the kernel with licenses other than GPLv2, you do not need
the license "boilerplate" text at all in files. That's what the SPDX
line is for.
thanks,
greg k-h
On Mon, Sep 27, 2021 at 09:38:01AM -0700, Luis Chamberlain wrote:
If one ends up expanding on this line checkpatch will complain that the
combination S_IRWXU|S_IRUGO|S_IXUGO should just be replaced with the
octal 0755. Do that.
This makes no functional changes.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
On Mon, Sep 27, 2021 at 09:37:55AM -0700, Luis Chamberlain wrote:
Two selftests drivers exist under the copyleft-next license.
These drivers were added prior to SPDX practice taking full swing
in the kernel. Now that we have an SPDX tag for copylef-next-0.3.1
documented, embrace it and remove the boiler plate.
Cc: Goldwyn Rodrigues <redacted>
Cc: Kuno Woudt <redacted>
Cc: Richard Fontana <redacted>
Cc: copyleft-next@lists.fedorahosted.org
Cc: Ciaran Farrell <redacted>
Cc: Christopher De Nicolo <redacted>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Thomas Gleixner <redacted>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Thorsten Leemhuis <linux@leemhuis.info>
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
You're the primary author, and it cleans up boilerplate, so LGTM.
Reviewed-by: Kees Cook <redacted>
--
Kees Cook
-----Original Message-----
From: Greg KH <gregkh@linuxfoundation.org>
Sent: Tuesday, October 5, 2021 8:17 AM
To: Luis Chamberlain <mcgrof@kernel.org>
Cc: tj@kernel.org; akpm@linux-foundation.org; minchan@kernel.org; jeyu@kernel.org; shuah@kernel.org; bvanassche@acm.org;
dan.j.williams@intel.com; joe@perches.com; tglx@linutronix.de; keescook@chromium.org; rostedt@goodmis.org; linux-
spdx@vger.kernel.org; linux-doc@vger.kernel.org; linux-block@vger.kernel.org; linux-fsdevel@vger.kernel.org; linux-
kselftest@vger.kernel.org; linux-kernel@vger.kernel.org
Subject: Re: [PATCH v8 03/12] selftests: add tests_sysfs module
On Mon, Sep 27, 2021 at 09:37:56AM -0700, Luis Chamberlain wrote:
quoted
--- /dev/null+++ b/lib/test_sysfs.c
@@ -0,0 +1,921 @@+// SPDX-License-Identifier: GPL-2.0-or-later OR copyleft-next-0.3.1+/*+*sysfstestdriver+*+*Copyright(C)2021LuisChamberlain<mcgrof@kernel.org>+*+*Thisprogramisfreesoftware;youcanredistributeitand/ormodifyit+*underthetermsoftheGNUGeneralPublicLicenseaspublishedbytheFree+*SoftwareFoundation;eitherversion2oftheLicense,oratyouroptionany+*laterversion;or,whendistributedseparatelyfromtheLinuxkernelor+*whenincorporatedintoothersoftwarepackages,subjecttothefollowing+*license:
This is a very strange license grant, which I'm not sure is covered by any
current SPDX syntax.
" when distributed separately from the Linux kernel or when incorporated into
other software packages, subject to the following license:"
Why would we care about the license used when the code is used in a non-kernel
project? If it is desired for the code to be available outside the kernel under a
different license, then surely the easiest thing is to make it available separately
under that license. I'm not sure why the kernel needs to carry this license for
non-kernel use of the code.
I would recommend giving this a GPLv2 SPDX header, and maybe in the comment
at the top of the file put a reference to a git repository where the code can be
obtained under a different license.
Just my 2 cents.
-- Tim
quoted
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of copyleft-next (version 0.3.1 or later) as published
+ * at http://copyleft-next.org/.
Independant of the fact that I don't like sysfs code attempting to be
accessed in the kernel with licenses other than GPLv2, you do not need
the license "boilerplate" text at all in files. That's what the SPDX
line is for.
thanks,
greg k-h
On Mon, Sep 27, 2021 at 09:37:57AM -0700, Luis Chamberlain wrote:
This adds initial failure injection support to kernfs. We start
off with debug knobs which when enabled allow test drivers, such as
test_sysfs, to then make use of these to try to force certain
difficult races to take place with a high degree of certainty.
This only adds runtime code *iff* the new bool CONFIG_FAIL_KERNFS_KNOBS is
enabled in your kernel. If you don't have this enabled this provides
no new functional. When CONFIG_FAIL_KERNFS_KNOBS is disabled the new
routine kernfs_debug_should_wait() ends up being transformed to if
(false), and so the compiler should optimize these out as dead code
producing no new effective binary changes.
We start off with enabling failure injections in kernfs by allowing us to
alter the way kernfs_fop_write_iter() behaves. We allow for the routine
kernfs_fop_write_iter() to wait for a certain condition in the kernel to
occur, after which it will sleep a predefined amount of time. This lets
kernfs users to time exactly when it want kernfs_fop_write_iter() to
complete, allowing for developing race conditions and test for correctness
in kernfs.
You'd boot with this enabled on your kernel command line:
fail_kernfs_fop_write_iter=1,100,0,1
The values are <interval,probability,size,times>, we don't care for
size, so for now we ignore it. The above ensures a failure will trigger
only once.
*How* we allow for this routine to change behaviour is left to knobs we
expose under debugfs:
# ls -1 /sys/kernel/debug/kernfs/config_fail_kernfs_fop_write_iter/
I'd expect this to live under /sys/kernel/debug/fail_kernfs, like the
other fault injectors.
quoted hunk
wait_after_active
wait_after_mutex
wait_at_start
wait_before_mutex
A debugfs entry also exists to allow us to sleep a configurabler amount
of time after the completion:
/sys/kernel/debug/kernfs/sleep_after_wait_ms
These two sets of knobs allow us to construct races and demonstrate
how the kernfs active reference should suffice to project against
races.
Enabling CONFIG_FAULT_INJECTION_DEBUG_FS enables us to configure the
differnt fault injection parametres for the new fail_kernfs_fop_write_iter
fault injection at run time:
ls -1 /sys/kernel/debug/kernfs/fail_kernfs_fop_write_iter/
interval
probability
space
times
task-filter
verbose
verbose_ratelimit_burst
verbose_ratelimit_interval_ms
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
.../fault-injection/fault-injection.rst | 22 +++++
MAINTAINERS | 2 +-
fs/kernfs/Makefile | 1 +
fs/kernfs/failure-injection.c | 91 +++++++++++++++++++
fs/kernfs/file.c | 13 +++
fs/kernfs/kernfs-internal.h | 72 +++++++++++++++
include/linux/kernfs.h | 5 +
lib/Kconfig.debug | 10 ++
8 files changed, 215 insertions(+), 1 deletion(-)
create mode 100644 fs/kernfs/failure-injection.c
@@ -28,6 +28,28 @@ Available fault injection capabilities injects kernel RPC client and server failures.+- fail_kernfs_fop_write_iter++ Allows for failures to be enabled inside kernfs_fop_write_iter(). Enabling+ this does not immediately enable any errors to occur. You must configure+ how you want this routine to fail or change behaviour by using the debugfs+ knobs for it:++ # ls -1 /sys/kernel/debug/kernfs/config_fail_kernfs_fop_write_iter/+ wait_after_active+ wait_after_mutex+ wait_at_start+ wait_before_mutex
This should be split up and detailed in the "debugfs entries" section
below here.
quoted hunk
+
+ You can also configure how long to sleep after a wait under
+
+ /sys/kernel/debug/kernfs/sleep_after_wait_ms
+
+ If you enable CONFIG_FAULT_INJECTION_DEBUG_FS the fail_add_disk failure
+ injection parameters are placed under:
+
+ /sys/kernel/debug/kernfs/fail_kernfs_fop_write_iter/
+
- fail_make_request
injects disk IO errors on devices permitted by setting
I'd name this fault_inject.c, which matches the more common case:
$ find . -type f -name '*fault*inject*.c'
./fs/nfsd/fault_inject.c
./drivers/nvme/host/fault_inject.c
./drivers/scsi/ufs/ufs-fault-injection.c
./lib/fault-inject.c
./lib/fault-inject-usercopy.c
Every caller ends up doing the wait, so how about just including that
here instead? It should make things much less intrusive and more readable.
And for the naming, other fault injectors use "should_fail_$topic", so
maybe better here would be something like may_wait_kernfs(...).
+
+DECLARE_COMPLETION(kernfs_debug_wait_completion);
+EXPORT_SYMBOL_NS_GPL(kernfs_debug_wait_completion, KERNFS_DEBUG_PRIVATE);
+
+void kernfs_debug_wait(void)
+{
+ unsigned long timeout;
+
+ timeout = wait_for_completion_timeout(&kernfs_debug_wait_completion,
+ msecs_to_jiffies(3000));
+ if (!timeout)
+ pr_info("%s waiting for kernfs_debug_wait_completion timed out\n",
+ __func__);
+ else
+ pr_info("%s received completion with time left on timeout %u ms\n",
+ __func__, jiffies_to_msecs(timeout));
+
+ /**
+ * The goal is wait for an event, and *then* once we have
+ * reached it, the other side will try to do something which
+ * it thinks will break. So we must give it some time to do
+ * that. The amount of time is configurable.
+ */
+ msleep(kernfs_config_fail.sleep_after_wait_ms);
+ pr_info("%s ended\n", __func__);
+}
All the uses of "__func__" here seems redundant; I would drop them.
So this could just be:
may_wait_kernfs(kernfs_fop_write_iter, at_start);
quoted hunk
+
if (of->atomic_write_len) {
if (len > of->atomic_write_len)
return -E2BIG;
@@ -280,17 +283,27 @@ static ssize_t kernfs_fop_write_iter(struct kiocb *iocb, struct iov_iter *iter) } buf[len] = '\0'; /* guarantee string termination */+ if (kernfs_debug_should_wait(kernfs_fop_write_iter, before_mutex))+ kernfs_debug_wait();+ /* * @of->mutex nests outside active ref and is used both to ensure that * the ops aren't called concurrently for the same open file. */ mutex_lock(&of->mutex);++ if (kernfs_debug_should_wait(kernfs_fop_write_iter, after_mutex))+ kernfs_debug_wait();+ if (!kernfs_get_active(of->kn)) { mutex_unlock(&of->mutex); len = -ENODEV; goto out_free; }+ if (kernfs_debug_should_wait(kernfs_fop_write_iter, after_active))+ kernfs_debug_wait();+ ops = kernfs_ops(of->kn); if (ops->write) len = ops->write(of, buf, len, iocb->ki_pos);
@@ -1902,6 +1902,16 @@ config FAULT_INJECTION_USERCOPYProvidesfault-injectioncapabilitytoinjectfailuresinusercopyfunctions(copy_from_user(),get_user(),...).+configFAIL_KERNFS_KNOBS+bool"Fault-injection support in kernfs"+depends onFAULT_INJECTION+help+Providefault-injectioncapabilityforkernfs.Thisonlyenables+theerrorinjectionfunctionality.Touseityoumustconfigurewhich+whichpathyouwanttotriggeronerroronusingdebugfsunder+/sys/kernel/debug/kernfs/config_fail_kernfs_fop_write_iter/.By+defaultallofthesearedisabled.+configFAIL_MAKE_REQUESTbool"Fault-injection capability for disk IO"depends onFAULT_INJECTION&&BLOCK
On Mon, Sep 27, 2021 at 09:37:58AM -0700, Luis Chamberlain wrote:
quoted hunk
This extends test_sysfs with support for using the failure injection
wait completion and knobs to force a few race conditions which
demonstrates that kernfs active reference protection is sufficient
for kobject / device protection at higher layers.
This adds 4 new tests which tries to remove the device attribute
store operation in 4 different situations:
1) at the start of kernfs_kernfs_fop_write_iter()
2) before the of->mutex is held in kernfs_kernfs_fop_write_iter()
3) after the of->mutex is held in kernfs_kernfs_fop_write_iter()
4) after the kernfs node active reference is taken
A write fails in call cases except the last one, test number #32. There
is a good explanation for this: *once* kernfs_get_active() gets called
we have a guarantee that the kernfs entry cannot be removed. If
kernfs_get_active() succeeds that entry cannot be removed and so
anything trying to remove that entry will have to wait. It is perhaps
not obvious but since a sysfs write will trigger eventually a
kernfs_get_active() call, and *only* if this succeeds will the sysfs
op be called, this and the fact that you cannot remove the kernfs
entry while the kenfs entry is active implies that a module that
created the respective sysfs / kernfs entry *cannot* possibly be
removed during a sysfs operation. And test number 32 provides us with
proof of this. If it were not true test #32 should crash.
No null dereferences are reproduced, even though this has been observed
in some complex testing cases [0]. If this issue really exists we should
have enough tools on the sysfs_test toolbox now to try to reproduce
this easily without having to poke around other drivers. It very likley
was the case that the issue reported [0] was possibly a side issue after
the first bug which was zram specific. This is why it is important to
isolate the issue and try to reproduce it in a generic form using the
test_sysfs driver.
[0] https://lkml.kernel.org/r/20210623215007.862787-1-mcgrof@kernel.org
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
lib/Kconfig.debug | 3 +
lib/test_sysfs.c | 31 +++++
tools/testing/selftests/sysfs/config | 3 +
tools/testing/selftests/sysfs/sysfs.sh | 175 +++++++++++++++++++++++++
4 files changed, 212 insertions(+)
@@ -904,10 +922,23 @@ static int __init test_sysfs_init(void) } module_init(test_sysfs_init);+#ifdef CONFIG_FAIL_KERNFS_KNOBS+/* The goal is to race our device removal with a pending kernfs -> store call */+static void test_sysfs_kernfs_send_completion_rmmod(void)+{+ if (!enable_completion_on_rmmod)+ return;+ complete(&kernfs_debug_wait_completion);+}+#else+static inline void test_sysfs_kernfs_send_completion_rmmod(void) {}+#endif+ static void __exit test_sysfs_exit(void) { if (enable_debugfs) debugfs_remove(debugfs_dir);+ test_sysfs_kernfs_send_completion_rmmod(); if (delay_rmmod_ms) msleep(delay_rmmod_ms); unregister_test_dev_sysfs(first_test_dev);
@@ -197,6 +212,63 @@ debugfs_reset_first_test_dev_ignore_errors()echo-n"1">"$SYSFS_DEBUGFS_DIR"/reset_first_test_dev}+debugfs_kernfs_kernfs_fop_write_iter_exists()+{+KNOB_DIR="${KERNFS_DEBUGFS_DIR}/config_fail_kernfs_fop_write_iter"+if[[!-d$KNOB_DIR]];then+echo"kernfs debugfs does not exist $KNOB_DIR"+return0;+fi+KNOB_DEBUGFS="${KERNFS_DEBUGFS_DIR}/fail_kernfs_fop_write_iter"+if[[!-d$KNOB_DEBUGFS]];then+echo-n"kernfs debugfs for coniguring fail_kernfs_fop_write_iter "+echo"does not exist $KNOB_DIR"+return0;+fi+return1+}++debugfs_kernfs_kernfs_fop_write_iter_set_fail_once()+{+KNOB_DEBUGFS="${KERNFS_DEBUGFS_DIR}/fail_kernfs_fop_write_iter"+echo1>$KNOB_DEBUGFS/interval+echo100>$KNOB_DEBUGFS/probability+echo0>$KNOB_DEBUGFS/space+# Disable verbose messages on the kernel ring buffer which may+# confuse developers with a kernel panic.+echo0>$KNOB_DEBUGFS/verbose++# Fail only once+echo1>$KNOB_DEBUGFS/times+}++debugfs_kernfs_kernfs_fop_write_iter_set_fail_never()+{+KNOB_DEBUGFS="${KERNFS_DEBUGFS_DIR}/fail_kernfs_fop_write_iter"+echo0>$KNOB_DEBUGFS/times+}++debugfs_kernfs_set_wait_ms()+{+SLEEP_AFTER_WAIT_MS="${KERNFS_DEBUGFS_DIR}/sleep_after_wait_ms"+echo$1>$SLEEP_AFTER_WAIT_MS+}++debugfs_kernfs_disable_wait_kernfs_fop_write_iter()+{+ENABLE_WAIT_KNOB="${KERNFS_DEBUGFS_DIR}/config_fail_kernfs_fop_write_iter/wait_"+forKNOBin${ENABLE_WAIT_KNOB}*;do+echo0>$KNOB+done+}++debugfs_kernfs_enable_wait_kernfs_fop_write_iter()+{+ENABLE_WAIT_KNOB="${KERNFS_DEBUGFS_DIR}/config_fail_kernfs_fop_write_iter/wait_$1"+echo-n"1">$ENABLE_WAIT_KNOB+return$?+}+ set_orig(){if[[!-z$TARGET]]&&[[!-z$ORIG]];then
@@ -972,6 +1044,105 @@ sysfs_test_0028()fi}+sysfs_race_kernfs_kernfs_fop_write_iter()+{+TARGET="${DIR}/$(get_test_target$1)"+WAIT_AT=$2+EXPECT_WRITE_RETURNS=$3+MSDELAY=$4++modprobe_reset_enable_completion+ORIG=$(cat"${TARGET}")+TEST_STR=$(($ORIG+1))++echo-n"Test racing removal of sysfs store op with kernfs $WAIT_AT ... "++ifdebugfs_kernfs_kernfs_fop_write_iter_exists;then+echo-n"skipping test as CONFIG_FAIL_KERNFS_KNOBS "+echo" or CONFIG_FAULT_INJECTION_DEBUG_FS is disabled"+return$ksft_skip+fi++# Allow for failing the kernfs_kernfs_fop_write_iter call once,+# we'll provide exact context shortly afterwards.+debugfs_kernfs_kernfs_fop_write_iter_set_fail_once++# First disable all waits+debugfs_kernfs_disable_wait_kernfs_fop_write_iter++# Enable a wait_for_completion(&kernfs_debug_wait_completion) at the+# specified location inside the kernfs_fop_write_iter() routine+debugfs_kernfs_enable_wait_kernfs_fop_write_iter$WAIT_AT++# Configure kernfs so that after its wait_for_completion() it+# will msleep() this amount of time and schedule(). We figure this+# will be sufficient time to allow for our module removal to complete.+debugfs_kernfs_set_wait_ms$MSDELAY++# Now we trigger a kernfs write op, which will run kernfs_fop_write_iter,+# but will wait until our driver sends a respective completion+set_test_ignore_errors&+write_pid=$!++# At this point kernfs_fop_write_iter() hasn't run our op, its+# waiting for our completion at the specified time $WAIT_AT.+# We now remove our module which will send a+# complete(&kernfs_debug_wait_completion) right before we deregister+# our device and the sysfs device attributes are removed.+#+# After the completion is sent, the test_sysfs driver races with+# kernfs to do the device deregistration with the kernfs msleep+# and schedule(). This should mean we've forced trying to remove the+# module prior to allowing kernfs to run our store operation. If the+# race did happen we'll panic with a null dereference on the store op.+#+# If no race happens we should see no write operation triggered.+modprobe-r$TEST_DRIVER>/dev/null2>&1++debugfs_kernfs_kernfs_fop_write_iter_set_fail_never++wait$write_pid+if[[$?-eq$EXPECT_WRITE_RETURNS]];then+echo"ok"+else+echo"FAIL">&2+fi+}++sysfs_test_0029()+{+fordelayin02481632641282465121024;do+echo"Using delay-after-completion: $delay"+sysfs_race_kernfs_kernfs_fop_write_iter0029at_start1$delay+done+}++sysfs_test_0030()+{+fordelayin02481632641282465121024;do+echo"Using delay-after-completion: $delay"+sysfs_race_kernfs_kernfs_fop_write_iter0030before_mutex1$delay+done+}++sysfs_test_0031()+{+fordelayin02481632641282465121024;do+echo"Using delay-after-completion: $delay"+sysfs_race_kernfs_kernfs_fop_write_iter0031after_mutex1$delay+done+}++# A write only succeeds *iff* a module removal happens *after* the+# kernfs active reference is obtained with kernfs_get_active().+sysfs_test_0032()+{+fordelayin02481632641282465121024;do+echo"Using delay-after-completion: $delay"+sysfs_race_kernfs_kernfs_fop_write_iter0032after_active0$delay+done+}+ test_gen_desc(){echo-n"$1 x $(get_test_count$1)"
@@ -1013,6 +1184,10 @@ list_tests()echo"$(test_gen_desc0026) - block test writing y larger delay and resetting device"echo"$(test_gen_desc0027) - test rmmod deadlock while writing x ... "echo"$(test_gen_desc0028) - test rmmod deadlock using rtnl_lock while writing x ..."+echo"$(test_gen_desc0029) - racing removal of store op with kernfs at start"+echo"$(test_gen_desc0030) - racing removal of store op with kernfs before mutex"+echo"$(test_gen_desc0031) - racing removal of store op with kernfs after mutex"+echo"$(test_gen_desc0032) - racing removal of store op with kernfs after active"} usage()
On Mon, Sep 27, 2021 at 09:37:59AM -0700, Luis Chamberlain wrote:
quoted hunk
There is quite a bit of tribal knowledge around proper use of
try_module_get() and that it must be used only in a context which
can ensure the module won't be gone during the operation. Document
this little bit of tribal knowledge.
I'm extending this tribal knowledge with new developments which it
seems some folks do not yet believe to be true: we can be sure a
module will exist during the lifetime of a sysfs file operation.
For proof, refer to test_sysfs test #32:
./tools/testing/selftests/sysfs/sysfs.sh -t 0032
Without this being true, the write would fail or worse,
a crash would happen, in this test. It does not.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
include/linux/module.h | 34 ++++++++++++++++++++++++++++++++--
1 file changed, 32 insertions(+), 2 deletions(-)
@@ -609,10 +609,40 @@ void symbol_put_addr(void *addr);tohandletheerrorcase(whichonlyhappenswithrmmod--wait).*/externvoid__module_get(structmodule*module);-/* This is the Right Way to get a module: if it fails, it's being removed,-*sopretendit'snotthere.*/+/**+*try_module_get()-yieldstomoduleremovalandbumpsrefcntotherwise
I find this hard to parse. How about:
"Take module refcount unless module is being removed"
+ * @module: the module we should check for
+ *
+ * This can be used to try to bump the reference count of a module, so to
+ * prevent module removal. The reference count of a module is not allowed
+ * to be incremented if the module is already being removed.
This I understand.
+ *
+ * Care must be taken to ensure the module cannot be removed during the call to
+ * try_module_get(). This can be done by having another entity other than the
+ * module itself increment the module reference count, or through some other
+ * means which guarantees the module could not be removed during an operation.
+ * An example of this later case is using try_module_get() in a sysfs file
+ * which the module created. The sysfs store / read file operations are
+ * gauranteed to exist through the use of kernfs's active reference (see
+ * kernfs_active()). If a sysfs file operation is being run, the module which
+ * created it must still exist as the module is in charge of removing the same
+ * sysfs file being read. Also, a sysfs / kernfs file removal cannot happen
+ * unless the same file is not active.
I can't understand this paragraph at all. "Care must be taken ..."? Why?
Shouldn't callers of try_module_get() be satisfied with the results? I
don't follow the example at all. It seems to just say "sysfs store/read
functions don't need try_module_get() because whatever opened the sysfs
file is already keeping the module referenced." ?
+ *
+ * One of the real values to try_module_get() is the module_is_live() check
+ * which ensures this the caller of try_module_get() can yield to userspace
+ * module removal requests and fail whatever it was about to process.
Please document the return value explicitly.
+ */
extern bool try_module_get(struct module *module);
+/**
+ * module_put() - release a reference count to a module
+ * @module: the module we should release a reference count for
+ *
+ * If you successfully bump a reference count to a module with try_module_get(),
+ * when you are finished you must call module_put() to release that reference
+ * count.
+ */
extern void module_put(struct module *module);
#else /*!CONFIG_MODULE_UNLOAD*/
--
2.30.2
On Mon, Sep 27, 2021 at 09:38:00AM -0700, Luis Chamberlain wrote:
If one ends up extending this line checkpatch will complain about the
use of S_IRWXUGO suggesting it is not preferred and that 0777
should be used instead. Take the tip from checkpatch and do that
change before we do our subsequent changes.
This makes no functional changes.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
On Mon, Sep 27, 2021 at 09:38:02AM -0700, Luis Chamberlain wrote:
When driver sysfs attributes use a lock also used on module removal we
can race to deadlock. This happens when for instance a sysfs file on
a driver is used, then at the same time we have module removal call
trigger. The module removal call code holds a lock, and then the
driver's sysfs file entry waits for the same lock. While holding the
lock the module removal tries to remove the sysfs entries, but these
cannot be removed yet as one is waiting for a lock. This won't complete
as the lock is already held. Likewise module removal cannot complete,
and so we deadlock.
This can now be easily reproducible with our sysfs selftest as follows:
./tools/testing/selftests/sysfs/sysfs.sh -t 0027
This uses a local driver lock. Test 0028 can also be used, that uses
the rtnl_lock():
./tools/testing/selftests/sysfs/sysfs.sh -t 0028
To fix this we extend the struct kernfs_node with a module reference
and use the try_module_get() after kernfs_get_active() is called. As
I would agree: kernfs must know about the module containing the ops
structure it has been given. (Without this, there are, at the very least,
removal races for looking at kernfs_ops structures.)
In other places in the kernel, function callback dependencies are more
explicit in that if code is holding such things, it has already taken a
module reference, etc. But kernfs is special in the sense that just
because a kernfs entry exists, we don't want to pin the module use count
too.
But simple locking isn't workable to solve this because kernfs_remove()
must be able to be called from a module_exit routine without deadlocking.
(i.e. we would create exactly the situation that caused this condition
to get noticed in the first place.)
documented in the prior patch, we now know that once kernfs_get_active()
is called the module is implicitly guarded to exist and cannot be removed.
This is because the module is the one in charge of removing the same
sysfs file it created, and removal of sysfs files on module exit will wait
until they don't have any active references. By using a try_module_get()
after kernfs_get_active() we yield to let module removal trump calls to
process a sysfs operation, while also preventing module removal if a sysfs
operation is in already progress. This prevents the deadlock.
This deadlock was first reported with the zram driver, however the live
patching folks have acknowledged they have observed this as well with
live patching, when a live patch is removed. I was then able to
reproduce easily by creating a dedicated selftest for it.
A sketch of how this can happen follows, consider foo a local mutex
part of a driver, and used on the driver's module exit routine and
on one of its sysfs ops:
foo.c:
static DEFINE_MUTEX(foo);
static ssize_t foo_store(struct device *dev,
struct device_attribute *attr,
const char *buf, size_t count)
{
...
mutex_lock(&foo);
...
mutex_lock(&foo);
...
}
static DEVICE_ATTR_RW(foo);
...
void foo_exit(void)
{
mutex_lock(&foo);
...
mutex_unlock(&foo);
}
module_exit(foo_exit);
And this can lead to this condition:
CPU A CPU B
foo_store()
foo_exit()
mutex_lock(&foo)
mutex_lock(&foo)
del_gendisk(some_struct->disk);
device_del()
device_remove_groups()
Please expand this further, where does device_remove_groups() end up
waiting for that never happens?
quoted hunk
In this situation foo_store() is waiting for the mutex foo to
become unlocked, but that won't happen until module removal is complete.
But module removal won't complete until the sysfs file being poked at
completes which is waiting for a lock already held.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
arch/x86/kernel/cpu/resctrl/rdtgroup.c | 4 +-
fs/kernfs/dir.c | 44 ++++++++++++++++++----
fs/kernfs/file.c | 6 ++-
fs/kernfs/kernfs-internal.h | 3 +-
fs/kernfs/symlink.c | 3 +-
fs/sysfs/dir.c | 2 +-
fs/sysfs/file.c | 6 ++-
fs/sysfs/group.c | 3 +-
include/linux/kernfs.h | 14 ++++---
include/linux/sysfs.h | 52 ++++++++++++++++++++------
kernel/cgroup/cgroup.c | 2 +-
11 files changed, 105 insertions(+), 34 deletions(-)
The special casing in here makes me think this isn't happening the right
place. (i.e this looks like an open-coded version of kernfs_put_active())
quoted hunk
+
if (kernfs_lockdep(kn))
rwsem_acquire_read(&kn->dep_map, 0, 1, _RET_IP_);
return kn;
@@ -442,6 +460,13 @@ void kernfs_put_active(struct kernfs_node *kn) if (kernfs_lockdep(kn)) rwsem_release(&kn->dep_map, _RET_IP_); v = atomic_dec_return(&kn->active);++ /*+ * We prevent module exit *until* we know for sure all possible+ * kernfs ops are done.+ */+ module_put(kn->owner);+ if (likely(v != KN_DEACTIVATED_BIAS)) return;
What I don't understand, however, is what kernfs_get/put_active() is
intending to do -- it looks like it's trying to provide an interruption
point for open kernfs file operations?
This all seems extremely complex for what seems like it should just be a
global "am I being removed?" bool?
Regardless, while I do see the logic of associating the module get/put
with get/put of kernfs "active", why is it not better tied to strictly
kernfs open/close? That would seem to be much simpler and not require
any special handling?
For example, why does this not work?
On Mon, Sep 27, 2021 at 09:38:04AM -0700, Luis Chamberlain wrote:
Provide a simple state machine to fix races with driver exit where we
remove the CPU multistate callbacks and re-initialization / creation of
new per CPU instances which should be managed by these callbacks.
The zram driver makes use of cpu hotplug multistate support, whereby it
associates a struct zcomp per CPU. Each struct zcomp represents a
compression algorithm in charge of managing compression streams per
CPU. Although a compiled zram driver only supports a fixed set of
compression algorithms, each zram device gets a struct zcomp allocated
per CPU. The "multi" in CPU hotplug multstate refers to these per
cpu struct zcomp instances. Each of these will have the CPU hotplug
callback called for it on CPU plug / unplug. The kernel's CPU hotplug
multistate keeps a linked list of these different structures so that
it will iterate over them on CPU transitions.
By default at driver initialization we will create just one zram device
(num_devices=1) and a zcomp structure then set for the now default
lzo-rle comrpession algorithm. At driver removal we first remove each
zram device, and so we destroy the associated struct zcomp per CPU. But
since we expose sysfs attributes to create new devices or reset /
initialize existing zram devices, we can easily end up re-initializing
a struct zcomp for a zram device before the exit routine of the module
removes the cpu hotplug callback. When this happens the kernel's CPU
hotplug will detect that at least one instance (struct zcomp for us)
exists. This can happen in the following situation:
CPU 1 CPU 2
disksize_store(...);
class_unregister(...);
idr_for_each(...);
zram_debugfs_destroy();
idr_destroy(...);
unregister_blkdev(...);
cpuhp_remove_multi_state(...);
So this is strictly separate from the sysfs/module unloading race?
-Kees
quoted hunk
The warning comes up on cpuhp_remove_multi_state() when it sees that the
state for CPUHP_ZCOMP_PREPARE does not have an empty instance linked list.
In this case, that a struct zcom still exists, the driver allowed its
creation per CPU even though we could have just freed them per CPU
though a call on another CPU, and we are then later trying to remove the
hotplug callback.
Fix all this by providing a zram initialization boolean
protected the shared in the driver zram_index_mutex, which we
can use to annotate when sysfs attributes are safe to use or
not -- once the driver is properly initialized. When the driver
is going down we also are sure to not let userspace muck with
attributes which may affect each per cpu struct zcomp.
This also fixes a series of possible memory leaks. The
crashes and memory leaks can easily be caused by issuing
the zram02.sh script from the LTP project [0] in a loop
in two separate windows:
cd testcases/kernel/device-drivers/zram
while true; do PATH=$PATH:$PWD:$PWD/../../../lib/ ./zram02.sh; done
You end up with a splat as follows:
kernel: zram: Removed device: zram0
kernel: zram: Added device: zram0
kernel: zram0: detected capacity change from 0 to 209715200
kernel: Adding 104857596k swap on /dev/zram0. <etc>
kernel: zram0: detected capacitky change from 209715200 to 0
kernel: zram0: detected capacity change from 0 to 209715200
kernel: ------------[ cut here ]------------
kernel: Error: Removing state 63 which has instances left.
kernel: WARNING: CPU: 7 PID: 70457 at \
kernel/cpu.c:2069 __cpuhp_remove_state_cpuslocked+0xf9/0x100
kernel: Modules linked in: zram(E-) zsmalloc(E) <etc>
kernel: CPU: 7 PID: 70457 Comm: rmmod Tainted: G \
E 5.12.0-rc1-next-20210304 #3
kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), \
BIOS 1.14.0-2 04/01/2014
kernel: RIP: 0010:__cpuhp_remove_state_cpuslocked+0xf9/0x100
kernel: Code: <etc>
kernel: RSP: 0018:ffffa800c139be98 EFLAGS: 00010282
kernel: RAX: 0000000000000000 RBX: ffffffff9083db58 RCX: ffff9609f7dd86d8
kernel: RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffff9609f7dd86d0
kernel: RBP: 0000000000000000i R08: 0000000000000000 R09: ffffa800c139bcb8
kernel: R10: ffffa800c139bcb0 R11: ffffffff908bea40 R12: 000000000000003f
kernel: R13: 00000000000009d8 R14: 0000000000000000 R15: 0000000000000000
kernel: FS: 00007f1b075a7540(0000) GS:ffff9609f7dc0000(0000) knlGS:<etc>
kernel: CS: 0010 DS: 0000 ES 0000 CR0: 0000000080050033
kernel: CR2: 00007f1b07610490 CR3: 00000001bd04e000 CR4: 0000000000350ee0
kernel: Call Trace:
kernel: __cpuhp_remove_state+0x2e/0x80
kernel: __do_sys_delete_module+0x190/0x2a0
kernel: do_syscall_64+0x33/0x80
kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae
The "Error: Removing state 63 which has instances left" refers
to the zram per CPU struct zcomp instances left.
[0] https://github.com/linux-test-project/ltp.git
Acked-by: Minchan Kim <minchan@kernel.org>
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
drivers/block/zram/zram_drv.c | 63 ++++++++++++++++++++++++++++++-----
1 file changed, 55 insertions(+), 8 deletions(-)
@@ -1785,7 +1810,8 @@ static ssize_t reset_store(struct device *dev,/* Do not reset an active device or claimed device */if(bdev->bd_openers||zram->claim){mutex_unlock(&bdev->bd_disk->open_mutex);-return-EBUSY;+len=-EBUSY;+gotoout;}/* From now on, anyone can't open /dev/zram[0-9] */
On Mon, Sep 27, 2021 at 09:38:05AM -0700, Luis Chamberlain wrote:
The ATTRIBUTE_GROUPS is typically used to avoid boiler plate
code which is used in many drivers. Embracing ATTRIBUTE_GROUPS was
long due on the zram driver, however a recent fix for sysfs allows
users of ATTRIBUTE_GROUPS to also associate a module to the group
attribute.
Does this mean that other modules using sysfs but _not_
ATTRIBUTE_GROUPS() are still vulnerable to potential use-after-free of
the kernfs fops?
-Kees
quoted hunk
In zram's case this also means it allows us to fix a race which triggers
a deadlock on the zram driver. This deadlock happens when a sysfs attribute
use a lock also used on module removal. This happens when for instance a
sysfs file on a driver is used, then at the same time we have module
removal call trigger. The module removal call code holds a lock, and then
the sysfs file entry waits for the same lock. While holding the lock the
module removal tries to remove the sysfs entries, but these cannot be
removed yet as one is waiting for a lock. This won't complete as the lock
is already held. Likewise module removal cannot complete, and so we
deadlock.
Sysfs fixes this when the group attributes have a module associated to
it, sysfs will *try* to get a refcount to the module when a shared
lock is used, prior to mucking with a sysfs attribute. If this fails we
just give up right away.
This deadlock was first reported with the zram driver, a sketch of how
this can happen follows:
CPU A CPU B
whatever_store()
module_unload
mutex_lock(foo)
mutex_lock(foo)
del_gendisk(zram->disk);
device_del()
device_remove_groups()
In this situation whatever_store() is waiting for the mutex foo to
become unlocked, but that won't happen until module removal is complete.
But module removal won't complete until the sysfs file being poked
completes which is waiting for a lock already held.
This issue can be reproduced easily on the zram driver as follows:
Loop 1 on one terminal:
while true;
do modprobe zram;
modprobe -r zram;
done
Loop 2 on a second terminal:
while true; do
echo 1024 > /sys/block/zram0/disksize;
echo 1 > /sys/block/zram0/reset;
done
Without this patch we end up in a deadlock, and the following
stack trace is produced which hints to us what the issue was:
INFO: task bash:888 blocked for more than 120 seconds.
Tainted: G E 5.12.0-rc1-next-20210304+ #4
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:bash state:D stack: 0 pid: 888 ppid: 887 flags:<etc>
Call Trace:
__schedule+0x2e4/0x900
schedule+0x46/0xb0
schedule_preempt_disabled+0xa/0x10
__mutex_lock.constprop.0+0x2c3/0x490
? _kstrtoull+0x35/0xd0
reset_store+0x6c/0x160 [zram]
kernfs_fop_write_iter+0x124/0x1b0
new_sync_write+0x11c/0x1b0
vfs_write+0x1c2/0x260
ksys_write+0x5f/0xe0
do_syscall_64+0x33/0x80
entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x7f34f2c3df33
RSP: 002b:00007ffe751df6e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f34f2c3df33
RDX: 0000000000000002 RSI: 0000561ccb06ec10 RDI: 0000000000000001
RBP: 0000561ccb06ec10 R08: 000000000000000a R09: 0000000000000001
R10: 0000561ccb157590 R11: 0000000000000246 R12: 0000000000000002
R13: 00007f34f2d0e6a0 R14: 0000000000000002 R15: 00007f34f2d0e8a0
INFO: task modprobe:1104 can't die for more than 120 seconds.
task:modprobe state:D stack: 0 pid: 1104 ppid: 916 flags:<etc>
Call Trace:
__schedule+0x2e4/0x900
schedule+0x46/0xb0
__kernfs_remove.part.0+0x228/0x2b0
? finish_wait+0x80/0x80
kernfs_remove_by_name_ns+0x50/0x90
remove_files+0x2b/0x60
sysfs_remove_group+0x38/0x80
sysfs_remove_groups+0x29/0x40
device_remove_attrs+0x4a/0x80
device_del+0x183/0x3e0
? mutex_lock+0xe/0x30
del_gendisk+0x27a/0x2d0
zram_remove+0x8a/0xb0 [zram]
? hot_remove_store+0xf0/0xf0 [zram]
zram_remove_cb+0xd/0x10 [zram]
idr_for_each+0x5e/0xd0
destroy_devices+0x39/0x6f [zram]
__do_sys_delete_module+0x190/0x2a0
do_syscall_64+0x33/0x80
entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x7f32adf727d7
RSP: 002b:00007ffc08bb38a8 EFLAGS: 00000206 ORIG_RAX: 00000000000000b0
RAX: ffffffffffffffda RBX: 000055eea23cbb10 RCX: 00007f32adf727d7
RDX: 0000000000000000 RSI: 0000000000000800 RDI: 000055eea23cbb78
RBP: 000055eea23cbb10 R08: 0000000000000000 R09: 0000000000000000
R10: 00007f32adfe5ac0 R11: 0000000000000206 R12: 000055eea23cbb78
R13: 0000000000000000 R14: 0000000000000000 R15: 000055eea23cbc20
[0] https://lkml.kernel.org/r/20210401235925.GR4332@42.do-not-panic.com
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
drivers/block/zram/zram_drv.c | 11 ++---------
1 file changed, 2 insertions(+), 9 deletions(-)
From: Miroslav Benes <mbenes@suse.cz> Date: 2021-10-07 14:23:29
On Mon, 27 Sep 2021, Luis Chamberlain wrote:
This adds a new selftest module which can be used to test sysfs, which
would otherwise require using an existing driver. This lets us muck
with a template driver to test breaking things without affecting
system behaviour or requiring the dependencies of a real device
driver.
A series of 28 tests are added. Support for using two device types are
supported:
* misc
* block
I suppose the selftests will run for more than 45 seconds (default
kselftest timeout), so you probably also want to set timeout to something
sensible in tools/testing/selftests/sysfs/settings file (0 would disable
it).
Miroslav
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-10-11 17:38:53
On Tue, Oct 05, 2021 at 04:16:46PM +0200, Greg KH wrote:
On Mon, Sep 27, 2021 at 09:37:56AM -0700, Luis Chamberlain wrote:
quoted
--- /dev/null+++ b/lib/test_sysfs.c
@@ -0,0 +1,921 @@+// SPDX-License-Identifier: GPL-2.0-or-later OR copyleft-next-0.3.1+/*+*sysfstestdriver+*+*Copyright(C)2021LuisChamberlain<mcgrof@kernel.org>+*+*Thisprogramisfreesoftware;youcanredistributeitand/ormodifyit+*underthetermsoftheGNUGeneralPublicLicenseaspublishedbytheFree+*SoftwareFoundation;eitherversion2oftheLicense,oratyouroptionany+*laterversion;or,whendistributedseparatelyfromtheLinuxkernelor+*whenincorporatedintoothersoftwarepackages,subjecttothefollowing+*license:+*+*Thisprogramisfreesoftware;youcanredistributeitand/ormodifyit+*underthetermsofcopyleft-next(version0.3.1orlater)aspublished+*athttp://copyleft-next.org/.
Independant of the fact that I don't like sysfs code attempting to be
accessed in the kernel with licenses other than GPLv2, you do not need
the license "boilerplate" text at all in files. That's what the SPDX
line is for.
Sure, I'll remove the boilerplate, sorry for missing that again, I
thought I had removed it.
Luis
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-10-11 17:40:52
On Tue, Oct 05, 2021 at 04:57:55PM +0000, Tim.Bird@sony.com wrote:
quoted
-----Original Message-----
From: Greg KH <gregkh@linuxfoundation.org>
Sent: Tuesday, October 5, 2021 8:17 AM
To: Luis Chamberlain <mcgrof@kernel.org>
Cc: tj@kernel.org; akpm@linux-foundation.org; minchan@kernel.org; jeyu@kernel.org; shuah@kernel.org; bvanassche@acm.org;
dan.j.williams@intel.com; joe@perches.com; tglx@linutronix.de; keescook@chromium.org; rostedt@goodmis.org; linux-
spdx@vger.kernel.org; linux-doc@vger.kernel.org; linux-block@vger.kernel.org; linux-fsdevel@vger.kernel.org; linux-
kselftest@vger.kernel.org; linux-kernel@vger.kernel.org
Subject: Re: [PATCH v8 03/12] selftests: add tests_sysfs module
On Mon, Sep 27, 2021 at 09:37:56AM -0700, Luis Chamberlain wrote:
quoted
--- /dev/null+++ b/lib/test_sysfs.c
@@ -0,0 +1,921 @@+// SPDX-License-Identifier: GPL-2.0-or-later OR copyleft-next-0.3.1+/*+*sysfstestdriver+*+*Copyright(C)2021LuisChamberlain<mcgrof@kernel.org>+*+*Thisprogramisfreesoftware;youcanredistributeitand/ormodifyit+*underthetermsoftheGNUGeneralPublicLicenseaspublishedbytheFree+*SoftwareFoundation;eitherversion2oftheLicense,oratyouroptionany+*laterversion;or,whendistributedseparatelyfromtheLinuxkernelor+*whenincorporatedintoothersoftwarepackages,subjecttothefollowing+*license:
This is a very strange license grant, which I'm not sure is covered by any
current SPDX syntax.
" when distributed separately from the Linux kernel or when incorporated into
other software packages, subject to the following license:"
drivers/xen/events/events_fifo.c has that same language.
Why would we care about the license used when the code is used in a non-kernel
project? If it is desired for the code to be available outside the kernel under a
different license, then surely the easiest thing is to make it available separately
under that license. I'm not sure why the kernel needs to carry this license for
non-kernel use of the code.
I would recommend giving this a GPLv2 SPDX header, and maybe in the comment
at the top of the file put a reference to a git repository where the code can be
obtained under a different license.
Keeping the dual let's new updates directly on the kernel benefit from
evolution. A fork would stagnate it in place and would require updates
separately.
Luis
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-10-11 18:27:40
On Tue, Oct 05, 2021 at 01:55:35PM -0700, Kees Cook wrote:
On Mon, Sep 27, 2021 at 09:38:04AM -0700, Luis Chamberlain wrote:
quoted
Provide a simple state machine to fix races with driver exit where we
remove the CPU multistate callbacks and re-initialization / creation of
new per CPU instances which should be managed by these callbacks.
The zram driver makes use of cpu hotplug multistate support, whereby it
associates a struct zcomp per CPU. Each struct zcomp represents a
compression algorithm in charge of managing compression streams per
CPU. Although a compiled zram driver only supports a fixed set of
compression algorithms, each zram device gets a struct zcomp allocated
per CPU. The "multi" in CPU hotplug multstate refers to these per
cpu struct zcomp instances. Each of these will have the CPU hotplug
callback called for it on CPU plug / unplug. The kernel's CPU hotplug
multistate keeps a linked list of these different structures so that
it will iterate over them on CPU transitions.
By default at driver initialization we will create just one zram device
(num_devices=1) and a zcomp structure then set for the now default
lzo-rle comrpession algorithm. At driver removal we first remove each
zram device, and so we destroy the associated struct zcomp per CPU. But
since we expose sysfs attributes to create new devices or reset /
initialize existing zram devices, we can easily end up re-initializing
a struct zcomp for a zram device before the exit routine of the module
removes the cpu hotplug callback. When this happens the kernel's CPU
hotplug will detect that at least one instance (struct zcomp for us)
exists. This can happen in the following situation:
CPU 1 CPU 2
disksize_store(...);
class_unregister(...);
idr_for_each(...);
zram_debugfs_destroy();
idr_destroy(...);
unregister_blkdev(...);
cpuhp_remove_multi_state(...);
So this is strictly separate from the sysfs/module unloading race?
It is only related in the sense that the sysfs/module unloading race
happened *after* this other issue, but addressing these through
separate threads created a break in conversation and focus. For
instance, a theoretical race was mentioned in one thread, which
I worked to prove/disprove and then I disproved it was not possible.
But at this point, yes, this is a purely separate issue, and this
patch *should* be picked up already.
Andrew, can you merge this? It already has the respective maintainer
Ack, and I can continue to work on the rest of the patches. The only
issue I can think of would be a conflict with the last patch but
that's a oneliner, I think chances are low that would create a conflict
if its all merged separately, and if so, it should be an easy fix for
a merge conflict.
Luis
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-10-11 18:28:49
On Tue, Oct 05, 2021 at 01:57:00PM -0700, Kees Cook wrote:
On Mon, Sep 27, 2021 at 09:38:05AM -0700, Luis Chamberlain wrote:
quoted
The ATTRIBUTE_GROUPS is typically used to avoid boiler plate
code which is used in many drivers. Embracing ATTRIBUTE_GROUPS was
long due on the zram driver, however a recent fix for sysfs allows
users of ATTRIBUTE_GROUPS to also associate a module to the group
attribute.
Does this mean that other modules using sysfs but _not_
ATTRIBUTE_GROUPS() are still vulnerable to potential use-after-free of
the kernfs fops?
The issue is not UAF, its the possible deadlock, but in that sense, yes.
If they don't use ATTRIBUTE_GROUPS() then there is no information being
provided to sysfs about the module owner.
Luis
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-10-11 19:11:45
On Thu, Oct 07, 2021 at 04:23:22PM +0200, Miroslav Benes wrote:
On Mon, 27 Sep 2021, Luis Chamberlain wrote:
quoted
This adds a new selftest module which can be used to test sysfs, which
would otherwise require using an existing driver. This lets us muck
with a template driver to test breaking things without affecting
system behaviour or requiring the dependencies of a real device
driver.
A series of 28 tests are added. Support for using two device types are
supported:
* misc
* block
I suppose the selftests will run for more than 45 seconds (default
kselftest timeout), so you probably also want to set timeout to something
sensible in tools/testing/selftests/sysfs/settings file (0 would disable
it).
Good catch, I'll use a default of 200, in practice for me this runs in
much less than that, about 110 seconds, so 200 should be good wiggle
room.
Luis
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-10-11 20:44:30
On Tue, Oct 05, 2021 at 12:47:22PM -0700, Kees Cook wrote:
On Mon, Sep 27, 2021 at 09:37:57AM -0700, Luis Chamberlain wrote:
quoted
This adds initial failure injection support to kernfs. We start
off with debug knobs which when enabled allow test drivers, such as
test_sysfs, to then make use of these to try to force certain
difficult races to take place with a high degree of certainty.
This only adds runtime code *iff* the new bool CONFIG_FAIL_KERNFS_KNOBS is
enabled in your kernel. If you don't have this enabled this provides
no new functional. When CONFIG_FAIL_KERNFS_KNOBS is disabled the new
routine kernfs_debug_should_wait() ends up being transformed to if
(false), and so the compiler should optimize these out as dead code
producing no new effective binary changes.
We start off with enabling failure injections in kernfs by allowing us to
alter the way kernfs_fop_write_iter() behaves. We allow for the routine
kernfs_fop_write_iter() to wait for a certain condition in the kernel to
occur, after which it will sleep a predefined amount of time. This lets
kernfs users to time exactly when it want kernfs_fop_write_iter() to
complete, allowing for developing race conditions and test for correctness
in kernfs.
You'd boot with this enabled on your kernel command line:
fail_kernfs_fop_write_iter=1,100,0,1
The values are <interval,probability,size,times>, we don't care for
size, so for now we ignore it. The above ensures a failure will trigger
only once.
*How* we allow for this routine to change behaviour is left to knobs we
expose under debugfs:
# ls -1 /sys/kernel/debug/kernfs/config_fail_kernfs_fop_write_iter/
I'd expect this to live under /sys/kernel/debug/fail_kernfs, like the
other fault injectors.
@@ -28,6 +28,28 @@ Available fault injection capabilities injects kernel RPC client and server failures.+- fail_kernfs_fop_write_iter++ Allows for failures to be enabled inside kernfs_fop_write_iter(). Enabling+ this does not immediately enable any errors to occur. You must configure+ how you want this routine to fail or change behaviour by using the debugfs+ knobs for it:++ # ls -1 /sys/kernel/debug/kernfs/config_fail_kernfs_fop_write_iter/+ wait_after_active+ wait_after_mutex+ wait_at_start+ wait_before_mutex
This should be split up and detailed in the "debugfs entries" section
below here.
I'd name this fault_inject.c, which matches the more common case:
$ find . -type f -name '*fault*inject*.c'
./fs/nfsd/fault_inject.c
./drivers/nvme/host/fault_inject.c
./drivers/scsi/ufs/ufs-fault-injection.c
./lib/fault-inject.c
./lib/fault-inject-usercopy.c
Every caller ends up doing the wait, so how about just including that
here instead? It should make things much less intrusive and more readable.
And for the naming, other fault injectors use "should_fail_$topic", so
maybe better here would be something like may_wait_kernfs(...).
In case anyone is reading Hail Mary by Andy Weir: "Yes yes yes!"
Indeed, that's a great idea. Changed!
quoted
+
+DECLARE_COMPLETION(kernfs_debug_wait_completion);
+EXPORT_SYMBOL_NS_GPL(kernfs_debug_wait_completion, KERNFS_DEBUG_PRIVATE);
+
+void kernfs_debug_wait(void)
+{
+ unsigned long timeout;
+
+ timeout = wait_for_completion_timeout(&kernfs_debug_wait_completion,
+ msecs_to_jiffies(3000));
+ if (!timeout)
+ pr_info("%s waiting for kernfs_debug_wait_completion timed out\n",
+ __func__);
+ else
+ pr_info("%s received completion with time left on timeout %u ms\n",
+ __func__, jiffies_to_msecs(timeout));
+
+ /**
+ * The goal is wait for an event, and *then* once we have
+ * reached it, the other side will try to do something which
+ * it thinks will break. So we must give it some time to do
+ * that. The amount of time is configurable.
+ */
+ msleep(kernfs_config_fail.sleep_after_wait_ms);
+ pr_info("%s ended\n", __func__);
+}
All the uses of "__func__" here seems redundant; I would drop them.
Alright, and I also added the pr_fmt define which I forgot.
@@ -609,10 +609,40 @@ void symbol_put_addr(void *addr);tohandletheerrorcase(whichonlyhappenswithrmmod--wait).*/externvoid__module_get(structmodule*module);-/* This is the Right Way to get a module: if it fails, it's being removed,-*sopretendit'snotthere.*/+/**+*try_module_get()-yieldstomoduleremovalandbumpsrefcntotherwise
I find this hard to parse. How about:
"Take module refcount unless module is being removed"
Sure.
quoted
+ * @module: the module we should check for
+ *
+ * This can be used to try to bump the reference count of a module, so to
+ * prevent module removal. The reference count of a module is not allowed
+ * to be incremented if the module is already being removed.
This I understand.
quoted
+ *
+ * Care must be taken to ensure the module cannot be removed during the call to
+ * try_module_get(). This can be done by having another entity other than the
+ * module itself increment the module reference count, or through some other
+ * means which guarantees the module could not be removed during an operation.
+ * An example of this later case is using try_module_get() in a sysfs file
+ * which the module created. The sysfs store / read file operations are
+ * gauranteed to exist through the use of kernfs's active reference (see
+ * kernfs_active()). If a sysfs file operation is being run, the module which
+ * created it must still exist as the module is in charge of removing the same
+ * sysfs file being read. Also, a sysfs / kernfs file removal cannot happen
+ * unless the same file is not active.
I can't understand this paragraph at all. "Care must be taken ..."? Why?
Because the routine try_module_get() assumes the struct module pointer
is valid for the entire call. That can only be true if at least one
reference is held prior to this call.
Shouldn't callers of try_module_get() be satisfied with the results?
Yes but only with the above care addressed.
I don't follow the example at all. It seems to just say "sysfs store/read
functions don't need try_module_get() because whatever opened the sysfs
file is already keeping the module referenced." ?
That is exactly what I intended to clarify with that example, yes, a
reference is held but this is done implicitly. *If* a kernfs op is
active module removal waits for that active reference to go down. So
while a kernfs file is being used it is simply not possible for the
module to disappear underneath us. And the reason is that the module
that created the sysfs file must obviously destroy that same sysfs file.
But since kernfs ensures that sysfs file cannot be removed if a sysfs
file is being used, this implicitly holds a module reference.
Let me know if y ou can think of a better way to phrase this.
quoted
+ *
+ * One of the real values to try_module_get() is the module_is_live() check
+ * which ensures this the caller of try_module_get() can yield to userspace
+ * module removal requests and fail whatever it was about to process.
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-10-11 21:25:57
On Tue, Oct 05, 2021 at 05:24:18PM +0800, Ming Lei wrote:
On Mon, Sep 27, 2021 at 09:38:02AM -0700, Luis Chamberlain wrote:
quoted
When driver sysfs attributes use a lock also used on module removal we
can race to deadlock. This happens when for instance a sysfs file on
a driver is used, then at the same time we have module removal call
trigger. The module removal call code holds a lock, and then the
driver's sysfs file entry waits for the same lock. While holding the
lock the module removal tries to remove the sysfs entries, but these
cannot be removed yet as one is waiting for a lock. This won't complete
as the lock is already held. Likewise module removal cannot complete,
and so we deadlock.
This can now be easily reproducible with our sysfs selftest as follows:
./tools/testing/selftests/sysfs/sysfs.sh -t 0027
This uses a local driver lock. Test 0028 can also be used, that uses
the rtnl_lock():
./tools/testing/selftests/sysfs/sysfs.sh -t 0028
To fix this we extend the struct kernfs_node with a module reference
and use the try_module_get() after kernfs_get_active() is called. As
documented in the prior patch, we now know that once kernfs_get_active()
is called the module is implicitly guarded to exist and cannot be removed.
This is because the module is the one in charge of removing the same
sysfs file it created, and removal of sysfs files on module exit will wait
until they don't have any active references. By using a try_module_get()
after kernfs_get_active() we yield to let module removal trump calls to
process a sysfs operation, while also preventing module removal if a sysfs
operation is in already progress. This prevents the deadlock.
This deadlock was first reported with the zram driver, however the live
Looks not see the lock pattern you mentioned in zram driver, can you
share the related zram code?
I recommend to not look at the zram driver, instead look at the
test_sysfs driver as that abstracts the issue more clearly and uses
two different locks as an example. The point is that if on module
removal *any* lock is used which is *also* used on the sysfs file
created by the module, you can deadlock.
quoted
And this can lead to this condition:
CPU A CPU B
foo_store()
foo_exit()
mutex_lock(&foo)
mutex_lock(&foo)
del_gendisk(some_struct->disk);
device_del()
device_remove_groups()
I guess the deadlock exists if foo_exit() is called anywhere. If yes,
look the issue may not be related with removing module directly, right?
No, the reason this can deadlock is that the module exit routine will
patiently wait for the sysfs / kernfs files to be stop being used,
but clearly they cannot if the exit routine took the mutex also used
by the sysfs ops. That is, the special condition here is the removal of
the sysfs files, and the sysfs files using a lock also used on module
exit.
Luis
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-10-11 22:26:15
On Tue, Oct 05, 2021 at 01:50:31PM -0700, Kees Cook wrote:
On Mon, Sep 27, 2021 at 09:38:02AM -0700, Luis Chamberlain wrote:
quoted
A sketch of how this can happen follows, consider foo a local mutex
part of a driver, and used on the driver's module exit routine and
on one of its sysfs ops:
foo.c:
static DEFINE_MUTEX(foo);
static ssize_t foo_store(struct device *dev,
struct device_attribute *attr,
const char *buf, size_t count)
{
...
mutex_lock(&foo);
...
mutex_lock(&foo);
...
}
static DEVICE_ATTR_RW(foo);
...
void foo_exit(void)
{
mutex_lock(&foo);
...
mutex_unlock(&foo);
}
module_exit(foo_exit);
And this can lead to this condition:
CPU A CPU B
foo_store()
foo_exit()
mutex_lock(&foo)
mutex_lock(&foo)
del_gendisk(some_struct->disk);
device_del()
device_remove_groups()
Please expand this further, where does device_remove_groups() end up
waiting for that never happens?
Sure. How about:
Furthermore, device_remove_groups() will just go on trying to remove
the sysfs files, which are kernfs entries. The way kernfs deals with
removal is that it will wait until all active references for the files
being removed are done. The active reference is obtained through
kernfs_get_active(). Removal ends up waiting through kernfs_drain()
for the active references to be done, and that only happens if the
kernfs file ops can complete. If these kernfs ops / sysfs files
are waiting for a mutex which taken by the module's exit routine
prior to trying to remove the sysfs files we deadlock.
quoted
In this situation foo_store() is waiting for the mutex foo to
become unlocked, but that won't happen until module removal is complete.
But module removal won't complete until the sysfs file being poked at
completes which is waiting for a lock already held.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
arch/x86/kernel/cpu/resctrl/rdtgroup.c | 4 +-
fs/kernfs/dir.c | 44 ++++++++++++++++++----
fs/kernfs/file.c | 6 ++-
fs/kernfs/kernfs-internal.h | 3 +-
fs/kernfs/symlink.c | 3 +-
fs/sysfs/dir.c | 2 +-
fs/sysfs/file.c | 6 ++-
fs/sysfs/group.c | 3 +-
include/linux/kernfs.h | 14 ++++---
include/linux/sysfs.h | 52 ++++++++++++++++++++------
kernel/cgroup/cgroup.c | 2 +-
11 files changed, 105 insertions(+), 34 deletions(-)
The special casing in here makes me think this isn't happening the right
place. (i.e this looks like an open-coded version of kernfs_put_active())
No, well you see, in effect the special care taken in
kernfs_put_active() *is* the right way to inform a waiter that
that the *taken* reference right above *also* is no longer active.
The special casing here is because we took the active reference
before the try_module_get() in the above atomic_inc_unless_negative()
call. Outside callers deal with this through kernfs_put_active().
We are special casing to deal with the deadlock case.
quoted
+
if (kernfs_lockdep(kn))
rwsem_acquire_read(&kn->dep_map, 0, 1, _RET_IP_);
return kn;
@@ -442,6 +460,13 @@ void kernfs_put_active(struct kernfs_node *kn) if (kernfs_lockdep(kn)) rwsem_release(&kn->dep_map, _RET_IP_); v = atomic_dec_return(&kn->active);++ /*+ * We prevent module exit *until* we know for sure all possible+ * kernfs ops are done.+ */+ module_put(kn->owner);+ if (likely(v != KN_DEACTIVATED_BIAS)) return;
What I don't understand, however, is what kernfs_get/put_active() is
intending to do -- it looks like it's trying to provide an interruption
point for open kernfs file operations?
It is essentially ensuring that removal does not happen if any ops
are being used.
This all seems extremely complex for what seems like it should just be a
global "am I being removed?" bool?
It used to be worse :) And Tejun has cleaned this up over time. Yes,
perhaps we can improve that more but, given how sensible this code
is I think such improvements should be made separately.
Regardless, while I do see the logic of associating the module get/put
with get/put of kernfs "active", why is it not better tied to strictly
kernfs open/close?
It's not just files, consider kernfs_iop_mkdir() which also calls
kernfs_get_active(). How about kernfs_fop_mmap()? And so, the common
denominator is actually kernfs_get_active().
That would seem to be much simpler and not require
any special handling?
Yes true, but it I think this would still leave open some other possible
deadlocks.
For example, why does this not work?
It does for the write case for sure, but I haven't written tests for the
other odd cases, but suspect that would deadlock as well.
Luis
On Mon, Oct 11, 2021 at 02:25:46PM -0700, Luis Chamberlain wrote:
On Tue, Oct 05, 2021 at 05:24:18PM +0800, Ming Lei wrote:
quoted
On Mon, Sep 27, 2021 at 09:38:02AM -0700, Luis Chamberlain wrote:
quoted
When driver sysfs attributes use a lock also used on module removal we
can race to deadlock. This happens when for instance a sysfs file on
a driver is used, then at the same time we have module removal call
trigger. The module removal call code holds a lock, and then the
driver's sysfs file entry waits for the same lock. While holding the
lock the module removal tries to remove the sysfs entries, but these
cannot be removed yet as one is waiting for a lock. This won't complete
as the lock is already held. Likewise module removal cannot complete,
and so we deadlock.
This can now be easily reproducible with our sysfs selftest as follows:
./tools/testing/selftests/sysfs/sysfs.sh -t 0027
This uses a local driver lock. Test 0028 can also be used, that uses
the rtnl_lock():
./tools/testing/selftests/sysfs/sysfs.sh -t 0028
To fix this we extend the struct kernfs_node with a module reference
and use the try_module_get() after kernfs_get_active() is called. As
documented in the prior patch, we now know that once kernfs_get_active()
is called the module is implicitly guarded to exist and cannot be removed.
This is because the module is the one in charge of removing the same
sysfs file it created, and removal of sysfs files on module exit will wait
until they don't have any active references. By using a try_module_get()
after kernfs_get_active() we yield to let module removal trump calls to
process a sysfs operation, while also preventing module removal if a sysfs
operation is in already progress. This prevents the deadlock.
This deadlock was first reported with the zram driver, however the live
Looks not see the lock pattern you mentioned in zram driver, can you
share the related zram code?
I recommend to not look at the zram driver, instead look at the
test_sysfs driver as that abstracts the issue more clearly and uses
Looks test_sysfs isn't in linus tree, where can I find it? Also please
update your commit log about this wrong info if it can't be applied on
zram.
two different locks as an example. The point is that if on module
removal *any* lock is used which is *also* used on the sysfs file
created by the module, you can deadlock.
quoted
quoted
And this can lead to this condition:
CPU A CPU B
foo_store()
foo_exit()
mutex_lock(&foo)
mutex_lock(&foo)
del_gendisk(some_struct->disk);
device_del()
device_remove_groups()
I guess the deadlock exists if foo_exit() is called anywhere. If yes,
look the issue may not be related with removing module directly, right?
No, the reason this can deadlock is that the module exit routine will
patiently wait for the sysfs / kernfs files to be stop being used,
Can you share the code which waits for the sysfs / kernfs files to be
stop being used? And why does it make a difference in case of being
called from module_exit()?
Thanks,
Ming
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-10-12 21:18:44
On Tue, Oct 12, 2021 at 08:20:46AM +0800, Ming Lei wrote:
On Mon, Oct 11, 2021 at 02:25:46PM -0700, Luis Chamberlain wrote:
quoted
On Tue, Oct 05, 2021 at 05:24:18PM +0800, Ming Lei wrote:
quoted
On Mon, Sep 27, 2021 at 09:38:02AM -0700, Luis Chamberlain wrote:
quoted
When driver sysfs attributes use a lock also used on module removal we
can race to deadlock. This happens when for instance a sysfs file on
a driver is used, then at the same time we have module removal call
trigger. The module removal call code holds a lock, and then the
driver's sysfs file entry waits for the same lock. While holding the
lock the module removal tries to remove the sysfs entries, but these
cannot be removed yet as one is waiting for a lock. This won't complete
as the lock is already held. Likewise module removal cannot complete,
and so we deadlock.
This can now be easily reproducible with our sysfs selftest as follows:
./tools/testing/selftests/sysfs/sysfs.sh -t 0027
This uses a local driver lock. Test 0028 can also be used, that uses
the rtnl_lock():
./tools/testing/selftests/sysfs/sysfs.sh -t 0028
To fix this we extend the struct kernfs_node with a module reference
and use the try_module_get() after kernfs_get_active() is called. As
documented in the prior patch, we now know that once kernfs_get_active()
is called the module is implicitly guarded to exist and cannot be removed.
This is because the module is the one in charge of removing the same
sysfs file it created, and removal of sysfs files on module exit will wait
until they don't have any active references. By using a try_module_get()
after kernfs_get_active() we yield to let module removal trump calls to
process a sysfs operation, while also preventing module removal if a sysfs
operation is in already progress. This prevents the deadlock.
This deadlock was first reported with the zram driver, however the live
Looks not see the lock pattern you mentioned in zram driver, can you
share the related zram code?
I recommend to not look at the zram driver, instead look at the
test_sysfs driver as that abstracts the issue more clearly and uses
Looks test_sysfs isn't in linus tree, where can I find it?
Also please
update your commit log about this wrong info if it can't be applied on
zram.
It does apply to zram, it is just that I have other fixes for zram in
my pipeline which will change the zram driver further, and so what makes
more sense is to abstract the issue into a selftest driver to
demonstrate the issue more clearly.
To reproduce the deadlock revert the patch in this thread and then run
either of these two tests as root:
./tools/testing/selftests/sysfs/sysfs.sh -w 0027
./tools/testing/selftests/sysfs/sysfs.sh -w 0028
You will need to enable the test_sysfs driver.
quoted
two different locks as an example. The point is that if on module
removal *any* lock is used which is *also* used on the sysfs file
created by the module, you can deadlock.
quoted
quoted
And this can lead to this condition:
CPU A CPU B
foo_store()
foo_exit()
mutex_lock(&foo)
mutex_lock(&foo)
del_gendisk(some_struct->disk);
device_del()
device_remove_groups()
I guess the deadlock exists if foo_exit() is called anywhere. If yes,
look the issue may not be related with removing module directly, right?
No, the reason this can deadlock is that the module exit routine will
patiently wait for the sysfs / kernfs files to be stop being used,
Can you share the code which waits for the sysfs / kernfs files to be
stop being used?
How about a call trace of the two tasks which deadlock, here is one of
running test 0027:
kdevops login: [ 363.875459] INFO: task sysfs.sh:1271 blocked for more
than 120 seconds.
[ 363.878341] Tainted: G E
5.15.0-rc3-next-20210927+ #83
[ 363.881218] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 363.882255] task:sysfs.sh state:D stack: 0 pid: 1271 ppid:
1 flags:0x00000004
[ 363.882894] Call Trace:
[ 363.883091] <TASK>
[ 363.883259] __schedule+0x2fd/0x990
[ 363.883551] schedule+0x43/0xe0
[ 363.883800] schedule_preempt_disabled+0x14/0x20
[ 363.884160] __mutex_lock.constprop.0+0x249/0x470
[ 363.884524] test_dev_x_store+0xa5/0xc0 [test_sysfs]
[ 363.884915] kernfs_fop_write_iter+0x177/0x220
[ 363.885257] new_sync_write+0x11c/0x1b0
[ 363.885556] vfs_write+0x20d/0x2a0
[ 363.885821] ksys_write+0x5f/0xe0
[ 363.886081] do_syscall_64+0x38/0xc0
[ 363.886359] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 363.886748] RIP: 0033:0x7fee00f8bf33
[ 363.887029] RSP: 002b:00007ffd372c5d18 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 363.887633] RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007fee00f8bf33
[ 363.888217] RDX: 0000000000000003 RSI: 000055a4d14a0db0 RDI: 0000000000000001
[ 363.888761] RBP: 000055a4d14a0db0 R08: 000000000000000a R09: 0000000000000002
[ 363.889267] R10: 000055a4d1554ac0 R11: 0000000000000246 R12: 0000000000000003
[ 363.889983] R13: 00007fee0105c6a0 R14: 0000000000000003 R15: 00007fee0105c8a0
[ 363.890513] </TASK>
[ 363.890709] INFO: task modprobe:1276 blocked for more than 120 seconds.
[ 363.891185] Tainted: G E 5.15.0-rc3-next-20210927+ #83
[ 363.891781] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 363.892353] task:modprobe state:D stack: 0 pid: 1276 ppid: 1 flags:0x00004000
[ 363.892955] Call Trace:
[ 363.893141] <TASK>
[ 363.893457] __schedule+0x2fd/0x990
[ 363.893865] schedule+0x43/0xe0
[ 363.894246] __kernfs_remove.part.0+0x21e/0x2a0
[ 363.894704] ? do_wait_intr_irq+0xa0/0xa0
[ 363.895142] kernfs_remove_by_name_ns+0x50/0x90
[ 363.895632] remove_files+0x2b/0x60
[ 363.896035] sysfs_remove_group+0x38/0x80
[ 363.896470] sysfs_remove_groups+0x29/0x40
[ 363.896912] device_remove_attrs+0x5b/0x90
[ 363.897352] device_del+0x183/0x400
[ 363.897758] unregister_test_dev_sysfs+0x5b/0xaa [test_sysfs]
[ 363.898317] test_sysfs_exit+0x45/0xfb0 [test_sysfs]
[ 363.898833] __do_sys_delete_module+0x18d/0x2a0
[ 363.899329] ? fpregs_assert_state_consistent+0x1e/0x40
[ 363.899868] ? exit_to_user_mode_prepare+0x3a/0x180
[ 363.900390] do_syscall_64+0x38/0xc0
[ 363.900810] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 363.901330] RIP: 0033:0x7f21915c57d7
[ 363.901747] RSP: 002b:00007ffd90869fe8 EFLAGS: 00000206 ORIG_RAX: 00000000000000b0
[ 363.902442] RAX: ffffffffffffffda RBX: 000055ce676ffc30 RCX: 00007f21915c57d7
[ 363.903104] RDX: 0000000000000000 RSI: 0000000000000800 RDI: 000055ce676ffc98
[ 363.903782] RBP: 000055ce676ffc30 R08: 0000000000000000 R09: 0000000000000000
[ 363.904462] R10: 00007f2191638ac0 R11: 0000000000000206 R12: 000055ce676ffc98
[ 363.905128] R13: 0000000000000000 R14: 0000000000000000 R15: 000055ce676ffdf0
[ 363.905797] </TASK>
And gdb:
(gdb) l *(__kernfs_remove+0x21e)
0xffffffff8139288e is in __kernfs_remove (fs/kernfs/dir.c:476).
471 if (atomic_read(&kn->active) != KN_DEACTIVATED_BIAS)
472 lock_contended(&kn->dep_map, _RET_IP_);
473 }
474
475 /* but everyone should wait for draining */
476 wait_event(root->deactivate_waitq,
477 atomic_read(&kn->active) == KN_DEACTIVATED_BIAS);
478
479 if (kernfs_lockdep(kn)) {
480 lock_acquired(&kn->dep_map, _RET_IP_);
(gdb) l *(kernfs_remove_by_name_ns+0x50)
0xffffffff813938d0 is in kernfs_remove_by_name_ns (fs/kernfs/dir.c:1534).
1529
1530 kn = kernfs_find_ns(parent, name, ns);
1531 if (kn)
1532 __kernfs_remove(kn);
1533
1534 up_write(&kernfs_rwsem);
1535
1536 if (kn)
1537 return 0;
1538 else
The same happens for test 0028 except instead of a mutex
lock an rtnl_lock() is used.
Would this be better for the commit log?
And why does it make a difference in case of being
called from module_exit()?
Well because that is where we remove the sysfs files. *If*
a developer happens to use a lock on a sysfs op but it is
also used on module exit, this deadlock is bound to happen.
Luis
On Tue, Oct 12, 2021 at 02:18:28PM -0700, Luis Chamberlain wrote:
On Tue, Oct 12, 2021 at 08:20:46AM +0800, Ming Lei wrote:
quoted
On Mon, Oct 11, 2021 at 02:25:46PM -0700, Luis Chamberlain wrote:
quoted
On Tue, Oct 05, 2021 at 05:24:18PM +0800, Ming Lei wrote:
quoted
On Mon, Sep 27, 2021 at 09:38:02AM -0700, Luis Chamberlain wrote:
quoted
When driver sysfs attributes use a lock also used on module removal we
can race to deadlock. This happens when for instance a sysfs file on
a driver is used, then at the same time we have module removal call
trigger. The module removal call code holds a lock, and then the
driver's sysfs file entry waits for the same lock. While holding the
lock the module removal tries to remove the sysfs entries, but these
cannot be removed yet as one is waiting for a lock. This won't complete
as the lock is already held. Likewise module removal cannot complete,
and so we deadlock.
This can now be easily reproducible with our sysfs selftest as follows:
./tools/testing/selftests/sysfs/sysfs.sh -t 0027
This uses a local driver lock. Test 0028 can also be used, that uses
the rtnl_lock():
./tools/testing/selftests/sysfs/sysfs.sh -t 0028
To fix this we extend the struct kernfs_node with a module reference
and use the try_module_get() after kernfs_get_active() is called. As
documented in the prior patch, we now know that once kernfs_get_active()
is called the module is implicitly guarded to exist and cannot be removed.
This is because the module is the one in charge of removing the same
sysfs file it created, and removal of sysfs files on module exit will wait
until they don't have any active references. By using a try_module_get()
after kernfs_get_active() we yield to let module removal trump calls to
process a sysfs operation, while also preventing module removal if a sysfs
operation is in already progress. This prevents the deadlock.
This deadlock was first reported with the zram driver, however the live
Looks not see the lock pattern you mentioned in zram driver, can you
share the related zram code?
I recommend to not look at the zram driver, instead look at the
test_sysfs driver as that abstracts the issue more clearly and uses
Looks test_sysfs isn't in linus tree, where can I find it?
Also please
update your commit log about this wrong info if it can't be applied on
zram.
It does apply to zram, it is just that I have other fixes for zram in
my pipeline which will change the zram driver further, and so what makes
more sense is to abstract the issue into a selftest driver to
demonstrate the issue more clearly.
To reproduce the deadlock revert the patch in this thread and then run
either of these two tests as root:
./tools/testing/selftests/sysfs/sysfs.sh -w 0027
./tools/testing/selftests/sysfs/sysfs.sh -w 0028
You will need to enable the test_sysfs driver.
quoted
quoted
two different locks as an example. The point is that if on module
removal *any* lock is used which is *also* used on the sysfs file
created by the module, you can deadlock.
quoted
quoted
And this can lead to this condition:
CPU A CPU B
foo_store()
foo_exit()
mutex_lock(&foo)
mutex_lock(&foo)
del_gendisk(some_struct->disk);
device_del()
device_remove_groups()
I guess the deadlock exists if foo_exit() is called anywhere. If yes,
look the issue may not be related with removing module directly, right?
No, the reason this can deadlock is that the module exit routine will
patiently wait for the sysfs / kernfs files to be stop being used,
Can you share the code which waits for the sysfs / kernfs files to be
stop being used?
That doesn't show the deadlock is related with module_exit().
And gdb:
(gdb) l *(__kernfs_remove+0x21e)
0xffffffff8139288e is in __kernfs_remove (fs/kernfs/dir.c:476).
471 if (atomic_read(&kn->active) != KN_DEACTIVATED_BIAS)
472 lock_contended(&kn->dep_map, _RET_IP_);
473 }
474
475 /* but everyone should wait for draining */
476 wait_event(root->deactivate_waitq,
477 atomic_read(&kn->active) == KN_DEACTIVATED_BIAS);
478
479 if (kernfs_lockdep(kn)) {
480 lock_acquired(&kn->dep_map, _RET_IP_);
(gdb) l *(kernfs_remove_by_name_ns+0x50)
0xffffffff813938d0 is in kernfs_remove_by_name_ns (fs/kernfs/dir.c:1534).
1529
1530 kn = kernfs_find_ns(parent, name, ns);
1531 if (kn)
1532 __kernfs_remove(kn);
1533
1534 up_write(&kernfs_rwsem);
1535
1536 if (kn)
1537 return 0;
1538 else
The same happens for test 0028 except instead of a mutex
lock an rtnl_lock() is used.
Would this be better for the commit log?
quoted
And why does it make a difference in case of being
called from module_exit()?
Well because that is where we remove the sysfs files. *If*
a developer happens to use a lock on a sysfs op but it is
also used on module exit, this deadlock is bound to happen.
It is clearly one AA deadlock, what I meant was that it isn't related with
module exit cause lock & device_del() isn't always done in module exit, so
I doubt your fix with grabbing module refcnt is good or generic enough.
Except for your cooked test_sys module, how many real drivers do suffer the
problem? What are they? Why can't we fix the exact driver?
Thanks,
Ming
Can you share the code which waits for the sysfs / kernfs files to be
stop being used?
How about a call trace of the two tasks which deadlock, here is one of
running test 0027:
kdevops login: [ 363.875459] INFO: task sysfs.sh:1271 blocked for more
than 120 seconds.
<-- snip -->
That doesn't show the deadlock is related with module_exit().
Not directly no.
It is clearly one AA deadlock, what I meant was that it isn't related with
module exit cause lock & device_del() isn't always done in module exit, so
I doubt your fix with grabbing module refcnt is good or generic enough.
A device_del() *can* happen in other areas other than module exit sure,
but the issue is if a shared lock is used *before* device_del() and also
used on a sysfs op. Typically this can happen on module exit, and the
other common use case in my experience is on sysfs ops, such is the case
with the zram driver. Both cases are covered then by this fix.
If there are other areas, that is still driver specific, but of the
things we *can* generalize, definitely module exit is a common path.
Except for your cooked test_sys module, how many real drivers do suffer the
problem? What are they?
I only really seriously considered trying to generalize this after it
was hinted to me live patching was also affected, and so clearly
something generic was desirable.
There may be other drivers for sure, but a hunt for that with semantics
would require a bit complex coccinelle patch with iteration support.
Why can't we fix the exact driver?
You can try, the way the lock is used in zram is correct, specially
after my other fix in this series which addresses another unrelated bug
with cpu hotplug multistate support. So we then can proceed to either
take the position to say: "Thou shalt not use a shared lock on module
exit and a sysfs op" and try to fix all places, or we generalize a fix
for this. A generic fix seems more desirable.
Luis
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-10-13 12:42:05
On Mon, Oct 11, 2021 at 03:26:02PM -0700, Luis Chamberlain wrote:
On Tue, Oct 05, 2021 at 01:50:31PM -0700, Kees Cook wrote:
quoted
For example, why does this not work?
It does for the write case for sure,
I mispoke, just for the record, the changes you mentioned actually don't
suffice for the test cases in question for test_sysfs, the deadlock
still occurs with those changes. At first I thought it did but I had failed
to remove my own fix first on fs/kernfs/dir.c. After removing that and
just trying the proposed changes I confirm it does not fix the deadlock.
Luis
Can you share the code which waits for the sysfs / kernfs files to be
stop being used?
How about a call trace of the two tasks which deadlock, here is one of
running test 0027:
kdevops login: [ 363.875459] INFO: task sysfs.sh:1271 blocked for more
than 120 seconds.
<-- snip -->
quoted
That doesn't show the deadlock is related with module_exit().
Not directly no.
Then the patch title of 'sysfs: fix deadlock race with module removal'
is wrong.
quoted
It is clearly one AA deadlock, what I meant was that it isn't related with
module exit cause lock & device_del() isn't always done in module exit, so
I doubt your fix with grabbing module refcnt is good or generic enough.
A device_del() *can* happen in other areas other than module exit sure,
but the issue is if a shared lock is used *before* device_del() and also
used on a sysfs op. Typically this can happen on module exit, and the
other common use case in my experience is on sysfs ops, such is the case
with the zram driver. Both cases are covered then by this fix.
Again, can you share the related zram code about the issue? In
zram_drv.c of linus or next tree, I don't see any lock is held before
calling del_gendisk().
If there are other areas, that is still driver specific, but of the
things we *can* generalize, definitely module exit is a common path.
quoted
Except for your cooked test_sys module, how many real drivers do suffer the
problem? What are they?
I only really seriously considered trying to generalize this after it
IMO your generalization isn't good or correct because this kind of issue
is _not_ related with module exit at all. What matters is just that one lock is
held before calling device_del(), meantime the same lock is required
in the device's attribute show/store function().
There are many cases in which we call device_del() not from module_exit(),
such as scsi scan, scsi sysfs store(), or even handling event from
device side, nvme error handling, usb hotplug, ...
was hinted to me live patching was also affected, and so clearly
something generic was desirable.
It might be just the only two drivers(zram and live patch) with this bug, and
it is one simply AA bug in driver. Not mention I don't see such usage in
zram_drv.c.
There may be other drivers for sure, but a hunt for that with semantics
would require a bit complex coccinelle patch with iteration support.
quoted
Why can't we fix the exact driver?
You can try, the way the lock is used in zram is correct, specially
What is the lock in zram? Again can you share the related functions?
after my other fix in this series which addresses another unrelated bug
with cpu hotplug multistate support. So we then can proceed to either
take the position to say: "Thou shalt not use a shared lock on module
exit and a sysfs op" and try to fix all places, or we generalize a fix
for this. A generic fix seems more desirable.
What matters is that the lock is held before calling device_del()
instead of being held in module_exit().
Thanks,
Ming
Can you share the code which waits for the sysfs / kernfs files to be
stop being used?
How about a call trace of the two tasks which deadlock, here is one of
running test 0027:
kdevops login: [ 363.875459] INFO: task sysfs.sh:1271 blocked for more
than 120 seconds.
<-- snip -->
quoted
That doesn't show the deadlock is related with module_exit().
Not directly no.
Then the patch title of 'sysfs: fix deadlock race with module removal'
is wrong.
Well that is what it does though. The scope of the issue you are raising
is beyond module removal, but I do agree such races can exist outside of
module removal.
quoted
quoted
It is clearly one AA deadlock, what I meant was that it isn't related with
module exit cause lock & device_del() isn't always done in module exit, so
I doubt your fix with grabbing module refcnt is good or generic enough.
A device_del() *can* happen in other areas other than module exit sure,
but the issue is if a shared lock is used *before* device_del() and also
used on a sysfs op. Typically this can happen on module exit, and the
other common use case in my experience is on sysfs ops, such is the case
with the zram driver. Both cases are covered then by this fix.
Again, can you share the related zram code about the issue? In
zram_drv.c of linus or next tree, I don't see any lock is held before
calling del_gendisk().
There is another bug with CPU hotplug multistate support in the zram
driver which a patch in this series fixes, refer to the patch titled
"zram: fix crashes with cpu hotplug multistate". In zram's case we need
to contend a generic lock on certain sysfs attributes due to the way CPU
hotplug is used.
If we tried to generalize this on the block layer the closest we get is
the disk->fops->owner, however zram is an example driver where the
disk->fops is actually be even changed *after* module load, and so the
original disk->fops->owner can be dynamic. In zram's case the
fops->owner is the same, however we have no semantics to ensure this is
the case for all block drivers.
In the case for live patching, refer to the use of klp_mutex. The way
that was solved there was a combination of completions and deferred
works to solve it, so that all kobject_put calls are outside of the
critical sections, refer to commit 3ec24776bfd0 ("livepatch:
allow removal of a disabled patch").
And so it was encouraged a generic solution be sought after.
quoted
If there are other areas, that is still driver specific, but of the
things we *can* generalize, definitely module exit is a common path.
quoted
Except for your cooked test_sys module, how many real drivers do suffer the
problem? What are they?
I only really seriously considered trying to generalize this after it
IMO your generalization isn't good or correct because this kind of issue
is _not_ related with module exit at all. What matters is just that one lock is
held before calling device_del(), meantime the same lock is required
in the device's attribute show/store function().
Your point that a race for a deadlock still can exist beyond module
removal is valid but unfortunately there are no possible semantics I can
see to fix that generically at this time.
There are many cases in which we call device_del() not from module_exit(),
such as scsi scan, scsi sysfs store(), or even handling event from
device side, nvme error handling, usb hotplug, ...
These are really good points.
quoted
was hinted to me live patching was also affected, and so clearly
something generic was desirable.
It might be just the only two drivers(zram and live patch) with this bug, and
it is one simply AA bug in driver. Not mention I don't see such usage in
zram_drv.c.
Well... given what you say above about other uses cases other than
module removal which can remove sysfs files and having them be used,
the possibilities of this deadlock existing elsewhere should increase,
not decrease.
quoted
There may be other drivers for sure, but a hunt for that with semantics
would require a bit complex coccinelle patch with iteration support.
quoted
Why can't we fix the exact driver?
You can try, the way the lock is used in zram is correct, specially
What is the lock in zram? Again can you share the related functions?
If you git checked out the tree I mentioned try looking at the code
there with the fix for CPU hotplug multistate in mind.
quoted
after my other fix in this series which addresses another unrelated bug
with cpu hotplug multistate support. So we then can proceed to either
take the position to say: "Thou shalt not use a shared lock on module
exit and a sysfs op" and try to fix all places, or we generalize a fix
for this. A generic fix seems more desirable.
What matters is that the lock is held before calling device_del()
instead of being held in module_exit().
I agree the possibilities can include more than just module exit.
Unfortunately I can't see a way to generalize this further. I tried,
see below, and this moves the ideas from a module to the kobject, but
even with that, it does not get us any closer to fixing this
generically. The reason a fix works for module removal is the
try_module_get() call when getting the kernfs active reference
will trump the module exit call completely, and so we *do* prevent
the context which will issue the lock in this case if a sysfs
operation is in progress.
Outside of that call sequence I am afraid we'd need separate solutions
or side with the 'though shall not use a shared lock on a sysfs op
and when issuing a device_del(), other than module exit'.
Below is an attempt to generalize this further, but it does not work,
let me know if you have further ideas.
On Mon, Sep 27, 2021 at 09:38:04AM -0700, Luis Chamberlain wrote:
Provide a simple state machine to fix races with driver exit where we
remove the CPU multistate callbacks and re-initialization / creation of
new per CPU instances which should be managed by these callbacks.
The zram driver makes use of cpu hotplug multistate support, whereby it
associates a struct zcomp per CPU. Each struct zcomp represents a
compression algorithm in charge of managing compression streams per
CPU. Although a compiled zram driver only supports a fixed set of
compression algorithms, each zram device gets a struct zcomp allocated
per CPU. The "multi" in CPU hotplug multstate refers to these per
cpu struct zcomp instances. Each of these will have the CPU hotplug
callback called for it on CPU plug / unplug. The kernel's CPU hotplug
multistate keeps a linked list of these different structures so that
it will iterate over them on CPU transitions.
By default at driver initialization we will create just one zram device
(num_devices=1) and a zcomp structure then set for the now default
lzo-rle comrpession algorithm. At driver removal we first remove each
zram device, and so we destroy the associated struct zcomp per CPU. But
since we expose sysfs attributes to create new devices or reset /
initialize existing zram devices, we can easily end up re-initializing
a struct zcomp for a zram device before the exit routine of the module
removes the cpu hotplug callback. When this happens the kernel's CPU
hotplug will detect that at least one instance (struct zcomp for us)
exists. This can happen in the following situation:
CPU 1 CPU 2
disksize_store(...);
class_unregister(...);
idr_for_each(...);
zram_debugfs_destroy();
idr_destroy(...);
unregister_blkdev(...);
cpuhp_remove_multi_state(...);
The warning comes up on cpuhp_remove_multi_state() when it sees that the
state for CPUHP_ZCOMP_PREPARE does not have an empty instance linked list.
In this case, that a struct zcom still exists, the driver allowed its
creation per CPU even though we could have just freed them per CPU
though a call on another CPU, and we are then later trying to remove the
hotplug callback.
Fix all this by providing a zram initialization boolean
protected the shared in the driver zram_index_mutex, which we
can use to annotate when sysfs attributes are safe to use or
not -- once the driver is properly initialized. When the driver
is going down we also are sure to not let userspace muck with
attributes which may affect each per cpu struct zcomp.
This also fixes a series of possible memory leaks. The
crashes and memory leaks can easily be caused by issuing
the zram02.sh script from the LTP project [0] in a loop
in two separate windows:
cd testcases/kernel/device-drivers/zram
while true; do PATH=$PATH:$PWD:$PWD/../../../lib/ ./zram02.sh; done
You end up with a splat as follows:
kernel: zram: Removed device: zram0
kernel: zram: Added device: zram0
kernel: zram0: detected capacity change from 0 to 209715200
kernel: Adding 104857596k swap on /dev/zram0. <etc>
kernel: zram0: detected capacitky change from 209715200 to 0
kernel: zram0: detected capacity change from 0 to 209715200
kernel: ------------[ cut here ]------------
kernel: Error: Removing state 63 which has instances left.
kernel: WARNING: CPU: 7 PID: 70457 at \
kernel/cpu.c:2069 __cpuhp_remove_state_cpuslocked+0xf9/0x100
kernel: Modules linked in: zram(E-) zsmalloc(E) <etc>
kernel: CPU: 7 PID: 70457 Comm: rmmod Tainted: G \
E 5.12.0-rc1-next-20210304 #3
kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), \
BIOS 1.14.0-2 04/01/2014
kernel: RIP: 0010:__cpuhp_remove_state_cpuslocked+0xf9/0x100
kernel: Code: <etc>
kernel: RSP: 0018:ffffa800c139be98 EFLAGS: 00010282
kernel: RAX: 0000000000000000 RBX: ffffffff9083db58 RCX: ffff9609f7dd86d8
kernel: RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffff9609f7dd86d0
kernel: RBP: 0000000000000000i R08: 0000000000000000 R09: ffffa800c139bcb8
kernel: R10: ffffa800c139bcb0 R11: ffffffff908bea40 R12: 000000000000003f
kernel: R13: 00000000000009d8 R14: 0000000000000000 R15: 0000000000000000
kernel: FS: 00007f1b075a7540(0000) GS:ffff9609f7dc0000(0000) knlGS:<etc>
kernel: CS: 0010 DS: 0000 ES 0000 CR0: 0000000080050033
kernel: CR2: 00007f1b07610490 CR3: 00000001bd04e000 CR4: 0000000000350ee0
kernel: Call Trace:
kernel: __cpuhp_remove_state+0x2e/0x80
kernel: __do_sys_delete_module+0x190/0x2a0
kernel: do_syscall_64+0x33/0x80
kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae
The "Error: Removing state 63 which has instances left" refers
to the zram per CPU struct zcomp instances left.
[0] https://github.com/linux-test-project/ltp.git
Acked-by: Minchan Kim <minchan@kernel.org>
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
Hello Luis,
Can you test the following patch and see if the issue can be addressed?
Please see the idea from the inline comment.
Also zram_index_mutex isn't needed in zram disk's store() compared with
your patch, then the deadlock issue you are addressing in this series can
be avoided.
@@ -1985,11 +1985,17 @@ static int zram_remove(struct zram *zram)/* Make sure all the pending I/O are finished */fsync_bdev(bdev);-zram_reset_device(zram);pr_info("Removed device: %s\n",zram->disk->disk_name);del_gendisk(zram->disk);++/*+*resetdeviceaftergendiskisremoved,soanychangefromsysfs+*storewon'tcomein,thenwecanreallyresetdevicehere+*/+zram_reset_device(zram);+blk_cleanup_disk(zram->disk);kfree(zram);return0;
@@ -2073,7 +2079,12 @@ static int zram_remove_cb(int id, void *ptr, void *data)staticvoiddestroy_devices(void){class_unregister(&zram_control_class);++/* hold the global lock so new device can't be added */+mutex_lock(&zram_index_mutex);idr_for_each(&zram_index_idr,&zram_remove_cb,NULL);+mutex_unlock(&zram_index_mutex);+zram_debugfs_destroy();idr_destroy(&zram_index_idr);unregister_blkdev(zram_major,"zram");
On Thu, Oct 14, 2021 at 09:55:48AM +0800, Ming Lei wrote:
On Mon, Sep 27, 2021 at 09:38:04AM -0700, Luis Chamberlain wrote:
...
quoted hunk
Hello Luis,
Can you test the following patch and see if the issue can be addressed?
Please see the idea from the inline comment.
Also zram_index_mutex isn't needed in zram disk's store() compared with
your patch, then the deadlock issue you are addressing in this series can
be avoided.
@@ -1985,11 +1985,17 @@ static int zram_remove(struct zram *zram)/* Make sure all the pending I/O are finished */fsync_bdev(bdev);-zram_reset_device(zram);pr_info("Removed device: %s\n",zram->disk->disk_name);del_gendisk(zram->disk);++/*+*resetdeviceaftergendiskisremoved,soanychangefromsysfs+*storewon'tcomein,thenwecanreallyresetdevicehere+*/+zram_reset_device(zram);+blk_cleanup_disk(zram->disk);kfree(zram);return0;
@@ -2073,7 +2079,12 @@ static int zram_remove_cb(int id, void *ptr, void *data)staticvoiddestroy_devices(void){class_unregister(&zram_control_class);++/* hold the global lock so new device can't be added */+mutex_lock(&zram_index_mutex);idr_for_each(&zram_index_idr,&zram_remove_cb,NULL);+mutex_unlock(&zram_index_mutex);+
Actually zram_index_mutex isn't needed when calling zram_remove_cb()
since the zram-control sysfs interface has been removed, so userspace
can't add new device any more, then the issue is supposed to be fixed
by the following one line change, please test it:
@@ -1985,11 +1985,17 @@ static int zram_remove(struct zram *zram)/* Make sure all the pending I/O are finished */fsync_bdev(bdev);-zram_reset_device(zram);pr_info("Removed device: %s\n",zram->disk->disk_name);del_gendisk(zram->disk);++/*+*resetdeviceaftergendiskisremoved,soanychangefromsysfs+*storewon'tcomein,thenwecanreallyresetdevicehere+*/+zram_reset_device(zram);+blk_cleanup_disk(zram->disk);kfree(zram);return0;
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-10-14 20:24:45
On Thu, Oct 14, 2021 at 10:11:46AM +0800, Ming Lei wrote:
quoted hunk
On Thu, Oct 14, 2021 at 09:55:48AM +0800, Ming Lei wrote:
quoted
On Mon, Sep 27, 2021 at 09:38:04AM -0700, Luis Chamberlain wrote:
...
quoted
Hello Luis,
Can you test the following patch and see if the issue can be addressed?
Please see the idea from the inline comment.
Also zram_index_mutex isn't needed in zram disk's store() compared with
your patch, then the deadlock issue you are addressing in this series can
be avoided.
@@ -1985,11 +1985,17 @@ static int zram_remove(struct zram *zram)/* Make sure all the pending I/O are finished */fsync_bdev(bdev);-zram_reset_device(zram);pr_info("Removed device: %s\n",zram->disk->disk_name);del_gendisk(zram->disk);++/*+*resetdeviceaftergendiskisremoved,soanychangefromsysfs+*storewon'tcomein,thenwecanreallyresetdevicehere+*/+zram_reset_device(zram);+blk_cleanup_disk(zram->disk);kfree(zram);return0;
@@ -2073,7 +2079,12 @@ static int zram_remove_cb(int id, void *ptr, void *data)staticvoiddestroy_devices(void){class_unregister(&zram_control_class);++/* hold the global lock so new device can't be added */+mutex_lock(&zram_index_mutex);idr_for_each(&zram_index_idr,&zram_remove_cb,NULL);+mutex_unlock(&zram_index_mutex);+
Actually zram_index_mutex isn't needed when calling zram_remove_cb()
since the zram-control sysfs interface has been removed, so userspace
can't add new device any more, then the issue is supposed to be fixed
by the following one line change, please test it:
@@ -1985,11 +1985,17 @@ static int zram_remove(struct zram *zram)/* Make sure all the pending I/O are finished */fsync_bdev(bdev);-zram_reset_device(zram);pr_info("Removed device: %s\n",zram->disk->disk_name);del_gendisk(zram->disk);++/*+*resetdeviceaftergendiskisremoved,soanychangefromsysfs+*storewon'tcomein,thenwecanreallyresetdevicehere+*/+zram_reset_device(zram);+blk_cleanup_disk(zram->disk);kfree(zram);return0;
Sorry but nope, the cpu multistate issue is still present and we end up
eventually with page faults. I tried with both patches.
Oct 14 20:21:34 kdevops kernel: ------------[ cut here ]------------
Oct 14 20:21:34 kdevops kernel: Error: Removing state 65 which has
instances left.
Oct 14 20:21:34 kdevops kernel: WARNING: CPU: 4 PID: 3358 at
kernel/cpu.c:2151 __cpuhp_remove_state_cpuslocked+0xf9/0x100
Oct 14 20:21:34 kdevops kernel: Modules linked in: zram(E-) zstd(E)
zsmalloc(E) kvm_intel(E) kvm(E) irqbypass(E) crct10dif_pclmul(E)
crc32_pclmul(E) ghash_clmulni_intel(E) >
Oct 14 20:21:34 kdevops kernel: CPU: 4 PID: 3358 Comm: rmmod Tainted: G
E 5.15.0-rc3-next-20210927+ #89
Oct 14 20:21:34 kdevops kernel: Hardware name: QEMU Standard PC (i440FX
+ PIIX, 1996), BIOS 1.14.0-2 04/01/2014
Oct 14 20:21:34 kdevops kernel: RIP:
0010:__cpuhp_remove_state_cpuslocked+0xf9/0x100
Oct 14 20:21:34 kdevops kernel: Code: 21 00 48 c7 43 18 00 00 00 00 5b
5d 41 5c 41 5d 41 5e 41 5f e9 d8 17 84 00 0f 0b 44 89 e6 48 c7 c7 78 0c
8b ad e8 56 92 7f 00 <0f> 0b >
Oct 14 20:21:34 kdevops kernel: RSP: 0018:ffffaac980a1fe90 EFLAGS:
00010286
Oct 14 20:21:34 kdevops kernel: RAX: 0000000000000000 RBX:
ffffffffada3e208 RCX: 0000000000000000
Oct 14 20:21:34 kdevops kernel: RDX: 0000000000000001 RSI:
ffffffffad8efdb6 RDI: 00000000ffffffff
Oct 14 20:21:34 kdevops kernel: RBP: 0000000000000000 R08:
0000000000000000 R09: ffffaac980a1fcc0
Oct 14 20:21:34 kdevops kernel: R10: ffffaac980a1fcb8 R11:
ffffffffadac3c68 R12: 0000000000000041
Oct 14 20:21:34 kdevops kernel: R13: 0000000000000a28 R14:
0000000000000000 R15: 0000000000000000
Oct 14 20:21:34 kdevops kernel: FS: 00007fc0c2882580(0000)
GS:ffff9ed6f7d00000(0000) knlGS:0000000000000000
Oct 14 20:21:34 kdevops kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
0000000080050033
Oct 14 20:21:34 kdevops kernel: CR2: 00005621b0490b78 CR3:
000000011a538005 CR4: 0000000000370ee0
Oct 14 20:21:34 kdevops kernel: DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Oct 14 20:21:34 kdevops kernel: DR3: 0000000000000000 DR6:
00000000fffe0ff0 DR7: 0000000000000400
Oct 14 20:21:34 kdevops kernel: Call Trace:
Oct 14 20:21:34 kdevops kernel: <TASK>
Oct 14 20:21:34 kdevops kernel: __cpuhp_remove_state+0x4d/0xc0
Oct 14 20:21:34 kdevops kernel: __do_sys_delete_module+0x18d/0x2a0
Oct 14 20:21:34 kdevops kernel: ?
fpregs_assert_state_consistent+0x1e/0x40
Oct 14 20:21:34 kdevops kernel: ? exit_to_user_mode_prepare+0x3a/0x180
Oct 14 20:21:34 kdevops kernel: do_syscall_64+0x38/0xc0
Oct 14 20:21:34 kdevops kernel:
entry_SYSCALL_64_after_hwframe+0x44/0xae
Oct 14 20:21:34 kdevops kernel: RIP: 0033:0x7fc0c29a84a7
<etc>
Oct 14 20:21:35 kdevops kernel: sysfs: cannot create duplicate filename
'/devices/virtual/block/zram0'
Oct 14 20:21:35 kdevops kernel: CPU: 5 PID: 3388 Comm: modprobe Tainted:
G W E 5.15.0-rc3-next-20210927+ #89
Oct 14 20:21:35 kdevops kernel: Hardware name: QEMU Standard PC (i440FX
+ PIIX, 1996), BIOS 1.14.0-2 04/01/2014
Oct 14 20:21:35 kdevops kernel: Call Trace:
Oct 14 20:21:35 kdevops kernel: <TASK>
Oct 14 20:21:35 kdevops kernel: dump_stack_lvl+0x48/0x5e
Oct 14 20:21:35 kdevops kernel: sysfs_warn_dup.cold+0x17/0x24
Oct 14 20:21:35 kdevops kernel: sysfs_create_dir_ns+0xbc/0xd0
Oct 14 20:21:35 kdevops kernel: kobject_add_internal+0xbd/0x2b0
Oct 14 20:21:35 kdevops kernel: kobject_add+0x7e/0xb0
Oct 14 20:21:35 kdevops kernel: ? _raw_spin_unlock_irqrestore+0x25/0x40
Oct 14 20:21:35 kdevops kernel: ? preempt_count_add+0x68/0xa0
Oct 14 20:21:35 kdevops kernel: device_add+0x11a/0x980
Oct 14 20:21:35 kdevops kernel: ? dev_set_name+0x53/0x70
Oct 14 20:21:35 kdevops kernel: device_add_disk+0x9d/0x3a0
Oct 14 20:21:35 kdevops kernel: zram_add+0x1ad/0x200 [zram]
Oct 14 20:21:35 kdevops kernel: ? 0xffffffffc0c10000
Oct 14 20:21:35 kdevops kernel: zram_init+0xd7/0x1000 [zram]
Oct 14 20:21:35 kdevops kernel: do_one_initcall+0x41/0x200
Oct 14 20:21:35 kdevops kernel: ? _raw_spin_unlock_irqrestore+0x25/0x40
Oct 14 20:21:35 kdevops kernel: ? kmem_cache_alloc_trace+0x2ab/0x420
Oct 14 20:21:35 kdevops kernel: do_init_module+0x5c/0x270
Oct 14 20:21:35 kdevops kernel: __do_sys_finit_module+0xae/0x110
Oct 14 20:21:35 kdevops kernel: do_syscall_64+0x38/0xc0
Oct 14 20:21:35 kdevops kernel:
entry_SYSCALL_64_after_hwframe+0x44/0xae
Oct 14 20:21:35 kdevops kernel: RIP: 0033:0x7fca3aa555e9
Oct 14 20:21:35 kdevops kernel: Code: 00 c3 66 2e 0f 1f 84 00 00 00 00
00 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8
4c 8b 4c 24 08 0f 05 <48> 3d >
Oct 14 20:21:35 kdevops kernel: RSP: 002b:00007fff142417b8 EFLAGS:
00000246 ORIG_RAX: 0000000000000139
Oct 14 20:21:35 kdevops kernel: RAX: ffffffffffffffda RBX:
0000558ba9491bd0 RCX: 00007fca3aa555e9
Oct 14 20:21:35 kdevops kernel: RDX: 0000000000000000 RSI:
0000558ba9491f60 RDI: 0000000000000003
Oct 14 20:21:35 kdevops kernel: RBP: 0000000000040000 R08:
0000000000000000 R09: 0000558ba9491db0
Oct 14 20:21:35 kdevops kernel: R10: 0000000000000003 R11:
0000000000000246 R12: 0000558ba9491f60
Oct 14 20:21:35 kdevops kernel: R13: 0000000000000000 R14:
0000558ba9491d00 R15: 0000558ba9491bd0
Oct 14 20:21:35 kdevops kernel: </TASK>
<etc>
Oct 14 20:21:35 kdevops kernel: kobject_add_internal failed for zram0
with -EEXIST, don't try to register things with the same name in the
same directory.
Oct 14 20:21:35 kdevops kernel: ------------[ cut here ]------------
Oct 14 20:21:35 kdevops kernel: WARNING: CPU: 5 PID: 3388 at
block/genhd.c:537 device_add_disk+0x1b9/0x3a0
Oct 14 20:21:35 kdevops kernel: Modules linked in: zram(E+) zstd(E)
zsmalloc(E) kvm_intel(E) kvm(E) irqbypass(E) crct10dif_pclmul(E)
crc32_pclmul(E) ghash_clmulni_intel(E) >
Oct 14 20:21:35 kdevops kernel: CPU: 5 PID: 3388 Comm: modprobe Tainted:
G W E 5.15.0-rc3-next-20210927+ #89
Oct 14 20:21:35 kdevops kernel: Hardware name: QEMU Standard PC (i440FX
+ PIIX, 1996), BIOS 1.14.0-2 04/01/2014
Oct 14 20:21:35 kdevops kernel: RIP: 0010:device_add_disk+0x1b9/0x3a0
Oct 14 20:21:35 kdevops kernel: Code: 00 03 01 00 00 0f 85 32 ff ff ff
e9 1e ff ff ff 0f 0b 41 bc ea ff ff ff e9 29 ff ff ff 4c 89 ff e8 5c 45
1c 00 e9 ef fe ff ff <0f> 0b >
Oct 14 20:21:35 kdevops kernel: RSP: 0018:ffffaac980607d90 EFLAGS:
00010287
Oct 14 20:21:35 kdevops kernel: RAX: 0000000000000000 RBX:
0000000000000000 RCX: 0000000000023005
Oct 14 20:21:35 kdevops kernel: RDX: 0000000000022e05 RSI:
ffffffffacc4b710 RDI: 0000000000000000
Oct 14 20:21:35 kdevops kernel: RBP: ffff9ed5d788a600 R08:
0000000000000000 R09: ffffaac980607a98
Oct 14 20:21:35 kdevops kernel: R10: ffff9ed5c795ef00 R11:
ffffffffadac3c68 R12: 00000000ffffffef
Oct 14 20:21:35 kdevops kernel: R13: ffff9ed5d5600000 R14:
ffffffffc0a52100 R15: ffff9ed5d5600040
Oct 14 20:21:35 kdevops kernel: FS: 00007fca3a935580(0000)
GS:ffff9ed6f7d40000(0000) knlGS:0000000000000000
Oct 14 20:21:35 kdevops kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
0000000080050033
Oct 14 20:21:35 kdevops kernel: CR2: 00007fff1423e6d8 CR3:
0000000136752002 CR4: 0000000000370ee0
Oct 14 20:21:35 kdevops kernel: DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Oct 14 20:21:35 kdevops kernel: DR3: 0000000000000000 DR6:
00000000fffe0ff0 DR7: 0000000000000400
Oct 14 20:21:35 kdevops kernel: Call Trace:
Oct 14 20:21:35 kdevops kernel: <TASK>
Oct 14 20:21:35 kdevops kernel: zram_add+0x1ad/0x200 [zram]
Oct 14 20:21:35 kdevops kernel: ? 0xffffffffc0c10000
Oct 14 20:21:35 kdevops kernel: zram_init+0xd7/0x1000 [zram]
Oct 14 20:21:35 kdevops kernel: do_one_initcall+0x41/0x200
Oct 14 20:21:35 kdevops kernel: ? _raw_spin_unlock_irqrestore+0x25/0x40
Oct 14 20:21:35 kdevops kernel: ? kmem_cache_alloc_trace+0x2ab/0x420
Oct 14 20:21:35 kdevops kernel: do_init_module+0x5c/0x270
Oct 14 20:21:35 kdevops kernel: __do_sys_finit_module+0xae/0x110
Oct 14 20:21:35 kdevops kernel: do_syscall_64+0x38/0xc0
Oct 14 20:21:35 kdevops kernel:
entry_SYSCALL_64_after_hwframe+0x44/0xae
Oct 14 20:21:35 kdevops kernel: RIP: 0033:0x7fca3aa555e9
<etc>
Oct 14 20:21:35 kdevops kernel: ------------[ cut here ]------------
Oct 14 20:21:35 kdevops kernel: WARNING: CPU: 2 PID: 3457 at
block/genhd.c:564 del_gendisk+0x1a2/0x1d0
Oct 14 20:21:35 kdevops kernel: Modules linked in: 842(E)
842_decompress(E) 842_compress(E) zram(E-) zstd(E) zsmalloc(E)
kvm_intel(E) kvm(E) irqbypass(E) crct10dif_pclmul(E>
Oct 14 20:21:35 kdevops kernel: CPU: 2 PID: 3457 Comm: rmmod Tainted: G
W E 5.15.0-rc3-next-20210927+ #89
Oct 14 20:21:35 kdevops kernel: Hardware name: QEMU Standard PC (i440FX
+ PIIX, 1996), BIOS 1.14.0-2 04/01/2014
Oct 14 20:21:35 kdevops kernel: RIP: 0010:del_gendisk+0x1a2/0x1d0
Oct 14 20:21:35 kdevops kernel: Code: 48 8d 78 40 e8 8f 87 1d 00 48 8b
7b 40 5b 5d 41 5c 48 83 c7 40 e9 4e 47 1c 00 48 8b 70 40 eb ce f6 43 61
04 0f 85 85 fe ff ff <0f> 0b >
Oct 14 20:21:35 kdevops kernel: RSP: 0018:ffffaac9807cfe30 EFLAGS:
00010246
Oct 14 20:21:35 kdevops kernel: RAX: ffff9ed5d5600380 RBX:
ffff9ed5d788a600 RCX: 0000000000000000
Oct 14 20:21:35 kdevops kernel: RDX: 0000000000000000 RSI:
ffffffffad8efdb6 RDI: ffff9ed5d788a600
Oct 14 20:21:35 kdevops kernel: RBP: ffff9ed5d788b600 R08:
0000000000000000 R09: ffffaac9807cfc88
Oct 14 20:21:35 kdevops kernel: R10: ffffaac9807cfc80 R11:
ffffffffadac3c68 R12: ffff9ed5d5600000
Oct 14 20:21:35 kdevops kernel: R13: 0000000000000000 R14:
ffffffffc0a52360 R15: ffff9ed5c4a87b78
Oct 14 20:21:35 kdevops kernel: FS: 00007f292a2bb580(0000)
GS:ffff9ed6f7c80000(0000) knlGS:0000000000000000
Oct 14 20:21:35 kdevops kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
0000000080050033
Oct 14 20:21:35 kdevops kernel: CR2: 000056161b453b78 CR3:
000000013213e002 CR4: 0000000000370ee0
Oct 14 20:21:35 kdevops kernel: DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Oct 14 20:21:35 kdevops kernel: DR3: 0000000000000000 DR6:
00000000fffe0ff0 DR7: 0000000000000400
Oct 14 20:21:35 kdevops kernel: Call Trace:
Oct 14 20:21:35 kdevops kernel: <TASK>
Oct 14 20:21:35 kdevops kernel: zram_remove+0x96/0xc0 [zram]
Oct 14 20:21:35 kdevops kernel: ? hot_remove_store+0xe0/0xe0 [zram]
Oct 14 20:21:35 kdevops kernel: zram_remove_cb+0xd/0x10 [zram]
Oct 14 20:21:35 kdevops kernel: idr_for_each+0x5b/0xd0
Oct 14 20:21:35 kdevops kernel: destroy_devices+0x32/0x68 [zram]
Oct 14 20:21:35 kdevops kernel: __do_sys_delete_module+0x18d/0x2a0
Oct 14 20:21:35 kdevops kernel: ?
fpregs_assert_state_consistent+0x1e/0x40
Oct 14 20:21:35 kdevops kernel: ? exit_to_user_mode_prepare+0x3a/0x180
Oct 14 20:21:35 kdevops kernel: do_syscall_64+0x38/0xc0
Oct 14 20:21:35 kdevops kernel:
entry_SYSCALL_64_after_hwframe+0x44/0xae
Oct 14 20:21:35 kdevops kernel: RIP: 0033:0x7f292a3e14a7
<etc>
Oct 14 20:21:35 kdevops kernel: BUG: unable to handle page fault for
address: ffffffffc0a4e0ae
Oct 14 20:21:35 kdevops kernel: #PF: supervisor instruction fetch in
kernel mode
Oct 14 20:21:35 kdevops kernel: #PF: error_code(0x0010) - not-present
page
Oct 14 20:21:35 kdevops kernel: PGD 3ba0e067 P4D 3ba0e067 PUD 3ba10067
PMD 10526c067 PTE 0
Oct 14 20:21:35 kdevops kernel: Oops: 0010 [#1] PREEMPT SMP NOPTI
Oct 14 20:21:35 kdevops kernel: CPU: 6 PID: 3655 Comm: zram02.sh
Tainted: G W E 5.15.0-rc3-next-20210927+ #89
Oct 14 20:21:35 kdevops kernel: Hardware name: QEMU Standard PC (i440FX
+ PIIX, 1996), BIOS 1.14.0-2 04/01/2014
Oct 14 20:21:35 kdevops kernel: RIP: 0010:0xffffffffc0a4e0ae
Oct 14 20:21:35 kdevops kernel: Code: Unable to access opcode bytes at
RIP 0xffffffffc0a4e084.
Oct 14 20:21:35 kdevops kernel: RSP: 0018:ffffaac980687da8 EFLAGS:
00010286
Oct 14 20:21:35 kdevops kernel: RAX: 0000000000000000 RBX:
ffff9ed5c40be400 RCX: 0000000080400035
Oct 14 20:21:35 kdevops kernel: RDX: 0000000080400036 RSI:
fffffa3544561080 RDI: 0000000040000000
Oct 14 20:21:35 kdevops kernel: RBP: 0000000001900000 R08:
ffff9ed5d5842cc0 R09: 0000000080400035
Oct 14 20:21:35 kdevops kernel: R10: ffff9ed5d5842c00 R11:
ffff9ed5f1341350 R12: 0000000001900000
Oct 14 20:21:35 kdevops kernel: R13: ffff9ed5d5666c00 R14:
ffff9ed5c40be420 R15: ffff9ed5dfa8c8c0
Oct 14 20:21:35 kdevops kernel: FS: 00007f978fe2d5c0(0000)
GS:ffff9ed6f7d80000(0000) knlGS:0000000000000000
Oct 14 20:21:35 kdevops kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
0000000080050033
Oct 14 20:21:35 kdevops kernel: CR2: ffffffffc0a4e084 CR3:
0000000133fd4006 CR4: 0000000000370ee0
Oct 14 20:21:35 kdevops kernel: DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Oct 14 20:21:35 kdevops kernel: DR3: 0000000000000000 DR6:
00000000fffe0ff0 DR7: 0000000000000400
Oct 14 20:21:35 kdevops kernel: Call Trace:
Oct 14 20:21:35 kdevops kernel: <TASK>
Oct 14 20:21:35 kdevops kernel: ? kernfs_fop_write_iter+0x177/0x220
Oct 14 20:21:35 kdevops kernel: ? new_sync_write+0x11c/0x1b0
Oct 14 20:21:35 kdevops kernel: ? vfs_write+0x20d/0x2a0
Oct 14 20:21:35 kdevops kernel: ? ksys_write+0x5f/0xe0
Oct 14 20:21:35 kdevops kernel: ? do_syscall_64+0x38/0xc0
Oct 14 20:21:35 kdevops kernel: ?
entry_SYSCALL_64_after_hwframe+0x44/0xae
Oct 14 20:21:35 kdevops kernel: </TASK>
<etc, etc, etc, this goes on and on>
Luis
On Thu, Oct 14, 2021 at 01:24:32PM -0700, Luis Chamberlain wrote:
On Thu, Oct 14, 2021 at 10:11:46AM +0800, Ming Lei wrote:
quoted
On Thu, Oct 14, 2021 at 09:55:48AM +0800, Ming Lei wrote:
quoted
On Mon, Sep 27, 2021 at 09:38:04AM -0700, Luis Chamberlain wrote:
...
quoted
Hello Luis,
Can you test the following patch and see if the issue can be addressed?
Please see the idea from the inline comment.
Also zram_index_mutex isn't needed in zram disk's store() compared with
your patch, then the deadlock issue you are addressing in this series can
be avoided.
@@ -1985,11 +1985,17 @@ static int zram_remove(struct zram *zram)/* Make sure all the pending I/O are finished */fsync_bdev(bdev);-zram_reset_device(zram);pr_info("Removed device: %s\n",zram->disk->disk_name);del_gendisk(zram->disk);++/*+*resetdeviceaftergendiskisremoved,soanychangefromsysfs+*storewon'tcomein,thenwecanreallyresetdevicehere+*/+zram_reset_device(zram);+blk_cleanup_disk(zram->disk);kfree(zram);return0;
@@ -2073,7 +2079,12 @@ static int zram_remove_cb(int id, void *ptr, void *data)staticvoiddestroy_devices(void){class_unregister(&zram_control_class);++/* hold the global lock so new device can't be added */+mutex_lock(&zram_index_mutex);idr_for_each(&zram_index_idr,&zram_remove_cb,NULL);+mutex_unlock(&zram_index_mutex);+
Actually zram_index_mutex isn't needed when calling zram_remove_cb()
since the zram-control sysfs interface has been removed, so userspace
can't add new device any more, then the issue is supposed to be fixed
by the following one line change, please test it:
@@ -1985,11 +1985,17 @@ static int zram_remove(struct zram *zram)/* Make sure all the pending I/O are finished */fsync_bdev(bdev);-zram_reset_device(zram);pr_info("Removed device: %s\n",zram->disk->disk_name);del_gendisk(zram->disk);++/*+*resetdeviceaftergendiskisremoved,soanychangefromsysfs+*storewon'tcomein,thenwecanreallyresetdevicehere+*/+zram_reset_device(zram);+blk_cleanup_disk(zram->disk);kfree(zram);return0;
Sorry but nope, the cpu multistate issue is still present and we end up
eventually with page faults. I tried with both patches.
In theory disksize_store() can't come in after del_gendisk() returns,
then zram_reset_device() should cleanup everything, that is the issue
you described in commit log.
We need to understand the exact reason why there is still cpuhp node
left, can you share us the exact steps for reproducing the issue?
Otherwise we may have to trace and narrow down the reason.
thanks,
Ming
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-10-15 00:23:04
On Fri, Oct 15, 2021 at 07:52:04AM +0800, Ming Lei wrote:
On Thu, Oct 14, 2021 at 01:24:32PM -0700, Luis Chamberlain wrote:
quoted
On Thu, Oct 14, 2021 at 10:11:46AM +0800, Ming Lei wrote:
quoted
On Thu, Oct 14, 2021 at 09:55:48AM +0800, Ming Lei wrote:
quoted
On Mon, Sep 27, 2021 at 09:38:04AM -0700, Luis Chamberlain wrote:
...
quoted
Hello Luis,
Can you test the following patch and see if the issue can be addressed?
Please see the idea from the inline comment.
Also zram_index_mutex isn't needed in zram disk's store() compared with
your patch, then the deadlock issue you are addressing in this series can
be avoided.
@@ -1985,11 +1985,17 @@ static int zram_remove(struct zram *zram)/* Make sure all the pending I/O are finished */fsync_bdev(bdev);-zram_reset_device(zram);pr_info("Removed device: %s\n",zram->disk->disk_name);del_gendisk(zram->disk);++/*+*resetdeviceaftergendiskisremoved,soanychangefromsysfs+*storewon'tcomein,thenwecanreallyresetdevicehere+*/+zram_reset_device(zram);+blk_cleanup_disk(zram->disk);kfree(zram);return0;
@@ -2073,7 +2079,12 @@ static int zram_remove_cb(int id, void *ptr, void *data)staticvoiddestroy_devices(void){class_unregister(&zram_control_class);++/* hold the global lock so new device can't be added */+mutex_lock(&zram_index_mutex);idr_for_each(&zram_index_idr,&zram_remove_cb,NULL);+mutex_unlock(&zram_index_mutex);+
Actually zram_index_mutex isn't needed when calling zram_remove_cb()
since the zram-control sysfs interface has been removed, so userspace
can't add new device any more, then the issue is supposed to be fixed
by the following one line change, please test it:
@@ -1985,11 +1985,17 @@ static int zram_remove(struct zram *zram)/* Make sure all the pending I/O are finished */fsync_bdev(bdev);-zram_reset_device(zram);pr_info("Removed device: %s\n",zram->disk->disk_name);del_gendisk(zram->disk);++/*+*resetdeviceaftergendiskisremoved,soanychangefromsysfs+*storewon'tcomein,thenwecanreallyresetdevicehere+*/+zram_reset_device(zram);+blk_cleanup_disk(zram->disk);kfree(zram);return0;
Sorry but nope, the cpu multistate issue is still present and we end up
eventually with page faults. I tried with both patches.
In theory disksize_store() can't come in after del_gendisk() returns,
then zram_reset_device() should cleanup everything, that is the issue
you described in commit log.
We need to understand the exact reason why there is still cpuhp node
left, can you share us the exact steps for reproducing the issue?
Otherwise we may have to trace and narrow down the reason.
See my commit log for my own fix for this issue.
Luis
On Thu, Oct 14, 2021 at 05:22:40PM -0700, Luis Chamberlain wrote:
On Fri, Oct 15, 2021 at 07:52:04AM +0800, Ming Lei wrote:
...
quoted
We need to understand the exact reason why there is still cpuhp node
left, can you share us the exact steps for reproducing the issue?
Otherwise we may have to trace and narrow down the reason.
See my commit log for my own fix for this issue.
OK, thanks!
I can reproduce the issue, and the reason is that reset_store fails
zram_remove() when unloading module, then the warning is caused.
The top 3 patches in the following tree can fix the issue:
https://github.com/ming1/linux/commits/my_v5.15-blk-dev
Thanks,
Ming
On Fri, Oct 15, 2021 at 04:36:11PM +0800, Ming Lei wrote:
On Thu, Oct 14, 2021 at 05:22:40PM -0700, Luis Chamberlain wrote:
quoted
On Fri, Oct 15, 2021 at 07:52:04AM +0800, Ming Lei wrote:
...
quoted
quoted
We need to understand the exact reason why there is still cpuhp node
left, can you share us the exact steps for reproducing the issue?
Otherwise we may have to trace and narrow down the reason.
See my commit log for my own fix for this issue.
OK, thanks!
I can reproduce the issue, and the reason is that reset_store fails
zram_remove() when unloading module, then the warning is caused.
The top 3 patches in the following tree can fix the issue:
https://github.com/ming1/linux/commits/my_v5.15-blk-dev
At a quick glance, those look sane to me, nice work.
greg k-h
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-10-15 17:31:45
On Fri, Oct 15, 2021 at 04:36:11PM +0800, Ming Lei wrote:
On Thu, Oct 14, 2021 at 05:22:40PM -0700, Luis Chamberlain wrote:
quoted
On Fri, Oct 15, 2021 at 07:52:04AM +0800, Ming Lei wrote:
...
quoted
quoted
We need to understand the exact reason why there is still cpuhp node
left, can you share us the exact steps for reproducing the issue?
Otherwise we may have to trace and narrow down the reason.
See my commit log for my own fix for this issue.
OK, thanks!
I can reproduce the issue, and the reason is that reset_store fails
zram_remove() when unloading module, then the warning is caused.
The top 3 patches in the following tree can fix the issue:
https://github.com/ming1/linux/commits/my_v5.15-blk-dev
Thanks for trying an alternative fix! A crash stops yes, however this
also ends up leaving the driver in an unrecoverable state after a few
tries. Ie, you CTRL-C the scripts and try again over and over again and
the driver ends up in a situation where it just says:
zram: Can't change algorithm for initialized device
And the zram module can't be removed at that point.
Luis
On Fri, Oct 15, 2021 at 10:31:31AM -0700, Luis Chamberlain wrote:
On Fri, Oct 15, 2021 at 04:36:11PM +0800, Ming Lei wrote:
quoted
On Thu, Oct 14, 2021 at 05:22:40PM -0700, Luis Chamberlain wrote:
quoted
On Fri, Oct 15, 2021 at 07:52:04AM +0800, Ming Lei wrote:
...
quoted
quoted
We need to understand the exact reason why there is still cpuhp node
left, can you share us the exact steps for reproducing the issue?
Otherwise we may have to trace and narrow down the reason.
See my commit log for my own fix for this issue.
OK, thanks!
I can reproduce the issue, and the reason is that reset_store fails
zram_remove() when unloading module, then the warning is caused.
The top 3 patches in the following tree can fix the issue:
https://github.com/ming1/linux/commits/my_v5.15-blk-dev
Thanks for trying an alternative fix! A crash stops yes, however this
I doubt it is alternative since your patchset doesn't mention the exact
reason of 'Error: Removing state 63 which has instances left.', that is
simply caused by failing to remove zram because ->claim is set during
unloading module.
Yeah, you mentioned the race between disksize_store() vs. zram_remove(),
however I don't think it is reproduced easily in the test because the race
window is pretty small, also it can be fixed easily in my 3rd path
without any complicated tricks.
Not dig into details of your patchset via grabbing module reference
count during show/store attribute of kernfs which is done in your patch
9, but IMO this way isn't necessary:
1) any driver module has to cleanup anything which may refer to symbols
or data defined in module_exit of this driver
2) device_del() is often done in module_exit(), once device_del()
returns, no any new show/store on the device's kobject attribute
is possible.
3) it is _not_ a must or pattern for fixing bugs to hold one lock before
calling device_del(), meantime the lock is required in the device's
attribute show()/store(), which causes AA deadlock easily. Your approach
just avoids the issue by not releasing module until all show/store are
done.
Also the model of using module refcount is usually that if anyone will
use the module, grab one extra ref, and once the use is done, release
it. For example of block device, the driver's module refcnt is grabbed
when the disk/part is opened, and released when the disk/part is closed.
also ends up leaving the driver in an unrecoverable state after a few
tries. Ie, you CTRL-C the scripts and try again over and over again and
the driver ends up in a situation where it just says:
zram: Can't change algorithm for initialized device
It means the algorithm can't be changed for one initialized device
at the exact time. That is understandable because two zram02.sh are
running concurrently.
Your test script just runs two ./zram02.sh tasks concurrently forever,
so what is your expected result for the test? Of course, it can't be
over.
I can't reproduce the 'unrecoverable' state in my test, can you share the
stack trace log after that happens?
Is the zram02.sh still running or slept somewhere in the 'unrecoverable'
state? If it is still running, it means the current sleep point isn't
interruptable when running 'CTRL-C'. In my test, after several 'CTRL-C',
both the two zram02.sh started from two terminals can be terminated. If
it is slept somewhere forever, it can be one problem.
And the zram module can't be removed at that point.
It is just that systemd opens the zram or the disk is opened as swap
disk, and once systemd closes it or after you run swapoff, it can be
unloaded.
Thanks,
Ming
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-10-18 19:32:34
On Sat, Oct 16, 2021 at 07:28:39PM +0800, Ming Lei wrote:
On Fri, Oct 15, 2021 at 10:31:31AM -0700, Luis Chamberlain wrote:
quoted
On Fri, Oct 15, 2021 at 04:36:11PM +0800, Ming Lei wrote:
quoted
On Thu, Oct 14, 2021 at 05:22:40PM -0700, Luis Chamberlain wrote:
quoted
On Fri, Oct 15, 2021 at 07:52:04AM +0800, Ming Lei wrote:
...
quoted
quoted
We need to understand the exact reason why there is still cpuhp node
left, can you share us the exact steps for reproducing the issue?
Otherwise we may have to trace and narrow down the reason.
See my commit log for my own fix for this issue.
OK, thanks!
I can reproduce the issue, and the reason is that reset_store fails
zram_remove() when unloading module, then the warning is caused.
The top 3 patches in the following tree can fix the issue:
https://github.com/ming1/linux/commits/my_v5.15-blk-dev
Thanks for trying an alternative fix! A crash stops yes, however this
I doubt it is alternative since your patchset doesn't mention the exact
reason of 'Error: Removing state 63 which has instances left.', that is
simply caused by failing to remove zram because ->claim is set during
unloading module.
Well I disagree because it does explain how the race can happen, and it
also explains how since the sysfs interface is exposed until module
removal completes, it leaves exposed knobs to allow re-initializing of a
struct zcomp for a zram device before the exit.
Yeah, you mentioned the race between disksize_store() vs. zram_remove(),
however I don't think it is reproduced easily in the test because the race
window is pretty small, also it can be fixed easily in my 3rd path
without any complicated tricks.
Reproducing for me is... extremely easy.
Not dig into details of your patchset via grabbing module reference
count during show/store attribute of kernfs which is done in your patch
9, but IMO this way isn't necessary:
That's to address the deadlock only.
1) any driver module has to cleanup anything which may refer to symbols
or data defined in module_exit of this driver
Yes, and as the cpu multistate hotplug documentation warns (although
such documentation is kind of hidden) that driver authors need to be
careful with module removal too, refer to the warning at the end of
__cpuhp_remove_state_cpuslocked() about module removal.
2) device_del() is often done in module_exit(), once device_del()
returns, no any new show/store on the device's kobject attribute
is possible.
Right and if a syfs knob is exposed before device_del() completely
and is allowed to do things, the driver should take care to prevent
races for CPU multistate support. The small state machine I added ensures
we don't run over any expectations from cpu hotplug multistate support.
I've *never* suggested there cannot be alternatives to my solution with
the small state machine, but for you to say it is incorrect is simply
not right either.
3) it is _not_ a must or pattern for fixing bugs to hold one lock before
calling device_del(), meantime the lock is required in the device's
attribute show()/store(), which causes AA deadlock easily. Your approach
just avoids the issue by not releasing module until all show/store are
done.
Right, there are two approaches here:
a) Your approach is to accept the deadlock as a requirement and so
you would prefer to implement an alternative to using a shared lock
on module exit and sysfs op.
b) While I address such a deadlock head on as I think this sort of locking
be allowed for two reasons:
b1) as we never documented such requirement otherwise.
b2) There is a possibility that other drivers already exist too
which *do* use a shared lock on module removal and sysfs ops
(and I just confirmed this to be true)
By you only addressing the deadlock as a requirement on approach a) you are
forgetting that there *may* already be present drivers which *do* implement
such patterns in the kernel. I worked on addressing the deadlock because
I was informed livepatching *did* have that issue as well and so very
likely a generic solution to the deadlock could be beneficial to other
random drivers.
So I *really* don't think it is wise for us to simply accept this new
found deadlock as a *new* requirement, specially if we can fix it easily.
A cursory review using Coccinelle potential issues with mutex lock
directly used on module exit (so this doesn't cover drivers like zram
which uses a routine and then grabs the lock through indirection) and a
sysfs op shows these drivers are also affected by this deadlock:
* arch/powerpc/sysdev/fsl_mpic_timer_wakeup.c
* lib/test_firmware.c
Note that this cursory review does not cover spin_lock uses, and other
forms locks. Consider the case where a routine is used and then that
routine grabs a lock, so one level indirection. There are many levels
of indirections possible here. And likewise there are different types
of locks.
quoted
also ends up leaving the driver in an unrecoverable state after a few
tries. Ie, you CTRL-C the scripts and try again over and over again and
the driver ends up in a situation where it just says:
zram: Can't change algorithm for initialized device
It means the algorithm can't be changed for one initialized device
at the exact time. That is understandable because two zram02.sh are
running concurrently.
Indeed but with your patch it can get stuck and cannot be taken out of this
state.
Your test script just runs two ./zram02.sh tasks concurrently forever,
so what is your expected result for the test? Of course, it can't be
over.
I can't reproduce the 'unrecoverable' state in my test, can you share the
stack trace log after that happens?
Try a bit harder, cancel the scripts after running for a while randomly
(CTRL C a few times until the script finishes) and have them race again.
Do this a few times.
quoted
And the zram module can't be removed at that point.
It is just that systemd opens the zram or the disk is opened as swap
disk, and once systemd closes it or after you run swapoff, it can be
unloaded.
On Mon, Oct 18, 2021 at 12:32:11PM -0700, Luis Chamberlain wrote:
On Sat, Oct 16, 2021 at 07:28:39PM +0800, Ming Lei wrote:
quoted
On Fri, Oct 15, 2021 at 10:31:31AM -0700, Luis Chamberlain wrote:
quoted
On Fri, Oct 15, 2021 at 04:36:11PM +0800, Ming Lei wrote:
quoted
On Thu, Oct 14, 2021 at 05:22:40PM -0700, Luis Chamberlain wrote:
quoted
On Fri, Oct 15, 2021 at 07:52:04AM +0800, Ming Lei wrote:
...
quoted
quoted
We need to understand the exact reason why there is still cpuhp node
left, can you share us the exact steps for reproducing the issue?
Otherwise we may have to trace and narrow down the reason.
See my commit log for my own fix for this issue.
OK, thanks!
I can reproduce the issue, and the reason is that reset_store fails
zram_remove() when unloading module, then the warning is caused.
The top 3 patches in the following tree can fix the issue:
https://github.com/ming1/linux/commits/my_v5.15-blk-dev
Thanks for trying an alternative fix! A crash stops yes, however this
I doubt it is alternative since your patchset doesn't mention the exact
reason of 'Error: Removing state 63 which has instances left.', that is
simply caused by failing to remove zram because ->claim is set during
unloading module.
Well I disagree because it does explain how the race can happen, and it
also explains how since the sysfs interface is exposed until module
removal completes, it leaves exposed knobs to allow re-initializing of a
struct zcomp for a zram device before the exit.
quoted
Yeah, you mentioned the race between disksize_store() vs. zram_remove(),
however I don't think it is reproduced easily in the test because the race
window is pretty small, also it can be fixed easily in my 3rd path
without any complicated tricks.
Reproducing for me is... extremely easy.
In my observation, failing zram_remove() is extremely easy to trigger, which
is caused by reset_store() which sets ->reclaim as true, so
zram_remove() is failed and zram_reset_device() is bypassed , then the
failure of 'Error: Removing state 63 which has instances left.' is caused.
We are in same page?
quoted
Not dig into details of your patchset via grabbing module reference
count during show/store attribute of kernfs which is done in your patch
9, but IMO this way isn't necessary:
That's to address the deadlock only.
quoted
1) any driver module has to cleanup anything which may refer to symbols
or data defined in module_exit of this driver
Yes, and as the cpu multistate hotplug documentation warns (although
such documentation is kind of hidden) that driver authors need to be
careful with module removal too, refer to the warning at the end of
__cpuhp_remove_state_cpuslocked() about module removal.
It is zram's bug. zram has to clean everything in module_exit(),
unfortunately zram_remove() can be failed when calling from
module_exit() because ->claim is set as true by reset_store(), then
zram_reset_device()(->zcomp_destroy) isn't called, and this failure should
not happen when unloading module, should it?
quoted
2) device_del() is often done in module_exit(), once device_del()
returns, no any new show/store on the device's kobject attribute
is possible.
Right and if a syfs knob is exposed before device_del() completely
and is allowed to do things, the driver should take care to prevent
races for CPU multistate support. The small state machine I added ensures
What is the race for CPU multistate support? If you mean 'Error: Removing
state 63 which has instances left.', it is zram's bug since zram has to
cleanup everything in module_exit().
we don't run over any expectations from cpu hotplug multistate support.
I've *never* suggested there cannot be alternatives to my solution with
the small state machine, but for you to say it is incorrect is simply
not right either.
quoted
3) it is _not_ a must or pattern for fixing bugs to hold one lock before
calling device_del(), meantime the lock is required in the device's
attribute show()/store(), which causes AA deadlock easily. Your approach
just avoids the issue by not releasing module until all show/store are
done.
Right, there are two approaches here:
a) Your approach is to accept the deadlock as a requirement and so
you would prefer to implement an alternative to using a shared lock
on module exit and sysfs op.
wrt. in-tree zram, there is neither any deadlock in linus tree, nor after
applying my 3 patches. If you think there is, please share us the code
or lockdep warning.
b) While I address such a deadlock head on as I think this sort of locking
be allowed for two reasons:
b1) as we never documented such requirement otherwise.
b2) There is a possibility that other drivers already exist too
which *do* use a shared lock on module removal and sysfs ops
(and I just confirmed this to be true)
The 'deadlock' is actually caused by your out-of-tree patch of 'zram: fix
crashes with cpu hotplug multistate' which adds mutex_lock(zram_index_mutex)
in destroy_devices().
We can fix this issue easily without needing the global lock, please see the
attached(pre-V2) patch.
By you only addressing the deadlock as a requirement on approach a) you are
forgetting that there *may* already be present drivers which *do* implement
such patterns in the kernel. I worked on addressing the deadlock because
I was informed livepatching *did* have that issue as well and so very
likely a generic solution to the deadlock could be beneficial to other
random drivers.
In-tree zram doesn't have such deadlock, if livepatching has such AA deadlock,
just fixed it, and seems it has been fixed by 3ec24776bfd0.
So I *really* don't think it is wise for us to simply accept this new
found deadlock as a *new* requirement, specially if we can fix it easily.
A cursory review using Coccinelle potential issues with mutex lock
directly used on module exit (so this doesn't cover drivers like zram
which uses a routine and then grabs the lock through indirection) and a
sysfs op shows these drivers are also affected by this deadlock:
* arch/powerpc/sysdev/fsl_mpic_timer_wakeup.c
In fsl_wakeup_sys_exit(), device_remove_file() is called before
acquiring &sysfs_lock, so there shouldn't be such AA deadlock.
* lib/test_firmware.c
Yeah, there is the AA deadlock risk, but it should be fixed by moving
misc_deregister() out of &test_fw_mutex.
Note that this cursory review does not cover spin_lock uses, and other
forms locks. Consider the case where a routine is used and then that
routine grabs a lock, so one level indirection. There are many levels
of indirections possible here. And likewise there are different types
of locks.
quoted
quoted
also ends up leaving the driver in an unrecoverable state after a few
tries. Ie, you CTRL-C the scripts and try again over and over again and
the driver ends up in a situation where it just says:
zram: Can't change algorithm for initialized device
It means the algorithm can't be changed for one initialized device
at the exact time. That is understandable because two zram02.sh are
running concurrently.
Indeed but with your patch it can get stuck and cannot be taken out of this
state.
OK, I can keep current behavior: fail open() in case of removing or
resetting, meantime not hold open_mutex when sync bdev and reset device,
see attached patch.
quoted
Your test script just runs two ./zram02.sh tasks concurrently forever,
so what is your expected result for the test? Of course, it can't be
over.
I can't reproduce the 'unrecoverable' state in my test, can you share the
stack trace log after that happens?
Try a bit harder, cancel the scripts after running for a while randomly
(CTRL C a few times until the script finishes) and have them race again.
Do this a few times.
quoted
quoted
And the zram module can't be removed at that point.
It is just that systemd opens the zram or the disk is opened as swap
disk, and once systemd closes it or after you run swapoff, it can be
unloaded.
With my patch this issues does not happen.
It is because the patch 2 holds ->open_mutex() for sync bdev and reset
zram, so several 'CTRL-C' is needed for terminating the test script, then
zram02.sh's cleanup handler can be interrupted too. We can keep current
behavior easily.
Please try the following patch against upstream(linus or next) tree(basically
fold revised 2 and 3 of V1, and cover two issues: not fail zram_remove in
module_exit(), race between zram_remove() and disksize_store()), and see if
everything is fine for you:
@@ -1967,25 +1967,45 @@ static int zram_add(void)staticintzram_remove(structzram*zram){structblock_device*bdev=zram->disk->part0;+boolclaimed;mutex_lock(&bdev->bd_disk->open_mutex);-if(bdev->bd_openers||zram->claim){+if(bdev->bd_openers){mutex_unlock(&bdev->bd_disk->open_mutex);return-EBUSY;}-zram->claim=true;+claimed=zram->claim;+if(!claimed)+zram->claim=true;mutex_unlock(&bdev->bd_disk->open_mutex);zram_debugfs_unregister(zram);-/* Make sure all the pending I/O are finished */-fsync_bdev(bdev);-zram_reset_device(zram);+if(claimed){+/*+*Ifwewereclaimedbyreset_store(),del_gendisk()will+*waituntilsync&resetiscompleted,sodonothinghere.+*/+;+}else{+/* Make sure all the pending I/O are finished */+sync_blockdev(bdev);+zram_reset_device(zram);+}pr_info("Removed device: %s\n",zram->disk->disk_name);del_gendisk(zram->disk);++WARN_ON_ONCE(claimed&&zram->claim);++/*+*disksizestoremaycomeaftertheabovezram_reset_device+*returns,sorunthelastresettoavoidtherace+*/+zram_reset_device(zram);+blk_cleanup_disk(zram->disk);kfree(zram);return0;
From: Miroslav Benes <mbenes@suse.cz> Date: 2021-10-19 06:23:58
quoted
By you only addressing the deadlock as a requirement on approach a) you are
forgetting that there *may* already be present drivers which *do* implement
such patterns in the kernel. I worked on addressing the deadlock because
I was informed livepatching *did* have that issue as well and so very
likely a generic solution to the deadlock could be beneficial to other
random drivers.
In-tree zram doesn't have such deadlock, if livepatching has such AA deadlock,
just fixed it, and seems it has been fixed by 3ec24776bfd0.
I would not call it a fix. It is a kind of ugly workaround because the
generic infrastructure lacked (lacks) the proper support in my opinion.
Luis is trying to fix that.
Just my two cents.
Miroslav
On Tue, Oct 19, 2021 at 08:23:51AM +0200, Miroslav Benes wrote:
quoted
quoted
By you only addressing the deadlock as a requirement on approach a) you are
forgetting that there *may* already be present drivers which *do* implement
such patterns in the kernel. I worked on addressing the deadlock because
I was informed livepatching *did* have that issue as well and so very
likely a generic solution to the deadlock could be beneficial to other
random drivers.
In-tree zram doesn't have such deadlock, if livepatching has such AA deadlock,
just fixed it, and seems it has been fixed by 3ec24776bfd0.
I would not call it a fix. It is a kind of ugly workaround because the
generic infrastructure lacked (lacks) the proper support in my opinion.
Luis is trying to fix that.
What is the proper support of the generic infrastructure? I am not
familiar with livepatching's model(especially with module unload), you mean
livepatching have to do the following way from sysfs:
1) during module exit:
mutex_lock(lp_lock);
kobject_put(lp_kobj);
mutex_unlock(lp_lock);
2) show()/store() method of attributes of lp_kobj
mutex_lock(lp_lock)
...
mutex_unlock(lp_lock)
IMO, the above usage simply caused AA deadlock. Even in Luis's patch
'zram: fix crashes with cpu hotplug multistate', new/same AA deadlock
(hot_remove_store() vs. disksize_store() or reset_store()) is added
because hot_remove_store() isn't called from module_exit().
Luis tries to delay unloading module until all show()/store() are done. But
that can be obtained by the following way simply during module_exit():
kobject_del(lp_kobj); //all pending store()/show() from lp_kobj are done,
//no new store()/show() can come after
//kobject_del() returns
mutex_lock(lp_lock);
kobject_put(lp_kobj);
mutex_unlock(lp_lock);
Or can you explain your requirement on kobject/module unload in a bit
details?
Thanks,
Ming
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-10-19 15:28:38
On Tue, Oct 19, 2021 at 10:34:41AM +0800, Ming Lei wrote:
Please try the following patch against upstream(linus or next) tree(basically
fold revised 2 and 3 of V1, and cover two issues: not fail zram_remove in
module_exit(), race between zram_remove() and disksize_store()), and see if
everything is fine for you:
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-10-19 15:50:39
On Tue, Oct 19, 2021 at 10:34:41AM +0800, Ming Lei wrote:
On Mon, Oct 18, 2021 at 12:32:11PM -0700, Luis Chamberlain wrote:
quoted
On Sat, Oct 16, 2021 at 07:28:39PM +0800, Ming Lei wrote:
quoted
On Fri, Oct 15, 2021 at 10:31:31AM -0700, Luis Chamberlain wrote:
quoted
On Fri, Oct 15, 2021 at 04:36:11PM +0800, Ming Lei wrote:
quoted
On Thu, Oct 14, 2021 at 05:22:40PM -0700, Luis Chamberlain wrote:
quoted
On Fri, Oct 15, 2021 at 07:52:04AM +0800, Ming Lei wrote:
...
quoted
quoted
We need to understand the exact reason why there is still cpuhp node
left, can you share us the exact steps for reproducing the issue?
Otherwise we may have to trace and narrow down the reason.
See my commit log for my own fix for this issue.
OK, thanks!
I can reproduce the issue, and the reason is that reset_store fails
zram_remove() when unloading module, then the warning is caused.
The top 3 patches in the following tree can fix the issue:
https://github.com/ming1/linux/commits/my_v5.15-blk-dev
Thanks for trying an alternative fix! A crash stops yes, however this
I doubt it is alternative since your patchset doesn't mention the exact
reason of 'Error: Removing state 63 which has instances left.', that is
simply caused by failing to remove zram because ->claim is set during
unloading module.
Well I disagree because it does explain how the race can happen, and it
also explains how since the sysfs interface is exposed until module
removal completes, it leaves exposed knobs to allow re-initializing of a
struct zcomp for a zram device before the exit.
quoted
Yeah, you mentioned the race between disksize_store() vs. zram_remove(),
however I don't think it is reproduced easily in the test because the race
window is pretty small, also it can be fixed easily in my 3rd path
without any complicated tricks.
Reproducing for me is... extremely easy.
In my observation, failing zram_remove() is extremely easy to trigger, which
is caused by reset_store() which sets ->reclaim as true, so
zram_remove() is failed and zram_reset_device() is bypassed , then the
failure of 'Error: Removing state 63 which has instances left.' is caused.
We are in same page?
The actual first issue is the CPU hotplug remove callback is long gone and
in the meantime we allow a race to add a new "instance", in the zram
driver's case a cpu struct zcomp instance though the sysfs interface.
Regardless of if zram_remove() can fail or not, the above race needs to
be addressed.
quoted
quoted
Not dig into details of your patchset via grabbing module reference
count during show/store attribute of kernfs which is done in your patch
9, but IMO this way isn't necessary:
That's to address the deadlock only.
quoted
1) any driver module has to cleanup anything which may refer to symbols
or data defined in module_exit of this driver
Yes, and as the cpu multistate hotplug documentation warns (although
such documentation is kind of hidden) that driver authors need to be
careful with module removal too, refer to the warning at the end of
__cpuhp_remove_state_cpuslocked() about module removal.
It is zram's bug. zram has to clean everything in module_exit(),
unfortunately zram_remove() can be failed when calling from
module_exit() because ->claim is set as true by reset_store(), then
zram_reset_device()(->zcomp_destroy) isn't called, and this failure should
not happen when unloading module, should it?
You're addressing a possible failig zram_remove() while I address not
allowing entry to muck with the zram driver at all once we're bailing
on module removal.
quoted
quoted
2) device_del() is often done in module_exit(), once device_del()
returns, no any new show/store on the device's kobject attribute
is possible.
Right and if a syfs knob is exposed before device_del() completely
and is allowed to do things, the driver should take care to prevent
races for CPU multistate support. The small state machine I added ensures
What is the race for CPU multistate support? If you mean 'Error: Removing
state 63 which has instances left.', it is zram's bug since zram has to
cleanup everything in module_exit().
Yes. And it is what my out of tree yet Acked patch, 'zram: fix
crashes with cpu hotplug multistate' does.
quoted
we don't run over any expectations from cpu hotplug multistate support.
I've *never* suggested there cannot be alternatives to my solution with
the small state machine, but for you to say it is incorrect is simply
not right either.
quoted
3) it is _not_ a must or pattern for fixing bugs to hold one lock before
calling device_del(), meantime the lock is required in the device's
attribute show()/store(), which causes AA deadlock easily. Your approach
just avoids the issue by not releasing module until all show/store are
done.
Right, there are two approaches here:
a) Your approach is to accept the deadlock as a requirement and so
you would prefer to implement an alternative to using a shared lock
on module exit and sysfs op.
wrt. in-tree zram, there is neither any deadlock in linus tree, nor after
applying my 3 patches. If you think there is, please share us the code
or lockdep warning.
Right, 'zram: fix crashes with cpu hotplug multistate' is not yet
merged, my approach to fixing that does add a lock use on module removal
which does introduce a possible deadlock with syfs, which is later addressed
generically between sysfs and module removal for all drivers.
quoted
b) While I address such a deadlock head on as I think this sort of locking
be allowed for two reasons:
b1) as we never documented such requirement otherwise.
b2) There is a possibility that other drivers already exist too
which *do* use a shared lock on module removal and sysfs ops
(and I just confirmed this to be true)
The 'deadlock' is actually caused by your out-of-tree patch of 'zram: fix
crashes with cpu hotplug multistate' which adds mutex_lock(zram_index_mutex)
in destroy_devices().
Yes yes, but you are completely throwing out the window that other
possible deadlocks can exist in the kernel *and* that *new* cases of
the deadlock can easily also be added!
We can fix this issue easily without needing the global lock, please see the
attached(pre-V2) patch.
So far your patches do not fix the issues though...
quoted
So I *really* don't think it is wise for us to simply accept this new
found deadlock as a *new* requirement, specially if we can fix it easily.
A cursory review using Coccinelle potential issues with mutex lock
directly used on module exit (so this doesn't cover drivers like zram
which uses a routine and then grabs the lock through indirection) and a
sysfs op shows these drivers are also affected by this deadlock:
* arch/powerpc/sysdev/fsl_mpic_timer_wakeup.c
In fsl_wakeup_sys_exit(), device_remove_file() is called before
acquiring &sysfs_lock, so there shouldn't be such AA deadlock.
quoted
* lib/test_firmware.c
Yeah, there is the AA deadlock risk, but it should be fixed by moving
misc_deregister() out of &test_fw_mutex.
And just like that you are ignoring other possible uses in the kernel
which might have similar deadlocks.
So do you want to take the position:
Hey driver authors: you cannot use any shared lock on module removal and
on sysfs ops?
Luis
On Tue, Oct 19, 2021 at 08:50:24AM -0700, Luis Chamberlain wrote:
So do you want to take the position:
Hey driver authors: you cannot use any shared lock on module removal and
on sysfs ops?
Yes, I would not recommend using such a lock at all. sysfs operations
happen on a per-device basis, so you can lock the device structure.
Module removal happens on a driver basis, and I have no idea what you
want to lock there, but odds are it is NOT shared with your per-device
structures either, right?
If so, then yes, that is a bug, but a very rare one as drivers should do
almost nothing except register/unregister_driver() in their module
init/exit calls.
zram is not a "normal" driver at all here, so fixing this type of
problem up should be done in the zram code, it is not a generic
module/sysfs issue at all.
thanks,
greg k-h
On Tue, Oct 19, 2021 at 08:28:21AM -0700, Luis Chamberlain wrote:
On Tue, Oct 19, 2021 at 10:34:41AM +0800, Ming Lei wrote:
quoted
Please try the following patch against upstream(linus or next) tree(basically
fold revised 2 and 3 of V1, and cover two issues: not fail zram_remove in
module_exit(), race between zram_remove() and disksize_store()), and see if
everything is fine for you:
@@ -1704,12 +1704,12 @@ static void zram_reset_device(struct zram *zram)set_capacity_and_notify(zram->disk,0);part_stat_set_all(zram->disk->part0,0);-up_write(&zram->init_lock);/* I/O operation under all of CPU are done so let's free */zram_meta_free(zram,disksize);memset(&zram->stats,0,sizeof(zram->stats));zcomp_destroy(comp);reset_bdev(zram);+up_write(&zram->init_lock);}staticssize_tdisksize_store(structdevice*dev,
On Tue, Oct 19, 2021 at 08:50:24AM -0700, Luis Chamberlain wrote:
On Tue, Oct 19, 2021 at 10:34:41AM +0800, Ming Lei wrote:
quoted
On Mon, Oct 18, 2021 at 12:32:11PM -0700, Luis Chamberlain wrote:
quoted
On Sat, Oct 16, 2021 at 07:28:39PM +0800, Ming Lei wrote:
quoted
On Fri, Oct 15, 2021 at 10:31:31AM -0700, Luis Chamberlain wrote:
quoted
On Fri, Oct 15, 2021 at 04:36:11PM +0800, Ming Lei wrote:
quoted
On Thu, Oct 14, 2021 at 05:22:40PM -0700, Luis Chamberlain wrote:
quoted
On Fri, Oct 15, 2021 at 07:52:04AM +0800, Ming Lei wrote:
...
quoted
quoted
We need to understand the exact reason why there is still cpuhp node
left, can you share us the exact steps for reproducing the issue?
Otherwise we may have to trace and narrow down the reason.
See my commit log for my own fix for this issue.
OK, thanks!
I can reproduce the issue, and the reason is that reset_store fails
zram_remove() when unloading module, then the warning is caused.
The top 3 patches in the following tree can fix the issue:
https://github.com/ming1/linux/commits/my_v5.15-blk-dev
Thanks for trying an alternative fix! A crash stops yes, however this
I doubt it is alternative since your patchset doesn't mention the exact
reason of 'Error: Removing state 63 which has instances left.', that is
simply caused by failing to remove zram because ->claim is set during
unloading module.
Well I disagree because it does explain how the race can happen, and it
also explains how since the sysfs interface is exposed until module
removal completes, it leaves exposed knobs to allow re-initializing of a
struct zcomp for a zram device before the exit.
quoted
Yeah, you mentioned the race between disksize_store() vs. zram_remove(),
however I don't think it is reproduced easily in the test because the race
window is pretty small, also it can be fixed easily in my 3rd path
without any complicated tricks.
Reproducing for me is... extremely easy.
In my observation, failing zram_remove() is extremely easy to trigger, which
is caused by reset_store() which sets ->reclaim as true, so
zram_remove() is failed and zram_reset_device() is bypassed , then the
failure of 'Error: Removing state 63 which has instances left.' is caused.
We are in same page?
The actual first issue is the CPU hotplug remove callback is long gone and
in the meantime we allow a race to add a new "instance", in the zram
driver's case a cpu struct zcomp instance though the sysfs interface.
Regardless of if zram_remove() can fail or not, the above race needs to
be addressed.
quoted
quoted
quoted
Not dig into details of your patchset via grabbing module reference
count during show/store attribute of kernfs which is done in your patch
9, but IMO this way isn't necessary:
That's to address the deadlock only.
quoted
1) any driver module has to cleanup anything which may refer to symbols
or data defined in module_exit of this driver
Yes, and as the cpu multistate hotplug documentation warns (although
such documentation is kind of hidden) that driver authors need to be
careful with module removal too, refer to the warning at the end of
__cpuhp_remove_state_cpuslocked() about module removal.
It is zram's bug. zram has to clean everything in module_exit(),
unfortunately zram_remove() can be failed when calling from
module_exit() because ->claim is set as true by reset_store(), then
zram_reset_device()(->zcomp_destroy) isn't called, and this failure should
not happen when unloading module, should it?
You're addressing a possible failig zram_remove() while I address not
allowing entry to muck with the zram driver at all once we're bailing
on module removal.
quoted
quoted
quoted
2) device_del() is often done in module_exit(), once device_del()
returns, no any new show/store on the device's kobject attribute
is possible.
Right and if a syfs knob is exposed before device_del() completely
and is allowed to do things, the driver should take care to prevent
races for CPU multistate support. The small state machine I added ensures
What is the race for CPU multistate support? If you mean 'Error: Removing
state 63 which has instances left.', it is zram's bug since zram has to
cleanup everything in module_exit().
Yes. And it is what my out of tree yet Acked patch, 'zram: fix
crashes with cpu hotplug multistate' does.
Unfortunately that patch adds new deadlock between hot_remove_store() and
disksize_store() & others, see my below comment.
quoted
quoted
we don't run over any expectations from cpu hotplug multistate support.
I've *never* suggested there cannot be alternatives to my solution with
the small state machine, but for you to say it is incorrect is simply
not right either.
quoted
3) it is _not_ a must or pattern for fixing bugs to hold one lock before
calling device_del(), meantime the lock is required in the device's
attribute show()/store(), which causes AA deadlock easily. Your approach
just avoids the issue by not releasing module until all show/store are
done.
Right, there are two approaches here:
a) Your approach is to accept the deadlock as a requirement and so
you would prefer to implement an alternative to using a shared lock
on module exit and sysfs op.
wrt. in-tree zram, there is neither any deadlock in linus tree, nor after
applying my 3 patches. If you think there is, please share us the code
or lockdep warning.
Right, 'zram: fix crashes with cpu hotplug multistate' is not yet
merged, my approach to fixing that does add a lock use on module removal
which does introduce a possible deadlock with syfs, which is later addressed
generically between sysfs and module removal for all drivers.
quoted
quoted
b) While I address such a deadlock head on as I think this sort of locking
be allowed for two reasons:
b1) as we never documented such requirement otherwise.
b2) There is a possibility that other drivers already exist too
which *do* use a shared lock on module removal and sysfs ops
(and I just confirmed this to be true)
The 'deadlock' is actually caused by your out-of-tree patch of 'zram: fix
crashes with cpu hotplug multistate' which adds mutex_lock(zram_index_mutex)
in destroy_devices().
Yes yes, but you are completely throwing out the window that other
possible deadlocks can exist in the kernel *and* that *new* cases of
the deadlock can easily also be added!
quoted
We can fix this issue easily without needing the global lock, please see the
attached(pre-V2) patch.
So far your patches do not fix the issues though...
quoted
quoted
So I *really* don't think it is wise for us to simply accept this new
found deadlock as a *new* requirement, specially if we can fix it easily.
A cursory review using Coccinelle potential issues with mutex lock
directly used on module exit (so this doesn't cover drivers like zram
which uses a routine and then grabs the lock through indirection) and a
sysfs op shows these drivers are also affected by this deadlock:
* arch/powerpc/sysdev/fsl_mpic_timer_wakeup.c
In fsl_wakeup_sys_exit(), device_remove_file() is called before
acquiring &sysfs_lock, so there shouldn't be such AA deadlock.
quoted
* lib/test_firmware.c
Yeah, there is the AA deadlock risk, but it should be fixed by moving
misc_deregister() out of &test_fw_mutex.
And just like that you are ignoring other possible uses in the kernel
which might have similar deadlocks.
So do you want to take the position:
Hey driver authors: you cannot use any shared lock on module removal and
on sysfs ops?
IMO, yes, in your patch of 'zram: fix crashes with cpu hotplug multistate',
when you added mutex_lock(zram_index_mutex) to disksize_store() and
other attribute show() or store() method. You have added new deadlock
between hot_remove_store() and disksize_store() & others, which can't be
addressed by your approach of holding module refcnt.
So far not see ltp tests covers hot add/remove interface yet.
Thanks,
Ming
On Tue, Oct 19, 2021 at 09:30:05AM -0700, Luis Chamberlain wrote:
On Tue, Oct 19, 2021 at 06:25:18PM +0200, Greg KH wrote:
quoted
On Tue, Oct 19, 2021 at 08:50:24AM -0700, Luis Chamberlain wrote:
quoted
So do you want to take the position:
Hey driver authors: you cannot use any shared lock on module removal and
on sysfs ops?
Yes, I would not recommend using such a lock at all. sysfs operations
happen on a per-device basis, so you can lock the device structure.
All devices are going to be removed on module removal and so cannot be locked.
devices are not normally created by a driver, that is up to the bus
controller logic. A module will just disconnect itself from the device,
the device does not go away.
But yes, there are exceptions, and if you are doing something odd like
that, then you need to be aware of crazy things like this, so be
careful. But for all normal drivers, they do not have to worry about
this.
thanks,
greg k-h
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-10-19 19:37:02
On Wed, Oct 20, 2021 at 12:29:53AM +0800, Ming Lei wrote:
quoted hunk
On Tue, Oct 19, 2021 at 08:28:21AM -0700, Luis Chamberlain wrote:
quoted
On Tue, Oct 19, 2021 at 10:34:41AM +0800, Ming Lei wrote:
quoted
Please try the following patch against upstream(linus or next) tree(basically
fold revised 2 and 3 of V1, and cover two issues: not fail zram_remove in
module_exit(), race between zram_remove() and disksize_store()), and see if
everything is fine for you:
@@ -1704,12 +1704,12 @@ static void zram_reset_device(struct zram *zram)set_capacity_and_notify(zram->disk,0);part_stat_set_all(zram->disk->part0,0);-up_write(&zram->init_lock);/* I/O operation under all of CPU are done so let's free */zram_meta_free(zram,disksize);memset(&zram->stats,0,sizeof(zram->stats));zcomp_destroy(comp);reset_bdev(zram);+up_write(&zram->init_lock);}staticssize_tdisksize_store(structdevice*dev,
With this, it still ends up in a state where we loop and can't get out of:
zram: Can't change algorithm for initialized device
Luis
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-10-19 19:38:56
On Wed, Oct 20, 2021 at 12:39:22AM +0800, Ming Lei wrote:
On Tue, Oct 19, 2021 at 08:50:24AM -0700, Luis Chamberlain wrote:
quoted
So do you want to take the position:
Hey driver authors: you cannot use any shared lock on module removal and
on sysfs ops?
IMO, yes, in your patch of 'zram: fix crashes with cpu hotplug multistate',
when you added mutex_lock(zram_index_mutex) to disksize_store() and
other attribute show() or store() method. You have added new deadlock
between hot_remove_store() and disksize_store() & others, which can't be
addressed by your approach of holding module refcnt.
So far not see ltp tests covers hot add/remove interface yet.
Care to show what commands to use to cause this deadlock with my patches?
Luis
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-10-19 19:46:42
On Tue, Oct 19, 2021 at 07:28:35PM +0200, Greg KH wrote:
On Tue, Oct 19, 2021 at 09:30:05AM -0700, Luis Chamberlain wrote:
quoted
On Tue, Oct 19, 2021 at 06:25:18PM +0200, Greg KH wrote:
quoted
On Tue, Oct 19, 2021 at 08:50:24AM -0700, Luis Chamberlain wrote:
quoted
So do you want to take the position:
Hey driver authors: you cannot use any shared lock on module removal and
on sysfs ops?
Yes, I would not recommend using such a lock at all. sysfs operations
happen on a per-device basis, so you can lock the device structure.
All devices are going to be removed on module removal and so cannot be locked.
devices are not normally created by a driver, that is up to the bus
controller logic. A module will just disconnect itself from the device,
the device does not go away.
But yes, there are exceptions, and if you are doing something odd like
that, then you need to be aware of crazy things like this, so be
careful. But for all normal drivers, they do not have to worry about
this.
"Recommend" is a weak position to take given a possible deadlock with sysfs.
Do we want to at the very least document this is not a supported scheme?
If so I can also add a simple 1 level indirrection coccinelle patch to
detect these schemes and complain about them as wel, if we are going to
take this position.
But to simply disregard this as "not an issue", or we won't do anything
seems pretty counter productive given we *do* had drivers with this
issue before *and* still have them upstream, and can end up with more
drivers like this later.
Luis
On Tue, Oct 19, 2021 at 12:38:42PM -0700, Luis Chamberlain wrote:
On Wed, Oct 20, 2021 at 12:39:22AM +0800, Ming Lei wrote:
quoted
On Tue, Oct 19, 2021 at 08:50:24AM -0700, Luis Chamberlain wrote:
quoted
So do you want to take the position:
Hey driver authors: you cannot use any shared lock on module removal and
on sysfs ops?
IMO, yes, in your patch of 'zram: fix crashes with cpu hotplug multistate',
when you added mutex_lock(zram_index_mutex) to disksize_store() and
other attribute show() or store() method. You have added new deadlock
between hot_remove_store() and disksize_store() & others, which can't be
addressed by your approach of holding module refcnt.
So far not see ltp tests covers hot add/remove interface yet.
Care to show what commands to use to cause this deadlock with my patches?
Build a kernel with your patch 4,7,8,9,11 and 12(all others are test module or
document change), with lockdep enabled, run the following command, then you
will see the warning, and it is one real deadlock, not false warning.
BTW, your patch 9 can't be applied cleanly against both linus and next
tree, so I edited it manually, but that can't make difference wrt. this issue.
[root@ktest-09 ~]# lsblk | grep zram
zram0 253:0 0 0B 0 disk
cat /sys/class/zram-control/hot_add
[root@ktest-09 ~]# lsblk | grep zram
zram0 253:0 0 0B 0 disk
zram1 253:1 0 0B 0 disk
[root@ktest-09 ~]# echo 256M > /sys/block/zram1/disksize
[root@ktest-09 ~]# echo 1 > /sys/class/zram-control/hot_remove
[root@ktest-09 ~]# dmesg
...
[ 75.599882] ======================================================
[ 75.601355] WARNING: possible circular locking dependency detected
[ 75.602818] 5.15.0-rc3_zram_fix_luis+ #24 Not tainted
[ 75.604038] ------------------------------------------------------
[ 75.605512] bash/1154 is trying to acquire lock:
[ 75.606634] ffff91ce026cd428 (kn->active#237){++++}-{0:0}, at: __kernfs_remove+0x1ab/0x1e0
[ 75.608570]
but task is already holding lock:
[ 75.609955] ffffffff839e3ef0 (zram_index_mutex){+.+.}-{3:3}, at: hot_remove_store+0x52/0xf0
[ 75.611910]
which lock already depends on the new lock.
[ 75.613896]
the existing dependency chain (in reverse order) is:
[ 75.615830]
-> #1 (zram_index_mutex){+.+.}-{3:3}:
[ 75.617483] __lock_acquire+0x4d2/0x930
[ 75.618650] lock_acquire+0xbb/0x2d0
[ 75.619748] __mutex_lock+0x8e/0x8a0
[ 75.620854] disksize_store+0x38/0x180
[ 75.621996] kernfs_fop_write_iter+0x134/0x1d0
[ 75.623287] new_sync_write+0x122/0x1b0
[ 75.624442] vfs_write+0x23e/0x350
[ 75.625506] ksys_write+0x68/0xe0
[ 75.626550] do_syscall_64+0x3b/0x90
[ 75.627649] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 75.629070]
-> #0 (kn->active#237){++++}-{0:0}:
[ 75.630677] check_prev_add+0x91/0xc10
[ 75.631816] validate_chain+0x474/0x500
[ 75.632972] __lock_acquire+0x4d2/0x930
[ 75.634131] lock_acquire+0xbb/0x2d0
[ 75.635234] kernfs_drain+0x139/0x190
[ 75.636355] __kernfs_remove+0x1ab/0x1e0
[ 75.637532] kernfs_remove_by_name_ns+0x3f/0x80
[ 75.638843] remove_files+0x2b/0x60
[ 75.639926] sysfs_remove_group+0x38/0x80
[ 75.641120] sysfs_remove_groups+0x29/0x40
[ 75.642334] device_remove_attrs+0x5b/0x90
[ 75.643552] device_del+0x184/0x400
[ 75.644635] zram_remove+0xac/0xc0
[ 75.645700] hot_remove_store+0xa3/0xf0
[ 75.646856] kernfs_fop_write_iter+0x134/0x1d0
[ 75.648147] new_sync_write+0x122/0x1b0
[ 75.649311] vfs_write+0x23e/0x350
[ 75.650372] ksys_write+0x68/0xe0
[ 75.651412] do_syscall_64+0x3b/0x90
[ 75.652512] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 75.653929]
other info that might help us debug this:
[ 75.656054] Possible unsafe locking scenario:
[ 75.657637] CPU0 CPU1
[ 75.658833] ---- ----
[ 75.660020] lock(zram_index_mutex);
[ 75.661024] lock(kn->active#237);
[ 75.662549] lock(zram_index_mutex);
[ 75.664103] lock(kn->active#237);
[ 75.665072]
*** DEADLOCK ***
[ 75.666736] 4 locks held by bash/1154:
[ 75.667767] #0: ffff91ce06983470 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x68/0xe0
[ 75.669802] #1: ffff91ce4123d290 (&of->mutex){+.+.}-{3:3}, at: kernfs_fop_write_iter+0x100/0x1d0
[ 75.672050] #2: ffff91ce05a7ac40 (kn->active#238){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x108/0x1d0
[ 75.674383] #3: ffffffff839e3ef0 (zram_index_mutex){+.+.}-{3:3}, at: hot_remove_store+0x52/0xf0
[ 75.676595]
stack backtrace:
[ 75.677835] CPU: 2 PID: 1154 Comm: bash Not tainted 5.15.0-rc3_zram_fix_luis+ #24
[ 75.679768] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.14.0-1.fc33 04/01/2014
[ 75.681927] Call Trace:
[ 75.682674] dump_stack_lvl+0x57/0x7d
[ 75.683680] check_noncircular+0xff/0x110
[ 75.684758] ? stack_trace_save+0x4b/0x70
[ 75.685843] check_prev_add+0x91/0xc10
[ 75.686867] ? add_chain_cache+0x112/0x2d0
[ 75.687965] validate_chain+0x474/0x500
[ 75.689005] __lock_acquire+0x4d2/0x930
[ 75.690054] lock_acquire+0xbb/0x2d0
[ 75.691038] ? __kernfs_remove+0x1ab/0x1e0
[ 75.692131] ? __lock_release+0x179/0x2c0
[ 75.693212] ? kernfs_drain+0x5b/0x190
[ 75.694239] kernfs_drain+0x139/0x190
[ 75.695240] ? __kernfs_remove+0x1ab/0x1e0
[ 75.696341] __kernfs_remove+0x1ab/0x1e0
[ 75.697408] kernfs_remove_by_name_ns+0x3f/0x80
[ 75.698607] remove_files+0x2b/0x60
[ 75.699576] sysfs_remove_group+0x38/0x80
[ 75.700661] sysfs_remove_groups+0x29/0x40
[ 75.701770] device_remove_attrs+0x5b/0x90
[ 75.702870] device_del+0x184/0x400
[ 75.703835] zram_remove+0xac/0xc0
[ 75.704785] hot_remove_store+0xa3/0xf0
[ 75.705831] kernfs_fop_write_iter+0x134/0x1d0
[ 75.707004] new_sync_write+0x122/0x1b0
[ 75.708048] ? __do_fast_syscall_32+0xe0/0xf0
[ 75.709214] vfs_write+0x23e/0x350
[ 75.710161] ksys_write+0x68/0xe0
[ 75.711088] do_syscall_64+0x3b/0x90
[ 75.712078] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 75.713389] RIP: 0033:0x7fcc1893f927
[ 75.714381] Code: 0f 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 51 c3 48 83 ec 28 48 89 54 24 18 48 89 74 24
[ 75.718879] RSP: 002b:00007ffcd56d91a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 75.720832] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007fcc1893f927
[ 75.722592] RDX: 0000000000000002 RSI: 000055d7d33f78c0 RDI: 0000000000000001
[ 75.724352] RBP: 000055d7d33f78c0 R08: 0000000000000000 R09: 00007fcc189f44e0
[ 75.726123] R10: 00007fcc189f43e0 R11: 0000000000000246 R12: 0000000000000002
[ 75.727884] R13: 00007fcc18a395a0 R14: 0000000000000002 R15: 00007fcc18a397a0
Thanks,
Ming
On Tue, Oct 19, 2021 at 12:36:42PM -0700, Luis Chamberlain wrote:
On Wed, Oct 20, 2021 at 12:29:53AM +0800, Ming Lei wrote:
quoted
On Tue, Oct 19, 2021 at 08:28:21AM -0700, Luis Chamberlain wrote:
quoted
On Tue, Oct 19, 2021 at 10:34:41AM +0800, Ming Lei wrote:
quoted
Please try the following patch against upstream(linus or next) tree(basically
fold revised 2 and 3 of V1, and cover two issues: not fail zram_remove in
module_exit(), race between zram_remove() and disksize_store()), and see if
everything is fine for you:
@@ -1704,12 +1704,12 @@ static void zram_reset_device(struct zram *zram)set_capacity_and_notify(zram->disk,0);part_stat_set_all(zram->disk->part0,0);-up_write(&zram->init_lock);/* I/O operation under all of CPU are done so let's free */zram_meta_free(zram,disksize);memset(&zram->stats,0,sizeof(zram->stats));zcomp_destroy(comp);reset_bdev(zram);+up_write(&zram->init_lock);}staticssize_tdisksize_store(structdevice*dev,
With this, it still ends up in a state where we loop and can't get out of:
zram: Can't change algorithm for initialized device
Again, you are running two zram02.sh[1] on /dev/zram0, that isn't unexpected
behavior. Here the difference is just timing. In my test VM,
this message shows a while on one task, then it may be switched to
another task.
Just run your patches a while, nothing real difference here, and the
following message can be dumped from one task for long time:
can't set '107374182400' to /sys/block/zram0/disksize
Also you did not answer my question about your test expected result when
running the following script from two terminal concurrently:
while true; do
PATH=$PATH:$PWD:$PWD/../../../lib/ ./zram02.sh;
done
Thanks,
Ming
From: Miroslav Benes <mbenes@suse.cz> Date: 2021-10-20 06:43:41
On Tue, 19 Oct 2021, Ming Lei wrote:
On Tue, Oct 19, 2021 at 08:23:51AM +0200, Miroslav Benes wrote:
quoted
quoted
quoted
By you only addressing the deadlock as a requirement on approach a) you are
forgetting that there *may* already be present drivers which *do* implement
such patterns in the kernel. I worked on addressing the deadlock because
I was informed livepatching *did* have that issue as well and so very
likely a generic solution to the deadlock could be beneficial to other
random drivers.
In-tree zram doesn't have such deadlock, if livepatching has such AA deadlock,
just fixed it, and seems it has been fixed by 3ec24776bfd0.
I would not call it a fix. It is a kind of ugly workaround because the
generic infrastructure lacked (lacks) the proper support in my opinion.
Luis is trying to fix that.
What is the proper support of the generic infrastructure? I am not
familiar with livepatching's model(especially with module unload), you mean
livepatching have to do the following way from sysfs:
1) during module exit:
mutex_lock(lp_lock);
kobject_put(lp_kobj);
mutex_unlock(lp_lock);
2) show()/store() method of attributes of lp_kobj
mutex_lock(lp_lock)
...
mutex_unlock(lp_lock)
Yes, this was exactly the case. We then reworked it a lot (see
958ef1e39d24 ("livepatch: Simplify API by removing registration step"), so
now the call sequence is different. kobject_put() is basically offloaded
to a workqueue scheduled right from the store() method. Meaning that
Luis's work would probably not help us currently, but on the other hand
the issues with AA deadlock were one of the main drivers of the redesign
(if I remember correctly). There were other reasons too as the changelog
of the commit describes.
So, from my perspective, if there was a way to easily synchronize between
a data cleanup from module_exit callback and sysfs/kernfs operations, it
could spare people many headaches.
IMO, the above usage simply caused AA deadlock. Even in Luis's patch
'zram: fix crashes with cpu hotplug multistate', new/same AA deadlock
(hot_remove_store() vs. disksize_store() or reset_store()) is added
because hot_remove_store() isn't called from module_exit().
Luis tries to delay unloading module until all show()/store() are done. But
that can be obtained by the following way simply during module_exit():
kobject_del(lp_kobj); //all pending store()/show() from lp_kobj are done,
//no new store()/show() can come after
//kobject_del() returns
mutex_lock(lp_lock);
kobject_put(lp_kobj);
mutex_unlock(lp_lock);
kobject_del() already calls kobject_put(). Did you mean __kobject_del().
That one is internal though.
Or can you explain your requirement on kobject/module unload in a bit
details?
On Wed, Oct 20, 2021 at 08:43:37AM +0200, Miroslav Benes wrote:
On Tue, 19 Oct 2021, Ming Lei wrote:
quoted
On Tue, Oct 19, 2021 at 08:23:51AM +0200, Miroslav Benes wrote:
quoted
quoted
quoted
By you only addressing the deadlock as a requirement on approach a) you are
forgetting that there *may* already be present drivers which *do* implement
such patterns in the kernel. I worked on addressing the deadlock because
I was informed livepatching *did* have that issue as well and so very
likely a generic solution to the deadlock could be beneficial to other
random drivers.
In-tree zram doesn't have such deadlock, if livepatching has such AA deadlock,
just fixed it, and seems it has been fixed by 3ec24776bfd0.
I would not call it a fix. It is a kind of ugly workaround because the
generic infrastructure lacked (lacks) the proper support in my opinion.
Luis is trying to fix that.
What is the proper support of the generic infrastructure? I am not
familiar with livepatching's model(especially with module unload), you mean
livepatching have to do the following way from sysfs:
1) during module exit:
mutex_lock(lp_lock);
kobject_put(lp_kobj);
mutex_unlock(lp_lock);
2) show()/store() method of attributes of lp_kobj
mutex_lock(lp_lock)
...
mutex_unlock(lp_lock)
Yes, this was exactly the case. We then reworked it a lot (see
958ef1e39d24 ("livepatch: Simplify API by removing registration step"), so
now the call sequence is different. kobject_put() is basically offloaded
to a workqueue scheduled right from the store() method. Meaning that
Luis's work would probably not help us currently, but on the other hand
the issues with AA deadlock were one of the main drivers of the redesign
(if I remember correctly). There were other reasons too as the changelog
of the commit describes.
So, from my perspective, if there was a way to easily synchronize between
a data cleanup from module_exit callback and sysfs/kernfs operations, it
could spare people many headaches.
kobject_del() is supposed to do so, but you can't hold a shared lock
which is required in show()/store() method. Once kobject_del() returns,
no pending show()/store() any more.
The question is that why one shared lock is required for livepatching to
delete the kobject. What are you protecting when you delete one kobject?
quoted
IMO, the above usage simply caused AA deadlock. Even in Luis's patch
'zram: fix crashes with cpu hotplug multistate', new/same AA deadlock
(hot_remove_store() vs. disksize_store() or reset_store()) is added
because hot_remove_store() isn't called from module_exit().
Luis tries to delay unloading module until all show()/store() are done. But
that can be obtained by the following way simply during module_exit():
kobject_del(lp_kobj); //all pending store()/show() from lp_kobj are done,
//no new store()/show() can come after
//kobject_del() returns
mutex_lock(lp_lock);
kobject_put(lp_kobj);
mutex_unlock(lp_lock);
kobject_del() already calls kobject_put(). Did you mean __kobject_del().
That one is internal though.
kobject_del() is counter-part of kobject_add(), and kobject_put() will
call kobject_del() automatically() if it isn't deleted yet, but usually
kobject_put() is for releasing the object only. It is more often to
release kobject by calling kobject_del() and kobject_put().
quoted
Or can you explain your requirement on kobject/module unload in a bit
details?
Does the above makes sense?
I think now focus is the shared lock between kobject_del() and
show()/store() of the kobject's attributes.
Thanks,
Ming
From: Miroslav Benes <mbenes@suse.cz> Date: 2021-10-20 08:19:34
On Wed, 20 Oct 2021, Ming Lei wrote:
On Wed, Oct 20, 2021 at 08:43:37AM +0200, Miroslav Benes wrote:
quoted
On Tue, 19 Oct 2021, Ming Lei wrote:
quoted
On Tue, Oct 19, 2021 at 08:23:51AM +0200, Miroslav Benes wrote:
quoted
quoted
quoted
By you only addressing the deadlock as a requirement on approach a) you are
forgetting that there *may* already be present drivers which *do* implement
such patterns in the kernel. I worked on addressing the deadlock because
I was informed livepatching *did* have that issue as well and so very
likely a generic solution to the deadlock could be beneficial to other
random drivers.
In-tree zram doesn't have such deadlock, if livepatching has such AA deadlock,
just fixed it, and seems it has been fixed by 3ec24776bfd0.
I would not call it a fix. It is a kind of ugly workaround because the
generic infrastructure lacked (lacks) the proper support in my opinion.
Luis is trying to fix that.
What is the proper support of the generic infrastructure? I am not
familiar with livepatching's model(especially with module unload), you mean
livepatching have to do the following way from sysfs:
1) during module exit:
mutex_lock(lp_lock);
kobject_put(lp_kobj);
mutex_unlock(lp_lock);
2) show()/store() method of attributes of lp_kobj
mutex_lock(lp_lock)
...
mutex_unlock(lp_lock)
Yes, this was exactly the case. We then reworked it a lot (see
958ef1e39d24 ("livepatch: Simplify API by removing registration step"), so
now the call sequence is different. kobject_put() is basically offloaded
to a workqueue scheduled right from the store() method. Meaning that
Luis's work would probably not help us currently, but on the other hand
the issues with AA deadlock were one of the main drivers of the redesign
(if I remember correctly). There were other reasons too as the changelog
of the commit describes.
So, from my perspective, if there was a way to easily synchronize between
a data cleanup from module_exit callback and sysfs/kernfs operations, it
could spare people many headaches.
kobject_del() is supposed to do so, but you can't hold a shared lock
which is required in show()/store() method. Once kobject_del() returns,
no pending show()/store() any more.
The question is that why one shared lock is required for livepatching to
delete the kobject. What are you protecting when you delete one kobject?
I think it boils down to the fact that we embed kobject statically to
structures which livepatch uses to maintain data. That is discouraged
generally, but all the attempts to implement it correctly were utter
failures.
Miroslav
On Wed, Oct 20, 2021 at 10:19:27AM +0200, Miroslav Benes wrote:
On Wed, 20 Oct 2021, Ming Lei wrote:
quoted
On Wed, Oct 20, 2021 at 08:43:37AM +0200, Miroslav Benes wrote:
quoted
On Tue, 19 Oct 2021, Ming Lei wrote:
quoted
On Tue, Oct 19, 2021 at 08:23:51AM +0200, Miroslav Benes wrote:
quoted
quoted
quoted
By you only addressing the deadlock as a requirement on approach a) you are
forgetting that there *may* already be present drivers which *do* implement
such patterns in the kernel. I worked on addressing the deadlock because
I was informed livepatching *did* have that issue as well and so very
likely a generic solution to the deadlock could be beneficial to other
random drivers.
In-tree zram doesn't have such deadlock, if livepatching has such AA deadlock,
just fixed it, and seems it has been fixed by 3ec24776bfd0.
I would not call it a fix. It is a kind of ugly workaround because the
generic infrastructure lacked (lacks) the proper support in my opinion.
Luis is trying to fix that.
What is the proper support of the generic infrastructure? I am not
familiar with livepatching's model(especially with module unload), you mean
livepatching have to do the following way from sysfs:
1) during module exit:
mutex_lock(lp_lock);
kobject_put(lp_kobj);
mutex_unlock(lp_lock);
2) show()/store() method of attributes of lp_kobj
mutex_lock(lp_lock)
...
mutex_unlock(lp_lock)
Yes, this was exactly the case. We then reworked it a lot (see
958ef1e39d24 ("livepatch: Simplify API by removing registration step"), so
now the call sequence is different. kobject_put() is basically offloaded
to a workqueue scheduled right from the store() method. Meaning that
Luis's work would probably not help us currently, but on the other hand
the issues with AA deadlock were one of the main drivers of the redesign
(if I remember correctly). There were other reasons too as the changelog
of the commit describes.
So, from my perspective, if there was a way to easily synchronize between
a data cleanup from module_exit callback and sysfs/kernfs operations, it
could spare people many headaches.
kobject_del() is supposed to do so, but you can't hold a shared lock
which is required in show()/store() method. Once kobject_del() returns,
no pending show()/store() any more.
The question is that why one shared lock is required for livepatching to
delete the kobject. What are you protecting when you delete one kobject?
I think it boils down to the fact that we embed kobject statically to
structures which livepatch uses to maintain data. That is discouraged
generally, but all the attempts to implement it correctly were utter
failures.
Sounds like this is the real problem that needs to be fixed. kobjects
should always control the lifespan of the structure they are embedded
in. If not, then that is a design flaw of the user of the kobject :(
Where in the kernel is this happening? And where have been the attempts
to fix this up?
thanks,
greg k-h
On Wed, Oct 20, 2021 at 10:19:27AM +0200, Miroslav Benes wrote:
On Wed, 20 Oct 2021, Ming Lei wrote:
quoted
On Wed, Oct 20, 2021 at 08:43:37AM +0200, Miroslav Benes wrote:
quoted
On Tue, 19 Oct 2021, Ming Lei wrote:
quoted
On Tue, Oct 19, 2021 at 08:23:51AM +0200, Miroslav Benes wrote:
quoted
quoted
quoted
By you only addressing the deadlock as a requirement on approach a) you are
forgetting that there *may* already be present drivers which *do* implement
such patterns in the kernel. I worked on addressing the deadlock because
I was informed livepatching *did* have that issue as well and so very
likely a generic solution to the deadlock could be beneficial to other
random drivers.
In-tree zram doesn't have such deadlock, if livepatching has such AA deadlock,
just fixed it, and seems it has been fixed by 3ec24776bfd0.
I would not call it a fix. It is a kind of ugly workaround because the
generic infrastructure lacked (lacks) the proper support in my opinion.
Luis is trying to fix that.
What is the proper support of the generic infrastructure? I am not
familiar with livepatching's model(especially with module unload), you mean
livepatching have to do the following way from sysfs:
1) during module exit:
mutex_lock(lp_lock);
kobject_put(lp_kobj);
mutex_unlock(lp_lock);
2) show()/store() method of attributes of lp_kobj
mutex_lock(lp_lock)
...
mutex_unlock(lp_lock)
Yes, this was exactly the case. We then reworked it a lot (see
958ef1e39d24 ("livepatch: Simplify API by removing registration step"), so
now the call sequence is different. kobject_put() is basically offloaded
to a workqueue scheduled right from the store() method. Meaning that
Luis's work would probably not help us currently, but on the other hand
the issues with AA deadlock were one of the main drivers of the redesign
(if I remember correctly). There were other reasons too as the changelog
of the commit describes.
So, from my perspective, if there was a way to easily synchronize between
a data cleanup from module_exit callback and sysfs/kernfs operations, it
could spare people many headaches.
kobject_del() is supposed to do so, but you can't hold a shared lock
which is required in show()/store() method. Once kobject_del() returns,
no pending show()/store() any more.
The question is that why one shared lock is required for livepatching to
delete the kobject. What are you protecting when you delete one kobject?
I think it boils down to the fact that we embed kobject statically to
structures which livepatch uses to maintain data. That is discouraged
generally, but all the attempts to implement it correctly were utter
failures.
OK, then it isn't one common usage, in which kobject covers the release
of the external object. What is the exact kobject in livepatching?
But kobject_del() won't release the kobject, you shouldn't need the lock
to delete kobject first. After the kobject is deleted, no any show() and
store() any more, isn't such sync[1] you expected?
Thanks,
Ming
@@ -1704,12 +1704,12 @@ static void zram_reset_device(struct zram *zram)set_capacity_and_notify(zram->disk,0);part_stat_set_all(zram->disk->part0,0);-up_write(&zram->init_lock);/* I/O operation under all of CPU are done so let's free */zram_meta_free(zram,disksize);memset(&zram->stats,0,sizeof(zram->stats));zcomp_destroy(comp);reset_bdev(zram);+up_write(&zram->init_lock);}staticssize_tdisksize_store(structdevice*dev,
With this, it still ends up in a state where we loop and can't get out of:
zram: Can't change algorithm for initialized device
Again, you are running two zram02.sh[1] on /dev/zram0, that isn't unexpected
You mean that it is not expected? If so then yes, of course.
behavior. Here the difference is just timing.
Right, but that is what helped reproduce a difficutl to re-produce customer
bug. Once you find an easy way to reproduce a reported issue you stick
with it and try to make the situation worse to ensure no more bugs are
present.
Also you did not answer my question about your test expected result when
running the following script from two terminal concurrently:
while true; do
PATH=$PATH:$PWD:$PWD/../../../lib/ ./zram02.sh;
done
If you run this, you should see no failures.
Once you start a second script that one should cause odd issues on both
sides but never crash or stall the module.
A second series of tests is hitting CTRL-C on either randonly and
restarting testing once again randomly.
Again, neither should crash the kernel or stall the module.
In the end of these tests you should be able to run the script alone
just once and not see issues.
Luis
@@ -1704,12 +1704,12 @@ static void zram_reset_device(struct zram *zram)set_capacity_and_notify(zram->disk,0);part_stat_set_all(zram->disk->part0,0);-up_write(&zram->init_lock);/* I/O operation under all of CPU are done so let's free */zram_meta_free(zram,disksize);memset(&zram->stats,0,sizeof(zram->stats));zcomp_destroy(comp);reset_bdev(zram);+up_write(&zram->init_lock);}staticssize_tdisksize_store(structdevice*dev,
With this, it still ends up in a state where we loop and can't get out of:
zram: Can't change algorithm for initialized device
Again, you are running two zram02.sh[1] on /dev/zram0, that isn't unexpected
You mean that it is not expected? If so then yes, of course.
My meaning is clear: it is not unexpected, so it is expected.
quoted
behavior. Here the difference is just timing.
Right, but that is what helped reproduce a difficutl to re-produce customer
bug. Once you find an easy way to reproduce a reported issue you stick
with it and try to make the situation worse to ensure no more bugs are
present.
quoted
Also you did not answer my question about your test expected result when
running the following script from two terminal concurrently:
while true; do
PATH=$PATH:$PWD:$PWD/../../../lib/ ./zram02.sh;
done
If you run this, you should see no failures.
OK, not see any failure when running single zram02.sh after applying my
patch V2.
Once you start a second script that one should cause odd issues on both
sides but never crash or stall the module.
crash can't be observed with my patch V2, what do you mean 'stall'
the module? Is that 'zram' can't be unloaded after the test is
terminated via multiple 'ctrl-c'?
A second series of tests is hitting CTRL-C on either randonly and
restarting testing once again randomly.
ltp/zram02.sh has cleanup handler via trap to clean everything(swapoff/umount/reset/
rmmod), ctrl-c will terminate current forground task and cause shell to run the
cleanup handler first, but further 'ctrl-c' will terminate the cleanup handler,
then the cleanup won't be done completely, such as zram disk is left as swap
device and zram can't be unloaded. The idea can be observed via the following
script:
#!/bin/bash
trap 'echo "enter trap"; sleep 20; echo "exit trap";' INT
sleep 30
After the above script is run foreground, when 1st ctrl-c is pressed, 'sleep 30'
is terminated, then the trap command is run, so you can see "enter trap"
dumped. Then if you pressed 2nd ctrl-c, 'sleep 20' is terminated immediately.
So 'swapoff' from zram02.sh's trap function can be terminated in this way.
zram disk being left as swap disk can be observed with your patch too
after terminating via multiple ctrl-c which has to be done this way because
the test is dead loop.
So it is hard to cleanup everything completely after multiple 'CTRL-C' is
involved, and it should be impossible. It needs violent multiple ctrl-c to
terminate the dealoop test.
So it isn't reasonable to expect that zram can be always unloaded successfully
after the test script is terminated via multiple ctrl-c.
But zram can be unloaded after running swapoff manually, from driver
viewpoint, nothing is wrong.
Again, neither should crash the kernel or stall the module.
In the end of these tests you should be able to run the script alone
just once and not see issues.