From: Luis Chamberlain <mcgrof@kernel.org> Date: 2022-05-09 16:13:14
On Mon, May 09, 2022 at 09:23:39PM +1000, Michael Ellerman wrote:
Herbert Xu [off-list ref] writes:
quoted
Hi:
There are some code paths in the kernel where you can reliably
trigger a request_module of a non-existant module. For example,
if you attempt to load a non-existent crypto algorithm, or create
a socket of a non-existent network family, it will result in a
request_module call that is guaranteed to fail.
As user-space can do this repeatedly, it can quickly overwhelm
the concurrency limit in kmod. This in itself is expected,
however, at least on some platforms this appears to result in
a live-lock. Here is an example triggered by stress-ng on ppc64:
[ 529.853264] request_module: kmod_concurrent_max (0) close to 0 (max_modprobes: 50), for module crypto-aegis128l, throttling...
If you run stress-ng with a timeout does the system eventually recover?
OK the respective stress-ng test should be something like:
./stress-ng --af-alg 8192
I had left this running overnight on x86_64 without issues:
sudo ./tools/testing/selftests/kmod/kmod.sh -t 0009
Going to leave the above stress-ng call running in a loop to see
if I can reproduce the live lock on x86_64.
Luis
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2022-05-11 16:36:12
On Mon, May 09, 2022 at 09:13:03AM -0700, Luis Chamberlain wrote:
On Mon, May 09, 2022 at 09:23:39PM +1000, Michael Ellerman wrote:
quoted
Herbert Xu [off-list ref] writes:
quoted
Hi:
There are some code paths in the kernel where you can reliably
trigger a request_module of a non-existant module. For example,
if you attempt to load a non-existent crypto algorithm, or create
a socket of a non-existent network family, it will result in a
request_module call that is guaranteed to fail.
As user-space can do this repeatedly, it can quickly overwhelm
the concurrency limit in kmod. This in itself is expected,
however, at least on some platforms this appears to result in
a live-lock. Here is an example triggered by stress-ng on ppc64:
[ 529.853264] request_module: kmod_concurrent_max (0) close to 0 (max_modprobes: 50), for module crypto-aegis128l, throttling...
If you run stress-ng with a timeout does the system eventually recover?
OK the respective stress-ng test should be something like:
./stress-ng --af-alg 8192
I had left this running overnight on x86_64 without issues:
sudo ./tools/testing/selftests/kmod/kmod.sh -t 0009
Going to leave the above stress-ng call running in a loop to see
if I can reproduce the live lock on x86_64.
The following loop has been running on 5.18.0-rc5-next-20220506 since
May 9 without any issues on x86_64:
while true; do sudo ./stress-ng --af-alg 8192; done
Can someone try this on ppc64le system? At this point I am not convinced
this issue is generic.
Luis
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2022-05-12 07:36:39
Luis Chamberlain [off-list ref] writes:
On Mon, May 09, 2022 at 09:13:03AM -0700, Luis Chamberlain wrote:
quoted
On Mon, May 09, 2022 at 09:23:39PM +1000, Michael Ellerman wrote:
quoted
Herbert Xu [off-list ref] writes:
quoted
Hi:
There are some code paths in the kernel where you can reliably
trigger a request_module of a non-existant module. For example,
if you attempt to load a non-existent crypto algorithm, or create
a socket of a non-existent network family, it will result in a
request_module call that is guaranteed to fail.
As user-space can do this repeatedly, it can quickly overwhelm
the concurrency limit in kmod. This in itself is expected,
however, at least on some platforms this appears to result in
a live-lock. Here is an example triggered by stress-ng on ppc64:
[ 529.853264] request_module: kmod_concurrent_max (0) close to 0 (max_modprobes: 50), for module crypto-aegis128l, throttling...
If you run stress-ng with a timeout does the system eventually recover?
OK the respective stress-ng test should be something like:
./stress-ng --af-alg 8192
I had left this running overnight on x86_64 without issues:
sudo ./tools/testing/selftests/kmod/kmod.sh -t 0009
Going to leave the above stress-ng call running in a loop to see
if I can reproduce the live lock on x86_64.
The following loop has been running on 5.18.0-rc5-next-20220506 since
May 9 without any issues on x86_64:
while true; do sudo ./stress-ng --af-alg 8192; done
I ran the above on a ppc64le system here, no issues. But it's only a 32
CPU machine.
Can someone try this on ppc64le system? At this point I am not convinced
this issue is generic.
Does your x86 system have at least 784 CPUs?
I don't know where the original report came from, but the trace shows
"CPU 784", which would usually indicate a system with at least that many
CPUs.
I would hope we can handle that many CPUs banging on a spin lock without
throwing traces, but maybe not. It could also be that the system is
under load and the hypervisor has scheduled us off for too long.
cheers
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2022-05-12 12:07:42
Michael Ellerman [off-list ref] writes:
Luis Chamberlain [off-list ref] writes:
...
quoted
Can someone try this on ppc64le system? At this point I am not convinced
this issue is generic.
Does your x86 system have at least 784 CPUs?
I don't know where the original report came from, but the trace shows
"CPU 784", which would usually indicate a system with at least that many
CPUs.
Update, apparently the report originally came from IBM, so I'll chase it
up internally.
I think you're right that there's probably no issue in the module code,
sorry to waste your time.
cheers
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2022-05-12 17:43:38
On Thu, May 12, 2022 at 10:07:26PM +1000, Michael Ellerman wrote:
Michael Ellerman [off-list ref] writes:
quoted
Luis Chamberlain [off-list ref] writes:
...
quoted
quoted
Can someone try this on ppc64le system? At this point I am not convinced
this issue is generic.
Does your x86 system have at least 784 CPUs?
I don't know where the original report came from, but the trace shows
"CPU 784", which would usually indicate a system with at least that many
CPUs.
Update, apparently the report originally came from IBM, so I'll chase it
up internally.
I think you're right that there's probably no issue in the module code,
sorry to waste your time.
It gives me testing happiness to know that may be the case :)
Luis