Re: WARNING: at kernel/workqueue.c:1473 __queue_work+0x3b8/0x3d0

On Mon, Mar 02, 2020 at 06:00:10PM +0000, Robin Murphy wrote:

On 02/03/2020 5:25 pm, Daniel Jordan wrote:
Something smelled familiar about this discussion, and sure enough that merge
contains c4741b230597 ("crypto: run initcalls for generic implementations
earlier"), which has raised its head before[1].

Yep, that looks suspicious.

The bisect didn't point to that specific commit, even though my version of git
tries commits in the merge.  I'm probably missing something.

quoted

Does this fix it?  I can't verify but figure it's worth trying the simplest
explanation first, which is that the work isn't initialized by the time it's
queued.

The relative initcall levels would appear to explain the symptom - I guess
the question is whether this represents a bug in a particular test/algorithm
(as with the unaligned accesses) or a fundamental problem in the
infrastructure now being able to poke the module loader too early.

I'm not familiar with the crypto code.  Could it be that the commit moved some
request_module() calls before modules_wq_init()?

And, is it "too early" or just "earlier"?  When is it too early for modprobe?

Barring other ideas, Corentin, would you be willing to boot with

    trace_event=initcall:*,module:* trace_options=stacktrace

and

diff --git a/kernel/module.c b/kernel/module.c
index 33569a01d6e1..393be6979a27 100644
--- a/kernel/module.c
+++ b/kernel/module.c

@@ -3604,8 +3604,11 @@ static noinline int do_init_module(struct module *mod)
 	 * be cleaned up needs to sync with the queued work - ie
 	 * rcu_barrier()
 	 */
-	if (llist_add(&freeinit->node, &init_free_list))
+	if (llist_add(&freeinit->node, &init_free_list)) {
+		pr_warn("%s: schedule_work for mod=%s\n", __func__, mod->name);
+		dump_stack();
 		schedule_work(&init_free_wq);
+	}
 
 	mutex_unlock(&module_mutex);
 	wake_up_all(&module_wq);

but not my earlier fix and share the dmesg and ftrace output to see if the
theory holds?

Also, could you attach your config?  Curious now what your crypto options look
like after fiddling with some of them today while trying and failing to see
this on x86.

thanks,
Daniel