Re: [ANNOUNCE] v5.14-rc4-rt4

On 2021-08-04 09:39:30 [-0600], Jens Axboe wrote:

I'm confused, the waitqueue locks are always IRQ disabling.

spin_lock_irq() does not disable interrupts on -RT. The patch above
produces:

| BUG: sleeping function called from invalid context at kernel/locking/spinlock_rt.c:35
| in_atomic(): 0, irqs_disabled(): 1, non_block: 0, pid: 2020, name: iou-wrk-2018
| 1 lock held by iou-wrk-2018/2020:
|  #0: ffff888111a47de8 (&hash->wait){+.+.}-{0:0}, at: io_worker_handle_work+0x443/0x630
| irq event stamp: 10
| hardirqs last  enabled at (9): [<ffffffff81c47818>] _raw_spin_unlock_irqrestore+0x28/0x70
| hardirqs last disabled at (10): [<ffffffff81c4769e>] _raw_spin_lock_irq+0x3e/0x40
| softirqs last  enabled at (0): [<ffffffff81077238>] copy_process+0x8f8/0x2020
| softirqs last disabled at (0): [<0000000000000000>] 0x0
| CPU: 5 PID: 2020 Comm: iou-wrk-2018 Tainted: G        W         5.14.0-rc4-rt4+ #97
| Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.14.0-2 04/01/2014
| Call Trace:
|  dump_stack_lvl+0x45/0x59
|  ___might_sleep.cold+0xa6/0xb6
|  rt_spin_lock+0x35/0xc0
|  ? io_worker_handle_work+0x443/0x630
|  io_worker_handle_work+0x443/0x630
|  io_wqe_worker+0xb4/0x340
|  ? lockdep_hardirqs_on_prepare+0xd4/0x170
|  ? _raw_spin_unlock_irqrestore+0x28/0x70
|  ? _raw_spin_unlock_irqrestore+0x28/0x70
|  ? io_worker_handle_work+0x630/0x630
|  ? rt_mutex_slowunlock+0x2ba/0x310
|  ? io_worker_handle_work+0x630/0x630
|  ret_from_fork+0x22/0x30


But indeed, you are right, my snippet breaks non-RT. So this then maybe:

diff --git a/fs/io-wq.c b/fs/io-wq.c
index 57d3cdddcdb3e..0b931ac3c83e6 100644
--- a/fs/io-wq.c
+++ b/fs/io-wq.c

@@ -384,7 +384,7 @@ static void io_wait_on_hash(struct io_wqe *wqe, unsigned int hash)
 {
 	struct io_wq *wq = wqe->wq;
 
-	spin_lock(&wq->hash->wait.lock);
+	spin_lock_irq(&wq->hash->wait.lock);
 	if (list_empty(&wqe->wait.entry)) {
 		__add_wait_queue(&wq->hash->wait, &wqe->wait);
 		if (!test_bit(hash, &wq->hash->map)) {

@@ -392,7 +392,7 @@ static void io_wait_on_hash(struct io_wqe *wqe, unsigned int hash)
 			list_del_init(&wqe->wait.entry);
 		}
 	}
-	spin_unlock(&wq->hash->wait.lock);
+	spin_unlock_irq(&wq->hash->wait.lock);
 }
 
 static struct io_wq_work *io_get_next_work(struct io_wqe *wqe)

@@ -430,9 +430,9 @@ static struct io_wq_work *io_get_next_work(struct io_wqe *wqe)
 	}
 
 	if (stall_hash != -1U) {
-		raw_spin_unlock(&wqe->lock);
+		raw_spin_unlock_irq(&wqe->lock);
 		io_wait_on_hash(wqe, stall_hash);
-		raw_spin_lock(&wqe->lock);
+		raw_spin_lock_irq(&wqe->lock);
 	}
 
 	return NULL;

(this is on-top of the patch you sent earlier and Daniel Cc: me on after
I checked that the problem/warning still exists).

Sebastian