nvmf regression with mq-deadline
Subsystems:
block layer , the rest
4 messages,
2 authors,
2017-02-27 · open the first message on its own page
Hey Jens,
I'm getting a regression in nvme-rdma/nvme-loop with for-linus [1]
with a small script to trigger it.
The reason seems to be that the sched_tags does not take into account
the tag_set reserved tags.
This solves it for me, any objections on this?
-- diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c
index 98c7b061781e..46ca965fff5c 100644
--- a/block/blk-mq-sched.c
+++ b/block/blk-mq-sched.c @@ -454,7 +454,8 @@ int blk_mq_sched_setup(struct request_queue *q)
*/
ret = 0 ;
queue_for_each_hw_ctx ( q , hctx , i ) {
- hctx -> sched_tags = blk_mq_alloc_rq_map ( set , i , q->nr_requests, 0);
+ hctx->sched_tags = blk_mq_alloc_rq_map(set, i,
+ q->nr_requests, set->reserved_tags);
if (!hctx->sched_tags) {
ret = -ENOMEM;
break;
--
[1]:
--
[ 94.819701] ------------[ cut here ]------------
[ 94.821639] WARNING: CPU: 0 PID: 729 at block/blk-mq-tag.c:114
blk_mq_get_tag+0x21e/0x260
[ 94.825201] Modules linked in: nvme_loop nvme_fabrics nvme_core
nvmet_rdma nvmet rdma_cm iw_cm null_blk mlx5_ib iscsi_target_mod ib_srpt
ib_cm ib_core tcm_loop tcm_fc libfc tcm_qla2xxx qla2xxx
scsi_transport_fc usb_f_tcm tcm_usb_gadget libcomposite udc_core
vhost_scsi vhost target_core_file target_core_iblock target_core_pscsi
target_core_mod configfs ppdev kvm_intel kvm irqbypass crct10dif_pclmul
crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd
glue_helper cryptd input_leds joydev serio_raw parport_pc i2c_piix4
parport mac_hid sunrpc autofs4 8139too cirrus ttm psmouse drm_kms_helper
syscopyarea floppy sysfillrect mlx5_core ptp pps_core sysimgblt
fb_sys_fops 8139cp drm mii pata_acpi
[ 94.849215] CPU: 0 PID: 729 Comm: bash Not tainted 4.10.0+ #114
[ 94.850761] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
[ 94.853183] Call Trace:
[ 94.853183] dump_stack+0x63/0x90
[ 94.853183] __warn+0xcb/0xf0
[ 94.853183] warn_slowpath_null+0x1d/0x20
[ 94.853183] blk_mq_get_tag+0x21e/0x260
[ 94.853183] ? wake_atomic_t_function+0x60/0x60
[ 94.853183] __blk_mq_alloc_request+0x1b/0xc0
[ 94.853183] blk_mq_sched_get_request+0x1d4/0x290
[ 94.853183] blk_mq_alloc_request+0x63/0xb0
[ 94.853183] nvme_alloc_request+0x53/0x60 [nvme_core]
[ 94.853183] __nvme_submit_sync_cmd+0x31/0xd0 [nvme_core]
[ 94.853183] nvmf_connect_admin_queue+0x11d/0x180 [nvme_fabrics]
[ 94.853183] ? blk_mq_init_allocated_queue+0x472/0x4a0
[ 94.853183] nvme_loop_configure_admin_queue+0xf5/0x1c0 [nvme_loop]
[ 94.853183] nvme_loop_create_ctrl+0x13c/0x550 [nvme_loop]
[ 94.853183] ? nvmf_dev_write+0x50c/0x8de [nvme_fabrics]
[ 94.853183] nvmf_dev_write+0x75a/0x8de [nvme_fabrics]
[ 94.853183] __vfs_write+0x28/0x140
[ 94.853183] ? apparmor_file_permission+0x1a/0x20
[ 94.853183] ? security_file_permission+0x3b/0xc0
[ 94.853183] ? rw_verify_area+0x4e/0xb0
[ 94.853183] vfs_write+0xb8/0x1b0
[ 94.853183] SyS_write+0x46/0xa0
[ 94.853183] ? __close_fd+0x96/0xc0
[ 94.853183] entry_SYSCALL_64_fastpath+0x1e/0xad
[ 94.853183] RIP: 0033:0x7f7be3e74a10
[ 94.853183] RSP: 002b:00007ffca6ac59c8 EFLAGS: 00000246 ORIG_RAX:
0000000000000001
[ 94.853183] RAX: ffffffffffffffda RBX: 0000000000000000 RCX:
00007f7be3e74a10
[ 94.853183] RDX: 000000000000001c RSI: 0000000001b90808 RDI:
0000000000000001
[ 94.853183] RBP: 0000000000000001 R08: 00007f7be4143780 R09:
00007f7be478b700
[ 94.853183] R10: 000000000000001b R11: 0000000000000246 R12:
0000000000000000
[ 94.853183] R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
[ 94.875340] ---[ end trace b820e053982d7057 ]---
--
[2]:
--
#!/bin/bash
CFGFS=/sys/kernel/config/nvmet
NQN=test
modprobe nvme_loop
mkdir $CFGFS/ports/1
echo "loop" > $CFGFS/ports/1/addr_trtype
mkdir $CFGFS/subsystems/$NQN
echo 1 > $CFGFS/subsystems/$NQN/attr_allow_any_host
ln -s $CFGFS/subsystems/$NQN $CFGFS/ports/1/subsystems/
echo "transport=loop,nqn=test" > /dev/nvme-fabrics
--
_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
quoted hunk Hey Jens,
I'm getting a regression in nvme-rdma/nvme-loop with for-linus [1]
with a small script to trigger it.
The reason seems to be that the sched_tags does not take into account
the tag_set reserved tags.
This solves it for me, any objections on this?
-- diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c
index 98c7b061781e..46ca965fff5c 100644
--- a/block/blk-mq-sched.c
+++ b/block/blk-mq-sched.c @@ -454,7 +454,8 @@ int blk_mq_sched_setup(struct request_queue *q)
*/
ret = 0 ;
queue_for_each_hw_ctx ( q , hctx , i ) {
- hctx -> sched_tags = blk_mq_alloc_rq_map ( set , i , q->nr_requests, 0);
+ hctx->sched_tags = blk_mq_alloc_rq_map(set, i,
+ q->nr_requests, set->reserved_tags);
if (!hctx->sched_tags) {
ret = -ENOMEM;
break;
--
Now I'm getting a NULL deref with nvme-rdma [1].
For some reason blk_mq_tag_to_rq() is returning NULL on
tag 0x0 which is io queue connect.
I'll try to see where this is coming from.
This does not happen with loop though...
--
[ 30.431889] nvme nvme0: creating 2 I/O queues.
[ 30.465458] nvme nvme0: tag 0x0 on QP 0x84 not found
[ 36.060168] BUG: unable to handle kernel NULL pointer dereference at
0000000000000030
[ 36.063277] IP: bt_iter+0x31/0x50
[ 36.064088] PGD 0
[ 36.064088] Oops: 0000 [#1] SMP
[ 36.064088] Modules linked in: nvme_rdma nvme_fabrics nvme_core
mlx5_ib ppdev kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul
ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper
cryptd i2c_piix4 joydev input_leds serio_raw parport_pc parport mac_hid
ib_iser rdma_cm iw_cm ib_cm ib_core configfs iscsi_tcp libiscsi_tcp
libiscsi sunrpc scsi_transport_iscsi autofs4 cirrus ttm drm_kms_helper
syscopyarea sysfillrect mlx5_core sysimgblt fb_sys_fops psmouse drm
floppy ptp pata_acpi pps_core
[ 36.064088] CPU: 0 PID: 186 Comm: kworker/0:1H Not tainted 4.10.0+ #115
[ 36.064088] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
[ 36.064088] Workqueue: kblockd blk_mq_timeout_work
[ 36.064088] task: ffff95f6393a0080 task.stack: ffffb826803ac000
[ 36.064088] RIP: 0010:bt_iter+0x31/0x50
[ 36.064088] RSP: 0018:ffffb826803afda0 EFLAGS: 00010202
[ 36.064088] RAX: ffffb826803afdd0 RBX: ffff95f63c036800 RCX:
0000000000000001
[ 36.064088] RDX: ffff95f635ff0798 RSI: 0000000000000000 RDI:
ffff95f63c036800
[ 36.064088] RBP: ffffb826803afe18 R08: 0000000000000000 R09:
0000000000000001
[ 36.064088] R10: 0000000000000000 R11: 0000000000000000 R12:
0000000000000000
[ 36.064088] R13: ffff95f635d7c240 R14: 0000000000000000 R15:
ffff95f63c47ff00
[ 36.064088] FS: 0000000000000000(0000) GS:ffff95f63fc00000(0000)
knlGS:0000000000000000
[ 36.064088] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 36.064088] CR2: 0000000000000030 CR3: 000000003c8db000 CR4:
00000000003406f0
[ 36.064088] Call Trace:
[ 36.064088] ? blk_mq_queue_tag_busy_iter+0x191/0x1d0
[ 36.064088] ? blk_mq_rq_timed_out+0x70/0x70
[ 36.064088] ? blk_mq_rq_timed_out+0x70/0x70
[ 36.064088] blk_mq_timeout_work+0xba/0x160
[ 36.064088] process_one_work+0x16b/0x480
[ 36.064088] worker_thread+0x4b/0x500
[ 36.064088] kthread+0x101/0x140
[ 36.064088] ? process_one_work+0x480/0x480
[ 36.064088] ? kthread_create_on_node+0x40/0x40
[ 36.064088] ret_from_fork+0x2c/0x40
[ 36.064088] Code: 89 d0 48 8b 3a 0f b6 48 18 48 8b 97 08 01 00 00 84
c9 75 03 03 72 04 48 8b 92 80 00 00 00 89 f6 48 8b 34 f2 48 8b 97 98 00
00 00 <48> 39 56 30 74 06 b8 01 00 00 00 c3 55 48 8b 50 10 48 89 e5 ff
[ 36.064088] RIP: bt_iter+0x31/0x50 RSP: ffffb826803afda0
[ 36.064088] CR2: 0000000000000030
[ 36.064088] ---[ end trace 469df54df5f3cd87 ]---
--
_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
Now I'm getting a NULL deref with nvme-rdma [1].
For some reason blk_mq_tag_to_rq() is returning NULL on
tag 0x0 which is io queue connect.
I'll try to see where this is coming from.
This does not happen with loop though...
That's because the loop driver does not rely on the
cqe.command_id to resolve the submitted request (I'll
fix that).
Looks like blk_mq_alloc_request_hctx was overlooked when
the back assignment of the request to the rq_map...
This patch solves the issue for fabrics:
-- diff --git a/block/blk-mq.c b/block/blk-mq.c
index d84c66fb37b7..9611cd9920e9 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c @@ -312,6 +312,7 @@ struct request *blk_mq_alloc_request_hctx(struct request_queue *q, int rw,
ret = -EWOULDBLOCK;
goto out_queue_exit;
}
+ alloc_data.hctx->tags->rqs[rq->tag] = rq;
return rq;
--
If its agreed with everyone I'll send a proper patch
for this and the blk_mq_sched_setup fix?
On Mon, Feb 27 2017, Sagi Grimberg wrote: quoted hunk quoted Now I'm getting a NULL deref with nvme-rdma [1].
For some reason blk_mq_tag_to_rq() is returning NULL on
tag 0x0 which is io queue connect.
I'll try to see where this is coming from.
This does not happen with loop though...
That's because the loop driver does not rely on the
cqe.command_id to resolve the submitted request (I'll
fix that).
Looks like blk_mq_alloc_request_hctx was overlooked when
the back assignment of the request to the rq_map...
This patch solves the issue for fabrics:
-- diff --git a/block/blk-mq.c b/block/blk-mq.c
index d84c66fb37b7..9611cd9920e9 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c @@ -312,6 +312,7 @@ struct request *blk_mq_alloc_request_hctx(struct request_queue *q, int rw,
ret = -EWOULDBLOCK;
goto out_queue_exit;
}
+ alloc_data.hctx->tags->rqs[rq->tag] = rq;
return rq;
--
If its agreed with everyone I'll send a proper patch
for this and the blk_mq_sched_setup fix?
Thanks Sagi, yes please send a proper patch for those two conditions!
--
Jens Axboe