Hi Ming, Dmitry,

Ming,

Also the following patch fixes one issue in remove path.

	http://marc.info/?l=linux-block&m=149498450028434&w=2

So could you test v4.12-rc1(d3cfb2a0 is merged) with the above patch?

Thanks for the suggestion but it still resulted in the same BUG.




Dmitry,

This is common bug which happens if device dies under our feet.
bh becomes invalidated and unmapped.
My proposed fix is here:
https://www.spinics.net/lists/kernel/msg2483231.html
Full patchset was not accepted, I'll update it and try again soon.

I was able to apply 1-4 on 4.12-rc1 but 5/5 couldnt apply clean. It
looks like an optimization however so I continued with 1-4.

It did improve the reliability a bit. I was able to do my test several
times before I hit a different bug [1]. I agree with Christoph's reply
to 1 that it seems like a fix that covers up a deeper issue, but it did
help here...


[1]:
[  331.467807] blk_update_request: I/O error, dev nvme5n1, sector 4978432
[  331.481582]
==================================================================
[  331.481596] BUG: KASAN: use-after-free in
swiotlb_unmap_sg_attrs+0x39/0x80
[  331.481601] Read of size 4 at addr ffff88025e28a398 by task
kworker/0:1/174
[  331.481603]
[  331.481610] CPU: 0 PID: 174 Comm: kworker/0:1 Not tainted
4.12.0-rc1-hr+ #68
[  331.481614] Hardware name: Intel Corporation PURLEY/PURLEY, BIOS
PLYDCRB1.86B.0121.R04.1702012027 02/01/2017
[  331.481624] Workqueue: pciehp-0 pciehp_power_thread
[  331.481627] Call Trace:
[  331.481636]  dump_stack+0x63/0x8d
[  331.481645]  print_address_description+0x7b/0x290
[  331.481651]  kasan_report+0x138/0x240
[  331.481657]  ? swiotlb_unmap_sg_attrs+0x39/0x80
[  331.481663]  ? swiotlb_unmap_sg_attrs+0x39/0x80
[  331.481673]  __asan_load4+0x61/0x80
[  331.481678]  swiotlb_unmap_sg_attrs+0x39/0x80
[  331.481686]  vmd_unmap_sg+0x9b/0xc0
[  331.481698]  nvme_pci_complete_rq+0x18b/0x250 [nvme]
[  331.481707]  __blk_mq_complete_request+0x13b/0x290
[  331.481713]  blk_mq_complete_request+0x16/0x20
[  331.481731]  nvme_cancel_request+0x7e/0xe0 [nvme_core]
[  331.481746]  ? nvme_complete_rq+0x170/0x170 [nvme_core]
[  331.481752]  bt_tags_iter+0x88/0xa0
[  331.481759]  blk_mq_tagset_busy_iter+0x18b/0x390
[  331.481774]  ? nvme_complete_rq+0x170/0x170 [nvme_core]
[  331.481790]  ? nvme_complete_rq+0x170/0x170 [nvme_core]
[  331.481799]  nvme_dev_disable+0x1c7/0x590 [nvme]
[  331.481811]  nvme_remove+0x146/0x150 [nvme]
[  331.481817]  pci_device_remove+0x61/0x110
[  331.481827]  device_release_driver_internal+0x1b6/0x2c0
[  331.481834]  device_release_driver+0x12/0x20
[  331.481841]  pci_stop_bus_device+0xc8/0xf0
[  331.481847]  pci_stop_and_remove_bus_device+0x12/0x20
[  331.481854]  pciehp_unconfigure_device+0xc3/0x2a0
[  331.481859]  ? kasan_slab_free+0x92/0xc0
[  331.481866]  pciehp_disable_slot+0x78/0x130
[  331.481872]  pciehp_power_thread+0xab/0xf0
[  331.481880]  process_one_work+0x297/0x5e0
[  331.481886]  worker_thread+0x89/0x6a0
[  331.481894]  kthread+0x18c/0x1e0
[  331.481899]  ? rescuer_thread+0x5f0/0x5f0
[  331.481905]  ? kthread_park+0xa0/0xa0
[  331.481913]  ret_from_fork+0x2c/0x40
[  331.481916]
[  331.481919] Allocated by task 762:
[  331.481927]  save_stack_trace+0x1b/0x20
[  331.481933]  save_stack+0x46/0xd0
[  331.481937]  kasan_kmalloc+0x93/0xc0
[  331.481942]  __kmalloc+0x12e/0x230
[  331.481950]  nvme_queue_rq+0x1db/0xdca [nvme]
[  331.481956]  __blk_mq_try_issue_directly+0x106/0x170
[  331.481961]  blk_mq_try_issue_directly+0x76/0x80
[  331.481966]  blk_mq_make_request+0x61a/0xa90
[  331.481972]  generic_make_request+0x1b5/0x430
[  331.481976]  submit_bio+0xb9/0x240
[  331.482092]  ext4_io_submit+0x6e/0x90 [ext4]
[  331.482169]  ext4_writepages+0x98e/0x1450 [ext4]
[  331.482177]  do_writepages+0x34/0xb0
[  331.482184]  __writeback_single_inode+0x6a/0x490
[  331.482189]  writeback_sb_inodes+0x271/0x650
[  331.482194]  __writeback_inodes_wb+0xac/0x100
[  331.482199]  wb_writeback+0x40c/0x430
[  331.482203]  wb_workfn+0x2b1/0x590
[  331.482208]  process_one_work+0x297/0x5e0
[  331.482212]  worker_thread+0x89/0x6a0
[  331.482217]  kthread+0x18c/0x1e0
[  331.482221]  ret_from_fork+0x2c/0x40
[  331.482222]
[  331.482224] Freed by task 762:
[  331.482229]  save_stack_trace+0x1b/0x20
[  331.482234]  save_stack+0x46/0xd0
[  331.482238]  kasan_slab_free+0x7c/0xc0
[  331.482244]  kfree+0x97/0x190
[  331.482252]  nvme_free_iod+0x163/0x1c0 [nvme]
[  331.482260]  nvme_queue_rq+0x406/0xdca [nvme]
[  331.482265]  __blk_mq_try_issue_directly+0x106/0x170
[  331.482270]  blk_mq_try_issue_directly+0x76/0x80
[  331.482275]  blk_mq_make_request+0x61a/0xa90
[  331.482280]  generic_make_request+0x1b5/0x430
[  331.482284]  submit_bio+0xb9/0x240
[  331.482363]  ext4_io_submit+0x6e/0x90 [ext4]
[  331.482439]  ext4_writepages+0x98e/0x1450 [ext4]
[  331.482444]  do_writepages+0x34/0xb0
[  331.482449]  __writeback_single_inode+0x6a/0x490
[  331.482454]  writeback_sb_inodes+0x271/0x650
[  331.482459]  __writeback_inodes_wb+0xac/0x100
[  331.482464]  wb_writeback+0x40c/0x430
[  331.482469]  wb_workfn+0x2b1/0x590
[  331.482473]  process_one_work+0x297/0x5e0
[  331.482477]  worker_thread+0x89/0x6a0
[  331.482482]  kthread+0x18c/0x1e0
[  331.482486]  ret_from_fork+0x2c/0x40
[  331.482487]
[  331.482492] The buggy address belongs to the object at ffff88025e28a380
[  331.482492]  which belongs to the cache kmalloc-96 of size 96
[  331.482497] The buggy address is located 24 bytes inside of
[  331.482497]  96-byte region [ffff88025e28a380, ffff88025e28a3e0)
[  331.482499] The buggy address belongs to the page:
[  331.482504] page:ffffea000978a280 count:1 mapcount:0 mapping:
 (null) index:0x0 compound_mapcount: 0
[  331.482512] flags: 0x2fffff80008100(slab|head)
[  331.482520] raw: 002fffff80008100 0000000000000000 0000000000000000
0000000180400040
[  331.482527] raw: dead000000000100 dead000000000200 ffff880275817540
0000000000000000
[  331.482528] page dumped because: kasan: bad access detected
[  331.482529]
[  331.482531] Memory state around the buggy address:
[  331.482535]  ffff88025e28a280: fb fb fb fb fb fb fb fb fb fb fb fb fc
fc fc fc
[  331.482540]  ffff88025e28a300: fb fb fb fb fb fb fb fb fb fb fb fb fc
fc fc fc
[  331.482544] >ffff88025e28a380: fb fb fb fb fb fb fb fb fb fb fb fb fc
fc fc fc
[  331.482546]                             ^
[  331.482550]  ffff88025e28a400: fc fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc
[  331.482554]  ffff88025e28a480: fc fc fc fc fc fc fc fc fc fc fc fc fc
fc fc fc
[  331.482556]
==================================================================