On Mon, Jan 20, 2020 at 9:24 PM John Garry [off-list ref] wrote:

On 15/01/2020 17:00, John Garry wrote:

quoted

On 15/01/2020 11:44, Ming Lei wrote:

quoted

Hi,

Thomas mentioned:
     "
      That was the constraint of managed interrupts from the very
beginning:
       The driver/subsystem has to quiesce the interrupt line and the
associated
       queue _before_ it gets shutdown in CPU unplug and not fiddle
with it
       until it's restarted by the core when the CPU is plugged in again.
     "

But no drivers or blk-mq do that before one hctx becomes inactive(all
CPUs for one hctx are offline), and even it is worse, blk-mq stills tries
to run hw queue after hctx is dead, see blk_mq_hctx_notify_dead().

This patchset tries to address the issue by two stages:

1) add one new cpuhp state of CPUHP_AP_BLK_MQ_ONLINE

- mark the hctx as internal stopped, and drain all in-flight requests
if the hctx is going to be dead.

2) re-submit IO in the state of CPUHP_BLK_MQ_DEAD after the hctx
becomes dead

- steal bios from the request, and resubmit them via
generic_make_request(),
then these IO will be mapped to other live hctx for dispatch

Please comment & review, thanks!

V5:
    - rename BLK_MQ_S_INTERNAL_STOPPED as BLK_MQ_S_INACTIVE
    - re-factor code for re-submit requests in cpu dead hotplug handler
    - address requeue corner case

V4:
    - resubmit IOs in dispatch list in case that this hctx is dead

V3:
    - re-organize patch 2 & 3 a bit for addressing Hannes's comment
    - fix patch 4 for avoiding potential deadlock, as found by Hannes

V2:
    - patch4 & patch 5 in V1 have been merged to block tree, so remove
      them
    - address comments from John Garry and Minwoo

Hi Ming,

Colleague Xiang Chen found this:

Starting 40 processes
Jobs: 40 (f=40)
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [8.3% done]
[2899MB/0KB/0KB /s] [742K/0/0 iops] [eta 00m:33s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [11.1% done]
[2892MB/0KB/0KB /s] [740K/0/0 iops] [eta 00m:32s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [13.9% done]
[2887MB/0KB/0KB /s] [739K/0/0 iops] [eta 00m:31s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [16.7% done]
[2850MB/0KB/0KB /s] [730K/0/0 iops] [eta 00m:30s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [19.4% done]
[2834MB/0KB/0KB /s] [726K/0/0 iops] [eta 00m:29s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [22.2% done]
[2827MB/0KB/0KB /s] [724K/0/0 iops] [eta 00m:28s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [25.0% done]
[2804MB/0KB/0KB /s] [718K/0/0 iops] [eta 00m:27s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [27.8% done]
[2783MB/0KB/0KB /s] [713K/0/0 iops] [eta 00m:26s]
[  112.990292] CPU16: shutdown
[  112.993113] psci: CPU16 killed (polled 0 ms)
[  113.029429] CPU15: shutdown
[  113.032245] psci: CPU15 killed (polled 0 ms)
[  113.073649] CPU14: shutdown
[  113.076461] psci: CPU14 killed (polled 0 ms)
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRR[  113.121258] CPU13:
shutdown
RRRRRRRRRR] [30.[  113.124247] psci: CPU13 killed (polled 0 ms)
6% done] [2772MB/0KB/0KB /s] [710K/0/0 iops] [eta 00m:25s]
[  113.177108] IRQ 600: no longer affine to CPU12
[  113.183185] CPU12: shutdown
[  113.186001] psci: CPU12 killed (polled 0 ms)
[  113.242555] CPU11: shutdown
[  113.245368] psci: CPU11 killed (polled 0 ms)
[  113.302598] CPU10: shutdown
[  113.305420] psci: CPU10 killed (polled 0 ms)
[  113.365757] CPU9: shutdown
[  113.368489] psci: CPU9 killed (polled 0 ms)
[  113.611904] IRQ 599: no longer affine to CPU8
[  113.620611] CPU8: shutdown
[  113.623340] psci: CPU8 killed (polled 0 ms)
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [33.3% done]
[2765MB/0KB/0KB /s] [708K/0/0 iops] [eta 00m:24s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [36.1% done]
[2767MB/0KB/0KB /s] [708K/0/0 iops] [eta 00m:23s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [38.9% done]
[2771MB/0KB/0KB /s] [709K/0/0 iops] [eta 00m:22s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [41.7% done]
[2780MB/0KB/0KB /s] [712K/0/0 iops] [eta 00m:21s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [44.4% done]
[2731MB/0KB/0KB /s] [699K/0/0 iops] [eta 00m:20s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [47.2% done]
[2757MB/0KB/0KB /s] [706K/0/0 iops] [eta 00m:19s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [50.0% done]
[2764MB/0KB/0KB /s] [708K/0/0 iops] [eta 00m:18s]
[  120.762141] ------------[ cut here ]------------
[  120.766760] WARNING: CPU: 0 PID: 10 at block/blk.h:299
generic_make_request_checks+0x480/0x4e8
[  120.775348] Modules linked in:
[  120.778397] CPU: 0 PID: 10 Comm: ksoftirqd/0 Not tainted 5.5.0-

Then this:

1158048319d:18h:40m:34s]
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [0.0% done]
[0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158049605d:23h:03m:30s]
[  141.915198] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[  141.921280] rcu:     96-...0: (4 GPs behind) idle=af6/0/0x1
softirq=90/90 fqs=2626
[  141.928658] rcu:     99-...0: (2 GPs behind)
idle=b82/1/0x4000000000000000 softirq=82/82 fqs=2627
[  141.937339]  (detected by 17, t=5254 jiffies, g=3925, q=545)
[  141.942985] Task dump for CPU 96:
[  141.946292] swapper/96      R  running task        0     0      1
0x0000002a
[  141.953321] Call trace:
[  141.955763]  __switch_to+0xbc/0x218
[  141.959244]  0x0
[  141.961079] Task dump for CPU 99:
[  141.964385] kworker/99:1H   R  running task        0  3473      2
0x0000022a
[  141.971417] Workqueue: kblockd blk_mq_run_work_fn
[  141.976109] Call trace:
[  141.978550]  __switch_to+0xbc/0x218
[  141.982029]  blk_mq_run_work_fn+0x1c/0x28
[  141.986027]  process_one_work+0x1e0/0x358
[  141.990025]  worker_thread+0x40/0x488
[  141.993678]  kthread+0x118/0x120
[  141.996897]  ret_from_fork+0x10/0x18

Hi John,

Thanks for your test!

Could you test the following patchset and only the last one is changed?

https://github.com/ming1/linux/commits/my_for_5.6_block

Thanks,
Ming Lei