On 31/01/2020 10:58, Ming Lei wrote:

On Fri, Jan 31, 2020 at 6:24 PM John Garry[off-list ref]  wrote:

quoted

quoted

quoted

[  141.976109] Call trace:
[  141.978550]  __switch_to+0xbc/0x218
[  141.982029]  blk_mq_run_work_fn+0x1c/0x28
[  141.986027]  process_one_work+0x1e0/0x358
[  141.990025]  worker_thread+0x40/0x488
[  141.993678]  kthread+0x118/0x120
[  141.996897]  ret_from_fork+0x10/0x18

Hi John,

Thanks for your test!

Hi Ming,

quoted

Could you test the following patchset and only the last one is changed?

https://github.com/ming1/linux/commits/my_for_5.6_block

For SCSI testing, I will ask my colleague Xiang Chen to test when he
returns to work. So I did not see this issue for my SCSI testing for
your original v5, but I was only using 1x as opposed to maybe 20x SAS disks.

BTW, did you test NVMe? For some reason I could not trigger a scenario
where we're draining the outstanding requests for a queue which is being
deactivated - I mean, the queues were always already quiesced.

I run cpu hotplug test on both NVMe and SCSI in KVM, and fio just runs
as expected.

NVMe is often 1:1 mapping, so it might be a bit difficult to trigger
draining in-flight IOs.

Hi Ming,

We got around to testing your my_for_5.6_block branch (Xiang Chen 
actually took the v5 series and applied the following on top only:
block: deactivate hctx when running queue in wrong CPU core
Revert "block: deactivate hctx when all its CPUs are offline when run…)

and we get this:

] IRQ 598: no longer affine to CPU4
[ 1077.396063] CPU4: shutdown
[ 1077.398769] psci: CPU4 killed (polled 0 ms)
[ 1077.457777] CPU3: shutdown
[ 1077.460495] psci: CPU3 killed (polled 0 ms)
[ 1077.499650] CPU2: shutdown
[ 1077.502357] psci: CPU2 killed (polled 0 ms)
[ 1077.546976] CPU1: shutdown
[ 1077.549690] psci: CPU1 killed (polled 0 ms)
it's running b  0
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [38.9% done] 
[1201MB/0KB/0KB /s] [307K/0/0 iops] [eta 00m:22s]
it's running b  1
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [41.7% done] 
[625.2MB/0KB/0KB /s] [160K/0/0 iops] [eta 00m:21s]
it's running b  2
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [44.4% done] 
[637.1MB/0KB/0KB /s] [163K/0/0 iops] [eta 00m:20s]
it's running b  3
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [47.2% done] 
[648.6MB/0KB/0KB /s] [166K/0/0 iops] [eta 00m:19s]
it's running b  4
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [50.0% done] 
[672.8MB/0KB/0KB /s] [172K/0/0 iops] [eta 00m:18s]
it's running b  5
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [52.8% done] 
[680.2MB/0KB/0KB /s] [174K/0/0 iops] [eta 00m:17s]
it's running b  6
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [55.6% done] 
[674.7MB/0KB/0KB /s] [173K/0/0 iops] [eta 00m:16s]
it's running b  7
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [58.3% done] 
[666.2MB/0KB/0KB /s] [171K/0/0 iops] [eta 00m:15s]
it's running b  8
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [61.1% done] 
[668.7MB/0KB/0KB /s] [171K/0/0 iops] [eta 00m:14s]
it's running b  9
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [63.9% done] 
[657.9MB/0KB/0KB /s] [168K/0/0 iops] [eta 00m:13s]
it's running b  10
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [66.7% done] 
[659.6MB/0KB/0KB /s] [169K/0/0 iops] [eta 00m:12s]
it's running b  11
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [69.4% done] 
[662.8MB/0KB/0KB /s] [170K/0/0 iops] [eta 00m:11s]
it's running b  12
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [72.2% done] 
[669.8MB/0KB/0KB /s] [171K/0/0 iops] [eta 00m:10s]
it's running b  13
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [75.0% done] 
[673.2MB/0KB/0KB /s] [172K/0/0 iops] [eta 00m:09s]
it's running b  14
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [77.8% done] 
[650.5MB/0KB/0KB /s] [167K/0/0 iops] [eta 00m:08s]
it's running b  15
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [80.6% done] 
[658.9MB/0KB/0KB /s] [169K/0/0 iops] [eta 00m:07s]
it's running b  16
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [83.3% done] 
[670.3MB/0KB/0KB /s] [172K/0/0 iops] [eta 00m:06s]
it's running b  17
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [86.1% done] 
[663.7MB/0KB/0KB /s] [170K/0/0 iops] [eta 00m:05s]
it's running b  18
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [88.9% done] 
[657.9MB/0KB/0KB /s] [168K/0/0 iops] [eta 00m:04s]
it's running b  19
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [91.7% done] 
[650.9MB/0KB/0KB /s] [167K/0/0 iops] [eta 00m:03s]
it's running b  20
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [94.4% done] 
[646.1MB/0KB/0KB /s] [166K/0/0 iops] [eta 00m:02s]
it's running b  21
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [97.2% done] 
[658.4MB/0KB/0KB /s] [169K/0/0 iops] [eta 00m:01s]
it's running b  22
Jobs: 40 (f=40): [RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR] [100.0% 
done] [649.4MB/0KB/0KB /s] [166K/0/0 iops] [eta 00m:00s]
it's running b  23
Jobs: 1 (f=1): [______________________R_________________] [2.6% done] 
[402.5MB/0KB/0KB /s] [103K/0/0 iops] [eta 22m:44s]
it's running b  24
Jobs: 1 (f=1): [______________________R_________________] [2.7% done] 
[0KB/0KB/0KB /s] [0/0/0 iops] [eta 22m:43s]
it's running b  25
Jobs: 1 (f=1): [______________________R_________________] [2.8% done] 
[0KB/0KB/0KB /s] [0/0/0 iops] [eta 22m:42s]
it's running b  26
Jobs: 1 (f=1): [______________________R_________________] [2.9% done] 
[0KB/0KB/0KB /s] [0/0/0 iops] [eta 22m:41s]
it's running b  27
Jobs: 1 (f=1): [______________________R_________________] [2.9% done] 
[0KB/0KB/0KB /s] [0/0/0 iops] [eta 22m:40s]
[ 1105.419335] sas: Enter sas_scsi_recover_host busy: 1 failed: 1
[ 1105.425185] sas: trying to find task 0x00000000f1b865f3
[ 1105.430409] sas: sas_scsi_find_task: aborting task 0x00000000f1b865f3
not running b  28
#

Looks like the queues are not properly drained as we're getting a single 
IO timeout. I'll have a look when I get a chance.

Cheers,
John