RE: [PATCH] blk-mq: update hctx->cpumask at cpu-hotplug(Internet mail)

Hi Ming,

-----Original Message-----
From: Ming Lei <redacted>
Sent: Tuesday, June 25, 2019 9:55 AM
To: Wenbin Zeng <redacted>
Cc: axboe@kernel.dk; keith.busch@intel.com; hare@suse.com; osandov@fb.com;
sagi@grimberg.me; bvanassche@acm.org; linux-block@vger.kernel.org;
linux-kernel@vger.kernel.org; wenbinzeng(曾文斌) [off-list ref]
Subject: Re: [PATCH] blk-mq: update hctx->cpumask at cpu-hotplug(Internet mail)

On Mon, Jun 24, 2019 at 11:24:07PM +0800, Wenbin Zeng wrote:

quoted

Currently hctx->cpumask is not updated when hot-plugging new cpus,
as there are many chances kblockd_mod_delayed_work_on() getting
called with WORK_CPU_UNBOUND, workqueue blk_mq_run_work_fn may run

There are only two cases in which WORK_CPU_UNBOUND is applied:

1) single hw queue

2) multiple hw queue, and all CPUs in this hctx become offline

For 1), all CPUs can be found in hctx->cpumask.

quoted

on the newly-plugged cpus, consequently __blk_mq_run_hw_queue()
reporting excessive "run queue from wrong CPU" messages because
cpumask_test_cpu(raw_smp_processor_id(), hctx->cpumask) returns false.

The message means CPU hotplug race is triggered.

Yeah, there is big problem in blk_mq_hctx_notify_dead() which is called
after one CPU is dead, but still run this hw queue to dispatch request,
and all CPUs in this hctx might become offline.

We have some discussion before on this issue:

https://lore.kernel.org/linux-block/CACVXFVN729SgFQGUgmu1iN7P6Mv5+puE78STz8hj
9J5bS828Ng@mail.gmail.com/

There is another scenario, you can reproduce it by hot-plugging cpus to kvm guests via qemu monitor (I believe virsh setvcpus --live can do the same thing), for example:
(qemu) cpu-add 1
(qemu) cpu-add 2
(qemu) cpu-add 3

In such scenario, cpu 1, 2 and 3 are not visible at boot, hctx->cpumask doesn't get synced when these cpus are added.

quoted

This patch added a cpu-hotplug handler into blk-mq, updating
hctx->cpumask at cpu-hotplug.

This way isn't correct, hctx->cpumask should be kept as sync with
queue mapping.

Please advise what should I do to deal with the above situation? Thanks a lot.

Thanks,
Ming