On Mon, Jan 25 2021 at 7:13am -0500,
Jeffle Xu [off-list ref] wrote:
Since currently we have no simple but efficient way to implement the
bio-based IO polling in the split-bio tracking style, this patch set
turns to the original implementation mechanism that iterates and
polls all underlying hw queues in polling mode. One optimization is
introduced to mitigate the race of one hw queue among multiple polling
instances.
I'm still open to the split bio tracking mechanism, if there's
reasonable way to implement it.
[Performance Test]
The performance is tested by fio (engine=io_uring) 4k randread on
dm-linear device. The dm-linear device is built upon nvme devices,
and every nvme device has one polling hw queue (nvme.poll_queues=1).
Test Case | IOPS in IRQ mode | IOPS in polling mode | Diff
| (hipri=0) | (hipri=1) |
--------------------------- | ---------------- | -------------------- | ----
3 target nvme, num_jobs = 1 | 198k | 276k | ~40%
3 target nvme, num_jobs = 3 | 608k | 705k | ~16%
6 target nvme, num_jobs = 6 | 1197k | 1347k | ~13%
3 target nvme, num_jobs = 6 | 1285k | 1293k | ~0%
As the number of polling instances (num_jobs) increases, the
performance improvement decreases, though it's still positive
compared to the IRQ mode.
I think there is serious room for improvement for DM's implementation;
but the block changes for this are all we'd need for DM in the longrun
anyway (famous last words). So on a block interface level I'm OK with
block patches 1-3.
I don't see why patch 5 is needed (said the same in reply to it; but I
just saw your reason below..).
Anyway, I can pick up DM patches 4 and 6 via linux-dm.git if Jens picks
up patches 1-3. Jens, what do you think?
[Optimization]
To mitigate the race when iterating all the underlying hw queues, one
flag is maintained on a per-hw-queue basis. This flag is used to
indicate whether this polling hw queue currently being polled on or
not. Every polling hw queue is exclusive to one polling instance, i.e.,
the polling instance will skip this polling hw queue if this hw queue
currently is being polled by another polling instance, and start
polling on the next hw queue.
This per-hw-queue flag map is currently maintained in dm layer. In
the table load phase, a table describing all underlying polling hw
queues is built and stored in 'struct dm_table'. It is safe when
reloading the mapping table.
changes since v1:
- patch 1,2,4 is the same as v1 and have already been reviewed
- patch 3 is refactored a bit on the basis of suggestions from
Mike Snitzer.
- patch 5 is newly added and introduces one new queue flag
representing if the queue is capable of IO polling. This mainly
simplifies the logic in queue_poll_store().
Ah OK, don't see why we want to eat a queue flag for that though!
- patch 6 implements the core mechanism supporting IO polling.
The sanity check checking if the dm device supports IO polling is
also folded into this patch, and the queue flag will be cleared if
it doesn't support, in case of table reloading.
Thanks,
Mike